Project JEDI - Issue Tracker
Mantis Bugtracker

Viewing Issue Simple Details Jump to Notes ] View Advanced ] Issue History ] Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0003730 [JEDI VCL] 00 JVCL Components major always 2006-05-28 07:34 2006-06-28 07:34
Reporter LiorF View Status public  
Assigned To ivan_ra
Priority normal Resolution fixed  
Status resolved   Product Version 3.10
Summary 0003730: JvInterpreter memory leak
Description We are exstensively using JvInterpreter to run scripts and if we leave our program to run for a long time, eventually it runs out of memory.
FastMM memory leak reporting points to JvInterpreter as the culprit. The memory gets clogged with huge amounts of strings. Stack trace reads:
[System][@GetMem]
[System][@NewAnsiString]
[System][@LStrFromPCharLen]
[JvInterpreterParser][TJvInterpreterParser.Token]
[JvInterpreter][TJvInterpreterExpression.ParseToken]
It is strange that strings aren't freed. Shouldn't there be reference counting?

Any solutions?
Additional Information We use Delphi6
Tags No tags attached.
Attached Files ? file icon JvInterpreter.pas.patch [^] (478 bytes) 2006-05-29 03:52
? file icon JvInterpreter.pas.patch1 [^] (3,365 bytes) 2006-06-05 10:25
? file icon JvInterpreter.pas.patch2 [^] (3,924 bytes) 2006-06-09 05:49
zip file icon Leak.zip [^] (2,879 bytes) 2006-06-20 06:32
jpg file icon NoLeak.JPG [^] (115,017 bytes) 2006-06-27 04:39

- Relationships
related to 0002073resolvedivan_ra OLE automation support does not work correctly in JvInterpreter 

-  Notes
(0009358)
ivan_ra (developer)
2006-05-28 11:20

strings in TJvInterpreterParser.Token are allocated by SetString procedure. If so, they are automatically free by system string manager, and there should not be memory leak in function TJvInterpreterParser.Token.
It's may be:
1) leak in your application
2) wrong strings/variants in D6
Could You post some code where called JvInterpreter functions? Do You create TJvInterpreter for every call of script, or use 1 instance?
(0009359)
remkobonte (developer)
2006-05-28 16:31

If I use the following script:

var
  A: array [0..10] of Integer;
begin
  Result := A[1];
end;

and test it with the demo program in examples\RaLib\RaInterpreter, I also get a memory leak. The allocation is at the same spot as LiorF reported, specifically the last NextToken call in TJvInterpreterFunction.InterpretIdentifier.

Something seems to go wrong in the following InternalSetValue(Identifier) call. The code is hard to read for me, but I think the problem in this case is in the JvInterpreterVarCopy procedure (parameter Source is then the A array and Dest is the 'A' string, the cast bypasses the reference count stuff)

I think :)
(0009360)
ivan_ra (developer)
2006-05-29 03:52

Remko, you are right.
JvInterpreterVarCopy is leaky because it bypasses the reference count stuff for varArray and varRecord types.
To release reference we must call VariantClear for Dest variable (look at System._VarCopy as example). So, I suggest to add VarClear(Dest) into JvInterpreterVarCopy.
This is patch for JvInterpreter.pas. It works fine with my progs.

for LiorF: could You test this patch with your program?
(0009361)
LiorF (reporter)
2006-05-30 07:01

Seems to have fixed the problem. Great job and thanks for the fast response!
I will report if I see any additional problems.
(0009362)
ivan_ra (developer)
2006-05-30 07:22

This is now in SVN
(0009374)
LiorF (reporter)
2006-06-04 02:24
edited on: 2006-06-04 02:25

Well, looks like it isn't the end of the story. FastMM memory leak reporting reports nothing significant, but the program still leaks memory. We left it running for a few days and it ate all the memory. FastMM reported nothing of significance. However just looking at the program's memory allocation with Task Manager shows that the memory allocation linearly increases with time. This happens only when we use the script. Without it (and with the program performing the same operations as defind by the script) the memory allocation remains constant.

(0009381)
ivan_ra (developer)
2006-06-05 06:04

records in JvInterpreter are still leaky. Do You use any record types in your scripts ? (for example, TPoint)
(0009382)
LiorF (reporter)
2006-06-05 06:37

I don't use records. Here is an example script:

var
  r:variant;
begin
  r:=func('xyz');//Call to a function that returns a dynamic array of double
  Result:=r[0];
end;
(0009388)
ivan_ra (developer)
2006-06-05 10:24

This is new incremental patch (fixes memory leak with arrays and records when they returns from function). The key to solve problem is JvInterpreterVarFree call.
Please try this patch and report results.
Patch also includes issue 0002073 and some optimisation
(0009541)
ivan_ra (developer)
2006-06-09 05:48
edited on: 2006-06-09 05:53

This is script illustrating memory leak for records (for jvInterpreterTest program). Requires {$DEFINE JvInterpreter_DEBUG} in head of JvInterpreter.pas

unit a;

procedure main;
var
  pPoint: TPoint;
  i:integer;
begin
  for i:=0 to 10000 do
    pPoint:=Point(0,0);
end;

end.

there is no leak with patch. The same thing with functions returning arrays.
The last patch also include some paranoidal JvInterpreterVarFree calls. I think thats enough.

(0009598)
LiorF (reporter)
2006-06-19 03:37

I implemented all the changes in patch2 and the leak still persists. I didn't touch patch1. Patch2 replaces it, right?

Try running the script I posted earlier.
(0009600)
ivan_ra (developer)
2006-06-19 05:56
edited on: 2006-06-19 06:15

may be wrong adapter for "func"?
there is no leak in such script:

unit a;

procedure main;
var
  a:variant;
  i:integer;
begin
  for i:=0 to 100000 do
    a:=VarArrayCreate([0,9],varDouble);
end;
end.

please check latest revision. patch2 now in SVN

(0009610)
LiorF (reporter)
2006-06-20 02:22

I downloaded all the files from the SVN and compiled the package.
The script in your example produces a leak.
Create a loop in your program that runs the script. Run the script in your example (with or without the loop) and monitor the memory allocation with Task Manager. It rises linearly with time.
(0009612)
ivan_ra (developer)
2006-06-20 06:36

Cant reproduce. There is no leak in my example (Leak.zip). Tested in D5 and D9
(0009615)
LiorF (reporter)
2006-06-22 03:39

I tested your example program on several machines. On some of them it produces a leak, while on others it doesn't. It's very strange. In cases where the leak exists, it isn't reported by the program, but can only be seen in the task manager.
All the machines where the leak occurs have the same installation of Delphi 6 and JVCL. Also, no other program suffers from the leak. It happens only with JvInterpreter.
Any ideas?
(0009616)
anudedeus (reporter)
2006-06-22 04:38

LiorF, you first reported the issue as being in v.3.10, are you sure you are using the latest daily JVCL build in your project?
(0009617)
ivan_ra (developer)
2006-06-22 04:49

Is there any dependance between leak and:
1) operation system (or service pack)
2) processor model
3) something else?

could You test example with another compiler version? I have not D6 but tested it with D5,D7 and D9 - all without leak, winXP pro, home, win2003
(0009630)
LiorF (reporter)
2006-06-22 15:22

I will try to test it on more systems and reinstall JEDI and I will report the results.
Results so far:
System 1:
Athlon XP, Windows XP, Delphi 7, JVCL 3.00 unpatched - no leak
System 2:
Pentium 4, Windows XP, Delphi 6, JVCL 3.10 with all JvInterpreter files updated to the latest versions from SVN and package recompiled - leak
System 3:
Pentium 4, Windows XP, Delphi 6, JVCL 3.10 unpatched - leak
(0009633)
anudedeus (reporter)
2006-06-23 02:15

Well, then you must updated your files!
All the bug fixes you find here in the Issue Tracker are introduced to the source repository and there is a daily build of the whole library. So today there is already a newer version than the day before and so on.
Please get the latest one from
http://jvcl.sf.net/daily/ [^]
and check if the leaks persist.
(0009650)
LiorF (reporter)
2006-06-27 01:28

I downloaded the latest versions of jcl and jvcl. I had no luck installing the new jvcl though. It says "Delphi 6 Update 2 is not installed" and quits. It is installed however. Delphi says that it is installed and just in case, I reinstalled it twice. It didn't help.
(0009657)
LiorF (reporter)
2006-06-27 03:35

I disabled the version check (set it to always return update 2) and installed JVCL.
The leak persists.
(0009668)
ivan_ra (developer)
2006-06-27 04:49
edited on: 2006-06-27 04:50

I just tested it in D6 - there is no leak (look at screen)
D6 Enterprise build 6.240 update pack 2
JVCL 3.30 from SVN
WinXP Pro v.2002 Service Pack 2
Pentium 4 2.4 GHz
There was no problem with installing JCL and JVCL in D6

(0009669)
LiorF (reporter)
2006-06-27 04:50

I tested it on more systems. All of them were Pentium 4 with Delphi 6 Professional and Windows XP Professional. All systems experienced the leak.
Delphi 6 seems the most logial common link, although it can still be something else.
(0009670)
ivan_ra (developer)
2006-06-27 04:54

difference between D6 Enterprise and D6 Professional?
Could You test it with D6 Enterprise?
(0009671)
LiorF (reporter)
2006-06-27 05:20

I don't have Enterprise. Note that the leak is not reported by the program nor by FastMM. It can be seen from the task manager and from the fact that if you run it for long enough, the system runs out of memory. What can cause such a leak?
I might try tomorrow or in a few days testing it on a system with a different installation of Windows, but with the same installation of Delphi.
(0009672)
ivan_ra (developer)
2006-06-27 05:35

This is test example compiled with D6 Ent:
http://niisau.ru/leak/Project1.rar [^]
(0009673)
anudedeus (reporter)
2006-06-27 05:45

Guys, I just realised that I had something similar in the past and that wasn't an actual memory leak, but the way that the memory manager or the OS works. By that time I was using the standard Delphi 7 Enterprise memory manager, WinXP Pro.
What I believe is happening is that although the code has requested the memory to be freed, and got an ok as result from the OS, it (the OS) doesn't really fully deallocates it, for performance reasons. That is specially true for very small chunks of data, as it considers that the same memory will be soon used again by the same app.
In my case, I was creating a tstringlist and freeing it, several times in a loop.
Sleuth (the tool I use to find memory leaks) didn't report any leak at all, but looking at task manager it goes up and up and up during the cycle, and even after I got out of the loop (and the variable was local to it), the memory continued to be allocated to my app until the app finished, where everything would be flushed out. No memory leak reports from anybody.
So, I don't believe there is any problem with the code (Jedi), and that's a case of reporting that to the FastMM developers instead and see what they think.
(0009689)
LiorF (reporter)
2006-06-27 07:21

This is a possibility. There is however certainly a bug somewhere, as eventually the system runs out of memory and the program crashes. It could be that the bug is in Windows.
Interestingly on one of the systems I tested, there was no problem. It was an Athlon XP with Windows XP professional service pack 2 and Delphi 7.
I also tested it on 4 systems with Pentium 4, Windows XP professional service pack 1 and Delphi 6. All these had the problem. The problem occured with both the default memory manager and fastmm. Also the problem occurs only when we use JvInterpreter. We use lots of other programs and never experienced any such behavior. There must be something in JvInterpreter that triggers it.
(0009698)
ivan_ra (developer)
2006-06-27 08:37
edited on: 2006-06-28 02:21

1) JvInterpreter intensively uses variants and custom variants (for arrays, records, etc)
2) New Variants was firstly introduced in D6. I'm not sure, but perhaps they was leaky and was fixed in latest versions (in D6 Ent Update 2 too), but not in D6 Pro. Thats why I ask You to test leak in any other IDE.
You can also check size or Variants unit from D6 Ent:

Source\Rtl\Sys\Variants.pas: 4991 lines, 153678 bytes
lib\Variants.dcu: 80328 bytes

Source\Rtl\Sys\VarUtils.pas: 1964 lines, 65892 bytes
lib\VarUtils.dcu: 5886 bytes

(0009702)
LiorF (reporter)
2006-06-28 02:07

Well, we have much smaller files:
Variants.pas 115,954 bytes
VarUtils.pas 53,108 bytes
(0009707)
ivan_ra (developer)
2006-06-28 02:38

OK. This is size of source files from initial installation of D6 (without Update 2).
Please check dcu size
Variants.dcu - 49 402
VarUtils.dcu - 4 010
this is sizes of files from D6 CD
If You have not right Update2 files (Variants.dcu: 80328 bytes and VarUtils.dcu: 5886 bytes), I think this is not JVCL issue and it can be resolved
(0009709)
obones (administrator)
2006-06-28 02:58

Also note that Delphi 6 has two Runtime Library updates. So to be sure you have everything in order, you should have installed this:

Delphi 6
General Update 2
Runtime Library Update 2
Runtime Library Update 3

Variants issues are specifically addressed by RTL Update 2.
After this, I agree this issue should be set as "resolved".
(0009710)
LiorF (reporter)
2006-06-28 05:55

The thing is I installed these updates in the past. To be sure I now reinstalled them. The insatall goes normally, no errors of any kind. In the end it shows a text file with what was installed and indeed in rtl2 it says that it updated the variants. Variants.pas and variants.dcu however remain unchanged and they are nowhere near the sizes you quoted. Maybe someone can send me the files by mail?
(0009711)
ivan_ra (developer)
2006-06-28 06:59
edited on: 2006-06-28 07:29

I think You need to try google search, for example
http://www.google.com/search?q=Delphi+6+Runtime+Library+Update+3 [^]

:)

RTL 2 Files:
Source\Rtl\Sys\Variants.pas: 5557 lines, 169 260 bytes
lib\Variants.dcu: 86 796 bytes

Source\Rtl\Sys\VarUtils.pas: 2258 lines, 75 756 bytes
lib\VarUtils.dcu: 13 130 bytes

after installing all patches please report about leak to resolve this issue

(0009712)
LiorF (reporter)
2006-06-28 07:13

Problem solved!
Apparently the updates were installing absoultely nothing even though they reported as if everything was installed. They were unable to rewrite any files. I don't know what caused it. It could be antivirus or something running in the background. So I let them install to a empty Delphi directory tree and took all the new files from there. The leak disappeared.
Thank you for all the help.
(0009714)
ivan_ra (developer)
2006-06-28 07:33

fixed 06-19-06 (message 9600)

- Issue History
Date Modified Username Field Change
2006-05-28 07:34 LiorF New Issue
2006-05-28 11:20 ivan_ra Note Added: 0009358
2006-05-28 11:20 ivan_ra Status new => feedback
2006-05-28 16:31 remkobonte Note Added: 0009359
2006-05-29 03:52 ivan_ra Note Added: 0009360
2006-05-29 03:52 ivan_ra File Added: JvInterpreter.pas.patch
2006-05-30 07:01 LiorF Note Added: 0009361
2006-05-30 07:22 ivan_ra Status feedback => resolved
2006-05-30 07:22 ivan_ra Resolution open => fixed
2006-05-30 07:22 ivan_ra Assigned To => ivan_ra
2006-05-30 07:22 ivan_ra Note Added: 0009362
2006-05-30 08:44 ivan_ra Status resolved => feedback
2006-05-30 08:44 ivan_ra Resolution fixed => reopened
2006-05-30 08:44 ivan_ra Status feedback => resolved
2006-05-30 08:44 ivan_ra Fixed in Version => Daily / SVN
2006-05-30 08:44 ivan_ra Resolution reopened => fixed
2006-06-04 02:24 LiorF Status resolved => feedback
2006-06-04 02:24 LiorF Resolution fixed => reopened
2006-06-04 02:24 LiorF Note Added: 0009374
2006-06-04 02:25 LiorF Note Edited: 0009374
2006-06-05 06:04 ivan_ra Note Added: 0009381
2006-06-05 06:37 LiorF Note Added: 0009382
2006-06-05 10:24 ivan_ra Note Added: 0009388
2006-06-05 10:25 ivan_ra File Added: JvInterpreter.pas.patch1
2006-06-09 05:48 ivan_ra Note Added: 0009541
2006-06-09 05:49 ivan_ra File Added: JvInterpreter.pas.patch2
2006-06-09 05:53 ivan_ra Note Edited: 0009541
2006-06-09 05:53 ivan_ra Note Edited: 0009541
2006-06-09 05:57 LiorF Note Added: 0009544
2006-06-09 06:16 obones Relationship added related to 0002073
2006-06-19 03:37 LiorF Note Added: 0009598
2006-06-19 04:30 LiorF Note Deleted: 0009544
2006-06-19 05:56 ivan_ra Note Added: 0009600
2006-06-19 06:15 ivan_ra Note Edited: 0009600
2006-06-20 02:22 LiorF Note Added: 0009610
2006-06-20 06:32 ivan_ra File Added: Leak.zip
2006-06-20 06:36 ivan_ra Note Added: 0009612
2006-06-22 03:39 LiorF Note Added: 0009615
2006-06-22 04:38 anudedeus Note Added: 0009616
2006-06-22 04:49 ivan_ra Note Added: 0009617
2006-06-22 15:22 LiorF Note Added: 0009630
2006-06-23 02:15 anudedeus Note Added: 0009633
2006-06-27 01:28 LiorF Note Added: 0009650
2006-06-27 03:35 LiorF Note Added: 0009657
2006-06-27 04:39 ivan_ra File Added: NoLeak.JPG
2006-06-27 04:49 ivan_ra Note Added: 0009668
2006-06-27 04:50 LiorF Note Added: 0009669
2006-06-27 04:50 ivan_ra Note Edited: 0009668
2006-06-27 04:54 ivan_ra Note Added: 0009670
2006-06-27 05:20 LiorF Note Added: 0009671
2006-06-27 05:35 ivan_ra Note Added: 0009672
2006-06-27 05:45 anudedeus Note Added: 0009673
2006-06-27 07:21 LiorF Note Added: 0009689
2006-06-27 08:37 ivan_ra Note Added: 0009698
2006-06-28 02:07 LiorF Note Added: 0009702
2006-06-28 02:21 ivan_ra Note Edited: 0009698
2006-06-28 02:38 ivan_ra Note Added: 0009707
2006-06-28 02:58 obones Note Added: 0009709
2006-06-28 05:55 LiorF Note Added: 0009710
2006-06-28 06:59 ivan_ra Note Added: 0009711
2006-06-28 07:13 LiorF Note Added: 0009712
2006-06-28 07:29 ivan_ra Note Edited: 0009711
2006-06-28 07:33 ivan_ra Status feedback => resolved
2006-06-28 07:33 ivan_ra Resolution reopened => fixed
2006-06-28 07:33 ivan_ra Note Added: 0009714


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker