View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0003333JEDI VCL00 JVCL Componentspublic2005-11-30 06:222006-01-19 14:03
Reporterivan_raAssigned Toobones 
PrioritynormalSeverityblockReproducibilityalways
Status resolvedResolutionfixed 
Product Version3.00 
Target VersionFixed in Version3.20 
Summary0003333: JvInterpreter: wrong extending in 0003217
DescriptionAfter extending of JvInterpreterParser (0003217 by peter schraut) it cant identify keywords with upper case symbols, for example:
Unit, UNIT, Uses, Interface and so on.
It raises exception: 'unit' expected but 'Unit' found, or something else in other cases, and cant interpret this units
Additional Informationtemp solution is to lowercase symbols in function PaTokenizeTag, but more correct is rewrite introduced tables "AssoIndices" and "AssoValues"
TagsNo tags attached.

Activities

ivan_ra

2005-11-30 07:50

developer   ~0008180

Last edited: 2005-12-06 11:39

 I suggest:
1) replace all "63" in function PaTokenizeTag back to "$1F" (difference between upper and lower case is $20)
2) shrink AssoIndices to first 32 ($20) values - the others are unusable

After first step function PaTokenizeTag will work fine; step 2 is sequent of step 1 and is only optimization

ivan_ra

2005-12-01 08:29

developer   ~0008181

Last edited: 2005-12-02 03:45

 This is working code (Please add it to CVS):
   AssoIndices: array [0..31] of Integer = (
    { 0 1 2 3 4 5 6 7 8 9 }
    {00} 50, 80, 25, 13, 92, 71, 87, 61, 91, 99,
    {10} 73, 95, 27, 7, 16, 1, 96, 41, 91, 99,
    {20} 19, 15, 72, 71, 50, 30, 9, 6, 45, 27,
    {30} 79, 61);

   AssoValues: array [0..255] of Integer = (
    { 0 1 2 3 4 5 6 7 8 9 }
    {00} -1, -1, -1, -1, -1, -1, 44, 10, -1, -1,
    {10} 37, -1, -1, -1, -1, 7, -1, -1, -1, -1,
    {20} -1, -1, -1, 27, -1, -1, -1, -1, -1, -1,
    {30} -1, -1, 26, -1, -1, 20, -1, 25, -1, -1,
    {40} -1, 30, 39, -1, -1, -1, -1, 13, -1, -1,
    {50} -1, -1, -1, -1, -1, -1, -1, 1, -1, -1,
    {60} -1, -1, -1, -1, -1, 12, -1, -1, -1, -1,
    {70} -1, -1, 6, -1, -1, -1, -1, -1, -1, -1,
    {80} 34, -1, -1, -1, -1, -1, 3, -1, -1, 49,
    {90} -1, -1, 45, -1, -1, -1, -1, -1, -1, -1,
    {100} 2, 41, -1, -1, -1, -1, -1, 46, -1, 28,
    {110}-1, -1, 17, -1, -1, -1, 36, -1, 11, -1,
    {120}-1, -1, 35, 48, -1, -1, -1, -1, 8, -1,
    {130}-1, 32, -1, 19, -1, -1, -1, 5, -1, -1,
    {140}40, -1, -1, -1, -1, -1, -1, -1, 21, -1,
    {150}22, -1, 31, -1, -1, -1, -1, -1, -1, 16,
    {160}43, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {170}-1, -1, 18, -1, -1, -1, -1, 47, -1, -1,
    {180}-1, -1, -1, -1, -1, -1, -1, 42, -1, -1,
    {190}-1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {200}-1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {210}-1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {220}29, -1, -1, -1, 4, 15, 24, -1, -1, -1,
    {230}-1, -1, 33, -1, -1, 9, -1, 50, -1, 14,
    {240}-1, -1, -1, 23, -1, -1, 38, -1, -1, -1,
    {250}-1, -1, -1, -1, -1, 0);

procedure PaTokenizeTag is nearly the same as before 3217 (I optimized code):
function PaTokenizeTag(const TokenStr: string): TTokenKind;
var
  Len,i: Integer;
  HVal: Integer;
begin
  Result := P_UNKNOWN;
  HVal := -1;
  Len := Length(TokenStr);
  
  if (MIN_WORD_LENGTH <= Len) and (Len <= MAX_WORD_LENGTH) then
  begin
    HVal := Len;
    for i:=1 to Len do begin
      HVal := HVal + AssoIndices[(Byte(TokenStr[i]) - Byte('a')) and $1F];
      if i=3 then
        Break;
    end;
    HVal := HVal + AssoIndices[(Byte(TokenStr[Len]) - Byte('a')) and $1F];
    HVal := HVal and 255; {High(AssoValues)}
    HVal := AssoValues[HVal];
  end;

  if HVal <> -1 then
  begin
    if Cmp(WordList[HVal].Token, TokenStr) then
      Result := WordList[HVal].TTyp;
  end;
end;


I also attach project to find hash indices (IHashFinal)

2005-12-01 08:29

 

Ihash.zip (2,974 bytes)

2005-12-02 03:29

 

IhashGood.zip (3,074 bytes)

2005-12-02 03:44

 

IhashFinal.zip (2,978 bytes)

obones

2006-01-05 08:06

administrator   ~0008289

Can you check with the latest daily?
There were discussions about this in the newsgroups and I believe it has been fixed.

ivan_ra

2006-01-06 02:03

developer   ~0008306

Cant check bacause nothing new in CVS yet.
But I already use fix #0008181 in my projects (more than 1 month)

scilib

2006-01-08 14:58

reporter   ~0008353

The problem is still not fixed in CVS. The clever and optimised ivan_ra's fix 0008181 works well for me and I hope that it could be implementet in the JvInterpreterParser.pas soon. Without this fix the current daily version does not work with "Unit" etc.

obones

2006-01-17 13:13

administrator   ~0008397

This is now fixed in CVS.
But ivan, where does the MakeHash project go in CVS?

ivan_ra

2006-01-17 20:52

developer   ~0008400

MakeHash project generates random AssoIndices table and tests it for collisions using the same PaTokenizeTag function and identifiers table. This repeats until success. The output is AssoIndices and AssoValues tables.
You can generate your own tables or use it in the future to support new keywords in JvInterpreter.
If You want to add it to CVS, would be better to put it into "devtools" folder?

obones

2006-01-19 14:03

administrator   ~0008407

Ok, many thanks.
This is now all in CVS.

Issue History

Date Modified Username Field Change
2005-11-30 06:22 ivan_ra New Issue
2005-11-30 07:50 ivan_ra Note Added: 0008180
2005-12-01 08:29 ivan_ra Note Added: 0008181
2005-12-01 08:29 ivan_ra File Added: Ihash.zip
2005-12-01 09:11 ivan_ra Note Edited: 0008181
2005-12-02 03:27 ivan_ra Note Edited: 0008181
2005-12-02 03:29 ivan_ra File Added: IhashGood.zip
2005-12-02 03:30 ivan_ra Note Edited: 0008181
2005-12-02 03:44 ivan_ra File Added: IhashFinal.zip
2005-12-02 03:45 ivan_ra Note Edited: 0008181
2005-12-06 11:39 ivan_ra Note Edited: 0008180
2006-01-05 08:06 obones Note Added: 0008289
2006-01-05 08:06 obones Status new => feedback
2006-01-06 02:03 ivan_ra Note Added: 0008306
2006-01-08 14:58 scilib Note Added: 0008353
2006-01-17 13:13 obones Note Added: 0008397
2006-01-17 20:52 ivan_ra Note Added: 0008400
2006-01-19 14:03 obones Status feedback => resolved
2006-01-19 14:03 obones Resolution open => fixed
2006-01-19 14:03 obones Assigned To => obones
2006-01-19 14:03 obones Note Added: 0008407