Project JEDI - Issue Tracker
Mantis Bugtracker

Viewing Issue Simple Details Jump to Notes ] View Advanced ] Issue History ] Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0003333 [JEDI VCL] 00 JVCL Components block always 2005-11-30 06:22 2006-01-19 14:03
Reporter ivan_ra View Status public  
Assigned To obones
Priority normal Resolution fixed  
Status resolved   Product Version 3.00
Summary 0003333: JvInterpreter: wrong extending in 0003217
Description After extending of JvInterpreterParser (0003217 by peter schraut) it cant identify keywords with upper case symbols, for example:
Unit, UNIT, Uses, Interface and so on.
It raises exception: 'unit' expected but 'Unit' found, or something else in other cases, and cant interpret this units
Additional Information temp solution is to lowercase symbols in function PaTokenizeTag, but more correct is rewrite introduced tables "AssoIndices" and "AssoValues"
Tags No tags attached.
Attached Files zip file icon Ihash.zip [^] (2,974 bytes) 2005-12-01 08:29
zip file icon IhashGood.zip [^] (3,074 bytes) 2005-12-02 03:29
zip file icon IhashFinal.zip [^] (2,978 bytes) 2005-12-02 03:44

- Relationships

-  Notes
(0008180)
ivan_ra (developer)
2005-11-30 07:50
edited on: 2005-12-06 11:39

I suggest:
1) replace all "63" in function PaTokenizeTag back to "$1F" (difference between upper and lower case is $20)
2) shrink AssoIndices to first 32 ($20) values - the others are unusable

After first step function PaTokenizeTag will work fine; step 2 is sequent of step 1 and is only optimization

(0008181)
ivan_ra (developer)
2005-12-01 08:29
edited on: 2005-12-02 03:45

This is working code (Please add it to CVS):
   AssoIndices: array [0..31] of Integer = (
    { 0 1 2 3 4 5 6 7 8 9 }
    {00} 50, 80, 25, 13, 92, 71, 87, 61, 91, 99,
    {10} 73, 95, 27, 7, 16, 1, 96, 41, 91, 99,
    {20} 19, 15, 72, 71, 50, 30, 9, 6, 45, 27,
    {30} 79, 61);

   AssoValues: array [0..255] of Integer = (
    { 0 1 2 3 4 5 6 7 8 9 }
    {00} -1, -1, -1, -1, -1, -1, 44, 10, -1, -1,
    {10} 37, -1, -1, -1, -1, 7, -1, -1, -1, -1,
    {20} -1, -1, -1, 27, -1, -1, -1, -1, -1, -1,
    {30} -1, -1, 26, -1, -1, 20, -1, 25, -1, -1,
    {40} -1, 30, 39, -1, -1, -1, -1, 13, -1, -1,
    {50} -1, -1, -1, -1, -1, -1, -1, 1, -1, -1,
    {60} -1, -1, -1, -1, -1, 12, -1, -1, -1, -1,
    {70} -1, -1, 6, -1, -1, -1, -1, -1, -1, -1,
    {80} 34, -1, -1, -1, -1, -1, 3, -1, -1, 49,
    {90} -1, -1, 45, -1, -1, -1, -1, -1, -1, -1,
    {100} 2, 41, -1, -1, -1, -1, -1, 46, -1, 28,
    {110}-1, -1, 17, -1, -1, -1, 36, -1, 11, -1,
    {120}-1, -1, 35, 48, -1, -1, -1, -1, 8, -1,
    {130}-1, 32, -1, 19, -1, -1, -1, 5, -1, -1,
    {140}40, -1, -1, -1, -1, -1, -1, -1, 21, -1,
    {150}22, -1, 31, -1, -1, -1, -1, -1, -1, 16,
    {160}43, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {170}-1, -1, 18, -1, -1, -1, -1, 47, -1, -1,
    {180}-1, -1, -1, -1, -1, -1, -1, 42, -1, -1,
    {190}-1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {200}-1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {210}-1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
    {220}29, -1, -1, -1, 4, 15, 24, -1, -1, -1,
    {230}-1, -1, 33, -1, -1, 9, -1, 50, -1, 14,
    {240}-1, -1, -1, 23, -1, -1, 38, -1, -1, -1,
    {250}-1, -1, -1, -1, -1, 0);

procedure PaTokenizeTag is nearly the same as before 3217 (I optimized code):
function PaTokenizeTag(const TokenStr: string): TTokenKind;
var
  Len,i: Integer;
  HVal: Integer;
begin
  Result := P_UNKNOWN;
  HVal := -1;
  Len := Length(TokenStr);
  
  if (MIN_WORD_LENGTH <= Len) and (Len <= MAX_WORD_LENGTH) then
  begin
    HVal := Len;
    for i:=1 to Len do begin
      HVal := HVal + AssoIndices[(Byte(TokenStr[i]) - Byte('a')) and $1F];
      if i=3 then
        Break;
    end;
    HVal := HVal + AssoIndices[(Byte(TokenStr[Len]) - Byte('a')) and $1F];
    HVal := HVal and 255; {High(AssoValues)}
    HVal := AssoValues[HVal];
  end;

  if HVal <> -1 then
  begin
    if Cmp(WordList[HVal].Token, TokenStr) then
      Result := WordList[HVal].TTyp;
  end;
end;


I also attach project to find hash indices (IHashFinal)

(0008289)
obones (administrator)
2006-01-05 08:06

Can you check with the latest daily?
There were discussions about this in the newsgroups and I believe it has been fixed.
(0008306)
ivan_ra (developer)
2006-01-06 02:03

Cant check bacause nothing new in CVS yet.
But I already use fix #0008181 in my projects (more than 1 month)
(0008353)
scilib (reporter)
2006-01-08 14:58

The problem is still not fixed in CVS. The clever and optimised ivan_ra's fix 0008181 works well for me and I hope that it could be implementet in the JvInterpreterParser.pas soon. Without this fix the current daily version does not work with "Unit" etc.
(0008397)
obones (administrator)
2006-01-17 13:13

This is now fixed in CVS.
But ivan, where does the MakeHash project go in CVS?
(0008400)
ivan_ra (developer)
2006-01-17 20:52

MakeHash project generates random AssoIndices table and tests it for collisions using the same PaTokenizeTag function and identifiers table. This repeats until success. The output is AssoIndices and AssoValues tables.
You can generate your own tables or use it in the future to support new keywords in JvInterpreter.
If You want to add it to CVS, would be better to put it into "devtools" folder?
(0008407)
obones (administrator)
2006-01-19 14:03

Ok, many thanks.
This is now all in CVS.

- Issue History
Date Modified Username Field Change
2005-11-30 06:22 ivan_ra New Issue
2005-11-30 07:50 ivan_ra Note Added: 0008180
2005-12-01 08:29 ivan_ra Note Added: 0008181
2005-12-01 08:29 ivan_ra File Added: Ihash.zip
2005-12-01 09:11 ivan_ra Note Edited: 0008181
2005-12-02 03:27 ivan_ra Note Edited: 0008181
2005-12-02 03:29 ivan_ra File Added: IhashGood.zip
2005-12-02 03:30 ivan_ra Note Edited: 0008181
2005-12-02 03:44 ivan_ra File Added: IhashFinal.zip
2005-12-02 03:45 ivan_ra Note Edited: 0008181
2005-12-06 11:39 ivan_ra Note Edited: 0008180
2006-01-05 08:06 obones Note Added: 0008289
2006-01-05 08:06 obones Status new => feedback
2006-01-06 02:03 ivan_ra Note Added: 0008306
2006-01-08 14:58 scilib Note Added: 0008353
2006-01-09 13:40 scilib Issue Monitored: scilib
2006-01-17 13:13 obones Note Added: 0008397
2006-01-17 20:52 ivan_ra Note Added: 0008400
2006-01-19 14:03 obones Status feedback => resolved
2006-01-19 14:03 obones Resolution open => fixed
2006-01-19 14:03 obones Assigned To => obones
2006-01-19 14:03 obones Note Added: 0008407


Mantis 1.1.6[^]
Copyright © 2000 - 2008 Mantis Group
Powered by Mantis Bugtracker