View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0004025JEDI VCL00 JVCL Componentspublic2006-12-19 06:482008-05-28 14:14
ReportersourcemakerAssigned Tojfudickar 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Product VersionDaily / GIT 
Target VersionFixed in Version 
Summary0004025: JvCsvDataSet: single double quote in a string field corrupts reading
DescriptionThe file contained units like 1/2" what means a half zoll.
Example:
264648;19999;X-CLEAN REINIGER 1000 ML;SOMETHING;;;;0;0;WA;
350010;19999;WASSERSCHLAUCH 1/2" 20M;TURBOFLEX;;;;18,9;0;WA;

The first line has two descriptions: X-CLEAN REINIGER 1000 ML and SOMETHING
The second results in only one: WASSERSCHLAUCH 1/2" 20M;TURBOFLEX

Excel reads the file correctly.

Best regards

Frank
TagsNo tags attached.

Activities

adrianonantua

2007-02-08 13:23

reporter   ~0010722

I had the very same problem. Going through the source, I got it. After a few simulations, I learned that the TJvCsvDataSet sees a double-quote char in the middle of the column data as an escape character. And that’s the problem. Excel reads it right because it considers that if the column data does not start with double-quote char, any double-quote char found in between is DATA not ESCAPE CHAR. That way, Excel simplifies the matter by always prioritizing the Separator Char. If your data contains the separator char, it has to be inside double-quote chars, like that:

00001;Product 1;”Observation text; the problem is yet to be solved”;234.00

A correction in the StringToCsvRow function would correct it. We must notice, though, that this is a bug in the approach, rather than in the code. Any way, it should be resolved.

adrianonantua

2007-02-08 14:08

reporter   ~0010723

I managed to solve the problem. And it was quite simple. There's a block of code inside the StringToCsvRow function that goes like that:

    if RowString[I] = '"' then
    begin
      if PermitEscapeSequences then
      begin
        QuoteFlag := not QuoteFlag;
        if QuoteFlag and (CharsInColumn > 1) then
        begin
          // (rom) no OutputDebugString in production code
          {$IFDEF DEBUGINFO_ON}
          OutputDebugString('CsvDataSource.pas: StringToCsvRow - unescaped quote character in middle of string!');
          {$ENDIF DEBUGINFO_ON}
        end;
      end
      else
      begin
        // (rom) no OutputDebugString in production code
        {$IFDEF DEBUGINFO_ON}
        OutputDebugString('CsvDataSource.pas: StringToCsvRow - quote character found where no escape sequences are permitted!');
        {$ENDIF DEBUGINFO_ON}
      end;
    end;

These lines do not do much. As we can see this is the code that handles the double quote char. The thing is that this code accepts the double quote char as an escape char anywhere. So I changed the following line:

QuoteFlag := not QuoteFlag;

To:

QuoteFlag := (CharsInColumn = 1);

The first line toggles on/off the quote flag inside the column. In order to read the file reported by our friend sourcemaker, we have to embrace the Excel approach. It only turns on the QuoteFlag variable when it’s the first character in the column. Otherwise, it’s turned off. It IS that simple. Double quotes that come inside an unquoted column will be handle as data. The last double quote of a correctly quoted column will turn off the QuoteFlag because it’s not the first, so there’s no side effect here. Last, but not least, it does not interfere with reading double quotes inside a quoted column.

You JEDI guys could make this simple change an option, controlled by a new property in the component (something like DoubleQuoteApproach). That way you maintain backwards compatibility with the existing code and we gain this new feature/option.

jfudickar

2007-02-09 02:18

developer   ~0010755

Frank,

did you have a small sample app for this.

I'm not so familiar with the component, and this would make the testing of your problem a little bit easier.

Greetings and thanks
Jens

obones

2007-06-19 07:18

administrator   ~0013444

Please provide the zipped sources of a sample application

sourcemaker

2007-09-20 08:11

reporter   ~0013792

Many thanks adrianonantua,

your patch works good for me.

Best regards

Frank

obones

2007-10-12 08:41

administrator   ~0013938

Frank, can we get the test application so that we can try the patch before committing it?

2007-10-25 03:57

 

CSVTest.zip (3,627 bytes)

sourcemaker

2007-10-25 04:00

reporter   ~0013996

Hi obones,

here is the test application you want. Please look at the artbez columns.

I hope that is what you need.

Best regards and sorry for the delay

Frank

jfudickar

2008-05-28 14:13

developer   ~0014314

Done in Repository

Issue History

Date Modified Username Field Change
2006-12-19 06:48 sourcemaker New Issue
2007-02-08 13:23 adrianonantua Note Added: 0010722
2007-02-08 14:08 adrianonantua Note Added: 0010723
2007-02-09 02:18 jfudickar Note Added: 0010755
2007-06-19 07:18 obones Note Added: 0013444
2007-06-19 07:18 obones Status new => feedback
2007-09-20 08:11 sourcemaker Note Added: 0013792
2007-10-12 08:41 obones Note Added: 0013938
2007-10-25 03:57 sourcemaker File Added: CSVTest.zip
2007-10-25 04:00 sourcemaker Note Added: 0013996
2008-05-28 14:13 jfudickar Status feedback => resolved
2008-05-28 14:13 jfudickar Resolution open => fixed
2008-05-28 14:13 jfudickar Assigned To => jfudickar
2008-05-28 14:13 jfudickar Note Added: 0014314