csv import: 4 rule file accepts a record of only 3 fields

[agander]

If you're not sure this is a bug, or if some discussion would help,
**
Brief chat on hledger.irc Jan 7th 16:08-16:56
**

Have you looked at (the right version of) the user manual?
Yes

A small reproducible example (--anon can help anonymise data)

I created my CSV data file and inadvertently missed a ',' after the payee name in record 2.
CSV data:

2021-12-22,garo_bike ;  ebici,expenses:trasporto:ebici,-50.0
2021-12-23,caffe_siciliaexpenses:cibo:dolce,-10.5
2021-12-27,simone ; pane:simone,expenses:cibo:pane,-4.8

RULES file:

account1 assets:bank:checking
fields date, description, account2, amount

CMD and result:

I then did a --dry-run:
An noticed the strange output in the 2nd posting

$ hledger import --dry-run --rules-file=./inc_exp2018_ods.csv.rules csv_missing_delim.csv
; would import 3 new transactions from csv_missing_delim.csv:

2021-12-22 garo_bike ;  ebici
    assets:bank:checking               -50.0
    expenses:trasporto:ebici            50.0

2021-12-23 caffe_siciliaexpenses:cibo:dolce
    assets:bank:checking              %4
    -10.5                            %-4

2021-12-27 simone ; pane:simone
    assets:bank:checking            -4.8
    expenses:cibo:pane               4.8

The output of hledger --version
1.24, linux-x86_64

What platform you're on
Linux

[agander]

This is the output from hledger import ... --debug 7

ignored_delim_1.txt

[agander]

And this is with --debug 6:

$ hledger import --dry-run --rules-file=./inc_exp2018_ods.csv.rules csv_missing_delim.csv --debug=6      trying reader:"journal"
trying reader:"csv"
using conversion rules file:"./inc_exp2018_ods.csv.rules"
csv rules:
 CsvRules
   { rdirectives = []
   , rcsvfieldindexes =
     [ ( "date", 1 ), ( "description", 2 ), ( "account2", 3 ), ( "amount", 4 ) ]
   , rassignments =
     [
       ( "account1", "assets:bank:checking" )
     ,
       ( "account2", "%3" )
     ,
       ( "amount", "%4" )
     ,
       ( "date", "%1" )
     ,
       ( "description", "%2" )
     ]
   , rconditionalblocks = []
   }
using separator:','
first 3 csv records:
 [
   [ "2021-12-22", "garo_bike ;  ebici", "expenses:trasporto:ebici", "-50.0" ]
 ,
   [ "2021-12-23", "caffe_siciliaexpenses:cibo:dolce", "-10.5" ]
 ,
   [ "2021-12-27", "simone ; pane:simone", "expenses:cibo:pane", "-4.8" ]
 ]
newestfirst:False
mdataseemsnewestfirst:Just False
; would import 3 new transactions from csv_missing_delim.csv:

2021-12-22 garo_bike ;  ebici
    assets:bank:checking               -50.0
    expenses:trasporto:ebici            50.0

2021-12-23 caffe_siciliaexpenses:cibo:dolce
    assets:bank:checking              %4
    -10.5                            %-4

2021-12-27 simone ; pane:simone
    assets:bank:checking            -4.8
    expenses:cibo:pane               4.8

[Xitian9]

I've looked into this a little bit. The defined behaviour is that if there is a missing field, it will be left as a reference to the field position (in this case %4). This is then being interpreted as an amount, hence the somewhat silly %-4 balance.

The amount, on the other hand, is interpreted as the account name: -10.5. This is not a valid account name (they are required to have each part starting with an alphanumeric charcacter), but the parser accepts it for backward compatibility reasons. @simonmichael: Do we still need to maintain backward compatibility here? Can we tighten up our account name parser?

But even tightening the account name parser would not completely solve this problem, as a positive number is a completely valid account name.

[simonmichael]

Thanks for the report @agander.
@Xitian9 yes I see no reason not to tighten up the account parsing there.
And I think it's a bug that %4 is shown there, showing nothing would be more intuitive.

[simonmichael]

@agander, any thoughts on #1814 ?