$title CSV2GDX - Testing the Field Separator Tab (CSV2GDX9,SEQ=754)

$onText
The Tab field separator is tested in combination with different decimal separators.

Contributor Jan-Erik Justkowiak, May 2018
$offText

* Define the control Set and Parameter for comparison with gdxdiff
$onEcho > controlData1.gms
Set Dim1 / a, b, c /;
Parameter x / a 1.3, b 2.5, c 3.9 /;

$gdxOut controlData1
$unLoad Dim1 x
$gdxOut
$offEcho
$call gams controlData1.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with controlData1!
*-------------------------------------------------------------------------------

* 1. This should fail, because tabs are replaced by spaces inside GAMS IDE (and in many other editors)
$onEcho > TabsFalseP.csv
a        b       c
1.3      2.5     3.9
$offEcho

$log
$log 1. Tabs are not set correctly
$call csv2gdx TabsFalseP.csv id=x fieldSep=Tab decimalSep=period useHeader=y values=1..3 trace=0
$ifE errorLevel=0 $abort Error: Should have failed. There are no tabs in TabsFalse!
$log Expected Error: Value(s) column number exceeds column count!
$log
*-------------------------------------------------------------------------------

* 2. Tabs are explicitly defined
*    DecimalSep is set correctly.
*    This should succeed!
$onEcho > TabsRightP.csv
a%system.tab%b%system.tab%c
1.3%system.tab%2.5%system.tab%3.9
$offEcho

$log
$log 2. Tab separated <-> fieldSep=Tab, period decimals <-> decimalSep=period
$call csv2gdx TabsRightP.csv id=x fieldSep=Tab decimalSep=period useHeader=y values=1..3 trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded when running 2.!
$call gdxdiff TabsRightP.gdx controlData1.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running 2.!
$log
*-------------------------------------------------------------------------------

* 3. Tabs are explicitly defined.
*    DecimalSep is not set correctly
*    Note that periods are still interpreted as decimal separator!
*    This should succeed!
$log
$log 3. Tab separated <-> fieldSep=Tab, period decimals <-> decimalSep=comma
$call csv2gdx TabsRightP.csv id=x fieldSep=Tab useHeader=y values=1..3 decimalSep=comma trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded when running 3.!
$call gdxdiff TabsRightP.gdx controlData1.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running 3.!
$log
*-------------------------------------------------------------------------------

* 4. Tabs are explicitly defined.
*    DecimalSep is not set correctly
*    The data should not be read succesfully!
$onEcho > TabsRightC.csv
a%system.tab%b%system.tab%c
1,3%system.tab%2,5%system.tab%3,9
$offEcho

$log
$log 4. Tab separated <-> fieldSep=Tab, comma decimals <-> decimalSep=period
$call csv2gdx TabsRightC.csv id=x fieldSep=Tab decimalSep=period useHeader=y values=1..3 trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded when running 4.!
$call gdxdiff TabsRightC.gdx controlData1.gdx > %system.nullfile%
$ifE errorLevel=0 $abort Data should not be correct after running 4.!
$log
*-------------------------------------------------------------------------------

* 5. Tabs are explicitly defined.
*    DecimalSep is set correctly
*    This will succeed!
$log
$log 5. Tab separated <-> fieldSep=Tab, comma decimals <-> decimalSep=comma
$call csv2gdx TabsRightC.csv id=x fieldSep=Tab decimalSep=comma useHeader=y values=1..3 trace=0
$ifE errorLevel<>0 $abort Error: CSV2GDX should have succeeded when running 5.!
$call gdxdiff TabsRightC.gdx controlData1.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running 5.!
$log
*-------------------------------------------------------------------------------

* 6. Tabs are explicitly defined.
*    FieldSep is not set correctly
*    This should fail, since there is no comma in the header row!
$log
$log 6. Tab separated <-> fieldSep=comma
$call csv2gdx TabsRightP.csv id=x fieldSep=comma useHeader=y values=1..3 trace=0
$ifE errorLevel=0 $abort Error: CSV2GDX should have failed after running 6.!
$log Expected Error: Value(s) column number exceeds column count!
$log
*-------------------------------------------------------------------------------

* 7. Tabs are explicitly defined.
*    FieldSep is not set correctly
*    This should fail, since there is no semicolon in the header row!
$log
$log 7. Tab separated <-> fieldSep=semicolon
$call csv2gdx TabsRightP.csv id=x fieldSep=semicolon useHeader=y values=1..3 trace=0
$ifE errorLevel=0 $abort Error: CSV2GDX should have failed after running 7.!
$log Expected Error: Value(s) column number exceeds column count!
$log
*-------------------------------------------------------------------------------

* 8. Testing the fieldSep=tab on nasty data in combination with most of the other options
*    Also note that the field separator is enclosed by quotes in some index columns
*    --> This will result in three "Bad Uel Counts" (with option acceptBadUels=y).
$onEcho > specialValues.csv
one%system.tab%two%system.tab%three%system.tab%four%system.tab%five%system.tab%six
red%system.tab%red%system.tab%%system.tab%Undef%system.tab%'3.3'%system.tab%red
red%system.tab%red%system.tab%"4.4"%system.tab%5.5%system.tab%Eps%system.tab%green
"red %system.tab% 1"%system.tab%'green'%system.tab%7.7e+02%system.tab%8.8°%system.tab%-Inf%system.tab%blue
blue%system.tab%blue%system.tab%"10%"%system.tab%0%system.tab%NA%system.tab%"'purple %system.tab%'"
green%system.tab%blue%system.tab%"Eps"%system.tab%"NA"%system.tab%"Undef"%system.tab%"pur %system.tab% ple"
green%system.tab%blue%system.tab%"0"%system.tab%""%system.tab%"  "%system.tab%'green'
$offEcho

$onEcho > controlData2.gms
Set
   Dim1 / r1*r6 /
   Dim2 / red, green, "BAD_Line_5_Dim_ 1", blue /
   Dim3 / red, "'green'", blue/
   Dim4 / red, green, "'green'", "BAD_Line_6_Dim_ 3", "BAD_Line_7_Dim_ 3", blue /
   Dim5 / c1*c3 /;

Parameter A
$onUNDF
/
r1.red.red.red.c2 Undf
r1.red.red.red.c3 Undf
r2.red.red.green.c1 4.4
r2.red.red.green.c2 5.5
r2.red.red.green.c3 Eps
r3."BAD_Line_5_Dim_ 1"."'green'".blue.c1 770
r3."BAD_Line_5_Dim_ 1"."'green'".blue.c2 Undf
r3."BAD_Line_5_Dim_ 1"."'green'".blue.c3 -Inf
r4.blue.blue."BAD_Line_6_Dim_ 3".c1 Undf
r4.blue.blue."BAD_Line_6_Dim_ 3".c2 Eps
r4.blue.blue."BAD_Line_6_Dim_ 3".c3 NA
r5.green.blue."BAD_Line_7_Dim_ 3".c1 Eps
r5.green.blue."BAD_Line_7_Dim_ 3".c2 NA
r5.green.blue."BAD_Line_7_Dim_ 3".c3 Undf
r6.green.blue."'green'".c1 Eps
/;
$offUNDF

$gdxOut controlData2
$unLoad Dim1 Dim2 Dim3 Dim4 Dim5 A
$gdxOut
$offEcho
$call gams controlData2.gms lo=%GAMS.lo%
$ifE errorLevel<>0 $abort Something went wrong with controlData2!

$log
$log 8.
$call csv2gdx specialValues.csv id=A fieldSep=tab index=1,2,6 values=3..5 colCount=6 autoRow=r autoCol=c valueDim=y useHeader=y storeZero=y acceptBadUels=y trace=0
$ifE errorLevel<>0 $abort Problems reading data with CSV2GDX in 8.!
$call gdxdiff specialValues.gdx controlData2.gdx > %system.nullfile%
$ifE errorLevel<>0 $abort Data is not correct after running 8.!
*-------------------------------------------------------------------------------

* 9. Doing the same as 8. but w/o setting 'acceptBadUels=y'
*    --> This should lead to an error
$log
$log 9.
$call csv2gdx specialValues.csv id=A fieldSep=tab index=1,2,6 values=3..5 colCount=6 autoRow=r autoCol=c valueDim=y useHeader=y storeZero=y trace=0
$ifE errorLevel<>1 $abort Error expected reading data with CSV2GDX in 9.!
*-------------------------------------------------------------------------------