Allow cut on non-existent columns

[tseemann]

% csvtk version
csvtk v0.23.

% cat foo.csv
A,B
1,2
3,4

% csvtk cut -f A,C foo.csv

# THIS HAPPENS
[ERRO] column "C" not existed in file: a.csv

# DESIRED OPTION 1
% csvtk cut --allow-missing-col -f A,C foo.csv
A
1
3

# DESIRED OPTION 2
% csvtk cut --blank-missing-col  -f A,C foo.csv
A,C
1,
3,

This would be usefulf or cleaning large datatsets that I need to merge or get partial versions from different sources.

[shenwei356]

New flags

  -m, --allow-missing-col   allow missing column
  -b, --blank-missing-col   blank missing column

Tests

$ cat foo.csv | csvtk cut -f D,A,C,B -m | csvtk pretty 
A   B
-   -
1   2
3   4

$ cat foo.csv | csvtk cut -f D,A,C,B -m -b | csvtk pretty 
D   A   C   B
-   -   -   -
    1       2
    3       4

$ cat foo.csv | csvtk cut -f 5,2,4,1 -m | csvtk pretty 
B   A
-   -
2   1
4   3

$ cat foo.csv | csvtk cut -f 5,2,4,1 -m -b  
,B,,A
,2,,1
,4,,3

Binaries

• csvtk_darwin_amd64.tar.gz
• csvtk_linux_amd64.tar.gz
• csvtk_darwin_arm64.tar.gz

[tseemann]

Awesome! Thanks @shenwei356 !

[clydeugene]

Hi @shenwei356

I am trying to combine SRA run tables in such a way that the final output will have consistent columns with blank values where there is no information. So, I used the cut command with the -m -b flags, and I was not able to replicate what you did here:

$ cat foo.csv | csvtk cut -f D,A,C,B -m -b | csvtk pretty 
D   A   C   B
-   -   -   -
    1       2
    3       4

Here's my command:

$ csvtk cut -f Run,BioProject,sex,source_name,BioSample,Diet,treatment,cell_type,AGE,strain,tissue SraRunTable.txt -m -b | csvtk pretty
Run           BioProject    source_name         BioSample      treatment   tissue           
-----------   -----------   -----------------   ------------   ---------   -----------------
SRR24897950   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897951   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897952   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897953   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897954   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897955   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897956   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets
SRR24897957   PRJNA982793   Pancreatic islets   SAMN35715211   HAMS        Pancreatic islets

Note: even though the file extension is .txt this is in fact a csv file. It has a .txt extension because another application in the workflow expects .txt and not .csv

From what I can tell the -m flag is working fine but the -b is not.

$ csvtk --version
csvtk v0.33.0

I would appreciate your help with this :)

[shenwei356]

API changed since then, but -b was not adapted. Will check this on Sunday. Need to sleep now.

[shenwei356]

fixed.

$ cat foo.csv | csvtk cut -f D,A,C,B -m  | csvtk pretty 
A   B
-   -
1   2
3   4


$ cat foo.csv | csvtk cut -f D,A,C,B -m -b | csvtk pretty 
D   A   C   B
-   -   -   -
    1       2
    3       4

• csvtk_darwin_arm64.tar.gz
• csvtk_linux_amd64.tar.gz
• csvtk_windows_amd64.exe.tar.gz

[clydeugene]

Thank you @shenwei356

I tested the linux binary and -b is now working as expected. Might you have an estimate of when the updated binaries will be available for installation with pixi? I would want to avoid installing via a pixi task although that's possible now.

[shenwei356]

I'll tag a new release this week.