transformHeader renaming and ignoring not working as expected

[cjrobbertse-ob]

I was trying to filter out unwanted columns and rename relevant columns with transformHeader, but I could only achieve one or the other.

For example, using the following config, I was able to rename the headers that are in my headerMap, but I HAD to return header as well. as seen below.

const headerMap = {
  'Code Set': 'codeSet',
  'Code Name': 'codeName',
  'Code Value': 'codeValue',
}

const config = {
    header: true,
    skipEmptyLines: true,
    transformHeader: header => {
      if (header in headerMap) {
        return headerMap[header]
      }
      return header
    }
  }

output (as expected):

 Data: [
  {
    {codeSet: 'something'},
    {codeName: 'something else'},
    {codeValue: 'more things'},
    {description: 'some description'},
  },
    etc. etc.
]

However, when i tried to ignore the non-relevant columns by returning null when not found in my header map, the output renamed all headers to null. Even when debugging it goes into the positive if block, but still renames everything to null.

const config = {
    header: true,
    skipEmptyLines: true,
    transformHeader: header => {
      if (header in headerMap) {
        return headerMap[header]
      }
      return null
    }
  }

output looked something like:

 Data: [
  {
    {null: ''},
    {null: ''},
    {null: ''},
    {null: ''},
  },
    etc. etc.
]

Notably, when using an array of headers instead of a map and not renaming the values, the ignoring with null worked correctly.

[dboskovic]

I'm looking at transformHeader issues. Can you explain the last note. I haven't been able to write a test that verifies this behavior. You're saying the following scenario works for you somehow?

{
		description: "transformHeader returning null should filter out columns",
		input: 'A,B,C,D\r\na,b,c,d',
		config: {
			header: ['A', 'B', 'C', 'D'],
			transformHeader: function(header) {
				if (header === 'A' || header === 'C') {
					return header.toLowerCase();
				}
				return null;
			}
		},
		expected: {
			data: [{"a": "a", "c": "c"}],
			errors: []
		}
},

[cjrobbertse-ob]

Hi @dboskovic,

Not exactly. It only (kind of) works if you return header without the .toLowerCase(). Just returning header produces something like:

{"A": "a", "B": "b", null: ''}

Does that make sense?

My ideal solution would be to remove all the unwanted columns from the table - using the config. But I'm not sure how, other than reducing all unwanted columns to null. And then I can't rename the headers at the same time (using this method). Maybe there is a better way?

[dboskovic]

can you share what version you're using? The latest version has some other fatal errors when these null keys are present.

[dboskovic]

Can you take a look at the replication code I wrote in #1101 and PR a revised version if I got it wrong?