Skip to content

Commit 7a4d92a

Browse files
if0sif0s
authored
Improves (#16)
1 parent 89dd70c commit 7a4d92a

File tree

9 files changed

+497
-52
lines changed

9 files changed

+497
-52
lines changed

‎.circleci/build_slug.sh

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎.circleci/config.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,8 @@ workflows:
118118
jobs:
119119
- build:
120120
name: "Build and publish docker image"
121+
context:
122+
- componentspusher
121123
filters:
122124
branches:
123125
ignore: /.*/

‎CHANGELOG.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
## 3.2.0 (May 02, 2023)
2+
* Added new config fields to `Read CSV attachment` action:
3+
* `Skip empty lines`
4+
* `Comment char`
5+
* Added new config fields to `Create CSV From Message Stream` and `Create CSV From JSON Array` actions:
6+
* `New line delimiter`
7+
* `Escape formulae`
8+
19
## 3.1.6 (December 16, 2022)
210

311
* Update Sailor version to 2.7.1

‎README.md

Lines changed: 67 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -48,14 +48,37 @@ To configure this action the following fields can be used:
4848

4949
#### Config Fields
5050

51-
* `Emit Behavior` - this selector configures output behavior of the component. If the option is `Fetch All` - the component emits an array of messages; `Emit Individually` - the component emits a message per row; `Emit Batch` - component will produce a series of message where each message has an array of max length equal to the `Batch Size`;
51+
* `Emit Behavior` (dropdown, required) - this selector configures output behavior of the component.
52+
* `Fetch All` - the component emits an array of messages;
53+
* `Emit Individually` - the component emits a message per row;
54+
* `Emit Batch` - component will produce a series of message where each message has an array of max length equal to the `Batch Size`;
55+
* `Skip empty lines` (checkbox, optional) - by default, empty lines are parsed if checked they will be skipped
56+
* `Comment char` (string, optional) - if specified, skips lines starting with this string
5257

5358
#### Input Metadata
5459

5560
* `URL` - We will fetch this URL and parse it as CSV file
5661
* `Contains headers` - if true, the first row of parsed data will be interpreted as field names, false by default.
57-
* `Delimiter` - The delimiting character. Leave blank to auto-detect from a list of most common delimiters.
58-
* `Convert Data types` - numeric, date and boolean data will be converted to their type instead of remaining strings, false by default.
62+
* `Delimiter` - The delimiting character. Leave blank to auto-detect from a list of most common delimiters or provide your own
63+
<details><summary>Example</summary>
64+
if you use "$" as Delimiter, this CSV:
65+
66+
```
67+
a$b$c$d
68+
```
69+
70+
can be parsed into this JSON
71+
72+
``` json
73+
{
74+
"column0": "a",
75+
"column1": "b",
76+
"column2": "c",
77+
"column3": "d"
78+
}
79+
```
80+
</details>
81+
* `Convert Data types` - numeric data and boolean data will be converted to their type instead of remaining strings, false by default.
5982
If `Emit Behavior` equals to `Emit Batch` - new field appears: `Batch Size` - max length of array for each message
6083
6184
#### Output Metadata
@@ -75,9 +98,29 @@ and attached to the outgoing message.
7598
7699
#### Config Fields
77100
78-
* `Upload CSV as file to attachments` - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
79-
* `Separator` - A single char used to delimit the CSV file. Default to `,`
80-
* `Column Order` - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
101+
* `Upload CSV as file to attachments` (checkbox, optional) - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
102+
* `Separator` (string, optional) - A single char used to delimit the CSV file. Default to "`,`" but you can set any
103+
<details><summary>Example</summary>
104+
if you use "$" as Delimiter, this CSV:
105+
106+
```
107+
a$b$c$d
108+
```
109+
110+
can be parsed into this JSON
111+
112+
``` json
113+
{
114+
"column0": "a",
115+
"column1": "b",
116+
"column2": "c",
117+
"column3": "d"
118+
}
119+
```
120+
</details>
121+
* `Column Order` (string, optional) - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
122+
* `New line delimiter` (string, optional, defaults to `\r\n`) - The character used to determine newline sequence.
123+
* `Escape formulae` (checkbox, optional) - If checked, field values that begin with `=`, `+`, `-`, `@`, `\t`, or `\r`, will be prepended with a ` ` ` to defend against injection attacks, because Excel and LibreOffice will automatically parse such cells as formulae
81124
82125
#### Input Metadata
83126
@@ -103,9 +146,24 @@ This action will convert an incoming array into a CSV file
103146
104147
#### Config Fields
105148
106-
* `Upload CSV as file to attachments` - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
107-
* `Separator` - A single char used to delimit the CSV file. Default to `,`
108-
* `Column Order` - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic.
149+
* `Upload CSV as file to attachments` (checkbox, optional) - If checked store the generated CSV data as an attachment. If unchecked, place the CSV as a string in the outbound message.
150+
* `Separator` (string, optional) - A single char used to delimit the CSV file. Default to "`,`" but you can set any
151+
<details><summary>Example </summary>
152+
default:
153+
154+
```
155+
a,b,c,d
156+
```
157+
158+
using "`;`" as separator:
159+
160+
```
161+
a;b;c;d
162+
```
163+
</details>
164+
* `Column Order` (string, optional) - A string delimited with the separator indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic. Columns names will be trimmed (removed spaces in beginning and end of column name, for example: 'col 1,col 2 ,col 3, col 4' => ['col 1', 'col 2', 'col 3', 'col 4'])
165+
* `New line delimiter` (string, optional, defaults to `\r\n`) - The character used to determine newline sequence.
166+
* `Escape formulae` (checkbox, optional) - If checked, field values that begin with `=`, `+`, `-`, `@`, `\t`, or `\r`, will be prepended with a ` ` ` to defend against injection attacks, because Excel and LibreOffice will automatically parse such cells as formulae
109167
110168
#### Input Metadata
111169

‎component.json

Lines changed: 64 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"title": "CSV",
33
"description": "A comma-separated values (CSV) file stores tabular data (numbers and text) in plain-text form",
44
"docsUrl": "https://github.com/elasticio/csv-component",
5-
"version": "3.1.6",
5+
"version": "3.2.0",
66
"actions": {
77
"read_action": {
88
"main": "./lib/actions/read.js",
@@ -15,13 +15,32 @@
1515
"emitAll": {
1616
"label": "Emit Behavior",
1717
"required": true,
18+
"order": 80,
1819
"viewClass": "SelectView",
1920
"model": {
2021
"fetchAll": "Fetch All",
2122
"emitIndividually": "Emit Individually",
2223
"emitBatch": "Emit Batch"
2324
},
2425
"prompt": "Select Emit Behavior"
26+
},
27+
"skipEmptyLines": {
28+
"label": "Skip empty lines",
29+
"order": 70,
30+
"required": false,
31+
"viewClass": "CheckBoxView",
32+
"help": {
33+
"description": "By default, empty lines are parsed if checked they will be skipped"
34+
}
35+
},
36+
"comments": {
37+
"label": "Comment char",
38+
"order": 60,
39+
"required": false,
40+
"viewClass": "TextFieldView",
41+
"help": {
42+
"description": "If specified, skips lines starting with this string"
43+
}
2544
}
2645
},
2746
"dynamicMetadata": true
@@ -37,20 +56,42 @@
3756
"uploadToAttachment": {
3857
"label": "Upload CSV as file to attachments",
3958
"viewClass": "CheckBoxView",
59+
"required": false,
60+
"order": 80,
4061
"description": "If checked store the generated CSV data as an attachment",
4162
"prompt": "Include headers? Default Yes."
4263
},
4364
"separator": {
4465
"viewClass": "TextFieldView",
4566
"required": false,
67+
"order": 70,
4668
"label": "Separator",
4769
"note": "A single char used to delimit the CSV file. Default to ','"
4870
},
4971
"order": {
5072
"viewClass": "TextFieldView",
5173
"required": false,
74+
"order": 60,
5275
"label": "Column Order",
5376
"note": "A string delimited with the separator (use same as above) indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic."
77+
},
78+
"newline": {
79+
"label": "New line delimiter",
80+
"order": 50,
81+
"required": false,
82+
"viewClass": "TextFieldView",
83+
"help": {
84+
"description": "The character used to determine newline sequence. defaults to <b style='background-color:LightGray;'>\\r\\n</b>"
85+
}
86+
},
87+
"escapeFormulae": {
88+
"label": "Escape formulae",
89+
"order": 40,
90+
"required": false,
91+
"viewClass": "CheckBoxView",
92+
"help": {
93+
"description": "If selected, field values that begin with <b style='background-color:LightGray;'>=</b>, <b style='background-color:LightGray;'>+</b>, <b style='background-color:LightGray;'>-</b>, <b style='background-color:LightGray;'>@</b>, <b style='background-color:LightGray;'>\\t</b>, or <b style='background-color:LightGray;'>\\r</b>, will be prepended with a <b style='background-color:LightGray;'>'</b> to defend against injection attacks, because Excel and LibreOffice will automatically parse such cells as formula"
94+
}
5495
}
5596
},
5697
"dynamicMetadata": true
@@ -66,20 +107,42 @@
66107
"uploadToAttachment": {
67108
"label": "Upload CSV as file to attachments",
68109
"viewClass": "CheckBoxView",
110+
"required": false,
111+
"order": 80,
69112
"description": "If checked store the generated CSV data as an attachment",
70113
"prompt": "Include headers? Default Yes."
71114
},
72115
"separator": {
73116
"viewClass": "TextFieldView",
74117
"required": false,
118+
"order": 70,
75119
"label": "Separator",
76120
"note": "A single char used to delimit the CSV file. Default to ','"
77121
},
78122
"order": {
79123
"viewClass": "TextFieldView",
80124
"required": false,
125+
"order": 60,
81126
"label": "Column Order",
82127
"note": "A string delimited with the separator (use same as above) indicating which columns & in what order the columns should appear in the resulting file. If omitted, the column order in the resulting file will not be deterministic."
128+
},
129+
"newline": {
130+
"label": "New line delimiter",
131+
"order": 50,
132+
"required": false,
133+
"viewClass": "TextFieldView",
134+
"help": {
135+
"description": "The character used to determine newline sequence. defaults to <b style='background-color:LightGray;'>\\r\\n</b>"
136+
}
137+
},
138+
"escapeFormulae": {
139+
"label": "Escape formulae",
140+
"order": 40,
141+
"required": false,
142+
"viewClass": "CheckBoxView",
143+
"help": {
144+
"description": "If selected, field values that begin with <b style='background-color:LightGray;'>=</b>, <b style='background-color:LightGray;'>+</b>, <b style='background-color:LightGray;'>-</b>, <b style='background-color:LightGray;'>@</b>, <b style='background-color:LightGray;'>\\t</b>, or <b style='background-color:LightGray;'>\\r</b>, will be prepended with a <b style='background-color:LightGray;'>'</b> to defend against injection attacks, because Excel and LibreOffice will automatically parse such cells as formula"
145+
}
83146
}
84147
},
85148
"dynamicMetadata": true

‎lib/actions/read.js

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ async function errHelper(text) {
2828

2929
async function readCSV(msg, cfg) {
3030
const that = this
31-
const emitBehavior = cfg.emitAll;
31+
const { emitAll: emitBehavior, skipEmptyLines, comments } = cfg;
3232
const { body } = msg;
3333

3434
let batchSize;
@@ -63,7 +63,9 @@ async function readCSV(msg, cfg) {
6363
const parseOptions = {
6464
header: body.header,
6565
dynamicTyping: body.dynamicTyping,
66-
delimiter: body.delimiter
66+
delimiter: body.delimiter,
67+
skipEmptyLines,
68+
comments,
6769
}
6870

6971
// if set "Fetch All" create object with results

‎lib/actions/write.js

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,17 +15,28 @@ const formStream = (data) => {
1515
}
1616

1717
const TIMEOUT_BETWEEN_EVENTS = process.env.TIMEOUT_BETWEEN_EVENTS || 10000; // 10s;
18+
const escapeSpecialChars = {
19+
n: '\n', t: '\t', r: '\r', b: '\b', f: '\f', v: '\v'
20+
}
1821

1922
const rawData = []
2023
let timeout
2124

2225
async function proceedData(data, msg, cfg) {
2326
let csvString;
24-
const delimiter = cfg.separator ? cfg.separator : ','
27+
const {
28+
separator,
29+
header,
30+
newline,
31+
escapeFormulae
32+
} = cfg
33+
const delimiter = separator || ','
2534

2635
const unparseOptions = {
27-
header: cfg.header,
28-
delimiter
36+
header,
37+
delimiter,
38+
newline: newline ? newline.replace(/\\[ntrbfv]/gm, (m) => escapeSpecialChars[m.slice(-1)]) : '\r\n',
39+
escapeFormulae
2940
}
3041

3142
if (cfg.order) {

0 commit comments

Comments
 (0)