Skip to content

Commit 1e74980

Browse files
kirill-levitskiykirill-levitskiy
authored
New: Write CSV attachment from JSON action (#56)
* Add "Write CSV attachment from JSON Array" action * Add "Write CSV attachment from JSON Object" action * Update sailor version to 2.6.5
1 parent 81ba614 commit 1e74980

File tree

9 files changed

+638
-30
lines changed

9 files changed

+638
-30
lines changed

‎CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
## 2.1.0 (May 7, 2020)
2+
3+
* Add "Write CSV attachment from Array" action
4+
* Add "Write CSV attachment from JSON" action
5+
* Update sailor version to 2.6.5
6+
17
## 2.0.2 (December 24, 2019)
28

39
* Update sailor version to 2.5.4

‎README.md

Lines changed: 90 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,6 @@ a `JSON` object. To configure this action the following fields can be used:
6060

6161
![image](https://user-images.githubusercontent.com/40201204/60706373-fda1a380-9f11-11e9-8b5a-2acd2df33a87.png)
6262

63-
6463
### Write CSV attachment
6564

6665
* `Include Header` - this select configures output behavior of the component. If option is `Yes` or no value chosen than header of csv file will be written to attachment, this is default behavior. If value `No` selected than csv header will be omitted from attachment.
@@ -100,8 +99,98 @@ The output of the CSV Write component will be a message with an attachment. In
10099
order to access this attachment, the component following the CSV Write must be
101100
able to handle file attachments.
102101

102+
### Write CSV attachment from JSON Object
103+
104+
* `Include Header` - this select configures output behavior of the component. If option is `Yes` or no value chosen than header of csv file will be written to attachment, this is default behavior. If value `No` selected than csv header will be omitted from attachment.
105+
* `Separator` - this select configures type of CSV delimiter in an output file. There are next options: `Comma (,)`, `Semicolon (;)`, `Space ( )`, `Tab (\t)`.
106+
107+
This action will combine multiple incoming events into a CSV file until there is a gap
108+
of more than 10 seconds between events. Afterwards, the CSV file will be closed
109+
and attached to the outgoing message.
110+
111+
This action will convert an incoming array into a CSV file by following approach:
112+
113+
* Header inherits names of keys from the input message;
114+
* Payload will store data from Values of relevant Keys (Columns);
115+
* Undefined values of a JSON Object won't be joined to result set (`{ key: undefined }`);
116+
* False values of a JSON Object will be represented as empty string (`{ key: false }` => `""`).
117+
118+
Requirements:
119+
120+
* The inbound message is an JSON Object;
121+
* This JSON object has plain structure without nested levels (structured types `objects` and `arrays` are not supported as values). Only primitive types are supported: `strings`, `numbers`, `booleans` and `null`. Otherwise, the error message will be thrown: `Inbound message should be a plain Object. At least one of entries is not a primitive type`.
122+
123+
The keys of an input JSON will be published as the header in the first row. For each incoming
124+
event, the value for each header will be `stringified` and written as the value
125+
for that cell. All other properties will be ignored. For example, headers
126+
`foo,bar` along with the following JSON events:
127+
128+
```
129+
{"foo":"myfoo", "bar":"mybar"}
130+
{"foo":"myfoo", "bar":[1,2]}
131+
{"bar":"mybar", "baz":"mybaz"}
132+
```
133+
134+
will produce the following `.csv` file:
135+
136+
```
137+
foo,bar
138+
myfoo,mybar
139+
myfoo,"[1,2]"
140+
,mybar
141+
```
142+
143+
The output of the CSV Write component will be a message with an attachment. In
144+
order to access this attachment, the component following the CSV Write must be
145+
able to handle file attachments.
146+
147+
### Write CSV attachment from JSON Array
148+
149+
* `Include Header` - this select configures output behavior of the component. If option is `Yes` or no value chosen than header of csv file will be written to attachment, this is default behavior. If value `No` selected than csv header will be omitted from attachment.
150+
* `Separator` - this select configures type of CSV delimiter in an output file. There are next options: `Comma (,)`, `Semicolon (;)`, `Space ( )`, `Tab (\t)`.
151+
152+
This action will convert an incoming array into a CSV file by following approach:
153+
154+
* Header inherits names of keys from the input message;
155+
* Payload will store data from Values of relevant Keys (Columns);
156+
* Undefined values of a JSON Object won't be joined to result set (`{ key: undefined }`);
157+
* False values of a JSON Object will be represented as empty string (`{ key: false }` => `""`).
158+
159+
Requirements:
160+
161+
* The inbound message is an JSON Array of Objects with identical structure;
162+
* Each JSON object has plain structure without nested levels (structured types `objects` and `arrays` are not supported as values). Only primitive types are supported: `strings`, `numbers`, `booleans` and `null`. Otherwise, the error message will be thrown: `Inbound message should be a plain Object. At least one of entries is not a primitive type`.
163+
164+
The keys of an input JSON will be published as the header in the first row. For each incoming
165+
event, the value for each header will be `stringified` and written as the value
166+
for that cell. All other properties will be ignored. For example, headers
167+
`foo,bar` along with the following JSON events:
168+
169+
```
170+
[
171+
{"foo":"myfoo", "bar":"mybar"}
172+
{"foo":"myfoo", "bar":[1,2]}
173+
{"bar":"mybar", "baz":"mybaz"}
174+
]
175+
```
176+
177+
will produce the following `.csv` file:
178+
179+
```
180+
foo,bar
181+
myfoo,mybar
182+
myfoo2,[1,2]"
183+
,mybar
184+
```
185+
186+
The output of the CSV Write component will be a message with an attachment. In
187+
order to access this attachment, the component following the CSV Write must be
188+
able to handle file attachments.
189+
103190
### Limitations
104191

192+
#### General
193+
105194
1. You may get `Component run out of memory and terminated.` error during run-time, that means that component needs more memory, please add
106195
`EIO_REQUIRED_RAM_MB` environment variable with an appropriate value (e.g. value `512` means that 512 MB will be allocated) for the component in this case.
107196
2. You may get `Error: write after end` error, as a current workaround try increase value of environment variable: `TIMEOUT_BETWEEN_EVENTS`.

‎component.json

Lines changed: 104 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,15 @@
22
"title": "CSV",
33
"description": "A comma-separated values (CSV) file stores tabular data (numbers and text) in plain-text form",
44
"docsUrl": "https://github.com/elasticio/csv-component",
5-
"buildType" : "docker",
5+
"buildType": "docker",
66
"triggers": {
77
"read": {
88
"main": "./lib/triggers/read.js",
99
"title": "Read CSV file from URL",
10+
"help": {
11+
"description": "Fetch a CSV file from a given URL and store it in the attachment storage.",
12+
"link": "/components/csv/index.html#read-csv-file-from-url"
13+
},
1014
"type": "polling",
1115
"fields": {
1216
"url": {
@@ -25,10 +29,14 @@
2529
}
2630
}
2731
},
28-
"actions" : {
32+
"actions": {
2933
"read_action": {
3034
"main": "./lib/triggers/read.js",
3135
"title": "Read CSV attachment",
36+
"help": {
37+
"description": "Read a CSV attachment of an incoming message.",
38+
"link": "/components/csv/index.html#read-csv-attachment"
39+
},
3240
"fields": {
3341
"emitAll": {
3442
"label": "Emit all messages",
@@ -44,19 +52,21 @@
4452
}
4553
},
4654
"write_attachment": {
47-
"description":
48-
"Multiple incoming events can be combined into one CSV file with the write CSV action. See https://github.com/elasticio/csv-component/ for additional documentation.",
4955
"main": "./lib/actions/write.js",
5056
"title": "Write CSV attachment",
57+
"help": {
58+
"description": "Multiple incoming events can be combined into one CSV file with the write CSV action.",
59+
"link": "/components/csv/index.html#write-csv-attachment"
60+
},
5161
"fields": {
5262
"includeHeaders": {
53-
"label" : "Include Headers",
63+
"label": "Include Headers",
5464
"required": false,
55-
"viewClass" : "SelectView",
56-
"description" : "Default Yes",
65+
"viewClass": "SelectView",
66+
"description": "Default Yes",
5767
"model": {
58-
"Yes" : "Yes",
59-
"No" : "No"
68+
"Yes": "Yes",
69+
"No": "No"
6070
},
6171
"prompt": "Include headers? Default Yes."
6272
},
@@ -66,11 +76,93 @@
6676
},
6777
"metadata": {
6878
"in": {
69-
"type": "object",
70-
"properties": {}
79+
"type": "object",
80+
"properties": {}
81+
},
82+
"out": {}
83+
}
84+
},
85+
"write_attachment_from_json": {
86+
"main": "./lib/actions/writeFromJson.js",
87+
"title": "Write CSV attachment from JSON Object",
88+
"help": {
89+
"description": "Multiple incoming events can be combined into one CSV file with the write CSV action.",
90+
"link": "/components/csv/index.html#write-csv-attachment-from-json"
91+
},
92+
"fields": {
93+
"includeHeaders": {
94+
"label": "Include Headers",
95+
"required": true,
96+
"viewClass": "SelectView",
97+
"description": "Default Yes",
98+
"model": {
99+
"Yes": "Yes",
100+
"No": "No"
101+
},
102+
"prompt": "Include headers? Default Yes"
103+
},
104+
"separator": {
105+
"label": "Separators",
106+
"required": true,
107+
"viewClass": "SelectView",
108+
"description": "Default Yes",
109+
"model": {
110+
"comma": "Comma (,)",
111+
"semicolon": "Semicolon (;)",
112+
"space": "Space ( )",
113+
"tab": "Tab (\\t)"
114+
},
115+
"prompt": "Choose required CSV delimiter"
116+
}
117+
},
118+
"metadata": {
119+
"in": {
120+
"type": "object",
121+
"properties": {}
122+
},
123+
"out": {}
124+
}
125+
},
126+
"write_attachment_from_array": {
127+
"main": "./lib/actions/writeFromArray.js",
128+
"title": "Write CSV attachment from JSON Array",
129+
"help": {
130+
"description": "Incoming array can be converted into one CSV file with the write CSV action.",
131+
"link": "/components/csv/index.html#write-csv-attachment-from-array"
132+
},
133+
"fields": {
134+
"includeHeaders": {
135+
"label": "Include Headers",
136+
"required": true,
137+
"viewClass": "SelectView",
138+
"description": "Default Yes",
139+
"model": {
140+
"Yes": "Yes",
141+
"No": "No"
142+
},
143+
"prompt": "Include headers? Default Yes"
144+
},
145+
"separator": {
146+
"label": "Separators",
147+
"required": true,
148+
"viewClass": "SelectView",
149+
"description": "Default Yes",
150+
"model": {
151+
"comma": "Comma (,)",
152+
"semicolon": "Semicolon (;)",
153+
"space": "Space ( )",
154+
"tab": "Tab (\\t)"
155+
},
156+
"prompt": "Choose required CSV delimiter"
157+
}
158+
},
159+
"metadata": {
160+
"in": {
161+
"type": "array",
162+
"properties": {}
71163
},
72164
"out": {}
73165
}
74166
}
75167
}
76-
}
168+
}

‎lib/actions/writeFromArray.js

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
const axios = require('axios');
2+
const csv = require('csv');
3+
const _ = require('lodash');
4+
const { messages } = require('elasticio-node');
5+
const client = require('elasticio-rest-node')();
6+
const logger = require('@elastic.io/component-logger')();
7+
8+
const util = require('../util/util');
9+
10+
const REQUEST_TIMEOUT = process.env.REQUEST_TIMEOUT || 10000; // 10s
11+
const REQUEST_MAX_RETRY = process.env.REQUEST_MAX_RETRY || 7;
12+
const REQUEST_RETRY_DELAY = process.env.REQUEST_RETRY_DELAY || 7000; // 7s
13+
const REQUEST_MAX_CONTENT_LENGTH = process.env.REQUEST_MAX_CONTENT_LENGTH || 10485760; // 10MB
14+
15+
let stringifier;
16+
let signedUrl;
17+
let rowCount = 0;
18+
let ax;
19+
let putUrl;
20+
let options;
21+
22+
async function init(cfg) {
23+
let delimiter;
24+
switch (cfg.separator) {
25+
case 'comma': {
26+
delimiter = ',';
27+
break;
28+
}
29+
case 'semicolon': {
30+
delimiter = ';';
31+
break;
32+
}
33+
case 'space': {
34+
delimiter = ' ';
35+
break;
36+
}
37+
case 'tab': {
38+
delimiter = '\t';
39+
break;
40+
}
41+
default: {
42+
throw Error(`Unexpected separator type: ${cfg.separator}`);
43+
}
44+
}
45+
const header = cfg.includeHeaders !== 'No';
46+
logger.trace('Using delimiter: \'%s\'', delimiter);
47+
options = {
48+
header,
49+
delimiter,
50+
};
51+
52+
stringifier = csv.stringify(options);
53+
signedUrl = await client.resources.storage.createSignedUrl();
54+
putUrl = signedUrl.put_url;
55+
logger.trace('CSV file to be uploaded file to uri=%s', putUrl);
56+
ax = axios.create();
57+
util.addRetryCountInterceptorToAxios(ax);
58+
}
59+
async function ProcessAction(msg) {
60+
// eslint-disable-next-line consistent-this
61+
const self = this;
62+
let isError = false;
63+
let errorValue = '';
64+
65+
const columns = Object.keys(msg.body[0]);
66+
rowCount = msg.body.length;
67+
logger.trace('Configured column names:', columns);
68+
let row = {};
69+
70+
await _.each(msg.body, async (item) => {
71+
const entries = Object.values(msg.body);
72+
// eslint-disable-next-line no-restricted-syntax
73+
for (const entry of entries) {
74+
if (isError) {
75+
break;
76+
}
77+
const values = Object.values(entry);
78+
// eslint-disable-next-line no-restricted-syntax
79+
for (const value of values) {
80+
if (value !== null && value !== undefined && (typeof value === 'object' || Array.isArray(value))) {
81+
isError = true;
82+
errorValue = value;
83+
break;
84+
}
85+
}
86+
}
87+
row = _.pick(item, columns);
88+
await stringifier.write(row);
89+
});
90+
self.logger.info('The resulting CSV file contains %s rows', rowCount);
91+
92+
if (isError) {
93+
throw Error(`Inbound message should be a plain Object. At least one of entries is not a primitive type: ${JSON.stringify(errorValue)}`);
94+
}
95+
96+
ax.put(putUrl, stringifier, {
97+
method: 'PUT',
98+
timeout: REQUEST_TIMEOUT,
99+
retry: REQUEST_MAX_RETRY,
100+
delay: REQUEST_RETRY_DELAY,
101+
maxContentLength: REQUEST_MAX_CONTENT_LENGTH,
102+
});
103+
stringifier.end();
104+
105+
const messageToEmit = messages.newMessageWithBody({
106+
rowCount,
107+
});
108+
const fileName = `${messageToEmit.id}.csv`;
109+
messageToEmit.attachments[fileName] = {
110+
'content-type': 'text/csv',
111+
url: signedUrl.get_url,
112+
};
113+
self.logger.trace('Emitting message %j', messageToEmit);
114+
await self.emit('data', messageToEmit);
115+
await self.emit('end');
116+
}
117+
118+
exports.process = ProcessAction;
119+
exports.init = init;

0 commit comments

Comments
 (0)