Skip to content

Commit 06f3991

Browse files
jhorbulykzuker
authored andcommitted
Make CSV write dynamic (#14)
* Add write view. * Add in.data. * Add debug statement. * More debug. * Fix bug. * Add documentation.
1 parent 16469a9 commit 06f3991

File tree

3 files changed

+44
-34
lines changed

3 files changed

+44
-34
lines changed

README.md

Lines changed: 35 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,40 @@
11
# csv-component
22

3-
> A CSV component for the [elastic.io platform](http://www.elastic.io "elastic.io platform").
3+
A CSV component for the [elastic.io platform](http://www.elastic.io "elastic.io platform").
4+
5+
6+
## Write CSV attachment
7+
8+
Multiple incoming events can be combined into one CSV file with the write CSV
9+
action. Incoming events will be written into the CSV file until there is a gap
10+
of more than 10 seconds between events. As part of the component setup, one
11+
must specify the columns of the CSV file. These columns will be published as
12+
the header in the first row. For each incoming event, the value for each header
13+
will be stringified and written as the value for that cell. All other
14+
properties will be ignored. For example, headers ``foo,bar`` along with the
15+
following JSON events
16+
17+
{"foo":"myfoo", "bar":"mybar"}
18+
{"foo":"myfoo", "bar":[1,2]}
19+
{"bar":"mybar", "baz":"mybaz"}
20+
21+
will produce the following ``.csv`` file:
22+
23+
foo,bar
24+
myfoo,mybar
25+
myfoo,"[1,2]"
26+
,mybar
27+
28+
When columns are added in the UI, you will be presented with an opportunity to
29+
provide a JSONata expression per column. If you require number formatting that
30+
is specific to a locale, the JSONata expression should handle that concern.
31+
32+
![screenshot from 2017-10-17 09-28-04](https://user-images.githubusercontent.com/5710732/31651871-926b4530-b31d-11e7-936f-bcf3ff05f8e2.png)
33+
34+
The output of the CSV Write component will be a message with an attachment. In
35+
order to access this attachment, the component following the CSV Write must be
36+
able to handle file attachments such as the [SFTP
37+
component](https://github.com/elasticio/sftp-component).
438

539
This is an open source component to work with
640
[CSV](http://en.wikipedia.org/wiki/Comma-separated_values) files in your
@@ -58,32 +92,3 @@ $ git push elasticio master
5892
Obviously the naming of your team and repository is entirely up-to you and if
5993
you do not put any corresponding naming our system will auto generate it for you
6094
but the naming might not entirely correspond to your project requirements.
61-
62-
## Write CSV attachment
63-
64-
Multiple incoming events can be combined into one CSV file with the write CSV
65-
action. Incoming events will be written into the CSV file until there is a gap
66-
of more than 10 seconds between events. As part of the component setup, one
67-
must specify the columns of the CSV file. These columns will be published as
68-
the header in the first row. For each incoming event, the value for each header
69-
will be stringified and written as the value for that cell. All other
70-
properties will be ignored. For example, headers ``foo,bar`` along with the
71-
following JSON events
72-
73-
{"foo":"myfoo", "bar":"mybar"}
74-
{"foo":"myfoo", "bar":[1,2]}
75-
{"bar":"mybar", "baz":"mybaz"}
76-
77-
will produce the following ``.csv`` file:
78-
79-
foo,bar
80-
myfoo,mybar
81-
myfoo,"[1,2]"
82-
,mybar
83-
84-
Currently no formatting is performed as part of CSV Write.
85-
86-
The output of the CSV Write component will be a message with an attachment. In
87-
order to access this attachment, the component following the CSV Write must be
88-
able to handle file attachments such as the [SFTP
89-
component](https://github.com/elasticio/sftp-component).

component.json

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,14 @@
4545
"title": "Write CSV attachment",
4646
"fields": {
4747
"writer": {
48-
"viewClass": "CSVReadView"
48+
"viewClass": "CSVWriteView"
4949
}
5050
},
5151
"metadata": {
52-
"in": {},
52+
"in": {
53+
"type": "object",
54+
"properties": {}
55+
},
5356
"out": {}
5457
}
5558
}

lib/actions/write.js

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -76,11 +76,13 @@ exports.process = function ProcessAction(msg, cfg) {
7676
});
7777
}, TIMEOUT_BETWEEN_EVENTS);
7878

79-
let row = msg.body;
79+
let row = msg.body.writer;
80+
console.log(`Incoming data: ${JSON.stringify(row)}`);
8081
if (cfg.writer.columns) {
8182
const columns = Object.keys(_.keyBy(cfg.writer.columns, 'property'));
82-
row = _.pick(msg.body, columns);
83+
row = _.pick(row, columns);
8384
}
85+
console.log(`Writing Row: ${JSON.stringify(row)}`);
8486
stringifier.write(row);
8587
rowCount++;
8688
this.emit('end');

0 commit comments

Comments
 (0)