Why doesn't this write json to kafka?

[rocketraman]

I'm not sure why this doesn't write json to kafka? We basically take our nice structured log output, and destructure it.

[gavinmcnair]

Agree completely. Looking into solutions for this now.

[rocketraman]

Any updates on this? Thanks!

[gavinmcnair]

My solution was ugly

Put it back into JSON.

https://github.com/gavinmcnair/logspout-kafka

[OneCricketeer]

Could just format the template to do that yourself

KAFKA_TEMPLATE="{\"time\":\"{{.Time}}\",\"container_name\":\"{{.Container.Name}}\",\"source\":\"{{.Source}}\",\"data\":\"{{.Data}}\"}"

Or more simply

KAFKA_TEMPLATE="{\"msg\":\"{{.Data}}\"}"

[rocketraman]

Could just format the template to do that yourself

This is quite likely to result in invalid JSON output, given that it does no escaping whatsoever of the data fields.

[OneCricketeer]

Probably a way to escape it. Not too familiar with Go templates.

https://golang.org/pkg/text/template/

[rocketraman]

Well, yes, I mean, that is the obvious workaround. The point of the issue, however, is why we need to do:

JSON -> destructured -> JSON

when we could just output the original JSON in the first place.

[dylanmei]

We seem to be struggling with the way Logspout is designed: applying templates to Docker log objects.

Interestingly, the stock raw adapter does have a toJSON helper to aid with extracting values from the object from within the template. https://github.com/gliderlabs/logspout#raw-format

[OneCricketeer]

If the message is already JSON, then I think you would just do KAFKA_TEMPLATE="{{.Data}}". Otherwise, if you need to get the other fields of the container, source, and time, then you need a workaround or use the raw formatting.