Proposal: SMTP source

[candlerb]

Overview

I would like to like to propose an SMTP source: that is, one which accepts log messages over SMTP.

Rationale

Many systems send system status reports over E-mail: from automated package updates, from cron jobs, from monitoring daemons like smartmond and mdadm, and so forth. In fact, I would argue that most systems have no user E-mail handling at all, and the only E-mails they generate are these system messages.

E-mail is a terrible medium for these messages. Many people don't want them or filter them out, and even when you want them, delivery is unreliable. Messages often get classified as spam, given that they have default sender addresses like root@server1.example.com, and the content doesn't look like it's human written (because it isn't).

I want to get these messages directly into an observability pipeline, so they can be stored and analyzed centrally and automatically. The ideal way to do this is to configure all systems (which don't handle E-mail for humans) with a "smarthost" setting so that all outbound SMTP connections go directly to the logging system.

This is completely transparent to the sending system, and requires nothing more than what you'd need for E-mail delivery anyway: either a standard MTA with its own queue (e.g. postfix), or a stateless SMTP forwarder (e.g. ssmtp, msmtp).

Proposal

Vector.dev gains an SMTP listener on port 25, 465 and/or 587.

Each incoming message is treated as a log.

The envelope (MAIL FROM and RCPT TO) are taken as-is and put into message metadata (aside: note that RCPT TO can be multi-valued). No further processing is done on these fields - there is no email routing! The source_ip and HELO/EHLO hostname would also be recorded, much like the syslog source.

The DATA section is split into headers and body; the body is taken as the "message", and the headers are additional metadata.

A future option might be to parse MIME multipart messages, but for now, keeping the entire E-mail body as-is works for me.

In principle, message bodies whose charset is not UTF-8 ought to be transcoded to UTF-8. Without this, there's a risk of invalid UTF-8 codepoints in the body. Similarly, binary attachments are normally base64-encoded, but it's possible that binary messages are sent as-is using the 8BITMIME extension. (AFAICS, Vector requires logs to be UTF-8)

For safety, maybe there should be a max_length setting, which can be advertised using the SIZE SMTP extension.

Security and authentication

This is much like syslog: in the first instance, the receiver can be firewalled to accepted SMTP connections from trusted sender IPs only.

However, SMTP does have various standard security mechanisms which can be used:

• Client authentication using SASL or LOGIN (normally used on port 587)
• TLS, using either STARTTLS on port 25/587 or immediately on connection on port 465. The certificate identity can be used for authorization, in conjunction with SASL EXTERNAL

Alternatives I've considered

The MTA on the sending system could be replaced with a custom /usr/bin/sendmail which talks another protocol that Vector already has a source for, e.g. http_server, syslog, or the vector protobuf protocol. I think this is messier, since it involves greater changes at the sending system than just pointing the smarthost at a Vector instance.

It would be possible to run a central MTA (e.g. postfix) and write a plugin, for example which gets exec()d on each new message, and submits it to Vector. That's not unreasonable, and not a bad way to make a prototype, but it's an extra moving part in the observability pipeline, and has performance impact as MTAs typically write and sync each message to disk.

It should be possible to get an MTA to write to a mbox file, and for Vector to read it using the file source with a suitable multiline delimiter pattern (^From ...). That also has an extra moving part, plus a mailbox that needs to be periodically trimmed.

[jszwedko]

Hi @candlerb !

This is pretty interesting! I think it would be a good candidate for when we have a plugin system for components (something we keep thinking about). I don't think it'd see enough usage to have it in the Vector tree, maintained by us, but certainly it could be an interesting plugin to have in the ecosystem.

[lublak]

@jszwedko Hopefully it's okay to reply here.
General interest in a smtp data source and sink would also be on my side. There is no plugin system. Would a pull request for smtp development be accepted? Then I would like to try to implement it :)

[jszwedko]

cc/ @pront for thoughts here.

As a workaround you could setup an SMTP server that receives mail and emits it in a format Vector already supports (e.g. writing it to a file).

[lublak]

@jszwedko that would of course also be a possibility. possibly even as an “Exec” function. I'll have to see what's available as a smtp server.

[pront]

I would happy to review a PR introducing an SMTP integration.