General documentation improvements

[deajan]

Hello Sergey,

Currently deploying postfix_exporter 2.0.5 on a couple of Debian, RHEL and SuSE machines.

I have a hard time understanding your documentation.
What's the difference between postfix_status_replies_total and postfix_smtp_replies_total ?

Also, how can I filter incomming and outgoing traffic, knowing that subprogram lmtp/smtp maybe both used for outgoing traffic ?

Thank you for any clarifications.

[sergeymakinen]

Hi @deajan,

Yeah... probably options mentioned in https://github.com/sergeymakinen/postfix_exporter/blob/main/CONFIGURATION.md should have a more clear explanation "what it is" before describing configuration options.

Okay:

• postfix_smtp_replies_total allows to account 'something happened, not going to bother with that request right now' log lines from smtp
• postfix_status_replies_total captures status changes as seen from smtp/lmtp

As for separating traffic, honestly I don't know, in my personal setup lmtp/smtp subprogram names alone are enough to distinguish a direction.
You may want to share in/out line examples, so I could take a look.

[deajan]

There's a lot I don't really understand with the postfix_exporter configuration. I've actually read the CONFIGURATION.md before asking this.
As far as I understand, status_replies implies that there is a DSN, and smtp replies don't ?

Also, I'd need a bit of advice for how to setup multiple status sent:
For instance, enhanced code 2.6.0 is also sent, how to I "add" this to -statuses: send ?

Could I perhaps require a review of my postfix_exporter.yml file ? It's an improved version of what I did last year:

status_replies:
  - statuses:
    - sent
    regexp: 2\.0\.0
    match: enhanced_code
    text: sent
  - regexp: 2\.6\.0
    match: enhanced_code
    text: sent but maybe in spam folder
  - regexp: (?i)out of storage
    text: remote storage full
  - regexp: (?i)temporary (local )?problem|internal resource
    text: remote conf problem
  - regexp: (?i)spf|dkim|dns
    text: local conf problem
  - regexp: (?i)recipient address|mailbox (is )?unavailable
    text: remote user not existing
  - regexp: (?i)connection time(d)? out|timeout|unreachable
    text: remote not reachable
  - regexp: (?i)reputation|blacklist
    text: bad reputation
  - type: queue_status
    regexp: (?i)OK \(.+\)
    text: sent
  - regexp: invalid DMARC record.*domain=(.+?);
    text: domain $1 has invalid DMARC record

smtp_replies:
  - regexp: (?i)gr(a|e)ylist
    text: greylisting

noqueue_reject_replies:
  - regexp: (?i)client host rejected|cannot find your hostname|rejected by SPF
    text: spam host
  - regexp: (?i)recipient address rejected|Recipient address not found|User unknown
    text: user not existing
  - regexp: (?i)Message size exceeds fixed limit
    text: message size too big
# The following regexes are precise, gather sender host, sender mail and recipient, but would create lots of entries - Only to be enabled for diag purposes because it will fill way too much different metrics
#  - regexp: (i?)NOQUEUE(.+)RCPT from (.+): [0-9][0-9][0-9](.+)(.+)from=<?(.+?)>?,? to=<?(.+?)>?,? proto=
#    text: message from $6 to $7 not accepted from host $3

Thank you.

[sergeymakinen]

To be honest, I struggle to understand how I could exactly update docs to make them easier to understand but I'd like to. Let me know if you have any ideas.

As far as I understand, status_replies implies that there is a DSN, and smtp replies don't ?

Not exactly, status replies contain statuses as provided by Postfix, I suppose they are queue statuses. Forget my prior assumption about 'something happened, not going to bother with that request right now', I'm not sure I'm right, so let's keep it simple: status replies for replies from smtp with Postfix statuses, smtp replies for other smtp replies.

For instance, enhanced code 2.6.0 is also sent, how to I "add" this to -statuses: send ?

You don't need to add it into an existing condition, just add a new condition, since all those YAML sequences imply OR, so it's going to test matches until some is found:

status_replies:
  - statuses:
      - sent
    regexp: 2\.0\.0
    match: enhanced_code
  - regexp: 2\.6\.0
    match: enhanced_code

As for the config file, it looks sane from the first sight. It made me think it might be worth to add a CLI flag to run the exporter in a test mode with a file, so it will try to parse the whole file, not just start tailing from the end. Maybe it could be useful to debug configurations based on partial/redacted logs.

[deajan]

You don't need to add it into an existing condition, just add a new condition, since all those YAML sequences imply OR, so it's going to test matches until some is found:

Yes, but the point is getting "sent" and "sent but in spam" text

As for the config file, it looks sane from the first sight. It made me think it might be worth to add a CLI flag to run the exporter in a test mode with a file, so it will try to parse the whole file, not just start tailing from the end. Maybe it could be useful to debug configurations based on partial/redacted logs.

Sergey, this is a brilliant idea, which would really help alot in improving the log parsing.
I'd suggest to ad a --manual-parse option which would just parse "everything it can read" in journald or in a config file, and just spit out to stdout the metrics.
Not being forced to wait for logs to happen and being able to just grep around without any http suff would be greatly welcome.

I'll actually wait until you release this one to improve my regexes (which I'll happily make a PR on your repo if they prove worthy).

[sergeymakinen]

Yes, but the point is getting "sent" and "sent but in spam" text

Have I missed something? I guess your config has both:

status_replies:
  - statuses:
    - sent
    regexp: 2\.0\.0
    match: enhanced_code
    text: sent
  - regexp: 2\.6\.0
    match: enhanced_code
    text: sent but maybe in spam folder

I'd suggest to ad a --manual-parse option which would just parse "everything it can read" in journald or in a config file, and just spit out to stdout the metrics.

Yeah, sounds good, I think I'll get to it at the beginning of March.

[deajan]

Any update for the --manual-parse option ? This would be very helpful in writing the rules ;)

[sergeymakinen]

Yeah… allocated some time this weekend, it hopefully shouldn’t take long.

[deajan]

Thank you !

[sergeymakinen]

@deajan you might want to grab https://github.com/sergeymakinen/postfix_exporter/releases/tag/v2.1.0-beta.0 and test a new test mode. :)

[deajan]

Currently testing. Giving feedback as soon as I got the logs I needed ^^ Thank you.

[sergeymakinen]

Let's keep it open for now.