Gathering data from sources without official Vector source modules

[jerrac]

Hey all,

I'm in the middle of trying to slim down my logs and metrics footprint. Both when it comes to resource usage, and when it comes to maintenance. Part of that is trying to get to only using 1 agent for gathering data. Vector does have the essentials, but there are some data sources that I'd like to use that don't have obvious ways of ingesting via Vector.

I know all of these items can be monitored. I've seen most/all done in other tools.

So, if anyone has figured out how to gather data from any of these sources (maybe via the exec source and some log to metric transforms?), would you be willing to share?

• Docker Metrics
  • This one I'm already trying to get done via the exec source, but as the discussion I posted shows, I'm still not quite getting it right. It'd be nicer if we had an official source that pulled in the data from the api properly.
• systemd unit metrics
  • Includes name, cpu/ram/swap/etc. usage, running/sleeping/etc. status, and so on.
• Postfix metrics
  • What I want to know would be things like how many messages are being processed and what kind of responses are being received? How long does a message stay in the queue? And so on.
  • While it is email related, Proposal: SMTP source #19336 does not have anything like what I'm after.
• mariadb/mysql metrics, including Galera clusters
  • It's been mentioned. New mysql_metrics source #7238
  • Though it's been a while, and there is no mention of Galera.
  • I'd like to know if all my nodes have the same wsrep_cluster_state_uuid, which nodes are in the cluster, how well syncing data is performing, etc.
• RabbitMQ metrics.
  • I see that Vector can read data off AMPQ's like RabbitMQ, but I couldn't find anything about gathering metrics.
  • My workplace is actively using RabbitMQ and will being doing so more as time goes by, so this one is important.
• Memcached metrics.
• Tomcat metrics
• Varnish metrics
• libvirt/kvm/qemu based VM metrics
• Linux Sensors data
  • Related: could you pls add sensors info in host_metrics #21389
• Disk health metrics/logs for disks.
  • I know SMART exists for HDD's, I actually haven't dug into what exists for NVME yet...
  • The goal is to know when a disk needs replacing.
• Netflow
  • I'd like this one as a once in a while thing when networking people need to analyze the data.
  • Related: Network Flow Handling #7386
• File state
  • Basically just periodically records a hash of a file and if the hash changed.
  • Would allow me to alert when something that shouldn't change, does change.
• Data about HTTPS certificates. Like how long until they expire.

(Side note: That's a long list, so I didn't want to post a topic per item. Should I have?)

Thanks in advance!

[pront]

Hi @jerrac, this is an interesting and wide topic.

That's a long list, so I didn't want to post a topic per item. Should I have?

You can create new feature requests (if one doesn't exist) to help us track them better and gauge community interest.