syslog source not emitting anything?

[vedantroy]

I'm trying to get a simple setup running, where vector echoes my syslog.
Here's what I have:

[sources.generate_syslog]
  type = "syslog" # must be: "syslog"
  mode = "tcp" # example, enum
  address = "127.0.0.1:514"

[transforms.remap_syslog]
inputs = [ "generate_syslog"]
type = "remap"
source = '''
  structured = parse_syslog!(.message)
  . = merge(., structured)
'''

[sinks.emit_syslog]
inputs = ["remap_syslog"]
type = "console"
encoding.codec = "json"

I run vector with:

sudo /home/vedantroy/.vector/bin/vector

Vector outputs the following logs:

2022-10-13T23:46:23.276701Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-10-13T23:46:23.276793Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2022-10-13T23:46:23.279731Z  INFO vector::topology::running: Running healthchecks.
2022-10-13T23:46:23.279986Z  INFO vector: Vector has started. debug="false" version="0.24.1" arch="x86_64" build_id="8935681 2022-09-12"
2022-10-13T23:46:23.280017Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-10-13T23:46:23.280137Z  INFO vector::topology::builder: Healthcheck: Passed.
2022-10-13T23:46:23.280271Z  INFO source{component_kind="source" component_id=generate_syslog component_type=syslog component_name=generate_syslog}: vector::sources::syslog: Listening. addr=127.0.0.1:514 type="udp"

however, there is nothing after that. I.e, there's no actual syslog messages.

In contrast, tail -f /var/log/syslog does seem to output the latest syslog messages.

Output of lsb_release -a:

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal

[jszwedko]

Hi @vedantroy !

What is feeding messages into 127.0.0.1:514 over TCP (e.g. rsyslog, syslog-ng, something else)? Would you be able to share the configuration of that?

[vedantroy]

I'm not sure. I'm just looking to monitor whatever goes into my Syslog, since for specific applications, I'll use a different source that directly reads from stdout/stderr.

I thought the syslog source was a way to get generic messages from any program on your system. Am I misunderstanding how it works?

Here's the output of tail -f /var/syslog if that helps:

Oct 13 20:32:29 vedant-ThinkPad-T580 dbus-daemon[2924]: [session uid=1000 pid=2924] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=2921 comm="/usr/libexec/tracker-miner-fs ")
Oct 13 20:32:29 vedant-ThinkPad-T580 systemd[2905]: Starting Tracker metadata database store and lookup manager...
Oct 13 20:32:29 vedant-ThinkPad-T580 dbus-daemon[2924]: [session uid=1000 pid=2924] Successfully activated service 'org.freedesktop.Tracker1'
Oct 13 20:32:29 vedant-ThinkPad-T580 systemd[2905]: Started Tracker metadata database store and lookup manager.
Oct 13 20:32:29 vedant-ThinkPad-T580 dbus-daemon[2924]: [session uid=1000 pid=2924] Activating service name='org.freedesktop.thumbnails.Cache1' requested by ':1.1' (uid=1000 pid=2921 comm="/usr/libexec/tracker-miner-fs ")
Oct 13 20:32:29 vedant-ThinkPad-T580 org.freedesktop.thumbnails.Cache1[290402]: Registered thumbnailer /usr/bin/gdk-pixbuf-thumbnailer -s %s %u %o
Oct 13 20:32:29 vedant-ThinkPad-T580 org.freedesktop.thumbnails.Cache1[290402]: Registered thumbnailer /usr/bin/gdk-pixbuf-thumbnailer -s %s %u %o
Oct 13 20:32:29 vedant-ThinkPad-T580 org.freedesktop.thumbnails.Cache1[290402]: Registered thumbnailer evince-thumbnailer -s %s %u %o
Oct 13 20:32:29 vedant-ThinkPad-T580 org.freedesktop.thumbnails.Cache1[290402]: Registered thumbnailer gnome-thumbnail-font --size %s %u %o
Oct 13 20:32:29 vedant-ThinkPad-T580 dbus-daemon[2924]: [session uid=1000 pid=2924] Successfully activated service 'org.freedesktop.thumbnails.Cache1'
Oct 13 20:32:36 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[290456]: Initializing nautilus-dropbox 2020.03.04
Oct 13 20:32:36 vedant-ThinkPad-T580 dbus-daemon[822]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.851' (uid=1000 pid=290456 comm="nautilus ")
Oct 13 20:32:36 vedant-ThinkPad-T580 systemd[1]: Starting Hostname Service...
Oct 13 20:32:36 vedant-ThinkPad-T580 dbus-daemon[822]: [system] Successfully activated service 'org.freedesktop.hostname1'
Oct 13 20:32:36 vedant-ThinkPad-T580 systemd[1]: Started Hostname Service.
Oct 13 20:32:43 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[290531]: [290526:290559:1013/203243.912590:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/portal/desktop: org.freedesktop.DBus.Error.InvalidArgs: No such interface “org.freedesktop.portal.FileChooser”
Oct 13 20:32:43 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[290531]: [290526:290559:1013/203243.912646:ERROR:select_file_dialog_linux_portal.cc(242)] Failed to read portal version property
Oct 13 20:32:43 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[290530]: Opening in existing browser session.
Oct 13 20:33:00 vedant-ThinkPad-T580 tracker-store[290371]: OK
Oct 13 20:33:00 vedant-ThinkPad-T580 systemd[2905]: tracker-store.service: Succeeded.
Oct 13 20:33:06 vedant-ThinkPad-T580 systemd[1]: systemd-hostnamed.service: Succeeded.
Oct 13 20:33:50 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[291016]: [291011:291044:1013/203350.594381:ERROR:object_proxy.cc(623)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/portal/desktop: org.freedesktop.DBus.Error.InvalidArgs: No such interface “org.freedesktop.portal.FileChooser”
Oct 13 20:33:50 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[291016]: [291011:291044:1013/203350.594448:ERROR:select_file_dialog_linux_portal.cc(242)] Failed to read portal version property
Oct 13 20:33:50 vedant-ThinkPad-T580 i3-gnome-flashback.desktop[291015]: Opening in existing browser session.
Oct 13 20:35:01 vedant-ThinkPad-T580 CRON[291327]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Oct 13 20:36:10 vedant-ThinkPad-T580 tailscaled[1262]: open-conn-track: timeout opening (TCP 100.107.78.81:49706 => 34.122.121.32:80); no associated peer node
Oct 13 20:36:13 vedant-ThinkPad-T580 tailscaled[1262]: open-conn-track: timeout opening (TCP 100.107.78.81:49706 => 34.122.121.32:80); no associated peer node
Oct 13 20:36:16 vedant-ThinkPad-T580 systemd[2905]: Started VTE child process 291744 launched by gnome-terminal-server process 265044.
Oct 13 20:36:17 vedant-ThinkPad-T580 dbus-daemon[2924]: [session uid=1000 pid=2924] Activating via systemd: service name='org.freedesktop.Tracker1' unit='tracker-store.service' requested by ':1.1' (uid=1000 pid=2921 comm="/usr/libexec/tracker-miner-fs ")
Oct 13 20:36:17 vedant-ThinkPad-T580 systemd[2905]: Starting Tracker metadata database store and lookup manager...
Oct 13 20:36:17 vedant-ThinkPad-T580 systemd[2905]: Started Tracker metadata database store and lookup manager.
Oct 13 20:36:17 vedant-ThinkPad-T580 dbus-daemon[2924]: [session uid=1000 pid=2924] Successfully activated service 'org.freedesktop.Tracker1'
Oct 13 20:36:19 vedant-ThinkPad-T580 systemd[2905]: vte-spawn-f76858fc-ee89-4b9c-be06-c770d923f24b.scope: Succeeded.
Oct 13 20:36:25 vedant-ThinkPad-T580 tailscaled[1262]: open-conn-track: timeout opening (TCP 100.107.78.81:49706 => 34.122.121.32:80); no associated peer node
Oct 13 20:36:48 vedant-ThinkPad-T580 tracker-store[291795]: OK
Oct 13 20:36:48 vedant-ThinkPad-T580 systemd[2905]: tracker-store.service: Succeeded.

In production, I'd use this to monitor things like error logs from NVIDIA drivers. But there would be no specific program that is feeding messages into that port.

[StephenWakely]

The syslog source just listens for messages on the specified tcp or udp port. It won't look in a specified file.

You have two options.

• Configure rsyslog or whichever syslog system you are using on your machine to forward messages to the given port (127.0.0.1:514).
• Use the file source to read from /var/syslog. You can setup a remap transform to take the events from the file source and call parse_syslog on the event to parse the message.

[bmallya]

@jszwedko , sorry to be bringing this thread back into discussion again, but would appreciate your help.
As above, I am trying to ingest all the logs that come to syslog using the following configuration using vector 0.22.0

type = "syslog"
mode = "udp"
max_length = 102400
address = "127.0.0.1:514"
path = "/tmp/syslogSock"

With this configuration, I get

thread 'vector-worker' panicked at 'Failed to bind to UDP listener socket: Os { code: 98, kind: AddrInUse, message: "Address already in use" }', src/sources/syslog.rs:228:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2023-01-18T16:12:33.721076Z  INFO source{component_kind="source" component_id=http_source component_type=http component_name=http_source}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:50332
2023-01-18T16:12:33.722357Z ERROR source{component_kind="source" component_id=syslogd component_type=syslog component_name=syslogd}: vector::topology: An error occurred that Vector couldn't handle.
2023-01-18T16:12:33.722466Z  INFO vector: Vector has stopped.

The part of the rsyslog configuration that is used is as follows:

#rsyslog v3 config file

# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

#### MODULES ####

$ModLoad imuxsock.so    # provides support for local system logging (e.g. via logger command)
$ModLoad imklog.so      # provides kernel logging support (previously done by rklogd)
#$ModLoad immark.so     # provides --MARK-- message capability

# Provides UDP syslog reception
$ModLoad imudp.so
$UDPServerRun 514

Can I ingest the logs/events coming into syslog through vector while also letting syslog continue to do its own thing (if the port is already used?)

[spencergilbert]

Hey @bmallya it sounds like that may be possible with this but I don't know if rsyslog supports that, or what bugs that may cause having two distinct applications listening on the same port.

In practice ports should only be bound once and for a single application.

[bmallya]

Thanks for the quick response @spencergilbert.
I understand that generally the port should be bound to only one application.
How does source syslog work? Should rsyslog not be listening on that port anymore?

[spencergilbert]

Correct, Vector should be able to replace rsyslog.