Problems with wildcards in config path

[StefanSa]

Hi there,
since the upgrade to the last version of vector, i see this wildcards problem.

vector -c /etc/vector/vector.toml -c /etc/vector/modules/sophos-xg/*.toml
error: Found argument '/etc/vector/modules/sophos-xg/sophos-xg-aspam.toml' which wasn't expected, or isn't valid in this context

vector -c /etc/vector/vector.toml -c '/etc/vector/modules/sophos-xg/*.toml'
2022-05-27T13:14:43.445709Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-05-27T13:14:43.446028Z  INFO vector::app: Loading configs. paths=["/etc/vector/modules/sophos-xg/all.toml", "/etc/vector/modules/sophos-xg/sophos-xg-aspam.toml", "/etc/vector/modules/sophos-xg/sophos-xg-atp.toml", "/etc/vector/modules/sophos-xg/sophos-xg-av.toml", "/etc/vector/modules/sophos-xg/sophos-xg-cfilter.toml", "/etc/vector/modules/sophos-xg/sophos-xg-event.toml", "/etc/vector/modules/sophos-xg/sophos-xg-firewall.toml", "/etc/vector/modules/sophos-xg/sophos-xg-idp.toml", "/etc/vector/modules/sophos-xg/sophos-xg-pipeline.toml", "/etc/vector/modules/sophos-xg/sophos-xg-sandbox.toml", "/etc/vector/modules/sophos-xg/sophos-xg-sink.toml", "/etc/vector/modules/sophos-xg/sophos-xg-source.toml", "/etc/vector/modules/sophos-xg/sophos-xg-waf.toml", "/etc/vector/vector.toml"]
2022-05-27T13:14:43.578646Z  WARN vector::config::loading: Transform "route_ip_waf._unmatched" has no consumers
2022-05-27T13:14:43.578683Z  WARN vector::config::loading: Transform "route_sophos_xg._unmatched" has no consumers
2022-05-27T13:14:43.578700Z  WARN vector::config::loading: Transform "route_ip_idp._unmatched" has no consumers
2022-05-27T13:14:43.578711Z  WARN vector::config::loading: Transform "route_ip_fw._unmatched" has no consumers
2022-05-27T13:14:43.578721Z  WARN vector::config::loading: Transform "route_ip_event._unmatched" has no consumers
2022-05-27T13:14:43.578732Z  WARN vector::config::loading: Transform "route_ip_cfl._unmatched" has no consumers
2022-05-27T13:14:43.578745Z  WARN vector::config::loading: Transform "route_ip_av._unmatched" has no consumers
2022-05-27T13:14:43.578757Z  WARN vector::config::loading: Transform "route_ip_atp._unmatched" has no consumers
2022-05-27T13:14:43.578768Z  WARN vector::config::loading: Transform "route_ip_aspam._unmatched" has no consumers
2022-05-27T13:14:44.737208Z  WARN vector::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.
2022-05-27T13:14:44.761214Z  WARN vector::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.
2022-05-27T13:14:44.811173Z ERROR vector::topology: Configuration error. error=Transform "ecs_mapping":

We had not seen this behavior in the previous version.

[spencergilbert]

Without quoting the path, your shell will expand the path before Vector consumes it causing an error. -c takes a single argument and you should avoid shell expansion there.

[StefanSa]

@spencergilbert
Interestingly, this worked with the previous version.
What would be the right way to handle wildcards here, can you please show an example ?

[spencergilbert]

You can see the reasoning and proper handling here

[StefanSa]

@spencergilbert
OK, thanks for the hint, but somehow I'm doing something wrong or don't understand it right because it's soon the weekend.
No matter if i do it this way or that way, i always get the error message that no "consumers" exist.

ector -c /etc/vector/vector.toml -C /etc/vector/modules/sophos-xg
2022-05-27T14:23:50.550620Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-05-27T14:23:50.550835Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml", "/etc/vector/modules/sophos-xg"]
2022-05-27T14:23:50.684884Z  WARN vector::config::loading: Transform "route_sophos_xg._unmatched" has no consumers
2022-05-27T14:23:50.684924Z  WARN vector::config::loading: Transform "route_ip_waf._unmatched" has no consumers
2022-05-27T14:23:50.684946Z  WARN vector::config::loading: Transform "route_ip_idp._unmatched" has no consumers
2022-05-27T14:23:50.684960Z  WARN vector::config::loading: Transform "route_ip_fw._unmatched" has no consumers
2022-05-27T14:23:50.684978Z  WARN vector::config::loading: Transform "route_ip_event._unmatched" has no consumers
2022-05-27T14:23:50.684991Z  WARN vector::config::loading: Transform "route_ip_cfl._unmatched" has no consumers
2022-05-27T14:23:50.685009Z  WARN vector::config::loading: Transform "route_ip_av._unmatched" has no consumers
2022-05-27T14:23:50.685026Z  WARN vector::config::loading: Transform "route_ip_atp._unmatched" has no consumers
2022-05-27T14:23:50.685038Z  WARN vector::config::loading: Transform "route_ip_aspam._unmatched" has no consumers
2022-05-27T14:23:52.208924Z  WARN vector::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.
2022-05-27T14:23:52.240904Z  WARN vector::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.

vector -c "/etc/vector/vector.toml,/etc/vector/modules/sophos-xg/*.toml"
2022-05-27T14:24:27.051856Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,kube=info"
2022-05-27T14:24:27.052194Z  INFO vector::app: Loading configs. paths=["/etc/vector/modules/sophos-xg/all.toml", "/etc/vector/modules/sophos-xg/sophos-xg-aspam.toml", "/etc/vector/modules/sophos-xg/sophos-xg-atp.toml", "/etc/vector/modules/sophos-xg/sophos-xg-av.toml", "/etc/vector/modules/sophos-xg/sophos-xg-cfilter.toml", "/etc/vector/modules/sophos-xg/sophos-xg-event.toml", "/etc/vector/modules/sophos-xg/sophos-xg-firewall.toml", "/etc/vector/modules/sophos-xg/sophos-xg-idp.toml", "/etc/vector/modules/sophos-xg/sophos-xg-pipeline.toml", "/etc/vector/modules/sophos-xg/sophos-xg-sandbox.toml", "/etc/vector/modules/sophos-xg/sophos-xg-sink.toml", "/etc/vector/modules/sophos-xg/sophos-xg-source.toml", "/etc/vector/modules/sophos-xg/sophos-xg-waf.toml", "/etc/vector/vector.toml"]
2022-05-27T14:24:27.187617Z  WARN vector::config::loading: Transform "route_ip_waf._unmatched" has no consumers
2022-05-27T14:24:27.187663Z  WARN vector::config::loading: Transform "route_sophos_xg._unmatched" has no consumers
2022-05-27T14:24:27.187680Z  WARN vector::config::loading: Transform "route_ip_idp._unmatched" has no consumers
2022-05-27T14:24:27.187691Z  WARN vector::config::loading: Transform "route_ip_fw._unmatched" has no consumers
2022-05-27T14:24:27.187702Z  WARN vector::config::loading: Transform "route_ip_event._unmatched" has no consumers
2022-05-27T14:24:27.187712Z  WARN vector::config::loading: Transform "route_ip_cfl._unmatched" has no consumers
2022-05-27T14:24:27.187724Z  WARN vector::config::loading: Transform "route_ip_av._unmatched" has no consumers
2022-05-27T14:24:27.187738Z  WARN vector::config::loading: Transform "route_ip_atp._unmatched" has no consumers
2022-05-27T14:24:27.187751Z  WARN vector::config::loading: Transform "route_ip_aspam._unmatched" has no consumers
2022-05-27T14:24:28.429521Z  WARN vector::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.
2022-05-27T14:24:28.450848Z  WARN vector::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.

@spencergilbert
🙏 please please saving my weekend 😉

[spencergilbert]

Well - first, those are WARNs - not ERRORs so the pipelines should be working just fine. The route transform now includes an _unmatched output.

See the release notes for more details.

[StefanSa]

OK,
i see there is more work to do on the upgrade than i thought. I will try again next week.
Thanks for your help Spencer.