Vlan interfaces not able to be used

[Bdaman80]

Hello. I first hit the issue mentioned in this other one . I changed the Vlans creation to give them names like "vlan3" instead of "ixl0.3". But I'm still getting an issue where if ngbuddy is enabled to start at boot that it actually prevents the vlans from being created then fails to pull them in to setup Netgraph links and bridges as expected. If I have ngbuddy disabled at boot, let the OS come up normally and setup the vlans, THEN manually start ngbuddy it works fine, creates everything as expected with no issues.

Any ideas where to start to look for what is causing them to not all be picked up? Or how to force a wait until the interfaces exist before creating bridges?

Running FreeBSD 14.3

###################################################
Example of my rc.conf networking section.

`# Wired WAN
ifconfig_igc0="up"

Base Network

ifconfig_ixl0="inet 192.168.2.16/24"
defaultrouter="192.168.2.1"

VLans

vlans_ixl0="vlan3 vlan4 vlan5 vlan6 vlan7 vlan1024 vlan2046"

create_args_vlan3="vlan 3"
create_args_vlan4="vlan 4"
create_args_vlan5="vlan 5"
create_args_vlan6="vlan 6"
create_args_vlan7="vlan 7"
create_args_vlan1024="vlan 1024"
create_args_vlan2046="vlan 2046"

ifconfig_vlan3="inet 192.168.3.16/24"
ifconfig_vlan4="inet 192.168.4.16/24"
ifconfig_vlan5="inet 192.168.5.16/24"
ifconfig_vlan6="inet 192.168.6.16/24"
ifconfig_vlan7="inet 192.168.7.16/24"
ifconfig_vlan1024="inet 192.168.57.16/24"
ifconfig_vlan2046="inet 192.168.56.16/24"

NG Buddy Networking

ngbuddy_enable="YES"

ngbuddy_wan1_if="igc0"

ngbuddy_vnet2_if="ixl0"

ngbuddy_vnet3_if="vlan3"
ngbuddy_vnet4_if="vlan4"
ngbuddy_vnet5_if="vlan5"
ngbuddy_vnet6_if="vlan6"
ngbuddy_vnet7_if="vlan7"
ngbuddy_vnet1024_if="vlan1024"
ngbuddy_vnet2046_if="vlan2046"`

The Network interfaces are created in boot and work properly from the host.

# ifconfig -l ixl0 ixl1 igc0 igc1 lo0 vlan3 vlan4 vlan5 vlan6 vlan7 vlan1024 vlan2046

[bellhyve]

For rc timing, I had to pick before or after the interface serup+cloning step. Ideally I’d put it immediately after the cloning step in netif, but I don’t think it’s possible, so I put it before so the ifconfig_<ngbuddy_eifaces> would be processed correctly.

If I put it after the netif step, that fixes this bug, but then the interface configuration won’t happen.

I think the best solution might be to have ngbuddy run after netif, but also ingest netif functions and have ngbuddy call those ifconfig_ configuration functions for eifaces generated on start. Hmm actually that might have some cool side effects. Should be easy. I’ll cook up a PR as soon as I can.

Let me know if you have any other ideas for this.

[Bdaman80]

Since at the moment I'm only using this for Bhyve guests, I may throw in a basic script to enable then start ngbuddy that is kicked off using the vm-bhyve config "prestart" in the first vm that I boot.

I wont be able to test again till next weekend, but to you think putting the ngbuddy section after the interface cloning and creating args in rc.conf. I believe it does the actions in order from top to bottom on boot.

[siladi]

I just got hit by this. I make use of the public bridge on vlan with both bhyve and vnet jails connected to it. It leaves the network stack for the vlan interface in a blocked state (igb driver) on reboot when it is configured enabled in rc.conf.

The reliable solution I use at this moment to avoid this:

• moved all ngbuddy , vm-bhyve and jail into rc.conf.local with all three services enabled
• add disable sysrc calls at the bottom of rc.shutdown for all three services

sysrc -f /etc/rc.conf.local ngbuddy_enable="NO"
sysrc -f /etc/rc.conf.local vm_enable="NO"
sysrc -f /etc/rc.conf.local jail_enable="NO"

• add call to a script in rc.local to set things up

sleep 2
sysrc -f /etc/rc.conf.local ngbuddy_enable="YES"
sysrc -f /etc/rc.conf.local vm_enable="YES"
sysrc -f /etc/rc.conf.local jail_enable="YES"

service ngbuddy start
sleep 5
service vm start
sleep 5
service jail start

One thing to note, always use shutdown -r now and never plain reboot command. Keeps things working well.

[Bdaman80]

Thats good info. I'll for sure give that a try on my setup. Thank you for sharing that.

[bellhyve]

Here's my suggested fix:

• Ingest /etc/network.subr to use standard network setup functions
• Move the rcorder after netif but before routing, ensuring that cloned boot up interfaces already exist
• Configure the interface with the base ifn_start() function so ifconfig_XYZ[_ipv6] rcvars will be automatically applied

I put this quick & dirty fix in the dev branch https://github.com/bellhyve/ngbuddy/tree/dev

I'll tidy up the code and add proper shutdowns to the stop commands before pushing to main branch and then to ports. Of course I appreciate testing or suggestions in the meantime. Note that ifconfig_DEFAULT rcvar doesn't work with with additional ifconfig_XYZ rcvars, so mind that gotcha if testing with the official FreeBSD VM images.

CC: @antranigv @rlogwood