[Bug] device_tracker only added when Integration is reloaded

[Zarox666]

Describe the issue

When a device, lets say my mobile phone, is connecting to another AP no device_tracker is added.
The existing device_tracker turns to "not_home." When the Integration for that AP is reloaded while the phone is connected to another AP, it turns to "unavailable".
Only when I reload the Integration for the AP the phone is now connected with, a second device_tracker will be added.
When the device reverts back to the first AP, the device_tracker of the second AP will show not_home and the first still "unavailable".
With a second reload of the integration, the first AP then shows the presence of the phone.
But I if reload the intergatrion for the second AP then that one goes to "unavailable".

Is it even neccessary, sensible to create multiple device_trackers for the same device?

How to reproduce the issue

Get two Mikrotik APs that provide the same WLAN and roam between them.

Expected behavior

Devices registering with an AP should get a device_tracker and the device_tracker should get updated, without reloading the Integration.

Screenshots

see above

Software versions

Home Assistant 2023.10.0
Supervisor 2023.10.0
Operating System 10.5
Frontend 20231002.0 - latest
Mikrotik Router integration version: v.2.1.4
RouterOS version: both Routers in this test run v6.49.7 (stable) but I have a 3rd AP running v7.6 that is also affected.

Diagnostics data

Could not find anything in the logs that seems to be related.

Traceback/Error logs

see above

Additional context

This has worked at some point in the past, as I did not have to connect my phone (or other devices) to a certain AP (by switching of the other APs) and then reload the integration in order to get all 3 possible device_trackers into my config, when I originally set up the integration. This must have been prior to v.2.1.3.
However I cannot say, when this stopped working.
Originally the device trackers were showing as device_tracker.name_of_the_device. And the second and third as device_tracker.name_of_the_device_2 and device_tracker.name_of_the_device_3 respectively.
Since the Upgrade of the Mikrotik Integration to Version 2.1.3 device_trackers now show as device_tracker.MAC_Address.
This broke all my automations concerned about device trackers and I had to reconfigure everything with the new entity_ids.
For that it seems I am not the only one: #301
But in my Case its a bunch of Shelly Devices and my Android OS mobile Phone with the Companion App that was affected.

In essence the current state would mean, that I have to bring all possible devices into my network. Switch off 2 of the APs, reload the Integration for the 3rd one, then switch that off and the next one on and repeat the same there.

Generally I am wondering if multiple device_trackers for the same device provided by multiple APs are neccessary.
From a (laymans) logic perspective a single device_tracker per device and than an information in the Attribute which AP reported the Status would be good.

Please let me know if anything is unclear or further information required.

[tomaae]

If you have multiple AP on same network, you will get tracker per each, no way going around it as they are different devices and cannot affect each other on HA level.
Way to go here is probably using capsman? not sure, I dont use mikrotik wifi.
#301 was something different outside of integration control.

Are you waiting long enough for integration to get data when adding new device?
If so, I will need to see debug log capturing new device being added and waiting for at least 2 cycles.

[Zarox666]

I can live with one device_tracker per AP per device as long as that works reliably and doesnt change entity_ids.

Well this morning I brought my mobile down to the basement, disable / enabled wifi and let it sit there.
I went upstairs, verified in the APs Registration list, that the Device shows up there and made myself a coffee.
approx 10 minutes later I checked in HA and the second device_tracker did not show up. I reloaded the integration for the AP in the basement and immediately the new device tracker showed.
Reloading the integration then for the other AP on the groundfloor cause the corresponding device_tracker to become unavailable and afterwards, even if the device is connected to it, it stays in the "unavailable" state, even after hours.

I have a rescan intervall of 30 Seconds and devices are considered timeout after 180 seconds. So 10 Minutes should have been plenty of time.

I got you a debug logging, just quick and dirty from right now.

Tracing started at 14:56 (all times on the full minute)
Enabled WIFI on Mobile at 14:57 while directly next to the AP.
Stopped tracing at 15:04
Afterwards device_tracker.28_c2_1f_61_16_de provided by VHB-WLAN-1 still showed "unavailable"

See attached
home-assistant_mikrotik_router_2023-10-09T13-04-01.104Z.log

I looked through the log file and noticed two things (not sure if they are relevant):

1. The MAC address 28_c2_1f_61_16_de does not show up at all (independent how you write it).
   When looking at a trace of reloading the integration (GoodCase) I at least see a line
   "Add entity device_tracker.28_c2_1f_61_16_de_2"

2. According to the Registration table the Mobile is connected to VHB-WLAN-1-wlan2 (5Ghz Interface) and its in fact the only device connected to this interface.
   In the Log I see a device connected to this interface after a Query for /interface/wireless/registration-table at 14:57:19.
   2023-10-09 14:57:19.555 DEBUG (SyncWorker_4) [custom_components.mikrotik_router.apiparser] Processing entry {'.id': '*1F', 'interface': 'VHB-WLAN-1-wlan2', 'mac-address': '**REDACTED**', 'ap': False, 'wds': False, 'bridge': False, 'rx-rate': '243Mbps-40MHz/2S', 'tx-rate': '65Mbps
   So I would say the AP has correctly reported the registration of this new Wifi-Client.

But after doing some further API Queries it finaly says
Finished fetching VHB-WLAN-1 data in 5.170 seconds (success: True)
and that is that.

Hope this helps!

[Zarox666]

any news?

[Zarox666]

I took the liberty and looked at the code with my limitied coding skills and developed a theory on what goes wrong:

The AP reports the Device with its MAC Address.... This MAC Address is allways the same.
However if another AP has added a Device tracker with "device_tracker.MAC-address" allready, the entity ID that would need to be added is "device_tracker.MAC-address_2". I think this function comes to the incorrect conclusion that the device_tracker allready exists as it just tests for existance of "device_tracker.MAC-address".

from device_tracker.py @line 56

        async def async_check_exist(obj, coordinator, uid: None) -> None:
            """Check entity exists."""
            entity_registry = er.async_get(hass)
            if uid:
                unique_id = f"{obj._inst.lower()}-{obj.entity_description.key}-{slugify(str(obj._data[obj.entity_description.data_reference]).lower())}"
            else:
                unique_id = f"{obj._inst.lower()}-{obj.entity_description.key}"

            entity_id = entity_registry.async_get_entity_id(
                platform.domain, DOMAIN, unique_id
            )
            entity = entity_registry.async_get(entity_id)
            if entity is None or (
                (entity_id not in platform.entities) and (entity.disabled is False)
            ):
                _LOGGER.debug("Add entity %s", entity_id)
                await platform.async_add_entities([obj])

Just a theory.. not sure if I am correct.
But I hope it helps... the problem is pretty annoying.

[Zarox666]

ok Scratch that... I added some further Debug Output to the function and I only see it beeing executed on reload.
Furthermore the unique_id contains the name of the AP so its indeed unique.

So it seems on a normal refresh that funtion is not even executed!?

[tomaae]

that part of code is there register entities and avoid duplicates. its part of startup process.

This is a tough one, specially since we cannot pair devices using debug. We will most likely have to remove masking and redo debugs again. We need to identify the device in wireless registration table and track it down to ping. Specially to make sure IP address is being correctly updated for that device.
It would be helpful if you could put some comment in mikrotik on device you are testing it with, for easier identification.

If you are willing to do that, you can remove IP and MAC related entries inside of TO_REDACT hash in const.py file. That should be first 11 entries.

[Zarox666]

I am not sure I am getting what you are suggesting.

The repro is dead simple and even works with a single Mikrotik AP:

1. Test
   -- Connect a new Device to the Wifi
   -- Wait for the device tracker to show up in HA (it doesn't) <- first problem
   -- Reload the Mikrotik Router Integration
   -- Now the Device tracker was created and shows up as "home"

2. Test (after the first test or for devices with allready existing device tracker)
   -- disable WIFI on the Device (aka your phone)
   -- device tracker will show as "away" <-- correct
   -- reload the Mikrotik Router Integration
   -- device tracker shows as "unavailable" <--- arguably correct!? (imho it should show as "away")
   -- enable WIFI on the Device
   -- device tracker stays "unavailable" <-- second problem
   -- reload Mikrotik Router integration
   -- device tracker shows "home" <-- correct

So in essence new device_trackers are only added when the integration is reloaded while at the same time the device is connected to this specific AP.
Device trackers are lost (state "unavailable") while the integration is relaoded when the device IS NOT connected to this specific AP.

I think solving the first issue would also solve the second.
Logic wise I would expect something along the lines of comparing the list of registrations reported from the Mikrotik API vs. the list of device trackers that have been created.
IF there is a device registered that does not have a device tracker, create that device tracker.
IF there is a device registered that has a device tracker, set the state of that tracker to "home".
IF there is a device tracker and the corresponding device is not registered, set the device tracker to "away".

On the other hand...
When reloading the Integration, it does what it should!
The problem is that during normal Updates it seems to do things differently and thus not correctly refreshing/creating the device trackers.
So quick-and-dirty-fix could be, to just let the Code do (in relation to device trackers) the same that it does on a reload but just for the 30 Second update/refresh.

[Zarox666]

Is there any ETA on this?

I could also live with an automated way to reload the integration from HA, but despite many people asking for it, I could not find a way to achive this.

[ivanjx]

this is how i reload the mikrotik integration using HA:

but it would be far better for the device tracker list to be reloaded automatically when the mikrotik dhcp clients are changed because reloading the integration takes a couple of seconds in my case and every single entities are marked as unavailable at the meantime.

[Zarox666]

@ivanjx

I resorted to the same approach to just reload the entire integration, via an automation, when certain conditions are true (such as the number of Devices at home beeing not plausible).
But in light of [Bug] Integration causing constant flash writes on queried device
I wonder if that is such a good idea.
I do indeed see my Write Counters on all 3 APs go up, when reloading the corresponding Integration.....
And I just had my trusty old Core Switch/AP die on me with disk issues.