CS tries to assign to VM IP already assigned to virtual Router

[akrasnov-drv]

problem

I faced a weird issue when CloudStack assigns to VM IP used by vRouter.
I tried removing all VMs, removing vRouter, but it did not help.
My configuration - shared network with separate management and guest subnets
Guest network IPs:

Still on new VM start I'm getting

As you can see on attachments 10.10.126.13 is used by vRouter but still assigned to VM

versions

CloudStack 4.21.0.0
KVM
Ubuntu 24
vRouter Software 4.21.0.0, template 4.20.1
I can provide mysql data on request

The steps to reproduce the bug

1. Configured CS with shared network (dns, dhcp, metadata server) - v4.19
2. Defined guest subnet (no vlan)
3. upgraded to 4.20, then to 4.21 (I have not noticed if already had an isse on previous versions, but can't tell for sure it started just with 4.21)
   3.Faced the issue of VM not getting name and IP from vRouter, and finally found duplicate IP assignment.

What to do about it?

CS should not assign IPs in use. I hardly understand why it happens at all. As you can see CS is aware that the IP is allocated already.

[akrasnov-drv]

I think I see the issue.

After removing all user VMs and also vRouter I see in mysql that IPs are free:

| 113 | 573e095f-ef95-4f52-801e-8a1aa1d8b7de |          1 |         1 | 10.10.126.11      |              1 |          0 | 2025-09-03 15:12:25 |          2 |              0 |  NULL | Allocated |         123 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 114 | 0f17b747-08f5-4b25-a1cf-227bb0549cc5 |       NULL |      NULL | 10.10.126.12      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         124 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 115 | 47c5f17e-9d44-49ef-8355-730d877a0c77 |       NULL |      NULL | 10.10.126.13      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         125 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 116 | 40c92ce1-b9d0-4af8-9817-4ea90ce5a909 |       NULL |      NULL | 10.10.126.14      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         126 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |

I request to create VM, it fires new vRouter deployment.
VM is getting 10.10.126.12, new vRouter -10.10.126.13
But in mysql 10.10.126.13 is still Free, even when the vRouter is fully functional (with Guest IP address 10.10.126.13):

| 114 | 0f17b747-08f5-4b25-a1cf-227bb0549cc5 |          2 |         1 | 10.10.126.12      |              1 |          0 | 2025-09-10 15:51:08 |          2 |              0 |  NULL | Allocated |         124 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 115 | 47c5f17e-9d44-49ef-8355-730d877a0c77 |       NULL |      NULL | 10.10.126.13      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         125 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |

Here is vRouter console at the same time

[weizhouapache]

@akrasnov-drv
Can you share the logs of the job ?

[akrasnov-drv]

job? what do you mean?
logs related to vRouter creation? you suspect that mysql update failed?
The issue is reproducible, I tried several times removing vRouter and VMs and starting them again, each time vRouter got 10.10.126.13, and the 2nd VM tried to use the same IP.
I believe vRouter creation just does not mark IP as allocated. I see no single reason for mysql update to fail (if it's attempted).
Then I get this:

and at the same time

[weizhouapache]

job? what do you mean? logs related to vRouter creation?

yes, you can find the logs of the async job in management-server.log (search for "job-")

you suspect that mysql update failed? The issue is reproducible, I tried several times removing vRouter and VMs and starting them again, each time vRouter got 10.10.126.13, and the 2nd VM tried to use the same IP. I believe vRouter creation just does not mark IP as allocated. I see no single reason for mysql update to fail (if it's attempted). Then I get this:

I could not reproduce the issue with shared networks in advanced zone. The VR/VM IPs are all in Allocated state.

can you share some DB queries ?

select * from user_ip_address where public_ip_address like "10.10.126.1.%";
select * from nics where ip4_address ="10.10.126.13";

[akrasnov-drv]

First of all I want to confirm, I updated manually mysql raw for the IP (set it to allocated), and it fixed the issue.
Here is the log for job for router, if you want more than this, please let me know what filter to use. Full log is quite large.

/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:10,712 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Asking VirtualRouter to prepare for Nic {"broadcastUri":"vlan:\/\/untagged","deviceId":0,"iPv4Address":"10.10.126.13","id":30012,"instanceId":29960,"reservationId":"1bb73151-7a01-43aa-9530-19276c9054fa","uuid":"43dae93a-e114-4be1-88f3-7fd0d9c9902d"}
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:10,730 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-1086211935126425991: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:10,730 DEBUG [c.c.a.t.Request] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-1086211935126425991: Sending  { Cmd , MgmtId: 179113803344805, via: 48(kvm110), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"1e:00:8b:00:00:7d","vmIpAddress":"10.10.126.13","vmName":"dn-cl-tiny-720","defaultRouter":"10.10.127.254","defaultDns":"10.10.126.13","duid":"00:03:00:01:1e:00:8b:00:00:7d","isDefault":"true","executeInSequence":"false","remove":"false","accessDetails":{"router.name":"r-29959-VM","router.guest.ip":"10.10.126.13","router.ip":"169.254.71.88","zone.network.type":"Basic"},"wait":"0","bypassHostMaintenance":"false"}}] }
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:11,591 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-1086211935126425992: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:11,592 DEBUG [c.c.a.t.Request] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-1086211935126425992: Sending  { Cmd , MgmtId: 179113803344805, via: 48(kvm110), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.SavePasswordCommand":{,"vmIpAddress":"10.10.126.13","vmName":"dn-cl-tiny-720","executeInSequence":"false","accessDetails":{"router.name":"r-29959-VM","router.guest.ip":"10.10.126.13","router.ip":"169.254.71.88","zone.network.type":"Basic"},"wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.10.126.13","vmName":"dn-cl-tiny-720","executeInSequence":"false","accessDetails":{"router.name":"r-29959-VM","router.guest.ip":"10.10.126.13","router.ip":"169.254.71.88","zone.network.type":"Basic"},"wait":"0","bypassHostMaintenance":"false"}}] }
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:13,117 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-3149704989392250255: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:26:31,173 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-3149704989392250257: Routed from 179113803344805
/var/log/cloudstack/management/management-server.log:2025-09-11 00:28:25,208 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-119:[ctx-f4fedbbb, job-91161/job-91162, ctx-0423e0ee]) (logid:218f089f) Seq 48-3149704989392250260: Routed from 179113803344805

As there are a lot of IPs I limited the queue, also, as I wrote above, I update the raw, originally it was

mysql> SELECT * FROM user_ip_address where id=115;
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+-----------+------------+----------------+-------+-------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------+---------+------------+--------------+------------+
| id  | uuid                                 | account_id | domain_id | public_ip_address | data_center_id | source_nat | allocated | vlan_db_id | one_to_one_nat | vm_id | state | mac_address | source_network_id | network_id | physical_network_id | is_system | vpc_id | dnat_vmip | is_portable | display | removed | created | rule_state | forsystemvms | for_router |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+-----------+------------+----------------+-------+-------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------+---------+------------+--------------+------------+
| 115 | 47c5f17e-9d44-49ef-8355-730d877a0c77 |       NULL |      NULL | 10.10.126.13      |              1 |          0 | NULL      |          2 |              0 |  NULL | Free  |         125 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL    | NULL    | NULL       |            0 |          0 |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+-----------+------------+----------------+-------+-------+-------------+-------------------+------------+-------------

now it's

mysql> select count(*) from user_ip_address where public_ip_address like "10.10.126.%";
+----------+
| count(*) |
+----------+
|      246 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT * FROM user_ip_address WHERE id BETWEEN 111 AND 120;
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+---------------------+------------+----------------+-------+-----------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------------------+---------+------------+--------------+------------+
| id  | uuid                                 | account_id | domain_id | public_ip_address | data_center_id | source_nat | allocated           | vlan_db_id | one_to_one_nat | vm_id | state     | mac_address | source_network_id | network_id | physical_network_id | is_system | vpc_id | dnat_vmip | is_portable | display | removed             | created | rule_state | forsystemvms | for_router |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+---------------------+------------+----------------+-------+-----------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------------------+---------+------------+--------------+------------+
| 111 | 5effad56-c1e5-45e2-a057-f027ee0670c3 |       NULL |      NULL | 10.10.67.125      |              1 |          0 | NULL                |          1 |              0 |  NULL | Free      |         121 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | 2025-08-19 12:51:25 | NULL    | NULL       |            0 |          0 |
| 112 | 0e6dce6a-0a5b-4bf5-84c0-8ff2f579e8e1 |          1 |         1 | 10.10.126.10      |              1 |          0 | 2025-09-03 15:12:25 |          2 |              0 |  NULL | Allocated |         122 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 113 | 573e095f-ef95-4f52-801e-8a1aa1d8b7de |          1 |         1 | 10.10.126.11      |              1 |          0 | 2025-09-03 15:12:25 |          2 |              0 |  NULL | Allocated |         123 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 114 | 0f17b747-08f5-4b25-a1cf-227bb0549cc5 |          2 |         1 | 10.10.126.12      |              1 |          0 | 2025-09-10 15:51:08 |          2 |              0 |  NULL | Allocated |         124 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 115 | 47c5f17e-9d44-49ef-8355-730d877a0c77 |          2 |         1 | 10.10.126.13      |              1 |          0 | 2025-09-10 15:51:08 |          2 |              0 |  NULL | Allocated |         125 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 116 | 40c92ce1-b9d0-4af8-9817-4ea90ce5a909 |          2 |         1 | 10.10.126.14      |              1 |          0 | 2025-09-11 10:50:42 |          2 |              0 |  NULL | Allocated |         126 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 117 | 198c4873-5ee8-4a35-9b7a-aa91060fd869 |       NULL |      NULL | 10.10.126.15      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         127 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 118 | 73bcdd9c-0e6e-4d06-8e1d-a8efaca69503 |       NULL |      NULL | 10.10.126.16      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         128 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 119 | 4455169d-4af0-4b3c-950a-453847c6cbc8 |       NULL |      NULL | 10.10.126.17      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         129 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
| 120 | 5ffc36a6-913b-46ba-abb2-13229b56cc2c |       NULL |      NULL | 10.10.126.18      |              1 |          0 | NULL                |          2 |              0 |  NULL | Free      |         130 |               204 |       NULL |                 200 |         0 |   NULL | NULL      |           0 |       1 | NULL                | NULL    | NULL       |            0 |          0 |
+-----+--------------------------------------+------------+-----------+-------------------+----------------+------------+---------------------+------------+----------------+-------+-----------+-------------+-------------------+------------+---------------------+-----------+--------+-----------+-------------+---------+---------------------+---------+------------+--------------+------------+
10 rows in set (0.00 sec)


mysql> select * from nics where ip4_address ="10.10.126.13";
+-------+--------------------------------------+-------------+-------------------+--------------+---------------+---------------+---------+-----------------+------------+------+--------------+-------------+---------------------------+--------------------------------------+-----------+-------------+---------------+-------------+-------------+--------------+---------------------+---------------------+-------------+----------+--------------+-------------+------+
| id    | uuid                                 | instance_id | mac_address       | ip4_address  | netmask       | gateway       | ip_type | broadcast_uri   | network_id | mode | state        | strategy    | reserver_name             | reservation_id                       | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type      | created             | removed             | ip6_gateway | ip6_cidr | secondary_ip | display_nic | mtu  |
+-------+--------------------------------------+-------------+-------------------+--------------+---------------+---------------+---------+-----------------+------------+------+--------------+-------------+---------------------------+--------------------------------------+-----------+-------------+---------------+-------------+-------------+--------------+---------------------+---------------------+-------------+----------+--------------+-------------+------+
| 23489 | 75e8bb64-e1cc-4f60-a82f-572eb2b3070d |        NULL | NULL              | 10.10.126.13 | NULL          | NULL          | NULL    | NULL            |        204 | NULL | Reserved     | PlaceHolder | NULL                      | NULL                                 |         0 | NULL        | NULL          | NULL        |           0 | DomainRouter | 2025-08-20 14:17:29 | NULL                | NULL        | NULL     |            0 |           1 | NULL |
| 29825 | be2444f2-cab2-4319-9cab-ae06539f08cd |       29776 | 1e:00:c4:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | 7b6c80cc-bd05-4553-9a4c-e2ebd2561205 |         0 | NULL        | NULL          | NULL        |           1 | DomainRouter | 2025-09-08 12:37:30 | 2025-09-10 14:57:42 | NULL        | NULL     |            0 |           1 | 1500 |
| 29996 | 86a20417-e5c6-48b9-aa6d-bac47e66151e |       29946 | 1e:00:c2:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | d2679db3-2793-4879-8a58-f8bd6a3fa095 |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-10 10:58:35 | 2025-09-10 11:16:19 | NULL        | NULL     |            0 |           1 | NULL |
| 29999 | 33498cbf-bd4c-4079-b3cf-e623bf21eb7c |       29949 | 1e:00:dc:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | 3ad6f1de-0c4b-430a-8a10-32a627da253d |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-10 12:44:45 | 2025-09-10 13:22:05 | NULL        | NULL     |            0 |           1 | NULL |
| 30001 | 3884f72d-32f7-4003-b01d-2a82ad46dac1 |       29951 | 1e:00:51:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | 8771e24f-39ad-48be-bebe-f12e298eb0b0 |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-10 13:46:27 | 2025-09-10 14:20:41 | NULL        | NULL     |            0 |           1 | NULL |
| 30003 | 9d4fdd27-02cb-4d5a-8971-f1fc4f75e000 |       29953 | 1e:00:0e:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | 93ee6e30-8da1-43a6-a88d-e8a29f9e8c64 |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-10 14:28:52 | 2025-09-10 14:51:17 | NULL        | NULL     |            0 |           1 | NULL |
| 30006 | 6d4bc286-06ff-4241-a850-9ec7b27f95e3 |       29956 | 1e:00:bf:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | 10db5b63-0ca6-48c2-9b02-70aa00202ef5 |         0 | NULL        | NULL          | NULL        |           1 | DomainRouter | 2025-09-10 15:00:34 | 2025-09-10 15:39:00 | NULL        | NULL     |            0 |           1 | 1500 |
| 30008 | 8b3fdc44-0ef9-46e3-8829-4ba898e21320 |       29957 | 1e:00:91:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | edbc8002-f19e-4f19-a477-7f86223f1e12 |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-10 15:01:30 | 2025-09-10 15:39:00 | NULL        | NULL     |            0 |           1 | NULL |
| 30010 | a2a033c3-663f-474b-8611-f99467dcf973 |       29959 | 1e:00:50:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Reserved     | Start       | DirectPodBasedNetworkGuru | 3d40d5da-438d-4c0d-8cc6-0688e8cc6d35 |         0 | NULL        | NULL          | NULL        |           1 | DomainRouter | 2025-09-10 15:51:08 | NULL                | NULL        | NULL     |            0 |           1 | 1500 |
| 30012 | 43dae93a-e114-4be1-88f3-7fd0d9c9902d |       29960 | 1e:00:8b:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | 1bb73151-7a01-43aa-9530-19276c9054fa |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-11 00:26:10 | 2025-09-11 00:54:07 | NULL        | NULL     |            0 |           1 | NULL |
| 30014 | 0093ab0e-9a97-4aaa-86dc-1ee81ceb3567 |       29962 | 1e:00:a6:00:00:7d | 10.10.126.13 | 255.255.240.0 | 10.10.127.254 | Ip4     | vlan://untagged |        204 | Dhcp | Deallocating | Start       | DirectPodBasedNetworkGuru | d6637bff-c725-4f2b-8b57-c66fc2dd0673 |         0 | NULL        | NULL          | NULL        |           1 | User         | 2025-09-11 04:44:10 | 2025-09-11 05:12:39 | NULL        | NULL     |            0 |           1 | NULL |
+-------+--------------------------------------+-------------+-------------------+--------------+---------------+---------------+---------+-----------------+------------+------+--------------+-------------+---------------------------+--------------------------------------+-----------+-------------+---------------+-------------+-------------+--------------+---------------------+---------------------+-------------+----------+--------------+-------------+------+
11 rows in set (0.00 sec)

[weizhouapache]

First of all I want to confirm, I updated manually mysql raw for the IP (set it to allocated), and it fixed the issue. Here is the log for job for router, if you want more than this, please let me know what filter to use. Full log is quite large.

@akrasnov-drv
the IP should be updated to Allocated state when the placeholder nic was created (2025-08-20 14:17:29)

any error around the time ?

[akrasnov-drv]

You mean that IP is assigned and should be marked as allocated the first time the router is created, and then just reused?
The only errors I see in the log at that time:

2025-08-20 14:17:28,444 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-1:[ctx-e9b8eac1, job-75131, ctx-29b5c96b]) (logid:f4f64507) Trying to deploy VM [error decoding VM instance {"id":23456,"instanceName":"i-2-23456-VM","state":"Stopped","type":"User","uuid":"3ce19dd4-ddd5-43aa-9de2-b02148608e5d"}] and details: Plan [{"_dcId":1,"_podId":1,"_clusterId":1,"_hostId":15,"_recreateDisks":false,"preferredHostIds":[],"migrationPlan":false,"hostPriorities":{}}]; avoid list [{}] and planner: [].
2025-08-20 14:17:28,449 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-1:[ctx-e9b8eac1, job-75131, ctx-29b5c96b]) (logid:f4f64507) Adding pods [[]], clusters [[]] and hosts [[]] to the avoid list in the deploy process of user VM [error decoding VM instance {"id":23456,"instanceName":"i-2-23456-VM","state":"Stopped","type":"User","uuid":"3ce19dd4-ddd5-43aa-9de2-b02148608e5d"}], because this VM is not explicitly dedicated to these components.
2025-08-20 14:17:29,563 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Trying to deploy VM [error decoding VM instance {"id":23456,"instanceName":"i-2-23456-VM","state":"Starting","type":"User","uuid":"3ce19dd4-ddd5-43aa-9de2-b02148608e5d"}] and details: Plan [{"_dcId":1,"_podId":1,"_clusterId":1,"_hostId":15,"_recreateDisks":false,"preferredHostIds":[],"migrationPlan":false,"hostPriorities":{}}]; avoid list [] and planner: [].
2025-08-20 14:17:29,566 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Trying to deploy VM [error decoding VM instance {"id":23456,"instanceName":"i-2-23456-VM","state":"Starting","type":"User","uuid":"3ce19dd4-ddd5-43aa-9de2-b02148608e5d"}] and details: Plan [error decoding {"_clusterId":1,"_dcId":1,"_physicalNetworkId":null,"_podId":1,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [].
2025-08-20 14:17:29,570 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Adding pods [[]], clusters [[]] and hosts [[]] to the avoid list in the deploy process of user VM [error decoding VM instance {"id":23456,"instanceName":"i-2-23456-VM","state":"Starting","type":"User","uuid":"3ce19dd4-ddd5-43aa-9de2-b02148608e5d"}], because this VM is not explicitly dedicated to these components.
2025-08-20 14:17:29,731 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Trying to deploy VM [error decoding VM instance {"id":23457,"instanceName":"r-23457-VM","state":"Starting","type":"DomainRouter","uuid":"7c4316e2-4709-4264-b8ac-d73f2c6b6d91"}] and details: Plan [error decoding {"_clusterId":null,"_dcId":1,"_physicalNetworkId":null,"_podId":1,"_poolId":null,"migrationPlan":false}]; avoid list [{}] and planner: [].
2025-08-20 14:17:29,738 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Adding pods [[]], clusters [[]] and hosts [[]] to the avoid list in the deploy process of VR VM [error decoding VM instance {"id":23457,"instanceName":"r-23457-VM","state":"Starting","type":"DomainRouter","uuid":"7c4316e2-4709-4264-b8ac-d73f2c6b6d91"}], because this VM is not dedicated to this components.

here are more logs related to the event, let me know if more logs are required

2025-08-20 14:17:28,265 DEBUG [c.c.n.NetworkModelImpl] (qtp1404565079-326:[ctx-c09e1680, ctx-d968c29b, ctx-5de2269d]) (logid:b186334d) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:28,284 DEBUG [c.c.n.NetworkModelImpl] (qtp1404565079-326:[ctx-c09e1680, ctx-d968c29b, ctx-5de2269d]) (logid:b186334d) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:28,337 DEBUG [o.a.c.e.o.NetworkOrchestrator] (qtp1404565079-326:[ctx-c09e1680, ctx-d968c29b, ctx-5de2269d]) (logid:b186334d) Allocating nic for vm VM instance {"id":23456,"instanceName":"i-2-23456-VM","state":"Stopped","type":"User","uuid":"3ce19dd4-ddd5-43aa-9de2-b02148608e5d"} in network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8} with requested profile NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":0,"reservationId":null,"uuid":null,"vmId":0}
2025-08-20 14:17:28,344 DEBUG [c.c.n.NetworkModelImpl] (qtp1404565079-326:[ctx-c09e1680, ctx-d968c29b, ctx-5de2269d]) (logid:b186334d) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:28,391 DEBUG [c.c.n.NetworkModelImpl] (qtp1404565079-326:[ctx-c09e1680, ctx-d968c29b, ctx-5de2269d]) (logid:b186334d) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:28,393 DEBUG [c.c.n.NetworkModelImpl] (qtp1404565079-326:[ctx-c09e1680, ctx-d968c29b, ctx-5de2269d]) (logid:b186334d) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:28,430 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-1:[ctx-e9b8eac1, job-75131, ctx-29b5c96b]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:28,432 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-1:[ctx-e9b8eac1, job-75131, ctx-29b5c96b]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:29,613 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8} is already implemented
2025-08-20 14:17:29,619 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:29,639 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Changing active number of nics for network id=Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8} on 1
2025-08-20 14:17:29,676 DEBUG [c.c.n.r.NetworkHelperImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Adding nic for Virtual Router in Guest network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:29,686 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Allocating nic for vm VM instance {"id":23457,"instanceName":"r-23457-VM","state":"Stopped","type":"DomainRouter","uuid":"7c4316e2-4709-4264-b8ac-d73f2c6b6d91"} in network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8} with requested profile NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":0,"reservationId":null,"uuid":null,"vmId":0}
2025-08-20 14:17:29,692 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:29,722 DEBUG [o.a.c.n.r.d.RouterDeploymentDefinition] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Lock is released for network [id: 204] (Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}) as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(15)-Storage(Volume(23456|ROOT-->Pool(53))]
2025-08-20 14:17:29,795 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:29,807 DEBUG [c.c.n.g.DirectPodBasedNetworkGuru] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Saving placeholder nic with ip4 address 10.10.126.13 for the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:29,815 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:36,187 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Service SecurityGroup is not supported in the network Network {"id": 204, "name": "defaultGuestNetwork", "uuid": "01986b9a-e178-4a8d-ac9b-ebbefbef48b8", "networkofferingid": 8}
2025-08-20 14:17:36,242 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:[ctx-828df247, job-75131/job-75132, ctx-8c19ba15]) (logid:f4f64507) Seq 15-753789987631136791: Sending  { Cmd , MgmtId: 179113803344805, via: 15(kvm130), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":"23457","name":"r-23457-VM","state":"Starting","type":"DomainRouter","cpus":"1","minSpeed":"500","maxSpeed":"500","minRam":"(512.00 MB) 536870912","maxRam":"(512.00 MB) 536870912","arch":"x86_64","os":"Debian GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux 5","bootArgs":" template=domP name=r-23457-VM authorized_key=ZWNkc2Etc2hhMi1uaXN0cDI1NiBBQUFBRTJWalpITmhMWE5vWVRJdGJtbHpkSEF5TlRZQUFBQUlibWx6ZEhBeU5UWUFBQUJCQkJoa3NUUW5WbGkzUlNqREQxb3pGMFVtK2QzMDIwVytZdG0rMW1CUmNLWTNSMnFqMlVlSmNJVDcrT3BDRVlHeHRZOEtJZmVaViszbUtuZC8xdEh3SEZ3PSBjbG91ZEBoMjM5Cg== eth0ip=10.10.126.13 eth0mask=255.255.240.0 gateway=10.10.127.254 privateMtu=1500 domain=cs1cloud.internal cidrsize=20 dhcprange=10.10.112.1 eth1ip=169.254.91.86 eth1mask=255.255.0.0 type=dhcpsrvr disable_rp_filter=true dns1=100.64.15.2 dns2=100.64.15.3 baremetalnotificationsecuritykey=5yfMmhfgd3hTEA6zaOKiUZmaXWrN2D_Er08kFjgXjlM-2dziSG4XNPiN53SE57cC73H1VSlTLBAlkFtHVtXVFg baremetalnotificationapikey=dcg3MUyahz4S8GK8ZRHI0KTetnvLyiqC5H0BPo0h9PPtRJjwnu4ReNdrk3xe8OZ7rujKkAwdknvXzLcFfk-U_w host=10.10.67.1 port=8080 logrotatefrequency=*:00:00","enableHA":"true","limitCpuUse":"false","enableDynamicallyScaleVm":"false","params":{},"uuid":"7c4316e2-4709-4264-b8ac-d73f2c6b6d91","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"2868a901-ac4d-4de2-82c9-b413d9d82a4f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4acb163a-f70c-3d9c-baa7-ea64930f6c94","name":"h239-nfs1","id":"61","poolType":"NetworkFilesystem","host":"10.10.67.1","path":"/var/lib/libvirt/images/cloudstack/storage1","port":"2049","url":"NetworkFilesystem://10.10.67.1/var/lib/libvirt/images/cloudstack/storage1/?ROLE=Primary&STOREUUID=4acb163a-f70c-3d9c-baa7-ea64930f6c94","isManaged":"false"}},"name":"ROOT-23457","size":"(4.88 GB) 5242880000","path":"2868a901-ac4d-4de2-82c9-b413d9d82a4f","volumeId":"23457","vmName":"r-23457-VM","accountId":"1","format":"QCOW2","provisioningType":"THIN","poolId":"61","id":"23457","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"0","path":"2868a901-ac4d-4de2-82c9-b413d9d82a4f","type":"ROOT","_details":{"storageHost":"10.10.67.1","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"(4.88 GB) 5242880000"}}],"nics":[{"deviceId":"0","networkRateMbps":"65535","defaultNic":"true","pxeDisable":"true","nicUuid":"0f3f95e2-46fd-4cd3-8d20-14d6a15166a4","details":{"MacAddressChanges":"true","MacLearning":"false","PromiscuousMode":"false","ForgedTransmits":"true"},"dpdkEnabled":"false","mtu":"1500","networkId":"204","networkSegmentName":"D1-A1-Z1-S204","uuid":"01986b9a-e178-4a8d-ac9b-ebbefbef48b8","ip":"10.10.126.13","netmask":"255.255.240.0","gateway":"10.10.127.254","mac":"1e:00:aa:00:00:7d","dns1":"100.64.15.2","dns2":"100.64.15.3","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":"false"},{"deviceId":"1","networkRateMbps":"-1","defaultNic":"false","pxeDisable":"true","nicUuid":"96b8da56-1f73-4699-addd-c804eff3291e","details":{"MacAddressChanges":"true","MacLearning":"false","PromiscuousMode":"false","ForgedTransmits":"true"},"dpdkEnabled":"false","networkId":"202","networkSegmentName":"D1-A1-Z1-S202","uuid":"f109d794-d960-464e-86fa-43527f204f2d","ip":"169.254.91.86","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:5b:56","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":"false"}],"vcpuMaxLimit":"1","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{},"metadataManufacturer":"Apache Software Foundation","metadataProductName":"CloudStack KVM Hypervisor"},"hostIp":"10.10.67.2","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.91.86","port":"3922","interval":"6","retries":"100","name":"r-23457-VM","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-23457-VM","router.ip":"169.254.91.86"},"wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.NetworkUsageCommand":{"privateIP":"169.254.91.86","domRName":"r-23457-VM","option":"create","forVpc":"false","wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Start","accessDetails":{"router.name":"r-23457-VM","router.ip":"169.254.91.86","router.guest.ip":"10.10.126.13"},"wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.SetMonitorServiceCommand":{"services":[{"id":"0","service":"dhcp","processname":"dnsmasq","serviceName":"dnsmasq","servicePath":"/var/run/dnsmasq/dnsmasq.pid","pidFile":"/var/run/dnsmasq/dnsmasq.pid","isDefault":"false"},{"id":"0","service":"ssh","processname":"sshd","serviceName":"ssh","servicePath":"/var/run/sshd.pid","pidFile":"/var/run/sshd.pid","isDefault":"true"},{"id":"0","service":"webserver","processname":"apache2","serviceName":"apache2","servicePath":"/var/run/apache2/apache2.pid","pidFile":"/var/run/apache2/apache2.pid","isDefault":"true"}],"healthChecksConfig":{"gateways":"gatewaysIps=10.10.127.254 169.254.0.1 ","portForwarding":"","systemThresholds":"minDiskNeeded=100.0,maxCpuUsage=100.0,maxMemoryUsage=100.0;","haproxyData":"","virtualMachines":""},"reconfigureAfterUpdate":"false","deleteFromProcessedCache":"false","accessDetails":{"router.health.checks.enabled":"true","router.guest.ip":"10.10.126.13","router.ip":"169.254.91.86","router.name":"r-23457-VM","router.health.checks.advanced.interval":"10","router.health.checks.basic.interval":"3","router.health.checks.excluded":""},"wait":"0","bypassHostMaintenance":"false"}},{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Finish","accessDetails":{"router.name":"r-23457-VM","router.ip":"169.254.91.86","router.guest.ip":"10.10.126.13"},"wait":"0","bypassHostMaintenance":"false"}}] }

[akrasnov-drv]

I tried luck with AI, and after analyzing CloudStack sources it stated:

The vRouter IP assignment and storage process in shared networks operates completely independently from the user_ip_address table. There is no automatic update to the user_ip_address table's allocation status when a vRouter receives an IP address in a shared network, as these serve different purposes in CloudStack's IP management system.

After additional clarification requests:

Based on my research, no, "Saving placeholder nic" for vRouter also does NOT update the user_ip_address table.
"Saving placeholder nic" Process:
What it does:
Creates a placeholder NIC entry in the nics table
Assigns an IPv4 address to the vRouter's network interface
Reserves the IP address for the vRouter within the shared network
What it does NOT do:
Does NOT update the user_ip_address table
Does NOT change any allocation status in user_ip_address

And from another AI:

What actually touches user_ip_address is the IP allocation path in IpAddressManagerImpl (e.g., when a new IP is needed). In 4.21, assignAndAllocateIpAddressEntry sets fields like allocatedToAccountId, associatedWithNetworkId, vpcId, state, and then calls markPublicIpAsAllocated—that’s the bit that updates the user_ip_address table. 

By contrast, the “placeholder NIC” flow in DirectPodBasedNetworkGuru#getIp() first tries to reuse an IP by reading user_ip_address (_ipAddressDao.findByIpAndSourceNetworkId), and only if no IP is present does it call the allocator. After that it logs “Saving placeholder nic with ip4 address …” and saves a NIC entry for the VR; this step doesn’t modify user_ip_address.

Could you verify that anything in vRouter IP assignment updates user_ip_address (according to AIs nothing)

And thank you for your involvement, very appreciated.

[weizhouapache]

I tried luck with AI, and after analyzing CloudStack sources it stated:

The vRouter IP assignment and storage process in shared networks operates completely independently from the user_ip_address table. There is no automatic update to the user_ip_address table's allocation status when a vRouter receives an IP address in a shared network, as these serve different purposes in CloudStack's IP management system.

After additional clarification requests:

Based on my research, no, "Saving placeholder nic" for vRouter also does NOT update the user_ip_address table.
"Saving placeholder nic" Process:
What it does:
Creates a placeholder NIC entry in the nics table
Assigns an IPv4 address to the vRouter's network interface
Reserves the IP address for the vRouter within the shared network
What it does NOT do:
Does NOT update the user_ip_address table
Does NOT change any allocation status in user_ip_address

And from another AI:

What actually touches user_ip_address is the IP allocation path in IpAddressManagerImpl (e.g., when a new IP is needed). In 4.21, assignAndAllocateIpAddressEntry sets fields like allocatedToAccountId, associatedWithNetworkId, vpcId, state, and then calls markPublicIpAsAllocated—that’s the bit that updates the user_ip_address table. 

By contrast, the “placeholder NIC” flow in DirectPodBasedNetworkGuru#getIp() first tries to reuse an IP by reading user_ip_address (_ipAddressDao.findByIpAndSourceNetworkId), and only if no IP is present does it call the allocator. After that it logs “Saving placeholder nic with ip4 address …” and saves a NIC entry for the VR; this step doesn’t modify user_ip_address.

Could you verify that anything in vRouter IP assignment updates user_ip_address (according to AIs nothing)

And thank you for your involvement, very appreciated.

@akrasnov-drv
saving placeholder nic does not update user_ip_address. however, right before it, ACS allocates the IP to the VR and update user_ip_address.

so, we can see two nics created at same time

mysql> select id,instance_id,mac_address,ip4_address,mode,state,strategy,created from nics where id>=45 and id <=46;
+----+-------------+-------------------+----------------+------+--------------+-------------+---------------------+
| id | instance_id | mac_address       | ip4_address    | mode | state        | strategy    | created             |
+----+-------------+-------------------+----------------+------+--------------+-------------+---------------------+
| 45 |        NULL | NULL              | 172.16.100.100 | NULL | Reserved     | PlaceHolder | 2025-09-11 10:00:33 |
| 46 |          24 | 1e:00:68:00:00:29 | 172.16.100.100 | Dhcp | Deallocating | Create      | 2025-09-11 10:00:33 |
+----+-------------+-------------------+----------------+------+--------------+-------------+---------------------+
2 rows in set (0.00 sec)

the Public IP is also Allocated at the same time

mysql> select id,uuid,account_id,domain_id,public_ip_address,allocated,state from user_ip_address where public_ip_address like "172.16.100.100";
+----+--------------------------------------+------------+-----------+-------------------+---------------------+-----------+
| id | uuid                                 | account_id | domain_id | public_ip_address | allocated           | state     |
+----+--------------------------------------+------------+-----------+-------------------+---------------------+-----------+
| 21 | 2c46b121-3cce-4257-b0a9-702464b9a99f |          1 |         1 | 172.16.100.100    | 2025-09-11 10:00:33 | Allocated |
+----+--------------------------------------+------------+-----------+-------------------+---------------------+-----------+
1 row in set (0.01 sec)

[weizhouapache]

You mean that IP is assigned and should be marked as allocated the first time the router is created, and then just reused?

yes @akrasnov-drv

do you use Basic zone ?
I do not find the nic with same IP for Domain Router after placeholder nic (id=23489)

[akrasnov-drv]

Yes, sorry, I though I wrote it.
It's basic shared network (DefaultSharedNetworkOffering).
Guest IPs are directly routable from the outside networks. I do not have public IPs in CS.
Beside guest traffic type I have just management.