Slice modify with broadcast link does not work properly

[kthare10]

Software issue for https://github.com/RENCI-NRIG/exogeni/issues/223

Scenario:

1. Create a slice with 3 VM instances on one rack connected to a broadcast link
2. Modify the slice by creating another VM instance on a different rack and connect it to existing broadcast link

Observation:

1. neuca_user_data reports the interface on 4th VM but the VM does not have the interface created on it

Expected:

1. interface should be created on the 4th VM

Investigation:
Initial analysis of the logs confirms that ec2/handler.xml join invokes neuca_quantum_add_iface but it fails with following error reported /var/log/orca/handler-vm.log on the AM

Snapshot from VM
[komalthareja@Node3 ~]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr FA:16:3E:16:77:42
inet addr:10.103.0.2 Bcast:10.103.0.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe16:7742/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5204 errors:0 dropped:0 overruns:0 frame:0
TX packets:5084 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1561915 (1.4 MiB) TX bytes:546359 (533.5 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

[komalthareja@Node3 ~]$ neuca-user-data
[global]
actor_id=89e7e930-b803-40e2-bf33-80e9e6f5259f
slice_id=d4061203-7937-472d-8d22-0b5bf4212112
reservation_id=91b7950a-8441-49cd-ad73-9455683bd922
unit_id=78bc5ed0-fd26-42c8-9c78-ff7341371d00
;router= Not Specified
;iscsi_initiator_iqn= Not Specified
slice_name=Komal
unit_url=http://geni-orca.renci.org/owl/97a6df7c-c643-45f9-8f6c-52155399deca#Node3
host_name=Node3
management_ip=162.244.229.76
physical_host=ciena-w1
nova_id=4dd54e13-58f1-422d-989f-f570a6c92247
[users]
root=no:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIxGUVBf24l4gSgUtQQaScP7S604CpXKh66cCMZB1GoXfGqyhRVO1xQUXGA2Oj8MeZf3bo4tjmrPnVeeTVfwTrxkkFNvekwY4QbGX7o8YPNnEFquLWMmkoLn9RFJI47Cj+JHWQN7sEW4WVnmHNITcw5lD3V+yw1bD5M0boUXvh/MnHTu59MEDRyLUyWY+N1FUxHrO0UgSISczRjFS31zF5WY83ssNWq+zxD0NM6GhLWg5Ynzat1J75NRvnMVkuj0VmFcJuHIl3jYCdL9uE7kCw08oh06p/VZBzUIDP6EB0e+H1udu0DvT7SunqBZnobrTCyj1Bma9BJEHPhocIIcPl komalthareja@dhcp152-54-6-178.wireless.europa.renci.org:
komalthareja=yes:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIxGUVBf24l4gSgUtQQaScP7S604CpXKh66cCMZB1GoXfGqyhRVO1xQUXGA2Oj8MeZf3bo4tjmrPnVeeTVfwTrxkkFNvekwY4QbGX7o8YPNnEFquLWMmkoLn9RFJI47Cj+JHWQN7sEW4WVnmHNITcw5lD3V+yw1bD5M0boUXvh/MnHTu59MEDRyLUyWY+N1FUxHrO0UgSISczRjFS31zF5WY83ssNWq+zxD0NM6GhLWg5Ynzat1J75NRvnMVkuj0VmFcJuHIl3jYCdL9uE7kCw08oh06p/VZBzUIDP6EB0e+H1udu0DvT7SunqBZnobrTCyj1Bma9BJEHPhocIIcPl komalthareja@dhcp152-54-6-178.wireless.europa.renci.org:
[interfaces]
fa163e002205=up:ipv4
[storage]
[routes]
[scripts]

Snapshot of /var/log/orca/handler-vm.log
2018-06-21 17:10:16,617 -- neuca-quantum-add-iface 20735 DEBUG : NOVA_CERT: /etc/orca/am+broker-12080/ec2/cacert.pem
2018-06-21 17:10:16,617 -- neuca-quantum-add-iface 20735 DEBUG : QUANTUM_NET_INSTANCE_NAME: 4dd54e13-58f1-422d-989f-f570a6c92247
2018-06-21 17:10:16,618 -- neuca-quantum-add-iface 20735 DEBUG : EC2_PRIVATE_KEY: /etc/orca/am+broker-12080/ec2/pk.pem
2018-06-21 17:10:16,618 -- neuca-quantum-add-iface 20735 DEBUG : neuca-quantum-add-iface: ['/etc/orca/am+broker-12080//packages/pkg/87DBB2BE-7977-4463-A0CB-F6043735DEF9/scripts//neuca-quantum-add-iface']
2018-06-21 17:10:16,618 -- neuca-quantum-add-iface 20735 DEBUG : run: args= ['quantum', 'list_nets_detail', 'geni-orca']
2018-06-21 17:10:16,776 -- neuca-quantum-add-iface 20735 DEBUG : rtncode: 0
2018-06-21 17:10:16,776 -- neuca-quantum-add-iface 20735 DEBUG : data_stdout: Virtual Networks for Tenant geni-orca
Network ipv4:vlan-storage:1009:10000000000:1250000 with ID: 8da46cbd-8c2f-4d6c-8934-e00c222606ef

2018-06-21 17:10:16,776 -- neuca-quantum-add-iface 20735 DEBUG : data_stderr: None
2018-06-21 17:10:16,777 -- neuca-quantum-add-iface 20735 ERROR : neuca-quantum-add-iface: no quantum network found for vlan: 2048, network: vlan-data

NOTE: Even though neuca-quantum-add-iface failed, BUILD was reported successful
Snapshot of /var/log/orca/am+broker-12080/orca-stdout.log
INFO | jvm 1 | 2018/06/21 17:10:16 | [echo] QUANTUM_TENANT_ID geni-orca
INFO | jvm 1 | 2018/06/21 17:10:16 | [echo] QUANTUM_NET_INSTANCE_NAME 4dd54e13-58f1-422d-989f-f570a6c92247
INFO | jvm 1 | 2018/06/21 17:10:16 | [echo] QUANTUM_NET_IFACE_MAC fa:16:3e:00:22:05
INFO | jvm 1 | 2018/06/21 17:10:16 | [echo] QUANTUM_NET_NETWORK vlan-data
INFO | jvm 1 | 2018/06/21 17:10:16 | [echo] QUANTUM_NET_VLAN 2048
INFO | jvm 1 | 2018/06/21 17:10:17 | [exec] Result: 1
INFO | jvm 1 | 2018/06/21 17:10:17 | [echo] result code: 1
INFO | jvm 1 | 2018/06/21 17:10:17 | [echo] output:
INFO | jvm 1 | 2018/06/21 17:10:17 | [echo] shirako.save.unit.quantum.ifaces:
INFO | jvm 1 | 2018/06/21 17:10:17 | [echo] EC2_SITE_PROPERTIES=/etc/orca/am+broker-12080/config/ec2.site.properties
INFO | jvm 1 | 2018/06/21 17:10:17 | [echo] update userdata instance: 4dd54e13-58f1-422d-989f-f570a6c92247
INFO | jvm 1 | 2018/06/21 17:10:18 | [echo] after update userdata: exit code 0, OK
INFO | jvm 1 | 2018/06/21 17:10:18 | [delete] Deleting: /tmp/neuca1934728267.ini
INFO | jvm 1 | 2018/06/21 17:10:18 | [echo] join exit code: 0
INFO | jvm 1 | 2018/06/21 17:10:18 |
INFO | jvm 1 | 2018/06/21 17:10:18 | BUILD SUCCESSFUL
INFO | jvm 1 | 2018/06/21 17:10:18 | Total time: 1 minute 36 seconds
INFO | jvm 1 | 2018/06/21 17:10:18 | Thu Jun 21 17:10:18 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.