load_hvm_image fails from remote test client, due to recent commits

[lincolnthomas]

In my testing of issue #376 on my fork https://github.com/lincolnthomas/eutester, I am running all the python eutester pieces that dev-test-regression-test runs, manually from my command line.

The testcases/cloud_user/images/load_hvm_image.py test was working, until I sync’d from the eutester repo, and got @shaon 's recent commits to that test, pull request #375 (and no other eutester changes; I was otherwise up-to-date). Now, with those changes, it fails with timeouts. See attached. The failure is at the end of the failure output file.

After discussion with him, @shaon is looking into it.
load_hvm_image__new_failure.txt
load_hvm_image__old_success.txt

My test system is a CentOS 6.7 VM running under VMware Workstation 11 on my Windows 7 work laptop from home. The same test does not fail within the Eucalyptus QA lab.

All other regression tests pass from my same VM, and this test passes without @shaon's recent commits.

[nephomaniac]

Thanks @lincolnthomas,
Maybe just network issues between the location running the test and the machines in this cloud? Network env changed? VPN + VMware maybe?

More info about the logged messages in 'new_failure'...
Before attempting to setup an SSH connection, eutester attempts a TCP socket connection to port 22 to each cloud host. This is mainly done for to provide debug, but some topology tests try to autodetect the network and ssh proxy through a working connection to machines behind it if needed.. Looks like those tcp socket connection tests fail as well as the final SSH attempt.

Error at the end of old_success:
All open ssh connections should be gracefully torn down before exiting. Assuming it's not happy about how Jenkins tears it down? I thought we had a ticket opened to do this as part of the default cleanup() method most tests use, but doesnt look like it.
Thanks!,
-M

[lincolnthomas]

Thanks @nephomaniac!

Maybe just network issues between the location running the test and the machines in this cloud?
Network env changed? VPN + VMware maybe?

None of the above, I believe. I can comment in/out those lines of code at will anytime, from the same client to the same CLC/cluster, and it always fails with the new code, and succeeds with the old code.

Error at the end of old_success:

Exception in thread Thread-4 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1777, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

I get that a lot in my testing, randomly, across lots of eutester test cases that pass otherwise. I wasn't worried about it, since it was during the shutdown of the python interpreter. I wasn't thinking it was relevant here.