Recently I was involved in a situation where a Red Hat Enterprise Linux build destined for Azure was not working as hoped, there was an .iso of an specialised build that had already been created, but every time it was built and provisioned in Azure, the VM would show that the Azure Agent was not responding and the team were not able to log into the VM using pre-configured credentials. Curiosity piqued I think to myself, how hard could this be? Well... it turns out that it took longer than I had anticipated!
So, to my investigations. My first port of call as a good technical chap was to replicate the issue that the team were seeing. I ingested their .ISO file into my Azure environment and then built a RHEL VM using it and yes indeed, the VM was exhibiting the same issue. At this point I know that I have a good baseline to work from. Having spoken to the team this was a relatively old build, so I then set about researching the 'right' settings that should be applied to an image & I found the following Microsoft Learn article Create and upload a Red Hat Enterprise Linux VHD for use in Azure - Azure Virtual Machines | Microsoft Learn which looked comprehensive and so I set about working through this. But as I did this, quickly realised that I wouldn't be able to diagnose what was going on, as it wasn't possible to login into the VM as the VM Agent wasn't working along with the diagnostics serial port. Given this I then transitioned to my local Hyper-V server where I was able to spin up VM's to my heart's content.
I then manually ran through the steps noted in the MS article and uploaded to Azure and.... failure how frustrating. After a few attempts I also quickly realise that uploading 500GB+ files was not going to be a quick process and was adding far too much time lag. At this point I transitioned from my home lab into Azure and I built out a nested virtualization Hyper-V server, so I can build the VM's there + setting up a virtual serial port for testing. An after a few more attempts I'm stumped, why are the instructions not working?! So I do more research and I then find this article: Deploying RHEL 9 on Microsoft Azure | Red Hat Product Documentation - this feels like a eureka moment! Whilst there is some commonality with the Microsoft article, there are several steps here that are different! Brilliant I think, so I start working through this and after a few more attempts, I get to a point where it looks like things are heading in the right way - I upload the .VHD into the storage account and create a managed disk & create the VM, the VM looks like it boots, but the serial port is still not working and I still can't login using SSH! Somewhat frustrated I try several different things and then realised that for some reason, my home IPS/IDS had decided that SSH was untrusted traffic! I really can't say how cross I was that I hadn't spotted this sooner! Remember folk, if it's not DNS, it's the firewall! 🤣
After giving the firewall a stern talking to, I can now SSH into the VM, success!! Well, sort of - it wasn't quite working as expected, the serial port was still not working plus there were a couple of other oddities that needed resolving. At this point I have reams of notepad/OneNote scribblings that I've been working from. I was systematically working through things, but realised perhaps not always in the exact same order etc. and I realise that I should create a script to make all the changes for me. This was actually a great exercise as it's been a fair old while since I've worked with Linux in anger and so building out a .sh script to make these changes was a super interesting thing to do. After several attempts and more experimentation I get to something that was working, but I still couldn't get the darned serial ports working!
At this point, out of frustration more than anything, I built out a RHEL Azure Gallery image and lo and behold, the settings for the serial port connection were different to the Microsoft & RHEL guidance! With a final update of the script with this and a kernel tweak, I build the VM and.... it worked! Excited I share the steps with the colleague who had the issue and he confirmed that yes, things worked as expected and brought to a close a particulary time sensative issue for them, success! 🎉
So, in the interests of saving folk going through the same exercise, I thought I'd share a version of the script that I created, I know it's not perfect and I'm a 100% sure it can be improved on (plus use at your own risk et.. etc.) but regardless, you can find it here on my GitHub, enjoy!