cloud-init fails to retrieve metadata over IPv6-only network

[rezaf28]

Cloud-init fails to retrieve instance metadata when operating in an IPv6-only environment. This leads to a complete failure of instance initialization in such setups, despite the metadata service being accessible over IPv6.

The issue has been observed even with the latest version of cloud-init and appears independent of any particular cloud provider. It affects environments that rely solely on IPv6 networking.

We welcome any assistance or collaboration in identifying and resolving the root cause of this problem.

A related report with more context is available here (All Informations):
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104426

Best regards.

[nmeyerhans]

See in particular this message in the Debian bug report for my debugging progress. It seems that cloud-init (or requests or urllib3) is not recognizing the default IPv6 IMDS endpoint as an IP literal and is attempted to resolve it in DNS. We can see this in the logs as

Exception(s) [UrlError('HTTPConnectionPool(host=\'fe80::a9fe:a9fe%25enp3s0\', port=80): Max retries exceeded with url: /openstack (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7fb44b82fe00>: Failed to resolve \'fe80::a9fe:a9fe%25enp3s0\' ([Errno -2] Name or service not known)"))')

I haven't dug any deeper than that yet, but the failure is straightforward to reproduce.

Note that this also seems to be a duplicate of #5893

[TheRealFalcon]

@rezaf28 , please don't post AI generated answers that you haven't tested. What you posted is not a solution.

@nmeyerhans

It seems that cloud-init (or requests or urllib3) is not recognizing the default IPv6 IMDS endpoint as an IP literal and is attempted to resolve it in DNS

Yes, it appears to be an issue in requests or something lower. If I pass your two test URLs directly to requests, I get the same result.

>>> import requests
>>> requests.get("http://[fe80::a9fe:a9fe%enp0s1]", timeout=1)
...
requests.exceptions.ConnectionError: HTTPConnectionPool(host='fe80::a9fe:a9fe%25enp0s1', port=80): Max retries exceeded with url: / (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7c3c69a0f230>: Failed to resolve 'fe80::a9fe:a9fe%25enp0s1' ([Errno -2] Name or service not known)"))

vs

>>> requests.get("http://[fd00:80db:0:5:34e5:8aff:fec5:b9bf]", timeout=1)
...
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='fd00:80db:0:5:34e5:8aff:fec5:b9bf', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c3c69a92fd0>, 'Connection to fd00:80db:0:5:34e5:8aff:fec5:b9bf timed out. (connect timeout=1)'))

[rezaf28]

@rezaf28 , please don't post AI generated answers that you haven't tested. What you posted is not a solution.

@nmeyerhans

It seems that cloud-init (or requests or urllib3) is not recognizing the default IPv6 IMDS endpoint as an IP literal and is attempted to resolve it in DNS

Yes, it appears to be an issue in requests or something lower. If I pass your two test URLs directly to requests, I get the same result.

>>> import requests
>>> requests.get("http://[fe80::a9fe:a9fe%enp0s1]", timeout=1)
...
requests.exceptions.ConnectionError: HTTPConnectionPool(host='fe80::a9fe:a9fe%25enp0s1', port=80): Max retries exceeded with url: / (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7c3c69a0f230>: Failed to resolve 'fe80::a9fe:a9fe%25enp0s1' ([Errno -2] Name or service not known)"))

vs

>>> requests.get("http://[fd00:80db:0:5:34e5:8aff:fec5:b9bf]", timeout=1)
...
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='fd00:80db:0:5:34e5:8aff:fec5:b9bf', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7c3c69a92fd0>, 'Connection to fd00:80db:0:5:34e5:8aff:fec5:b9bf timed out. (connect timeout=1)'))

You are right, but these discussions were a summary of what was raised in the Debian bug.

But before you accuse me, did you check the patch that was sent yesterday? Or are you guessing?
(just build and test)
If you have a better patch, I'm your servant.

You did not fix a bug that has been raised for a year and you are only here to manage، I'm sure you didn't even do a simple test to check the problem.

I appreciate your efforts, but your attitude is disappointing. This is a technical discussion, not your hatred of AI.

I deleted the post for your convenience.

[nmeyerhans]

See also psf/requests#6735 and psf/requests#6282

It seems like this has been a known issue for some time.

[TheRealFalcon]

@nmeyerhans

See also psf/requests#6735 and psf/requests#6282

Thanks, I saw 6282 but I can't reproduce it. I hadn't seen 6735 though, and that definitely appears to be the culprit. There's a PR up at psf/requests#6927 , but it appears to have gone stale. I commented to bump it. I'll leave this issue open, but I don't really think there's anything to do on the cloud-init side (unless that PR never merges).

@rezaf28

But before you accuse me, did you check the patch that was sent yesterday?

I did, and the "fix" wasn't even in the call path of the reproducer. It's not "hatred of AI" (I use AI for things too), it's posting a comment with a proposed fix as if you had figured it out yourself when it appears to be a hallucinated non-answer. It's misleading and it wastes the time of maintainers, especially since we weren't informed it was AI generated. Maintainers know how to ask AI things too, so these types of comments wind up being a detriment to the project. I appreciate the engagement and your desire to fix the problem though. In the future, I'd just ask that you check the solution first and preface the response. Something like "my AI gave me this solution, and I checked it and it appears to work" would be better.

[rezaf28]

@TheRealFalcon
Please clarify the situation completely.
1- Should we wait again?
2- Or temporary solution #6207 (I tested on Debian 13 and didn't see any problems)
3- Or do you have another plan?

[holmanb]

@rezaf28 Slow down, please.

@rezaf28 , please don't post AI generated answers that you haven't tested. What you posted is not a solution.

But before you accuse me, did you check the patch that was sent yesterday? Or are you guessing?

I did, and the "fix" wasn't even in the call path of the reproducer. It's not "hatred of AI" (I use AI for things too), it's posting a comment with a proposed fix as if you had figured it out yourself when it appears to be a hallucinated non-answer. It's misleading and it wastes the time of maintainers, especially since we weren't informed it was AI generated. Maintainers know how to ask AI things too, so these types of comments wind up being a detriment to the project. I appreciate the engagement and your desire to fix the problem though. In the future, I'd just ask that you check the solution first and preface the response. Something like "my AI gave me this solution, and I checked it and it appears to work" would be better.

As @TheRealFalcon already mentioned, a disclaimer of your understanding of the problem and methods used to generate your contributions is expected. I'll add that this is the bare minimum when you don't understand the problem and/or solution that you are proposing. Please read Daniel Stenberg's post for context.

(just build and test)
If you have a better patch, I'm your servant.
You did not fix a bug that has been raised for a year and you are only here to manage، I'm sure you didn't even do a simple test to check the problem.

I appreciate your efforts, but your attitude is disappointing. This is a technical discussion, not your hatred of AI.

@rezaf28 Your responses were uncharitable and rude - that is inappropriate. Please reconsider your communication style.

[rezaf28]

@TheRealFalcon
@holmanb

• For clarity, I emphasize that this post has nothing to do with artificial intelligence.
• This post is long, if you are not patient, do not read it.

Your responses were uncharitable and rude - that is inappropriate. Please reconsider your communication style.

You are right, I apologize.
I apologize again to @TheRealFalcon .

I emphasize that I have no precise information about the details of this project, I had to take action because this bug in Debian Cloud version has officially disabled the operating system and we have been waiting for a solution from you for a year.

Let's start from the beginning.

I first encountered this problem a year ago (I don't know if it was there from the beginning) and at that time I had no reaction because this bug was too big and I expected this problem to be a bug that would last for a few weeks and be fixed by you, but it didn't happen and it gradually became a joke like GitHub not supporting IPv6.

What exactly is the problem?
The problem of not being able to access metadata in Cloud-init is in IPv6-only.

What should I do?
Report a bug!
But the bug has already been reported, so why isn't it being fixed?

I have 3 solutions:
1- Wait for the bug to be fixed
2- Review and release a patch
3- Ask for help from others

I am in a hole about this bug, because I don't see any effective action, so what should I do?
1- My first action was a summary report of the comments given. (Summary of the AI ​​) which focused more on reviewing the comments of those whose names were even mentioned at the end of the post.
The text I had published was the summary of Chatgpt 4.5 of the problem raised and cited in the following links:
#1104426
#5893

But the fact that they raised this issue and that artificial intelligence had summarized it was not wrong, but they are absolutely right.

2- My second step was to investigate the problem itself and release a patch (I still had no precise information about the bug).
Here, after investigating the bug and doing a lot of testing, I realized that "requests" itself cannot directly understand hosts with iface
such as
[fe80::a9fe:a9fe%enp3s0]
and therefore I had to find a solution to solve the problem.

I had three choices:
1- Patch "requests" itself, which I realized by examining the project is involved in this issue and it is not clear whether it will be solved or not.
2- Edit the function that sends the request based on the solution proposed in this link
https://stackoverflow.com/questions/48996494/send-http-request-through-specific-network-interface
3- Change the entire request from "requests" to another method (new troubles...)

I chose the latter, which is a temporary solution until the problem is solved in the main package itself (since I have not this much knowledge)

For this reason, I first added the "requests-toolbelt" library to the project
Then I wrote a function called url_parser_plus that can be easier to parse links and used in other places in the future and is also upgradeable.
Then I edited the readurl function and only put the link host for the address and set iface via SO_BINDTODEVICE.
And I took a build and copied it to the Debian 13 image and tested a new VM from the same original image that was not edited and only its cloud-init executable file was changed to ipv6-only, it was completely successful.
I checked the logs and saw that there was no problem with ipv6.
And finally I published the patch #6207 .

This was the whole process I went through.
If I made a mistake, I don't know, if I went the whole way wrong, it was a lack of knowledge.

But I have a question, what is the situation now? What should I do?