failed to retrieve any content

[elizen]

Can not see any Video

[svilenmarkov]

Hey, could you provide the logs for this error? Are the requests timing out or returning an error?

[marcopaganini]

I'm not OP, but I'm having a similar problem and I can provide any information you need. Here's what I have:

I'm Using podman-compose inside an unprivileged linux-container (LXC) with 4G RAM and access to 4 CPUs. The machine seems to be very low on load.

Here's my docker-compose file:

# cat docker-compose.yml
services:
  glance:
    container_name: glance
    image: glanceapp/glance
    volumes:
      - /etc/glance:/app/config
    ports:
      - 8088:8080
    dns: 1.1.1.1
    dns_search: lan
    network_mode: "bridge"
    logging:
      driver: journald

The log for the container (journalctl -u) is attached (glance.log), but basically we see a lot timeouts waiting for the headers:

Mar 13 16:04:22 podman podman-compose[182540]: 2025/03/13 23:04:22 ERROR Failed to get RSS feed url=https://www.joshwcomeau.com/rss.xml error="Get \"https://www.joshwcomeau.com/rss.xml\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Trying the same operation using wget inside the container works fine:

# podman exec -it glance /bin/sh
/app # time wget -O/dev/null https://www.joshwcomeau.com/rss.xml
Connecting to www.joshwcomeau.com (76.76.21.98:443)
saving to '/dev/null'
null                 100% |*************************************************************************| 96245  0:00:00 ETA
'/dev/null' saved
real    0m 0.42s
user    0m 0.00s
sys     0m 0.00s

Running diagnose inside the container also shows everything OK (minus the API limitations of certain sites):

# podman exec glance /app/glance diagnose

Glance version: v0.7.4
Go version: go1.23.6
Platform: linux / amd64 / 2 CPUs
In Docker container: no

Checking network connectivity, this may take up to 10 seconds...

✓ Can resolve cloudflare.com through Cloudflare DoH | 261 bytes, {"Status":0,"TC":false,"RD":true,"RA":true,"AD":tr... | 95ms
✓ Can resolve cloudflare.com through Google DoH | 318 bytes, {"Status":0,"TC":false,"RD":true,"RA":true,"AD":tr... | 87ms
✓ Can resolve github.com | 140.82.116.3 | 5001ms
✓ Can resolve reddit.com | 151.101.129.140, 151.101.193.140, 151.101.1.140, 151.101.65.140, 2a04:4e42:400::396, 2a04:4e42:600::396, 2a04:4e42::396, 2a04:4e42:200::396 | 5027ms
✓ Can resolve twitch.tv | 151.101.66.167, 151.101.2.167, 151.101.130.167, 151.101.194.167 | 5002ms
✓ Can fetch data from YouTube RSS feed | 42605 bytes, <?xml version="1.0" encoding="UTF-8"?><feed xmlns:... | 5106ms
✓ Can fetch data from Twitch.tv GQL | 0 bytes | 5099ms
✓ Can fetch data from GitHub API | 2262 bytes, {"current_user_url":"https://api.github.com/user",... | 5216ms
✓ Can fetch data from Open-Meteo API | 3328 bytes, {"results":[{"id":2643743,"name":"London","latitud... | 5563ms
✗ Can't fetch data from Reddit API | 46 bytes, {"message": "Too Many Requests", "error": 429} | 5548ms
└╴ error: expected status code 200, got 429
✗ Can't fetch data from Yahoo finance API | 23 bytes, Edge: Too Many Requests | 5125ms
└╴ error: expected status code 200, got 429
✓ Can fetch data from Hacker News Firebase API | 4501 bytes, [43356314,43357955,43357687,43356995,43352531,4335... | 5182ms
✓ Can fetch data from Docker Hub API | 4257 bytes, {"creator":7,"id":2343,"images":[{"architecture":"... | 5394ms

Notes:

• The "In docker Container: No" above is interesting. Maybe it's not detecting podman? I don't know if that would make a difference.
• I've tried different DNS providers, including my internal one. The same problem happens.
• This same LXC container runs multiple pod (docker) containers without any problem.
• Interestingly, the problem seems to be intermittent. Sometimes, some sites will load, but never all.

Hope it helps

[marcopaganini]

Another interesting data point:

I've extracted the binary from the container with: podman cp glance:/app/glance /tmp and then copied my config file to /tmp. I ran glance directly on the machine (well, it's an LXC container) and everything seems to work!

There's definitely some interaction with podman (and apparently docker too).

[svilenmarkov]

@marcopaganini Looking at those response times, it seems like you're experiencing the exact same behavior as the one from this issue, which has something to do with DNS because the domain resolution by itself takes 5 seconds. It's very weird because you have set the DNS to 1.1.1.1, so presumably this potential solution wouldn't work.

I still have no idea what's causing this as the vast majority of people don't have this problem, and the ones that do probably have some configuration in common, I just can't figure out what that is.

[marcopaganini]

I remember some podman issues where, when multiple interfaces were used, packets would go out through the wrong interface. This has been fixed and I even forced it (again) on my system without success:

# sysctl net.ipv4.conf.all.rp_filter
net.ipv4.conf.all.rp_filter = 2

Let me test this in host mode to see what happens. I now have to find out how to change the port glance listens to since I have another docker container listening on 8080 on my machine.

[svilenmarkov]

Changing the port can be done in the config:

server:
  port: 8090

https://github.com/glanceapp/glance/blob/main/docs/configuration.md#server

[marcopaganini]

OK, after a lot of banging my head on the wall, I think I have a conclusion and a workaround.

Please note that I'm using podman, not docker. Not sure if this problem would happen with docker too.

I was originally using the following docker-compose.yml file:

services:
  glance:
    container_name: glance
    image: glanceapp/glance
    volumes:
      - /etc/glance:/app/config
    ports:
      - 8088:8080
    network_mode: bridge

With this configuration, podman does two things when it comes to networking:

1. It creates a <container_name>_default network in bridge mode (in this case, glance_default) and attaches it to the container.
2. It also attaches the container to the podman network which is shared with other containers using podman-compose.

In this configuration (two networks), things just don't work. I ran a tcpdump on the DNS traffic and I can see it going out on the podmanX interface on the host, but nothing comes back. The packets are coming back through the podman interface (apparently). The fix for this would be to set rp_filter to 2, which means "Loose" source filtering. However, I did that and it didn't work:

In /etc/sysctl.conf, I added:

net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.default.rp_filter=2

And rebooted the machine. Running sysctl -a | grep '\.rp_filter' shows all interfaces with rp_filter set to 2, as expected, but things still didn't work.

Next, I removed the network_mode: bridge from the docker-compose.yml file and tried again. Still no go. This is extremely puzzling as now we have a single network! There's no other place where packets could be going out of the container and coming back.

What "solved" the problem was to force the container to the default "podman" network (as if running with pure docker). For that, just add to the end of your docker_compose.yml file. This is the final working example:

services:
  glance:
    container_name: glance
    image: glanceapp/glance
    volumes:
      - /etc/glance:/app/config
    ports:
      - 8088:8080
    logging:
      driver: journald
networks:
  podman:
    external: true

I have to say that this is a very strange behavior. Only certain programs fail. I now understand why an old program of mine was failing (also written in Go, which does not use the standard resolver).

I have a friend who works closely with podman development. I'll ask him for clues and maybe post a question on some podman forum. This does not look like a healthy behavior.

[svilenmarkov]

Excellent work debugging and documenting this!

I'm afraid there's not much I can do on my end to resolve it and don't really have the knowledge to dig deep enough to understand the cause of it. Since it's happening on both Docker and Podman and for different Go applications, it sounds like it's either related to containerization at an OS level or maybe even a bug in Go's standard library since not all containers have this issue? No idea, hopefully your friend manages to point us in the right direction so we can get it fixed.

Do feel free to post updates and let me know if there's anything I can do to help.

[marcopaganini]

My pleasure thanks. I'll certainly post updates.

In the meantime, maybe a note in the installation instructions could help people. Something along the lines of "if you're using podman, add this to your configuration..."