Packages for Ubuntu focal for fluentd are failing with a 'File has an unexpected size'

[stejacks]

Describe the bug

Starting at 1320 UTC, we (at GitLab) started getting errors trying to download the Ubuntu package:

Hit:10 https://packages.gitlab.com/gitlab/pre-release/ubuntu focal InRelease                                                                                                                                                          
Get:11 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib all Packages [2,834 B]                                                                                                                                      
Get:12 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages [4,599 B]                                                                                                                                    
Err:12 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages                                                                                                                                              
  File has unexpected size (4302 != 4599). Mirror sync in progress? [IP: 18.173.166.102 443]                                                                                                                                          
  Hashes of expected file:                                                                                                                                                                                                            
   - Filesize:4599 [weak]                                                                                                                                                                                                             
   - SHA512:907296f5183eb31a1b490a503c22103d4bd238240f26a2a10a145d81dcb0f65e3606c4f6403161c06f1942217c1c84118a6cdadc549be72cf528656a1151a710                                                                                          
   - SHA256:1f8d6c0e8b58e4bd62b8cea2ca22f5538cd39ddad8fd059f877f76745751e47b                                                                                                                                                          
   - SHA1:c1c0ee04e1611d25c12905846e06ab0816e8edc3 [weak]                                                                                                                                                                             
   - MD5Sum:90a11d00260d4acc5f8268820a242fb3 [weak]                                                                                                                                                                                   
  Release file created at: Thu, 15 May 2025 05:18:39 +0000                                                                                                                                                                            
Fetched 10.9 kB in 1s (10.8 kB/s)                                                                                                                                                                                                     
Reading package lists... Done                                                                                                                                                                                                         
E: Failed to fetch https://packages.treasuredata.com/lts/5/ubuntu/focal/dists/focal/contrib/binary-amd64/Packages.bz2  File has unexpected size (4302 != 4599). Mirror sync in progress? [IP: 18.173.166.102 443]                     
   Hashes of expected file:                                                                                                                                                                                                           
    - Filesize:4599 [weak]                                                                                                                                                                                                            
    - SHA512:907296f5183eb31a1b490a503c22103d4bd238240f26a2a10a145d81dcb0f65e3606c4f6403161c06f1942217c1c84118a6cdadc549be72cf528656a1151a710                                                                                         
    - SHA256:1f8d6c0e8b58e4bd62b8cea2ca22f5538cd39ddad8fd059f877f76745751e47b                                                                                                                                                         
    - SHA1:c1c0ee04e1611d25c12905846e06ab0816e8edc3 [weak]                                                                                                                                                                            
    - MD5Sum:90a11d00260d4acc5f8268820a242fb3 [weak]                                                                                                                                                                                  
   Release file created at: Thu, 15 May 2025 05:18:39 +0000                                                                                                                                                                           
E: Some index files failed to download. They have been ignored, or old ones used instead.                                                                                                                                             
N: Usage of apt_auth.conf(5) should be preferred over embedding login information directly in the sources.list(5) entry for 'https://packages.gitlab.com/gitlab/pre-release/ubuntu' 

This is still going on, we're tracking it in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19833 and have reported it on the FluentD slack, but I thought it was worth opening here too.

To Reproduce

On a ubuntu focal host, set up the following:

cat /etc/apt/sources.list.d/fluentd.list

deb "http://packages.treasuredata.com/lts/5/ubuntu/focal" focal contrib

Run apt-get update.

apt-get update

Hit:1 http://us-east1.gce.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us-east1.gce.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us-east1.gce.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:4 https://packages.treasuredata.com/lts/5/ubuntu/focal focal InRelease [10.9 kB]
Hit:5 https://packages.cloud.google.com/apt cloud-sdk InRelease
Get:6 https://esm.ubuntu.com/infra/ubuntu focal-infra-security InRelease [7,462 B]
Get:8 https://esm.ubuntu.com/infra/ubuntu focal-infra-updates InRelease [7,461 B]
Hit:9 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:10 https://dpkg.wiz.io/projects/sensor-repos sensor-apt InRelease
Get:7 https://packages.gitlab.com/gitlab/pre-release/ubuntu focal InRelease [23.3 kB]
Get:11 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages [4,599 B]
Ign:11 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
Get:12 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib all Packages [2,834 B]
Ign:11 http://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
Ign:11 http://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
Err:11 http://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
File has unexpected size (4302 != 4599). Mirror sync in progress? [IP: 18.173.166.108 443]
Get:13 https://packages.gitlab.com/gitlab/pre-release/ubuntu focal/main amd64 Packages [1,728 kB]
Fetched 1,777 kB in 2s (1,068 kB/s)
Reading package lists... Done
E: Failed to fetch http://packages.treasuredata.com/lts/5/ubuntu/focal/dists/focal/contrib/binary-amd64/Packages File has unexpected size (4302 != 4599). Mirror sync in progress? [IP: 18.173.166.108 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.

Expected behavior

This should work.

Your Environment

- Fluentd version: fluentd 1.16.5 (95d130aaa44fc09b14a4e9686ee8018253185fc5)
- Package version: fluent-package 5.0.4 
- Operating system: NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)"
- Kernel version: 5.15.0-1068-gcp

Your Configuration

N/A

Your Error Log

apt-get update
Hit:1 http://us-east1.gce.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us-east1.gce.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us-east1.gce.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:4 https://packages.treasuredata.com/lts/5/ubuntu/focal focal InRelease [10.9 kB]
Hit:5 https://packages.cloud.google.com/apt cloud-sdk InRelease
Get:6 https://esm.ubuntu.com/infra/ubuntu focal-infra-security InRelease [7,462 B]
Get:8 https://esm.ubuntu.com/infra/ubuntu focal-infra-updates InRelease [7,461 B]
Hit:9 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:10 https://dpkg.wiz.io/projects/sensor-repos sensor-apt InRelease
Get:7 https://packages.gitlab.com/gitlab/pre-release/ubuntu focal InRelease [23.3 kB]
Get:11 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages [4,599 B]
Ign:11 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
Get:12 https://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib all Packages [2,834 B]
Ign:11 http://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
Ign:11 http://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
Err:11 http://packages.treasuredata.com/lts/5/ubuntu/focal focal/contrib amd64 Packages
  File has unexpected size (4302 != 4599). Mirror sync in progress? [IP: 18.173.166.108 443]
Get:13 https://packages.gitlab.com/gitlab/pre-release/ubuntu focal/main amd64 Packages [1,728 kB]
Fetched 1,777 kB in 2s (1,068 kB/s)
Reading package lists... Done
E: Failed to fetch http://packages.treasuredata.com/lts/5/ubuntu/focal/dists/focal/contrib/binary-amd64/Packages  File has unexpected size (4302 != 4599). Mirror sync in progress? [IP: 18.173.166.108 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.

Additional context

No response

[kenhys]

For upcoming fluent-package v5.0.7, package repository was updated recently, but not synced fully yet.
That is why the above error happens.
We will contact to solve this issue.

[kenhys]

We will contact to solve this issue.

As the Fluent repository cache was updated, the above issue was resolved for now.
But still similar problem will occur when the next release is done, so escalated it with infra team.

[kenhys]

@stejacks

By the way, I didn't know fluent-package LTS is used in GitLab infra.

If there are no obstacles, we want to note them in use-case here.

https://www.fluentd.org/testimonials

Could you feedback via GitHub issue template?
Please send your use-case via https://github.com/fluent/fluentd-website/issues/new?template=testimonials.yml