Skip to content

The unexpected exit of jsonfile.Encode is blocking the container's stdout and stderr. #4591

@zwtop

Description

@zwtop

Description

Any file write error will cause jsonfile.Encode to exit, stopping the copying of container logs. When buffer is full, container log writes will then block and hang.

Due to a full file system in our environment, a container hung on write logs. Despite our prompt cleanup, the issue persisted and was resolved by restarting the container finally.

relate containerd/containerd#12462

Container use json-file log driver have this issue.

if encErr != nil {
log.L.WithError(encErr).Errorf("failed to encode JSON")
return
}

Steps to reproduce the issue

  1. nerdctl run -d --name logging-test docker.io/library/busybox:latest ping 127.0.0.1 -i 0.1 -c 3000
  2. chattr +i $(nerdctl inspect logging-test -f {{.LogPath}}); sleep 1; chattr -i $(nerdctl inspect logging-test -f {{.LogPath}})
  3. The ping commands hangs forever.

Describe the results you received and expected

Received: container hang
Expected: container not hang

What version of nerdctl are you using?

Client:
 Version:       1.7.7
 OS/Arch:       linux/amd64
 Git commit:    23d838b457213765095992ddd2c1b26a0c402632.m
 buildctl:
  Version:

Server:
 containerd:
  Version:      1.6.32
  GitCommit:    8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:      1.1.12
  GitCommit:    v1.1.12-0-g51d5e94

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: 1.6.32
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.18.0-513.24.1.el8_9.v1.x86_64
 Operating System: Rocky Linux 8.9 (Green Obsidian)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.718GiB
 Name: osbuild
 ID: ef318042-eff5-4f1c-9609-b726bd1e0a19

WARNING: IPv4 forwarding is disabled
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions