nerdctl compose fails, profile 'nerdctl-default' does not exist #1237

nicolasespiau · 2022-07-15T16:40:24Z

nicolasespiau
Jul 15, 2022

Description

I'm developing a nodejs app suite with some microservices and an API.

In order to perform integration testing before any deployment, I have a step in my Gitlab CI that uses a docker-compose.yaml to spawn all these MS and API and launch some tests against the API.

All of this happens in a Kube cluster based on containerd, so I'm trying to use nerdctl compose.

When my CI steps run, it fails with some time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[90] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1" errors.

Steps to reproduce the issue

Describe the results you received and expected

Here is my docker-compose.yaml:

version: "3.9"
networks:
  testinteg:
    name: testinteg
services:
  mongosrv:
    container_name: "mongosrv"
    image: registry.gitlab.com/casetonpote1/ctp-utils/mongosrv:testinteg
    command: [ "--auth", "--dbpath", "/mongo/data/db" ]
    pull_policy: always
    networks:
      - testinteg
  users:
    container_name: "users"
    image: registry.gitlab.com/casetonpote1/backend-users:dev
    environment:
      NODE_ENV: "dev"
      PORT: 80
      LOG_LEVEL: "debug"
      LOG_FORMAT: "splat,simple"
      PASSWORD_JWT_SECRET: "anothersecurestring"
      PASSWORD_JWT_TTL: "30s"
      SSL_ENABLED: "false"
      MOCK_DB: "false"
      MONGO_DB: "casetespotes"
      MONGO_HOST: "mongosrv"
      MONGO_USER: "usersRW"
      MONGO_PASSWD: "password"
    depends_on:
      - mongosrv
    networks:
      - testinteg
  stories:
    container_name: "stories"
    image: registry.gitlab.com/casetonpote1/backend-stories:dev
    environment:
      NODE_ENV: "dev"
      PORT: 80
      LOG_LEVEL: "debug"
      LOG_FORMAT: "splat,simple"
      SSL_ENABLED: "false"
      MOCK_DB: "false"
      MONGO_DB: "casetespotes"
      MONGO_HOST: "mongosrv"
      MONGO_USER: "storiesRW"
      MONGO_PASSWD: "password"
    depends_on:
      - mongosrv
    networks:
      - testinteg
  profiles:
    container_name: "profiles"
    image: registry.gitlab.com/casetonpote1/backend-profiles:dev
    environment:
      NODE_ENV: "dev"
      PORT: 80
      LOG_LEVEL: "debug"
      LOG_FORMAT: "splat,simple"
      SSL_ENABLED: "false"
      MOCK_DB: "false"
      MONGO_DB: "casetespotes"
      MONGO_HOST: "mongosrv"
      MONGO_USER: "profilesRW"
      MONGO_PASSWD: "password"
      MOCK_GEO: "true"
    depends_on:
      - mongosrv
    networks:
      - testinteg
  sessions:
    container_name: "sessions"
    image: registry.gitlab.com/casetonpote1/backend-sessions:dev
    environment:
      NODE_ENV: "dev"
      PORT: 80
      LOG_LEVEL: "debug"
      LOG_FORMAT: "splat,simple"
      APP_JWT_SECRET: "anothersecurestring"
      APP_JWT_TTL: "30s"
      REFRESH_ALLOWED: "7d"
      SSL_ENABLED: "false"
      MOCK_DB: "false"
      MONGO_DB: "casetespotes"
      MONGO_HOST: "mongosrv"
      MONGO_USER: "authenticater"
      MONGO_PASSWD: "password"
      MS_USERS_ENDPOINT: "http://users"
    depends_on:
      - mongosrv
    networks:
      - testinteg
  linkings:
    container_name: "linkings"
    image: registry.gitlab.com/casetonpote1/backend-linkings:dev
    environment:
      NODE_ENV: "dev"
      PORT: 80
      LOG_LEVEL: "debug"
      LOG_FORMAT: "splat,simple"
      SSL_ENABLED: "false"
      MOCK_DB: "false"
      MONGO_DB: "casetespotes"
      MONGO_HOST: "mongosrv"
      MONGO_USER: "linkingsRW"
      MONGO_PASSWD: "password"
    depends_on:
      - mongosrv
    networks:
      - testinteg
  api-gateway:
    container_name: "api-gateway"
    image: registry.gitlab.com/casetonpote1/api-gateway-public:test
    environment:
      NODE_ENV: "dev"
      PORT: 80
      LOG_LEVEL: "debug"
      LOG_FORMAT: "splat,simple"
      SSL_ENABLED: "false"
      AUTH_SOURCE: "api-public-dev"
      MS_STORIES_ENDPOINT: "http://stories"
      MS_USERS_ENDPOINT: "http://users"
      MS_PROFILES_ENDPOINT: "http://profiles"
      MS_SESSIONS_ENDPOINT: "http://sessions"
      MS_LINKINGS_ENDPOINT: "http://linkings"
    depends_on:
      - users
      - stories
      - profiles
      - sessions
      - linkings
    networks:
      - testinteg

And here is my CI step:

integrationtesting:
  tags:
    - kubernetes
  image: ghcr.io/containerd/nerdctl:v0.21.0
  stage: tests
  before_script:
    - nerdctl compose -f test_integ/dependencies/docker-compose.yaml up -d --debug-full
  script:
    - nerdctl compose -f test_integ/dependencies/tester-compose.yaml up --debug-full
  after_script:
    - nerdctl compose -f test_integ/dependencies/docker-compose.yaml -f test_integ/dependencies/tester-compose.yaml down
  dependencies:
    - buildapitest
    - buildtestertest

Where dependencies are steps where required images are created. These steps are successful.

All the images pulling seems to be ok, but when creating the containers I get this error messages in the CI logs:

time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[80] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
108time="2022-07-15T16:17:33Z" level=debug msg="verification process skipped"
109time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[86] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
110time="2022-07-15T16:17:33Z" level=debug msg="verification process skipped"
111time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[82] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
112time="2022-07-15T16:17:33Z" level=debug msg="verification process skipped"
113time="2022-07-15T16:17:33Z" level=fatal msg="no such file or directory"
114time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[89] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
115time="2022-07-15T16:17:33Z" level=debug msg="verification process skipped"
116time="2022-07-15T16:17:33Z" level=fatal msg="no such file or directory"
117time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[90] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
118time="2022-07-15T16:17:33Z" level=debug msg="verification process skipped"
119time="2022-07-15T16:17:33Z" level=fatal msg="no such file or directory"
120time="2022-07-15T16:17:33Z" level=fatal msg="no such file or directory"
121time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[91] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
122time="2022-07-15T16:17:33Z" level=debug msg="verification process skipped"
123time="2022-07-15T16:17:34Z" level=fatal msg="no such file or directory"
124time="2022-07-15T16:17:34Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: \"[95] aa-exec: ERROR: profile 'nerdctl-default' does not exist\\n\"" error="exit status 1"
125time="2022-07-15T16:17:34Z" level=debug msg="verification process skipped"
126time="2022-07-15T16:17:34Z" level=fatal msg="no such file or directory"
127time="2022-07-15T16:17:34Z" level=fatal msg="no such file or directory"
128time="2022-07-15T16:17:34Z" level=fatal msg="error while creating container users: exit status 1"

What version of nerdctl are you using?

ghcr.io/containerd/nerdctl:v0.21.0

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

No response

Host information

$ nerdctl info
Client:
Namespace: default
Debug Mode: false
Server:
Server Version: v1.6.6
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Log: fluentd journald json-file
Storage: aufs native overlayfs
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.13.0-1017-aws
Operating System: Ubuntu 22.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.687GiB
Name: runner-dqoqvrt-project-25793245-concurrent-0ht8p8
ID: d4341431-1b20-4fc6-b55e-89dd1d3edd57
WARNING: IPv4 forwarding is disabled

Answered by AkihiroSuda

Jul 16, 2022

time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: "[90] aa-exec: ERROR: profile 'nerdctl-default' does not exist\n"" error="exit status 1"

This is just a negligible debug message, not a fatal error.

If your container doesn't work as expected, it's probably failing for other reason(s).

I'd suggest trying nerdctl run hello-world before composing up a complex YAML.

View full answer

Zheaoli · 2022-07-15T18:05:24Z

Zheaoli
Jul 15, 2022
Maintainer

Seems like you need to follow the following guideline to add AppArmor Profile for nerdctl

Applying an (existing) AppArmor profile to rootless containers: nerdctl run --security-opt apparmor=<PROFILE>. Use `sudo nerdctl apparmor load` to load the nerdctl-default profile.

https://github.com/containerd/nerdctl#features-present-in-nerdctl-but-not-present-in-docker

2 replies

nicolasespiau Jul 15, 2022
Author

I'm not trying to perform a run command but a compose.
The problem is that no profile is found.
My runner is executing as privileged, when I run nerdctl apparmor load in a before_script step of my CI I get the following error:

$ nerdctl apparmor load
31time="2022-07-15T22:04:45Z" level=info msg="Loading profile \"nerdctl-default\""
32time="2022-07-15T22:04:45Z" level=fatal msg="open /sys/kernel/security/apparmor/profiles: no such file or directory"

I tried to mount the profiles as a volume in my runner adding this section in the toml conf:

[[runners.kubernetes.volumes.host_path]]
  name = "nerdctlprofiles"
  host_path = "/sys/kernel/security/apparmor/profiles"
  mount_path = "/sys/kernel/security/apparmor/profiles"

but it creates a conflict with the execution of the runner and it never starts:

{"duration_s":6.25812877,"job":2729688867,"level":"error","msg":"Job failed (system failure): prepare environment: waiting for pod running: pod status is failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information","project":25793245,"runner":"xsjRNg35","time":"2022-07-15T22:09:34Z"}

I'm kind of stuck here.

Any clues about this mismatch between shell profile and nerdctl profile?

nicolasespiau Jul 15, 2022
Author

In fact, I logged into a container running ghcr.io/containerd/nerdctl:v0.21.0 and there is no apparmor inside.

How am I supposed to use it if it's not in the container?

AkihiroSuda · 2022-07-16T12:07:42Z

AkihiroSuda
Jul 16, 2022
Maintainer

time="2022-07-15T16:17:33Z" level=debug msg="failed to run [aa-exec -p nerdctl-default -- true]: "[90] aa-exec: ERROR: profile 'nerdctl-default' does not exist\n"" error="exit status 1"

This is just a negligible debug message, not a fatal error.

If your container doesn't work as expected, it's probably failing for other reason(s).

I'd suggest trying nerdctl run hello-world before composing up a complex YAML.

3 replies

nicolasespiau Jul 16, 2022
Author

I've followed your suggestion, there is no more information about the fatal error beside no such file or directory.
I've managed to log into the runner container by putting a sleep 3600 in the before_script and tried it manyally:

root@runner-ty3soix6-project-25793245-concurrent-05xnww:/# nerdctl run hello-world --debug
FATA[0000] no such file or directory

And that's all.

How can I have more information about what's blocking?

AkihiroSuda Jul 16, 2022
Maintainer

nerdctl run hello-world --debug

This should be nerdctl --debug run hello-world.
Could you open a separate GitHub Discussion, as the failure is unlikely related to AppArmor and Compose?

nicolasespiau Jul 16, 2022
Author

Yes sure!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

nerdctl compose fails, profile 'nerdctl-default' does not exist #1237

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

nerdctl compose fails, profile 'nerdctl-default' does not exist #1237

Uh oh!

nicolasespiau Jul 15, 2022

Description

Steps to reproduce the issue

Describe the results you received and expected

What version of nerdctl are you using?

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Host information

Replies: 2 comments · 5 replies

Uh oh!

Zheaoli Jul 15, 2022 Maintainer

Uh oh!

nicolasespiau Jul 15, 2022 Author

Uh oh!

nicolasespiau Jul 15, 2022 Author

Uh oh!

Uh oh!

AkihiroSuda Jul 16, 2022 Maintainer

Uh oh!

nicolasespiau Jul 16, 2022 Author

Uh oh!

AkihiroSuda Jul 16, 2022 Maintainer

Uh oh!

nicolasespiau Jul 16, 2022 Author

nicolasespiau
Jul 15, 2022

Replies: 2 comments 5 replies

Zheaoli
Jul 15, 2022
Maintainer

nicolasespiau Jul 15, 2022
Author

nicolasespiau Jul 15, 2022
Author

AkihiroSuda
Jul 16, 2022
Maintainer

nicolasespiau Jul 16, 2022
Author

AkihiroSuda Jul 16, 2022
Maintainer

nicolasespiau Jul 16, 2022
Author