Packer hangs on "Provisioning with shell script" when provisioning sysbox image

[makp0]

Overview of the Issue

While using sysbox-runc with Packer to add an internal image to a container, the process hangs indefinitely. This issue occurs consistently during the image build process.

Reproduction Steps

1. Use the provided packer.pkr.hcl build file.
2. Run the run-build.sh script to initiate the build.
3. Observe the build process hanging during the image addition step.
4. Inspect build.log to find an error

2024/08/30 13:02:50 packer-provisioner-shell plugin: [INFO] 155 bytes written for 'uploadData'
2024/08/30 13:02:50 [INFO] 155 bytes written for 'uploadData'
2024/08/30 13:02:50 packer-plugin-docker_v1.0.10_x5.0_linux_amd64 plugin: 2024/08/30 13:02:50 Copying to /tmp/script_7262.sh on container 505d3288b02cc7c3fdb2f74ee76b6415d0534ee1ea37b5fecaba44c5e572ce6a.
2024/08/30 13:02:50 packer-plugin-docker_v1.0.10_x5.0_linux_amd64 plugin: 2024/08/30 13:02:50 Copied 155 bytes for /tmp/script_7262.sh
2024/08/30 13:02:50 packer-provisioner-shell plugin: Retryable error: Error uploading script: Failed to upload to '/tmp/script_7262.sh' in container: Error response from daemon: Could not find the file /tmp in container 505d3288b02cc7c3fdb2f74ee76b6415d0534ee1ea37b5fecaba44c5e572ce6a
2024/08/30 13:02:50 packer-provisioner-shell plugin: . exit status 1.

Plugin and Packer version

Packer: 1.11.2
Packer Plugin Docker: 1.0.10
Sysbox: 0.6.4

Simplified Packer Buildfile

gist

Operating system and Environment details

Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
Arch: x86_64

Repo with logs

[lbajolet-hashicorp]

Hi @makp0,

Looking at the error, it seems the problem is that /tmp doesn't exist (or maybe it could be a permission error?) in your container. By default the shell provisioner uploads the inline script as a file into the guest VM/Container before executing it, and since it cannot copy the file, it fails.
Have you tried specifying a remote_folder to circumvent this problem? I'd suggest referring to our docs, alternatively the remote_path option might also do the trick in your case.

Let me know if that helps, and if so feel free to close this issue.

[makp0]

Hi @lbajolet-hashicorp, thanks for your suggestions.
I've tried all of them, folders exist it just fails on "docker cp" command here

	if err := localCmd.Wait(); err != nil {
		return fmt.Errorf("Failed to upload to '%s' in container: %s. %s.", dst, stderrOut, err)
	}

I've reproduced this issue manually using terminal.

and tried a workaround which fixed the issue:

Thanks for the analysis Cesar (@ctalledo).

In the meantime we've adjusted our docker cp statements as follows.

docker cp config.txt ${CONTAINER}:/config/config.txt

becomes

docker exec -i ${CONTAINER} bash -c 'cat > /config/config.txt' < config.txt

To embed this workaround into Communicator, the communicator should use the copying mechanism above for sysbox containers and ordinary "docker cp" for all others.
The mechanism can be picked based on Config.Runtime which is available in the communicator struct.