dvc exp run: fails when used with github actions and CML docker container

[tasdomas]

Description

dvc get fails in one of our example repos with the error response:

ERROR: unexpected error - Repository not found at /__w/stale-model-example/stale-model-example/.git/

None of the directories in that path are symlinks.

Running the dvc get command interactively (in an ssh session via action-tmate results in success - the action completes without error.

Running with the -vv flag gives the following output:

2022-09-28 18:08:49,331 TRACE: Namespace(all_pipelines=False, cd='.', checkpoint_resume=None, cmd='run', cprofile=False, cprofile_dump=None, downstream=False, dry=False, force=False, force_downstream=False, func=<class 'dvc.commands.exper
iments.run.CmdExperimentsRun'>, instrument=False, instrument_open=False, interactive=False, jobs=1, machine=None, metrics=False, name=None, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<clas
s 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False), pdb=False, pipeline=False, pull=False, queue=False, quiet=0, recursive=False, reset=False, run_all=False, set_param=[], single_item=False, targets=[], tmp_dir=
False, verbose=2, version=None, viztracer=False, viztracer_depth=None, yappi=False)                                                                                                                                                           
2022-09-28 18:08:49,709 ERROR: unexpected error - Repository not found at /__w/stale-model-example/stale-model-example/.git/
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/usr/local/lib/python3.8/dist-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/usr/local/lib/python3.8/dist-packages/dvc/commands/experiments/run.py", line 32, in run
    results = self.repo.experiments.run(
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/__init__.py", line 521, in run
    return run(self.repo, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/run.py", line 60, in run
    return repo.experiments.reproduce_one(
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/__init__.py", line 119, in reproduce_one
    self.queue_one(exp_queue, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/__init__.py", line 156, in queue_one
    return self.new(
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/utils.py", line 40, in wrapper
    return f(exp, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/__init__.py", line 269, in new
    return queue.put(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/queue/workspace.py", line 25, in put
    return self._stash_exp(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/dvc/repo/experiments/queue/base.py", line 315, in _stash_exp
    with self.scm.detach_head(client="dvc") as orig_head:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/__init__.py", line 404, in detach_head
    self.checkout(rev, detach=True, force=force)
  File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/__init__.py", line 283, in _backend_func
    for key, backend in self.backends.items():
  File "/usr/lib/python3.8/_collections_abc.py", line 744, in __iter__
    yield (key, self._mapping[key])
  File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/__init__.py", line 49, in __getitem__
    initialized = backend(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/scmrepo/git/backend/pygit2.py", line 97, in __init__
    self.repo = pygit2.Repository(path)
  File "/usr/local/lib/python3.8/dist-packages/pygit2/repository.py", line 1620, in __init__
    path_backend = init_file_backend(path, flags)
_pygit2.GitError: Repository not found at /__w/stale-model-example/stale-model-example/.git/
------------------------------------------------------------
2022-09-28 18:08:50,051 DEBUG: Version info for developers:
DVC version: 2.28.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.15.0-1020-azure-x86_64-with-glibc2.29
Subprojects:
        dvc_data = 0.13.0
        dvc_objects = 0.5.0
        dvc_render = 0.0.11
        dvc_task = 0.1.2
        dvclive = 0.11.0
        scmrepo = 0.1.1
Supports:
        gs (gcsfs = 2022.8.2),
        http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2022.8.2, boto3 = 1.24.59)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: s3, gs
Workspace directory: ext4 on /dev/root
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-09-28 18:08:50,053 DEBUG: Analytics is enabled.
2022-09-28 18:08:50,117 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp3gfjynwi']'
2022-09-28 18:08:50,119 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp3gfjynwi']'

Reproduce

The only way of reproducing this that I've found is via github actions.

Expected

The dvc get command should download the requested data.

Environment information

Output of dvc doctor:

DVC version: 2.30.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.15.0-1020-azure-x86_64-with-glibc2.29
Subprojects:
	dvc_data = 0.17.1
	dvc_objects = 0.7.0
	dvc_render = 0.0.12
	dvc_task = 0.1.3
	dvclive = 0.12.0
	scmrepo = 0.1.1
Supports:
	gs (gcsfs = 2022.8.2),
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2022.8.2, boto3 = 1.24.59)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: s3, gs
Workspace directory: ext4 on /dev/root
Repo: dvc, git

[pmrowla]

It looks like exp run is what's failing, not dvc get

[pmrowla]

The issue is probably due to how actions/checkout works:

When Git 2.18 or higher is not in your PATH, falls back to the REST API to download the files.

(see https://github.com/actions/checkout#checkout-v3)

I'm guessing that the CML docker image doesn't include git >2.18, so using actions/checkout inside that docker container will just download the source using the github web API (and does not create an actual git repo in the workspace), so dvc exp run will fail since it needs an actual git repo

[tasdomas]

The cml docker image include git==2.38 and the git repository is created normally.

Including additional debug information:

git version

+ git -v
git version 2.38.0

Directory contents

+ ls -alh
total 608K
drwxr-xr-x 9 1001  121 4.0K Oct 10 12:56 .
drwxr-xr-x 3 1001  121 4.0K Oct 10 12:56 ..
drwxr-xr-x 2 root root 4.0K Oct 10 12:56 data
drwxr-xr-x 4 root root 4.0K Oct 10 12:58 .dvc
-rw-r--r-- 1 root root  138 Oct 10 12:56 .dvcignore
-rw-r--r-- 1 root root 1.2K Oct 10 12:56 dvc.lock
-rw-r--r-- 1 root root  782 Oct 10 12:56 dvc.yaml
drwxr-xr-x 8 root root 4.0K Oct 10 12:56 .git
drwxr-xr-x 3 root root 4.0K Oct 10 12:56 .github
-rw-r--r-- 1 root root   73 Oct 10 12:56 .gitignore
drwxr-xr-x 2 root root 4.0K Oct 10 12:56 notebooks
-rw-r--r-- 1 root root   52 Oct 10 12:56 params.yaml
-rw-r--r-- 1 root root 130K Oct 10 12:56 prc.json
-rw-r--r-- 1 root root  603 Oct 10 12:56 README.md
drwxr-xr-x 2 root root 4.0K Oct 10 12:56 reports
-rw-r--r-- 1 root root  100 Oct 10 12:56 requirements.txt
-rw-r--r-- 1 root root 408K Oct 10 12:56 roc.json
-rw-r--r-- 1 root root   73 Oct 10 12:56 scores.json
drwxr-xr-x 2 root root 4.0K Oct 10 12:56 src

[pmrowla]

@tasdomas did you check that running the container yourself (or via ssh into it) or did you check it inside the actions workflow? It matters because the workflow PATH may not be the same

[pmrowla]

This issue is specific to using the CML container

import pygit2

pygit2.Repository(os.getcwd())

succeeds in regular GHA ubuntu-latest workflows but fails when using the CML container

see: https://github.com/pmrowla/gha-test/actions

[pmrowla]

The problem is that in the CML container the github workspace is created & owned by root but the actions are run as a different user. In order for git to work properly this means that you have to configure safe.directory (https://git-scm.com/docs/git-config/2.35.2#Documentation/git-config.txt-safedirectory`) to include your workspace path, otherwise both regular git and libgit2 will refuse to open the repository. In actions/checkout, they explicitly set this config var using a temporary global config (which only exists for the duration of the actions/checkout script step).

If you add

/usr/bin/git config --global --add safe.directory $GITHUB_WORKSPACE

to your workflow run (anywhere before dvc exp run) it should work as expected.

This is really something that should be probably be handled automatically in the CML container (perhaps via the --system level git config for the entire container)

see: https://github.com/pmrowla/gha-test/actions/runs/3219831064/jobs/5265728412

[pmrowla]

For reference, dulwich does not do the user permssion/safe.directory check at all which is why (almost) everything else in DVC works in the CML container other than exp run (which gets into some of the pygit only scmrepo git behavior)

[pmrowla]

Looking at it again I think exp run may actually fail unless you run it as root (with sudo) in this scenario (even if using safe.directory lets us open the repo in pygit). exp run needs to create new git commits and refs in .git/, but if it's owned by root and DVC is running as a different user in the workflow you will eventually hit a permissions error.

I haven't tested this, so I'm not sure what the behavior inside custom containers and GHA is for git write operations. But if this is the case (and we can't write commits) this is also something that would need to be addressed on the CML side. (Either users need to use sudo in GHA workflows or the workspace needs to be chown'd to the proper user in the container)

[tasdomas]

whoami indicates the action is run as root

[pmrowla]

@tasdomas it's an interaction w/ how the GHA workspace is mounted inside the docker container, even though whoami reports root, the user permissions won't match from git's perspective unless you set safe.directory

related: actions/runner#2033

[tasdomas]

Thanks @pmrowla - this looks like it's the cause.