Can I write rules that modify the working tree? I want to clone git repos so I can work on them

[patmaddox]

Hi there, I've been playing around with please and so far it's pretty interesting. I am able to do straightforward simple builds.

One thing I'd like to do - and I'm not sure if it's against the please way - is to write rules to clone git repos so that I can work on them. I have a monorepo for my own code, but it depends on a bunch of other repos that I need to fork and edit locally. In the past I've written a Makefile to clone the repo. Obviously I am trying to get away from make :)

Is that possible? Here's a rough example of what I want to do:

before

$ tree -L 1
.
└── BUILD

1 directory, 1 file

after

plz clone

$ tree -L 1
.
├── BUILD
├── master.jj
└── v17.12.4.jj

3 directories, 1 file

shell script implementation

The way to do this with a shell script is:

jj git clone https://github.com/thought-machine/please.git master.jj
jj -R master.jj workspace add --name v17.12.4 -r v17.12.4 v17.12.4.jj

an educational stroll down the wrong path

Here's a BUILD file using genrule... and what I learned is that it cleanly clones the repos into the plz-out/gen dir. When I'm doing a build, that repeatability is great. Here though, I want to actually modify the working tree.

genrule(
    name = "clone",
    srcs = [],
    outs = ["empty.jj"],
    cmd = "jj git clone https://github.com/patmaddox/empty.git empty.jj",
    deps = [],
)

genrule(
    name = "workspace_a",
    srcs = [],
    cmd = [
        "cd empty-example/empty.jj",
        "jj workspace add --name workspace_a ../workspace_a.jj",
        "printf '../../empty.jj/.jj/repo' > ../workspace_a.jj/.jj/repo",
    ],
    deps = [":clone"],
)

[patmaddox]

Okay I have the start of something... using sh_cmd and plz run :my_cmd I can run commands in the work tree.

I'm not sure how to set up a dependency so that genrule will run the sh_cmd first. Right now I have to do two steps: plz run :jj-clone-empty and plz build :empty.jj.

What I'm hoping I can eventually do is plz build and it will fetch all of the git dependencies for me in the worktree.

def jj_repo(name:str, repo:str):
    """Fetches a jj repo
    """
    target = f"jj-clone-{name}"
    dirname = f"$(dir :{target})"
    path = f"{dirname}/{name}.jj"
    
    cmd = f"if [ ! -d {path} ]; then jj git clone {repo} {path}; fi"
    
    sh_cmd(
        name = target,
        cmd = cmd
    )

    name = f"{name}.jj"

    genrule(
        name = name,
        cmd  = [f"mv {path} {name}"],
        srcs = [name],
        outs = [name],
        deps = [f":{target}"]
    )

def jj_workspace(repo:str, name:str):
    """Checks out a jj workspace for the given repo
    """
    target = f"jj-workspace-{name}"
    dirname = f"$(dir :{target})"
    repo_path = f"{dirname}/{repo}.jj"
    path = f"{dirname}/{name}.jj"

    command = f"if [ ! -d {path} ]; then jj -R {repo_path} workspace add --name {name} {path}; fi"
    
    sh_cmd(
        name = target,
        cmd = command
    )

    name = f"{name}.jj"

    genrule(
        name = name,
        cmd = [f"mv {path} {name}"],
        srcs = [name],
        outs = [name],
        deps = [f":{target}"]
    )
    

jj_repo(
    name = "empty",
    repo = "https://github.com/patmaddox/empty.git"
)

jj_workspace(
    repo = "empty",
    name = "workspace_a"
)

[patmaddox]

From what I can tell, these are not part of the build graph. So I don't know of a way to establish dependencies between them. Like if I want to clone multiple things, I have to do recursive make style:

sh_cmd(
    name = "clone",
    cmd = [
        "cd $(dir :clone)",
        "plz run :jj-clone-empty",
        "plz run :jj-workspace-workspace_a"
    ]
)

I think what I want is a plugin that is part of the build graph, but runs outside of the hermetic build dir.

[patmaddox]

Alright here's another attempt. There's still a bit of work to do to get it as clean as I'd like, but I'm getting a better grip on run vs build:

repos = [
    {"name": "empty", "repo": "https://github.com/patmaddox/empty"},
    {"name": "please", "repo": "https://github.com/thought-machine/please"}
]

workspaces = [
    {"repo": "empty", "name": "workspace_a"}
]

def jj_repo(name:str, repo:str):
    "Fetches a jj repo"

    target = f"jj-clone-{name}"
    dirname = f"./$(dir :{target})"
    path = f"{dirname}/{name}.jj"

    cmd = f"if [ ! -d {path} ]; then jj git clone {repo} {path}; fi"

    sh_cmd(
        name = target,
        cmd = cmd
    )

    return f":{target}"

def jj_workspace(repo:str, name:str):
    "Checks out a jj workspace for the given repo"

    target = f"jj-workspace-{name}"
    dirname = f"./$(dir :{target})"
    repo_path = f"{dirname}/{repo}.jj"
    path = f"{dirname}/{name}.jj"

    command = f"if [ ! -d {path} ]; then jj -R {repo_path} workspace add --name {name} {path}; fi"

    sh_cmd(
        name = target,
        cmd = command
    )

    return f":{target}"

repo_srcs = map(lambda r : jj_repo(name=r.name, repo=r.repo), repos)
workspace_srcs = map(lambda w : jj_workspace(repo=w.repo, name=w.name), workspaces)
all_srcs = repo_srcs + workspace_srcs
cmds = map(lambda t : f"sh $(out_location {t})", all_srcs)

sh_cmd(
    name = "fetch",
    srcs = all_srcs,
    cmd = cmds
)