Can we enable shallow cloning?

[BenSandeen]

Describe Request:

The code checkout step of this orb is the vast majority of the build time for this step in our CircleCI workflow. Shallow checkout (ie: using git clone --depth=1 would cut this step's duration by roughly 75% for our monolith codebase

Examples:

I don't know enough about how this orb works to provide code examples, but I may be willing to learn if it's not too involved

Supporting Documentation Links:

Here's our current task that uses this orb:

We use this shallow_checkout orb (https://circleci.com/developer/orbs/orb/datacamp/shallow-checkout) for the following steps of this build, and it routinely runs the code checkout step in 9-10 seconds, which would be a great benefit to speed up our builds a bit

[Peter-Darton-i2]

CircleCI is the first CI system I've used in years that doesn't do shallow clones by default.
The lack of support for shallow clones is very disappointing 😞

[eli-adept]

Also interested in this. There seems to be an old closed PR that implemented this using another orb

https://github.com/CircleCI-Public/path-filtering-orb/pull/48/files

I'm happy to revive that attempt or hack together another solution if the maintainers have any appetite for this change

[Fernando-Abreu]

I will add this to our backlog.

[BenSandeen]

Is there any chance we could just merge that PR? Do we know why @kynefuk closed it?

[krzema12]

Our repo takes over a minute to clone entirely. The tip of it is pretty lightweight now, clones within several seconds. Given that our CI worfklow uses path filtering, and then in other jobs we need to clone the repo again, it currently takes 2 extra minutes that we could save on cloning. While we have control over what happens in the other jobs, we can just optimize them. The initial step using the path-filtering orb is not possible to optimize (at least without rewriting it) because this ticket is still not addressed.

Please consider implementing this simple feature that will save compute resources (also your workers!) and improve productivity of multiple engineers.

[sebastian-lerner]

Hey folks, I'm a product manager at CircleCI, we are testing out a "blobless clone" which we hope will achieve a similar outcome as a shallow clone. If you would like to be included in early testing, email me at sebastian@circleci.com

[edoardo-bluframe]

Hey everybody

We really need this over here - like 2 min 45 sec each time of extra build when all we need is shallow??

This is a really cool orb but we do need that feature

Is there a way we can help?

Maybe we can have a look at that PR from @kynefuk?

Create a new one?

What kinda support do you need from the community to make this happen?? 😄

[mscrivo]

We forked this orb and made one that supports shallow cloning if folks are interested: https://circleci.com/developer/orbs/orb/affinity/path-filtering-orb. It's very similar to #48, but that PR was broken and didn't actually work. We've been using this for the past month or so and it's been working very well.

[edoardo-bluframe]

We forked this orb and made one that supports shallow cloning if folks are interested: https://circleci.com/developer/orbs/orb/affinity/path-filtering-orb. It's very similar to #48, but that PR was broken and didn't actually work. We've been using this for the past month or so and it's been working very well.

We wanna try it!

Does it work exactly the same? We just change the orb import?

[mscrivo]

Does it work exactly the same? We just change the orb import?

yep, just replace the circleci/path-filtering orb with: path-filtering: affinity/path-filtering-orb@0.9.1 in your orbs list and then add the shallow clone options to your path-filtering workflow, for example:

- path-filtering/filter:
    base-revision: master
    config-path: .circleci/continue_config.yml
    enable_shallow_checkout: true
    checkout_shallow_depth: 200 # adjust as necessary 
    checkout_shallow_fetch_depth: 200 # adjust as necessary

[Peter-Darton-i2]

Doesn't the blobless clone feature (mentioned above) eliminate the need for this entirely?

[mscrivo]

Doesn't the blobless clone feature (mentioned above) eliminate the need for this entirely?

It might? But it's not available yet. I believe they tried at some point last year and had to rollback because of issues? Anyhow this works now and brought down our clones from 30s to 7s.

[Peter-Darton-i2]

It might? But it's not available yet.

Oh dear.
We had been informed that it was all sorted last year as it's been something we've been asking for ever since we started using CircleCI; we were never informed of any rollback.
... in fact the official documentation still says blobless clones are happening; no mention of a rollback ☹

That said, it's not a total shock that there were "issues"; it is a "very clever" solution and (in my experience) anything "clever" that is very different to "what everyone else does" is rarely an unqualified success (otherwise "everyone else" would be doing it too), but it is sad that it was announced with great fanfare and then silently withdrawn (almost as sad, in fact, as the lack of shallow cloning support in the checkout command itself)

I guess this will have to go back on our "monthly discussions" agenda.