Seeking an effective way to page git log output when --all option is used

[axosoft-ramint]

When using git log with -n to limit the number of results, I can page the results by using the id/SHA of the last commit in the list to get more, i.e.:

git log -n10
git log -n10 <last commit SHA from previous results>^
...etc.

However, this logic seems to fall apart when using the --all option with git log, as it seems to bring in everything.

One workaround is to use -skip for paging, but it's a bit risky for my implementation (filtering out stash commits, for example, may affect the count).

Another workaround I've considered is to use commit date for paging i.e. --until/--before with the unix timestamp of the last commit date from the previous list, but I can see an edge case where if enough commits have the same timestamp (for example, a massive rebase), the paging would either not move forward or would skip commits in the next set that match the date timestamp, depending on how it's implemented.

Is there a more effective/elegant way of achieving paging git log output (possibly using a commit id) when using --all and limiting the number of results with -n?

[dscho]

One workaround is to use -skip for paging

That is not a workaround, but according to the implementation it does the exact thing that one would want for paging: it uses the very same traversal mechanism as the rest of the git log machinery to skip a given number of commits (note, though, that --skip seems to take effect only if you also provide a --max-count parameter).

[eamodio]

@dscho

I'm currently using git log -z --format=%H%x00%aN%x00%aE%x00%at%x00%ct%x00%P%x00%D%x00%B%x00 --date-order --all -n500 as my base query. And I have paging working using --before=<timestamp> --boundary and it seems to work almost all the time, but there are some odd edge cases where sometimes certain commits can go missing from a "page".

In one case, if I set --before to a specific time (lets call it T1) and --max-count=200, it returns 200 commits that start at T1, but if I run the exact same query but set --max-count=210, it returns a different set of the first 200 commits (actually the first ~190 of them are identical, but the --max-count=210 adds 3 extra commits that should have been there in the --max-count=200 query. And because these commits don't show up in the --max-count=200 query and the 199th commit timestamp is after the missing commits, they won't be included in my next call. Any clue as to why the commits could go missing like that?

I've since converted over to using --skip paging which seems to avoid this issue, but the challenge I'm having now is finding the best way to get all commits starting at the latest (e.g. --all) up to a specific commit sha. With the --before based paging, it was easy since I could just look up the timestamp of the specific commit, and then use --after on the main query. Which I could still do and just count the commits to use --skip from then on, but since --before can return a different set of commits I'm worried so would --after.

Any thoughts on how to best solve this?

FYI, this is all in support of the new Commit Graph feature in GitLens: gitkraken/vscode-gitlens#2158

[dscho]

[...] if I set --before to a specific time (lets call it T1) and --max-count=200, it returns 200 commits that start at T1, but if I run the exact same query but set --max-count=210, it returns a different set of the first 200 commits (actually the first ~190 of them are identical, but the --max-count=210 adds 3 extra commits that should have been there in the --max-count=200 query.

I bet this is due to --date-order. If you try --topo-order, it probably will work as expected (at the expense of some performance, unless the commit-graph feature is turned on).

If my hunch is correct, then there is date skew (or ambiguity) at work in the problematic scenario you've described.

[eamodio]

--topo-order seems to exhibit the same behavior where they match for the first ~200 commits and then diverge. It also seems weird that git log --format=%H --topo-order --all --before="1659558366" --boundary -n201 -- returned 211 commits and git log --format=%H --topo-order --all --before="1659558366" --boundary -n210 -- returned 221.

Also just FYI, for my follow-up question above to finding a specific commit in the history and getting all the commits before it and a bit after it -- I've switched to a streaming model and just ask for all the commits git log --date-order --all and then just kill git after I've found the commit I'm looking for (and some after buffer).