Checking out thousands of changes is slow for large repos

[Eveeifyeve]

Describe the bug
A clear and concise description of what the bug is.

Checking out between branches is way to slow for large branches.

To Reproduce
Steps to reproduce the behavior:

1. Use nixpkgs as an example
2. clone it
3. Add the 25.11 branch and checkout between them.

Expected behavior
A clear and concise description of what you expected to happen.

It should really not be as this slow.

Screenshots
If applicable, add screenshots to help explain your problem.

Version info:

• Lazygit: commit=, build date=, build source=nix, version=0.50.0, os=darwin, arch=arm64, git version=2.49.0
• Git: 2.49.0

Additional context

May 24 22:24:57 |DEBU| RunCommand command="git checkout master"
May 24 22:24:59 |INFO| git checkout master (1.683055292s)
May 24 22:24:59 |INFO| refreshing all scopes in block-ui mode
May 24 22:24:59 |INFO| Refresh took 10.042µs
May 24 22:24:59 |DEBU| RunCommand command="git stash list -z --pretty=%ct|%gs"
May 24 22:24:59 |DEBU| RunCommand command="git tag --list -n --sort=-creatordate"
May 24 22:24:59 |INFO| refreshed merge conflicts in 16.209µs
May 24 22:24:59 |DEBU| using cache for key status.showUntrackedFiles
May 24 22:24:59 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z --find-renames=50%"
May 24 22:24:59 |INFO| git stash list -z --pretty=%ct|%gs (11.563375ms)
May 24 22:24:59 |INFO| git tag --list -n --sort=-creatordate (11.591333ms)
May 24 22:24:59 |INFO| postRefreshUpdate for tags took 9.042µs
May 24 22:24:59 |INFO| refreshed tags in 11.860083ms
May 24 22:24:59 |INFO| git for-each-ref --sort=refname --format=%(refname:short) refs/remotes (11.621916ms)
May 24 22:24:59 |INFO| postRefreshUpdate for remotes took 19µs
May 24 22:24:59 |INFO| postRefreshUpdate for remoteBranches took 3.792µs
May 24 22:24:59 |INFO| refreshed remotes in 11.984583ms
May 24 22:24:59 |INFO| postRefreshUpdate for stash took 44.208µs
May 24 22:24:59 |INFO| refreshed stash in 12.026208ms
May 24 22:24:59 |INFO| git -c log.showSignature=false log -g --abbrev=40 --format=%h%x00%ct%x00%gs%x00%P (12.057125ms)
May 24 22:24:59 |INFO| postRefreshUpdate for reflogCommits took 213.917µs
May 24 22:25:00 |INFO| git status --untracked-files=all --porcelain -z --find-renames=50% (604.383375ms)
May 24 22:25:00 |INFO| refreshed files in 605.107708ms
May 24 22:25:00 |INFO| refreshed staging in 605.204208ms
May 24 22:25:00 |INFO| git log HEAD --topo-order --oneline --pretty=format:%H%x00%at%x00%aN%x00%ae%x00%D%x00%P%x00%m%x00%s --abbrev=40 -300 --no-show-signature -- (3.927187125s)
May 24 22:25:00 |DEBU| Error getting git config value for key: rebase.updateRefs. Error: the key is not found for [git config --get --null rebase.updateRefs]
May 24 22:25:00 |INFO| postRefreshUpdate for commits took 6.886709ms
May 24 22:25:00 |DEBU| using cache for key rebase.updateRefs
May 24 22:25:00 |DEBU| RunCommand command="git for-each-ref --sort=-committerdate --format=%(HEAD)%00%(refname:short)%00%(upstream:short)%00%(upstream:track)%00%(push:track)%00%(subject)%00%(objectname)%00%(committerdate:unix) refs/heads"
May 24 22:25:00 |DEBU| RunCommand command="git rev-parse --abbrev-ref --verify HEAD"
May 24 22:25:00 |INFO| git rev-parse --abbrev-ref --verify HEAD (5.811416ms)
May 24 22:25:00 |DEBU| RunCommand command="git merge-base master master@{u}"
May 24 22:25:00 |DEBU| RunCommand command="git merge-base HEAD refs/remotes/origin/master"
May 24 22:25:00 |INFO| git for-each-ref --sort=-committerdate --format=%(HEAD)%00%(refname:short)%00%(upstream:short)%00%(upstream:track)%00%(push:track)%00%(subject)%00%(objectname)%00%(committerdate:unix) refs/heads (8.345292ms)
May 24 22:25:00 |DEBU| RunCommand command="git worktree list --porcelain"
May 24 22:25:00 |INFO| git merge-base master master@{u} (7.074875ms)
May 24 22:25:00 |INFO| git merge-base HEAD refs/remotes/origin/master (8.179375ms)
May 24 22:25:00 |INFO| git worktree list --porcelain (5.933958ms)
May 24 22:25:00 |DEBU| RunCommand command="git -C /Users/eveeifyeve/github/eveeifyeve/nixpkgs rev-parse --path-format=absolute --absolute-git-dir"
May 24 22:25:00 |INFO| git -C /Users/eveeifyeve/github/eveeifyeve/nixpkgs rev-parse --path-format=absolute --absolute-git-dir (5.216458ms)
May 24 22:25:00 |INFO| postRefreshUpdate for worktrees took 53.083µs
May 24 22:25:00 |DEBU| RunCommand command="git log --graph --color=always --abbrev-commit --decorate --date=relative --pretty=medium refs/heads/master --"
May 24 22:25:00 |INFO| postRefreshUpdate for localBranches took 197.125µs
May 24 22:25:02 |INFO| refreshing the following scopes in sync mode: files
May 24 22:25:02 |INFO| Heap memory in use: 103.7 MB
May 24 22:25:02 |DEBU| using cache for key status.showUntrackedFiles
May 24 22:25:02 |INFO| refreshed merge conflicts in 26.625µs
May 24 22:25:02 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z --find-renames=50%"
May 24 22:25:03 |INFO| git status --untracked-files=all --porcelain -z --find-renames=50% (559.366209ms)
May 24 22:25:03 |INFO| refreshed files in 559.706625ms
May 24 22:25:03 |INFO| Refresh took 560.21425ms
May 24 22:25:03 |INFO| git log HEAD --topo-order --oneline --pretty=format:%H%x00%at%x00%aN%x00%ae%x00%D%x00%P%x00%m%x00%s --abbrev=40 -300 --no-show-signature -- (3.754516458s)
May 24 22:25:03 |DEBU| using cache for key rebase.updateRefs
May 24 22:25:03 |INFO| postRefreshUpdate for commits took 644.167µs
May 24 22:25:03 |INFO| refreshed commits and commit files in 4.430693458s
May 24 22:25:03 |DEBU| using cache for key rebase.updateRefs
May 24 22:25:03 |INFO| refreshed reflog and branches in 4.431357375s
May 24 22:25:03 |INFO| postRefreshUpdate for submodules took 3µs
May 24 22:25:03 |INFO| postRefreshUpdate for files took 3.917µs
May 24 22:25:03 |INFO| postRefreshUpdate for submodules took 917ns
May 24 22:25:03 |INFO| postRefreshUpdate for files took 1µs
May 24 22:25:03 |DEBU| using cache for key rebase.updateRefs
May 24 22:25:03 |DEBU| using cache for key rebase.updateRefs

[stefanhaller]

Can you compare the time it takes lazygit to checkout the branch versus the time it takes on the command line, i.e. git checkout master? If that's equally slow (which I suspect it is), then I'm not sure why you expect it to be any faster in lazygit.

[Eveeifyeve]

Can you compare the time it takes lazygit to checkout the branch versus the time it takes on the command line, i.e. git checkout master? If that's equally slow (which I suspect it is), then I'm not sure why you expect it to be any faster in lazygit.

How can I measure the time of that command?

[stefanhaller]

Stopwatch? 😄

[Eveeifyeve]

Stopwatch? 😄

[nix-shell:~/github/eveeifyeve/nixpkgs]$ hyperfine "git checkout master"

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
git checkout master	295.6 ± 471.4	111.1	1629.8	1.00

1.629 secs

[stefanhaller]

While hyperfine is a great benchmarking tool, it's no good in this case, because checking out the same branch again when it is already checked out is fast of course.

You might just use time git checkout master (if the other branch is checked out).

But the result seems to be pretty close to what lazygit's log said (1.6s). Are you saying that the interaction in lazygit (from pressing space until you see the new branch in the commits panel) is significantly slower than that?

[Eveeifyeve]

time git checkout master

[nix-shell:~/github/eveeifyeve/nixpkgs]$ time git checkout master
Updating files: 100% (12871/12871), done.
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

real	0m1.610s
user	0m0.426s
sys	0m1.399s

what is slowing it down is the git status --untracked-files=all --porcelain -z --find-renames=50%

[stefanhaller]

what is slowing it down is the git status --untracked-files=all --porcelain -z --find-renames=50%

Ok, but there's not much we can do about that. Lazygit needs to make that call in order to find out what modified files to display. nixpkgs is a repo with a large number of files, and in such repos git status is simply slow. I bet calling git status on the command line is as slow.

You might experiment with things like git config core.fsmonitor true to speed this up.

[wolfgangwalther]

I experience the same - and I don't think this was as slow as it is right now, still a few weeks back. Lazygit is essentially unusable for me with nixpkgs right now - every checkout or start of a rebase takes about 40 seconds via lazygit, but is instant (0,4 sec) from the command line.

time git status --untracked-files=all --porcelain -z --find-renames=50%

This takes about 0.4 secs for me.

You might experiment with things like git config core.fsmonitor true to speed this up.

I did that, but it didn't make a difference. It seems like this is not implemented for linux, yet?

% git fsmonitor--daemon status
fatal: fsmonitor--daemon not supported on this platform

I have been happily using lazygit with nixpkgs for a long time, so there must be a way to do this.

[wolfgangwalther]

The 40s seemed to have been an outlier, maybe there was some fetch going on at the same or so. But I consistently get something between 5-10s.

Some stuff from the logs:

May 26 12:40:16 |INFO| git log HEAD --date-order --oneline --pretty=format:%H%x00%at%x00%aN%x00%ae%x00%D%x00%P%x00%m%x00%s --abbrev=40 -300 --no-show-signature -- (5.836344487s)
[...]
May 26 12:40:16 |INFO| refreshed commits and commit files in 5.839295345s
[...]
May 26 12:40:18 |INFO| git for-each-ref --sort=refname --format=%(HEAD)%00%(refname:short)%00%(upstream:short)%00%(upstream:track)%00%(push:track)%00%(subject)%00%(objectname)%00%(committerdate:unix) refs/heads (7.884200316s)
[...]
May 26 12:40:18 |INFO| refreshed branches in 7.890493067s

The --date-order seems to be a problem for me. Removing that improves the git log command significantly - I explicitly need to set it to "default", not to topo-order, though. I can see that the log view loads much quicker on startup. I also updated from 0.50.0 to 0.51.1, which seems to have helped a little bit.

Still - entering a rebase or checking out a different branch freezes the UI for me for a few seconds. I noticed the "Local branches" view takes very long to load on initial startup. I assume that's related to the git for-each-ref above taking 7-8 seconds?

[wolfgangwalther]

I assume that's related to the git for-each-ref above taking 7-8 seconds?

When I manually run it, I can make this very fast by removing the upstream:track and push:track parts.

I tried removing a remote with ~900 branches, but that doesn't change it. I also tested in a fresh clone with only 1 local branch - that's instant. So this seems to be a function of the number of local branches, of which I currently have 43.

Especially the fact that the UI freezes while fetching that information is annoying. Maybe updating the tracking information could be separated from updating the other parts and run async?

[wolfgangwalther]

Turns out the git for-each-ref is slow when it deals with heavily outdated branches. I had two branches that I rebased locally, but didn't push. They were both ahead of upstream by about 8k commits. And a third branch that was waay behind upstream. Removing the remotes for those 3, makes the UI super fast again!

So, my takeaways for working with very big repos at the scale of nixpkgs:

• Remove --graph from branchLogCmd and allBranchesLogCmds - otherwise git log in the main view will be slow like a snail.
• git.log.order = default, otherwise the commit log will be really slow. No date-order, no topo-order :/
• Keep your branches up2date. The more they rot, the slower the UI will update.

I think it would still be good to not freeze on git for-each-ref, outdated branches shouldn't take lazygit down like that. If that could run async in the background... that would be much better.

[wolfgangwalther]

@Eveeifyeve note that you also have git log HEAD --topo-order in the log, which takes 3 seconds. So you might wanna try default ordering for the commit log. Maybe that helps you, too.

[stefanhaller]

Turns out the git for-each-ref is slow when it deals with heavily outdated branches.

This is a great finding, something to watch out for. Thanks.

• git.log.order = default, otherwise the commit log will be really slow. No date-order, no topo-order :/

This is a well-known one; the default sort order setting was introduced in lazygit to increase performance, not because somebody liked it better. 😄

However, I'm surprised that setting git.log.order = default in your user config file makes any difference. We store this setting in state.yml now, and the user config setting is only used when starting lazygit for the very first time (when there's no state.yml yet). To change this value, you need to go to the ctrl-l menu in the commits panel and change it there.

Side note: I think we should change this back. We moved the setting from user config to state.yml for consistency with other settings, and because we decided this is the new way of persisting changes that you can change in the UI (like the branches sort order, or the "ignore whitespace" switch). However, that was before we had repo-local config files. Now that we have those, it would actually be better to move it back to a user config so that you can set the default to topo-order for the general case, and override it to default only for those few giant repos where it is needed for performance.

I think it would still be good to not freeze on git for-each-ref, outdated branches shouldn't take lazygit down like that. If that could run async in the background... that would be much better.

We already do this for the divergence-from-base-branch information (not shown by default, but you can enable it with gui.showDivergenceFromBaseBranch: arrowAndNumber). When starting lazygit it takes a brief moment until these appear; when later refreshing the data, they stay on their previous values until they are updated (we do this to avoid flicker), which means they are stale for a moment.

For the blue arrows this is not a problem, because we only display them but never base any decisions on them. That's different for the yellow arrows: these are used for logic, e.g. to decide whether we need to prompt for force-pushing. It would be bad if we get this wrong because the values are stale. Hm, on the other hand, they can already be stale (with respect to what's actually on the remote) simply because you didn't fetch for a while, and we handle this case properly, so maybe it's fine to update them with a bit of delay. Anyway, I guess all I'm saying is that this is not a totally trivial change.

[wolfgangwalther]

However, I'm surprised that setting git.log.order = default in your user config file makes any difference. We store this setting in state.yml now, and the user config setting is only used when starting lazygit for the very first time (when there's no state.yml yet). To change this value, you need to go to the ctrl-l menu in the commits panel and change it there.

Well, I had git.log.order = date-order in my config, so that's where I started. Then I changed state.yml, where I found the same setting, to test (didn't bother to find the right shortcut to do that in app). It worked, then I changed it in my config. But good to know that changing it in the config file actually didn't do much.

(this was the global config, not the repo local config)

Side note: I think we should change this back. We moved the setting from user config to state.yml for consistency with other settings, and because we decided this is the new way of persisting changes that you can change in the UI (like the branches sort order, or the "ignore whitespace" switch). However, that was before we had repo-local config files. Now that we have those, it would actually be better to move it back to a user config so that you can set the default to topo-order for the general case, and override it to default only for those few giant repos where it is needed for performance.

Yes, that would be good. I'd think, those changed settings in the UI should not be persisted at all. I'd expect lazygit to start up with and respect whatever I have put in either the global or repo-local config. It's good to be able to temporarily change it, but the current situation, where I'd make a change to my global config file - without anything actually changing in practice, because there is a state that overrides it... is quite unintuitive.

That's different for the yellow arrows: these are used for logic, e.g. to decide whether we need to prompt for force-pushing. It would be bad if we get this wrong because the values are stale. Hm, on the other hand, they can already be stale (with respect to what's actually on the remote) simply because you didn't fetch for a while, and we handle this case properly, so maybe it's fine to update them with a bit of delay. Anyway, I guess all I'm saying is that this is not a totally trivial change.

A few thoughts:

• It would probably be good to indicate that the yellow arrows are currently loading, i.e. not showing the old state and then suddenly updating it when the request is over. Yellow loading-spinner in place of the arrows, maybe.
• It could also be possible to just block pushing while those are still being updated.
• If, instead of running a single git for-each-ref, this would be run for each branch separately.. then the heavily outdated branches would take longer, but all others would be updated instantly. The branches are outdated for a reason... because you're most likely not working on them right now. So this would very likely make everything the user normally does very fast.

[stefanhaller]

I'd think, those changed settings in the UI should not be persisted at all. I'd expect lazygit to start up with and respect whatever I have put in either the global or repo-local config. It's good to be able to temporarily change it, but the current situation, where I'd make a change to my global config file - without anything actually changing in practice, because there is a state that overrides it... is quite unintuitive.

We had long discussions about this, for example the one starting here. It was a deliberate decision to go with what we are doing now for diff context size, ignore whitespace, and branch sort order. That's not to say that it can't still be questioned again, of course.

[stefanhaller]

I'd think, those changed settings in the UI should not be persisted at all. I'd expect lazygit to start up with and respect whatever I have put in either the global or repo-local config. It's good to be able to temporarily change it, but the current situation, where I'd make a change to my global config file - without anything actually changing in practice, because there is a state that overrides it... is quite unintuitive.

We had long discussions about this, for example the one starting here. It was a deliberate decision to go with what we are doing now for diff context size, ignore whitespace, and branch sort order. That's not to say that it can't still be questioned again, of course.

I filed a separate issue about this: #4602