Skip to content

Commit b40fce3

Browse files
authored
Merge pull request #20280 from Kobzol/josh-sync
Switch to using josh-sync
2 parents 8d693ce + 00a47e3 commit b40fce3

File tree

6 files changed

+42
-229
lines changed

6 files changed

+42
-229
lines changed

Cargo.lock

Lines changed: 0 additions & 11 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/book/src/contributing/README.md

Lines changed: 42 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -252,18 +252,8 @@ Release steps:
252252
4. Commit & push the changelog.
253253
5. Run `cargo xtask publish-release-notes <CHANGELOG>` -- this will convert the changelog entry in AsciiDoc to Markdown and update the body of GitHub Releases entry.
254254
6. Tweet.
255-
7. Make a new branch and run `cargo xtask rustc-pull`, open a PR, and merge it.
256-
This will pull any changes from `rust-lang/rust` into `rust-analyzer`.
257-
8. Switch to `master`, pull, then run `cargo xtask rustc-push --rust-path ../rust-rust-analyzer --rust-fork matklad/rust`.
258-
Replace `matklad/rust` with your own fork of `rust-lang/rust`.
259-
You can use the token to authenticate when you get prompted for a password, since `josh` will push over HTTPS, not SSH.
260-
This will push the `rust-analyzer` changes to your fork.
261-
You can then open a PR against `rust-lang/rust`.
262-
263-
Note: besides the `rust-rust-analyzer` clone, the Josh cache (stored under `~/.cache/rust-analyzer-josh`) will contain a bare clone of `rust-lang/rust`.
264-
This currently takes about 3.5 GB.
265-
266-
This [HackMD](https://hackmd.io/7pOuxnkdQDaL1Y1FQr65xg) has details about how `josh` syncs work.
255+
7. Perform a subtree [pull](#performing-a-pull).
256+
8. Perform a subtree [push](#performing-a-push).
267257
268258
If the GitHub Actions release fails because of a transient problem like a timeout, you can re-run the job from the Actions console.
269259
If it fails because of something that needs to be fixed, remove the release tag (if needed), fix the problem, then start over.
@@ -288,3 +278,43 @@ There are two sets of people with extra permissions:
288278
If you don't feel like reviewing for whatever reason, someone else will pick the review up (but please speak up if you don't feel like it)!
289279
* The [rust-lang](https://github.com/rust-lang) team [t-rust-analyzer-contributors]([https://github.com/orgs/rust-analyzer/teams/triage](https://github.com/rust-lang/team/blob/master/teams/rust-analyzer-contributors.toml)).
290280
This team has general triaging permissions allowing to label, close and re-open issues.
281+
282+
## Synchronizing subtree changes
283+
`rust-analyzer` is a [josh](https://josh-project.github.io/josh/intro.html) subtree of the [rust-lang/rust](https://github.com/rust-lang/rust)
284+
repository. We use the [rustc-josh-sync](https://github.com/rust-lang/josh-sync) tool to perform synchronization between these two
285+
repositories. You can find documentation of the tool [here](https://github.com/rust-lang/josh-sync).
286+
287+
You can install the synchronization tool using the following commands:
288+
```
289+
cargo install --locked --git https://github.com/rust-lang/josh-sync
290+
```
291+
292+
Both pulls (synchronizing changes from rust-lang/rust into rust-analyzer) and pushes (synchronizing
293+
changes from rust-analyzer into rust-lang/rust) are performed from this repository.
294+
changes from rust-analyzer to rust-lang/rust) are performed from this repository.
295+
296+
Usually we first perform a pull, wait for it to be merged, and then perform a push.
297+
298+
### Performing a pull
299+
1) Checkout a new branch that will be used to create a PR against rust-analyzer
300+
2) Run the pull command
301+
```
302+
rustc-josh-sync pull
303+
```
304+
3) Push the branch to your fork of `rust-analyzer` and create a PR
305+
- If you have the `gh` CLI installed, `rustc-josh-sync` can create the PR for you.
306+
307+
### Performing a push
308+
309+
Wait for the previous pull to be merged.
310+
311+
1) Switch to `master` and pull
312+
2) Run the push command to create a branch named `<branch-name>` in a `rustc` fork under the `<gh-username>` account
313+
```
314+
rustc-josh-sync push <branch-name> <gh-username>
315+
```
316+
- The push will ask you to download a checkout of the `rust-lang/rust` repository.
317+
- If you get prompted for a password, see [this](https://github.com/rust-lang/josh-sync?tab=readme-ov-file#git-peculiarities).
318+
3) Create a PR from `<branch-name>` into `rust-lang/rust`
319+
320+
> Besides the `rust` checkout, the Josh cache (stored under `~/.cache/rustc-josh`) will contain a bare clone of `rust-lang/rust`. This currently takes several GBs.

xtask/Cargo.toml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@ rust-version.workspace = true
88

99
[dependencies]
1010
anyhow.workspace = true
11-
directories = "6.0"
1211
flate2 = "1.1.2"
1312
write-json = "0.1.4"
1413
xshell.workspace = true

xtask/src/flags.rs

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -59,20 +59,6 @@ xflags::xflags! {
5959
optional --dry-run
6060
}
6161

62-
cmd rustc-pull {
63-
/// rustc commit to pull.
64-
optional --commit refspec: String
65-
}
66-
67-
cmd rustc-push {
68-
/// rust local path, e.g. `../rust-rust-analyzer`.
69-
required --rust-path rust_path: String
70-
/// rust fork name, e.g. `matklad/rust`.
71-
required --rust-fork rust_fork: String
72-
/// branch name.
73-
optional --branch branch: String
74-
}
75-
7662
cmd dist {
7763
/// Use mimalloc allocator for server
7864
optional --mimalloc
@@ -121,8 +107,6 @@ pub enum XtaskCmd {
121107
Install(Install),
122108
FuzzTests(FuzzTests),
123109
Release(Release),
124-
RustcPull(RustcPull),
125-
RustcPush(RustcPush),
126110
Dist(Dist),
127111
PublishReleaseNotes(PublishReleaseNotes),
128112
Metrics(Metrics),
@@ -151,18 +135,6 @@ pub struct Release {
151135
pub dry_run: bool,
152136
}
153137

154-
#[derive(Debug)]
155-
pub struct RustcPull {
156-
pub commit: Option<String>,
157-
}
158-
159-
#[derive(Debug)]
160-
pub struct RustcPush {
161-
pub rust_path: String,
162-
pub rust_fork: String,
163-
pub branch: Option<String>,
164-
}
165-
166138
#[derive(Debug)]
167139
pub struct Dist {
168140
pub mimalloc: bool,

xtask/src/main.rs

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,6 @@ fn main() -> anyhow::Result<()> {
4242
flags::XtaskCmd::Install(cmd) => cmd.run(sh),
4343
flags::XtaskCmd::FuzzTests(_) => run_fuzzer(sh),
4444
flags::XtaskCmd::Release(cmd) => cmd.run(sh),
45-
flags::XtaskCmd::RustcPull(cmd) => cmd.run(sh),
46-
flags::XtaskCmd::RustcPush(cmd) => cmd.run(sh),
4745
flags::XtaskCmd::Dist(cmd) => cmd.run(sh),
4846
flags::XtaskCmd::PublishReleaseNotes(cmd) => cmd.run(sh),
4947
flags::XtaskCmd::Metrics(cmd) => cmd.run(sh),

xtask/src/release.rs

Lines changed: 0 additions & 175 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,5 @@
11
mod changelog;
22

3-
use std::process::{Command, Stdio};
4-
use std::thread;
5-
use std::time::Duration;
6-
7-
use anyhow::{Context as _, bail};
8-
use directories::ProjectDirs;
9-
use stdx::JodChild;
103
use xshell::{Shell, cmd};
114

125
use crate::{date_iso, flags, is_release_tag, project_root};
@@ -59,171 +52,3 @@ impl flags::Release {
5952
Ok(())
6053
}
6154
}
62-
63-
// git sync implementation adapted from https://github.com/rust-lang/miri/blob/62039ac/miri-script/src/commands.rs
64-
impl flags::RustcPull {
65-
pub(crate) fn run(self, sh: &Shell) -> anyhow::Result<()> {
66-
sh.change_dir(project_root());
67-
let commit = self.commit.map(Result::Ok).unwrap_or_else(|| {
68-
let rust_repo_head =
69-
cmd!(sh, "git ls-remote https://github.com/rust-lang/rust/ HEAD").read()?;
70-
rust_repo_head
71-
.split_whitespace()
72-
.next()
73-
.map(|front| front.trim().to_owned())
74-
.ok_or_else(|| anyhow::format_err!("Could not obtain Rust repo HEAD from remote."))
75-
})?;
76-
// Make sure the repo is clean.
77-
if !cmd!(sh, "git status --untracked-files=no --porcelain").read()?.is_empty() {
78-
bail!("working directory must be clean before running `cargo xtask pull`");
79-
}
80-
// This should not add any new root commits. So count those before and after merging.
81-
let num_roots = || -> anyhow::Result<u32> {
82-
Ok(cmd!(sh, "git rev-list HEAD --max-parents=0 --count")
83-
.read()
84-
.context("failed to determine the number of root commits")?
85-
.parse::<u32>()?)
86-
};
87-
let num_roots_before = num_roots()?;
88-
// Make sure josh is running.
89-
let josh = start_josh()?;
90-
91-
// Update rust-version file. As a separate commit, since making it part of
92-
// the merge has confused the heck out of josh in the past.
93-
// We pass `--no-verify` to avoid running any git hooks that might exist,
94-
// in case they dirty the repository.
95-
sh.write_file("rust-version", format!("{commit}\n"))?;
96-
const PREPARING_COMMIT_MESSAGE: &str = "Preparing for merge from rust-lang/rust";
97-
cmd!(sh, "git commit rust-version --no-verify -m {PREPARING_COMMIT_MESSAGE}")
98-
.run()
99-
.context("FAILED to commit rust-version file, something went wrong")?;
100-
101-
// Fetch given rustc commit.
102-
cmd!(sh, "git fetch http://localhost:{JOSH_PORT}/rust-lang/rust.git@{commit}{JOSH_FILTER}.git")
103-
.run()
104-
.inspect_err(|_| {
105-
// Try to un-do the previous `git commit`, to leave the repo in the state we found it it.
106-
cmd!(sh, "git reset --hard HEAD^")
107-
.run()
108-
.expect("FAILED to clean up again after failed `git fetch`, sorry for that");
109-
})
110-
.context("FAILED to fetch new commits, something went wrong (committing the rust-version file has been undone)")?;
111-
112-
// Merge the fetched commit.
113-
const MERGE_COMMIT_MESSAGE: &str = "Merge from rust-lang/rust";
114-
cmd!(sh, "git merge FETCH_HEAD --no-verify --no-ff -m {MERGE_COMMIT_MESSAGE}")
115-
.run()
116-
.context("FAILED to merge new commits, something went wrong")?;
117-
118-
// Check that the number of roots did not increase.
119-
if num_roots()? != num_roots_before {
120-
bail!("Josh created a new root commit. This is probably not the history you want.");
121-
}
122-
123-
drop(josh);
124-
Ok(())
125-
}
126-
}
127-
128-
impl flags::RustcPush {
129-
pub(crate) fn run(self, sh: &Shell) -> anyhow::Result<()> {
130-
let branch = self.branch.as_deref().unwrap_or("sync-from-ra");
131-
let rust_path = self.rust_path;
132-
let rust_fork = self.rust_fork;
133-
134-
sh.change_dir(project_root());
135-
let base = sh.read_file("rust-version")?.trim().to_owned();
136-
// Make sure the repo is clean.
137-
if !cmd!(sh, "git status --untracked-files=no --porcelain").read()?.is_empty() {
138-
bail!("working directory must be clean before running `cargo xtask push`");
139-
}
140-
// Make sure josh is running.
141-
let josh = start_josh()?;
142-
143-
// Find a repo we can do our preparation in.
144-
sh.change_dir(rust_path);
145-
146-
// Prepare the branch. Pushing works much better if we use as base exactly
147-
// the commit that we pulled from last time, so we use the `rust-version`
148-
// file to find out which commit that would be.
149-
println!("Preparing {rust_fork} (base: {base})...");
150-
if cmd!(sh, "git fetch https://github.com/{rust_fork} {branch}")
151-
.ignore_stderr()
152-
.read()
153-
.is_ok()
154-
{
155-
bail!(
156-
"The branch `{branch}` seems to already exist in `https://github.com/{rust_fork}`. Please delete it and try again."
157-
);
158-
}
159-
cmd!(sh, "git fetch https://github.com/rust-lang/rust {base}").run()?;
160-
cmd!(sh, "git push https://github.com/{rust_fork} {base}:refs/heads/{branch}")
161-
.ignore_stdout()
162-
.ignore_stderr() // silence the "create GitHub PR" message
163-
.run()?;
164-
println!();
165-
166-
// Do the actual push.
167-
sh.change_dir(project_root());
168-
println!("Pushing rust-analyzer changes...");
169-
cmd!(
170-
sh,
171-
"git push http://localhost:{JOSH_PORT}/{rust_fork}.git{JOSH_FILTER}.git HEAD:{branch}"
172-
)
173-
.run()?;
174-
println!();
175-
176-
// Do a round-trip check to make sure the push worked as expected.
177-
cmd!(
178-
sh,
179-
"git fetch http://localhost:{JOSH_PORT}/{rust_fork}.git{JOSH_FILTER}.git {branch}"
180-
)
181-
.ignore_stderr()
182-
.read()?;
183-
let head = cmd!(sh, "git rev-parse HEAD").read()?;
184-
let fetch_head = cmd!(sh, "git rev-parse FETCH_HEAD").read()?;
185-
if head != fetch_head {
186-
bail!(
187-
"Josh created a non-roundtrip push! Do NOT merge this into rustc!\n\
188-
Expected {head}, got {fetch_head}."
189-
);
190-
}
191-
println!(
192-
"Confirmed that the push round-trips back to rust-analyzer properly. Please create a rustc PR:"
193-
);
194-
// https://github.com/github-linguist/linguist/compare/master...octocat:linguist:master
195-
let fork_path = rust_fork.replace('/', ":");
196-
println!(
197-
" https://github.com/rust-lang/rust/compare/{fork_path}:{branch}?quick_pull=1&title=Subtree+update+of+rust-analyzer&body=r?+@ghost"
198-
);
199-
200-
drop(josh);
201-
Ok(())
202-
}
203-
}
204-
205-
/// Used for rustc syncs.
206-
const JOSH_FILTER: &str = ":rev(55d9a533b309119c8acd13061581b43ae8840823:prefix=src/tools/rust-analyzer):/src/tools/rust-analyzer";
207-
const JOSH_PORT: &str = "42042";
208-
209-
fn start_josh() -> anyhow::Result<impl Drop> {
210-
// Determine cache directory.
211-
let local_dir = {
212-
let user_dirs = ProjectDirs::from("org", "rust-lang", "rust-analyzer-josh").unwrap();
213-
user_dirs.cache_dir().to_owned()
214-
};
215-
216-
// Start josh, silencing its output.
217-
let mut cmd = Command::new("josh-proxy");
218-
cmd.arg("--local").arg(local_dir);
219-
cmd.arg("--remote").arg("https://github.com");
220-
cmd.arg("--port").arg(JOSH_PORT);
221-
cmd.arg("--no-background");
222-
cmd.stdout(Stdio::null());
223-
cmd.stderr(Stdio::null());
224-
let josh = cmd.spawn().context("failed to start josh-proxy, make sure it is installed")?;
225-
// Give it some time so hopefully the port is open. (100ms was not enough.)
226-
thread::sleep(Duration::from_millis(200));
227-
228-
Ok(JodChild(josh))
229-
}

0 commit comments

Comments
 (0)