Auto upgrading from clap v2 to v3

[pksunkara]

Some people might know that I have been trying to get a basic experiment working for a tool that auto upgrades user's code. For example, if the user depends on clap v2, they can run this tool and upgrade the code to work with clap v3. There are a lot more things to decide for this, but here is the result of the experiment.

https://github.com/pksunkara/cargo-up

It will replace Arg::help to Arg::about in normal builder usage of clap v2.

@CreepySkeleton @kbknapp Would like some feedback before I proceed.

[CreepySkeleton]

That's cool! I think I'll join you in week or so.

Is there some sort of documentation for ra_* stuff short of source code?

[pksunkara]

There are some dev/docs in the repo which talks about the overview of the codebase. From there, it is pretty easy to understand things.

[kbknapp]

Very cool! Is the plan to make this generic so anyone could potentially write "up" modules (for lack of a better term), or to hard code supported updates into cargo-up? Being able to write modules to upgrade code from one version to another could lead to being able to "chain" updates and would be a neat thing to be able to do!

[pksunkara]

Yes, the plan is to make it generic so any crate can write an up module. And yes, chaining updates is what the workflow is.

For example, our clap_up might have the following:

#[up(3.0.0-beta.1)]
rename_fn!("clap::args::arg", "Arg", "help", "about");

#[up(3.0.0-rc.0)]
rename_fn!("clap::arg", "Arg", "with_name", "new");

When the end user is using clap v2 and runs cargo up and chooses clap to upgrade to 3.0.0, then first all the 3.0.0-beta.1 changes are applied and then the 3.0.0-rc.0 changes after that and then any 3.0.0 changes after that.

[pksunkara]

I made a mistake, I was sleepy yesterday when I was writing the above comment.

Yes, we will have to do dynamic loading, but it's only to the extent of dependency injection. We can write a temporary main.rs and Cargo.toml which basically loads up the cargo-up and clap_up and then compile it and run it. It should not be issue. There is already prior work on this, https://github.com/rust-db/barrel.

[CreepySkeleton]

Um, as far as I figure, barrel doesn't load and interpret rust code at runtime. Or does it?

We can write a temporary main.rs and Cargo.toml which basically loads up the cargo-up and clap_up and then compile it and run it.

So, the large part of what we do is basically a library, not executable. This lib must be rlib type in order avoid recompiling rust-analyzer (which is a looong compilation), simultaneously allowing users to use rust API. cargo-up takes the script, builds a main.rs wrapper wrapper around it, runs cargo run.

Did I get it right?

[pksunkara]

load and interpret rust code at runtime

Can you expand on that? If you are talking about interpreting rust code of the crate we are working on upgrading, then it's just reading files and converting into AST. If you are talking about the rust code we are executing, then yes, barrel does do load and execute it during runtime using the wrapper method I mentioned above.

We will be doing both a library and executable, but executable's job also contains building another executable. I also want to incorporate a little bit of stuff from https://npmjs.com/package/npm-check.

I haven't thought about rlib, good idea.

Did I get it right?

Yes. Unless I misunderstood something else.

[CreepySkeleton]

If you are talking about the rust code we are executing

Yes, that's what I've been talking about.

I haven't thought about rlib, good idea.

I was actually getting to why this is a bad idea 😏 .

Rust doesn't have a standardized ABI yet and is unlikely to develop one in foreseeable future. This means, among other things, that every rlib compatible only with the rustc it was build with. What if we build our rlib with 1.43 and the user has 1.42 installed? Things may very well crumble and fall apart in a very cryptic way.

"Hold on! We can simply have the user rebuild rust-analyzer!", I hear you note. The problem is that users would have to recompile rust-analyzer for each and every project they'd want to update. Building it in release mode takes ~15 minutes on the build server which is not exactly a no-go, but it's absolutely unnecessary!

All these problems steam from two implications:

• We use a compiled language for what is essentially script. An interpreted language (with an interpreted bundled) would fit much better here.
• We rely on user's compiler instead of bundling it ourselves. It's much harder to bundle a compilator than an interpreter because interpreters are often purposely designed to be bundled and embedded.

That said, I propose to consider using an interpreted language, be it Python or Lua or Gluon or whatever.

[pksunkara]

We don't need release mode and it actually wasn't that big of timesink to compile rust-analyzer, (atleast that is what I observed). We can maintain a single target cache for all projects. We can always pivot to (or just add) the approach you are talking about. I will keep this in mind. Let's start with my proposed approach which allows more powerful modifications.

Thanks for talking about this, you raised some good points which I haven't considered.

[CreepySkeleton]

Concerns:

• Failures. Not every single update can be done flawlessly, sometimes there's no one-to-one mapping between versions, or the mapping is too complicated/sophisticated to perform automatically. Good descriptive error messages are vital.
• Interactivity. The process may be interactive in some cases: "there are two options, this and this, which one do you prefer?". Think git merge in presence of conflicts.
• Dependencies. "Apply change X only if Y has also been applied". Transitive deps, cycles, AND OR NOT relationship, combinations of those... I anticipate it's going to be FUN!
• Context awareness. "Rename Arg::help to Arg::about only if App::something is also called on this particular instance". I see we are going to hit a few cases like this in derive. Even more fun!
• Conflicts. "Rename App::help to App::about" and "Rename App::help to App::hint".

It's not going to be easy...

[pksunkara]

Yup, it's not going to be easy, but I think this is something that is desperately needed in the current tech landscape. We will start small and build up to big stuff. It is going to be challenging which makes me very interested in it 😄

DONT BE AFRAID TO UPGRADE is the motto.