Rust dependency propagation issues into downstream projects

[ft-filancore]

Bug description

The current way Rust dependencies are treated causes the following symptoms for downstream users:

• conflict on tokio versions (previously also on chrono versions) because of pins in this repository
• lockfile size explosion compared to an equal setup using stardust
• audit lint failures due to dependencies and transitive dependencies, some of which unused / not needed for actual downstream consumption

The first point makes working with the remaining ecosystem harder than usual, the latter two parts are major concerns for security-related products that need to keep audit their dependencies / software bills of materials.

This is somewhat related to iotaledger/identity#1665 but I could not find a corresponding issue in this repository so I'm opening this for upstream discussion.

I consider this a bug in this workspace's dependency management because it causes downstream problems not common in the Rust ecosystem.

Versions

• Rust version: 1.88
• library version number, commit, or branch: 1.2.3

Steps To reproduce the bug

There are a few ways:

• Scenario 1:

  • take an average up-to-date repository using iota-sdk 1.1.5 (stardust)
  • run cargo update && cargo upgrade
  • migrate to rebased (iota-sdk 1.2.3 and potentially identity.rs and/or notarization)
  • see that you have to downgrade dependencies like tokio and with that potentially other packages in your workspace using newer tokio to satisfy IOTA

• Scenario 2:

  • take an average up-to-date repository using iota-sdk 1.1.5 (stardust)
  • migrate to rebased (iota-sdk 1.2.3 and potentially identity.rs and/or notarization)
  • diff the lockfiles → there will be an explosion in dependencies (in our project it's a net increase of 4,000 lines in the Cargo.lock)

• Scenario 3:

  • take an average up-to-date repository using iota-sdk 1.1.5 (stardust)
  • migrate to rebased (iota-sdk 1.2.3 and potentially identity.rs and/or notarization)
  • run cargo deny check with a default config
  • notice that you have to allow more security advisories than before
  • notice that you have to allow git sources that are too large to really audit before using

(We stumbled across this in the migration of a medium-sized product from Stardust to Rebased.)

Expected behaviour

The IOTA workspace should use its dependencies so that

• the workspace does not depend on features and disables default features: Rust features are additive, you enable them for all crates in a workspace pulling in a dependency, e.g., you enable multiple axum feature where you may not need all those axum features in all crates or more relevant to downstream:
  • iota-sdk pulls in clap
  • the workspace activates clap's derive feature
  • now my library-consuming crate gets all the dependencies and proc macros for running a CLI application just for it to be compiled out later
• clearly separate dev-dependencies from dependencies: if you need a dependency or its feature only for dev (tests, …), only enable it in the dev-dependencies, that way it does not cause downstream propagation of this dependency or its transitive dependencies through features
• do not pin dependencies for downstream projects: pinning tokio as a central ecosystem crate in a library that is primarily meant as an HTTP client (the iota-sdk connecting to JSON-RPC nodes) means breaking the stack of basically all clients without a major reason
• reduce the use of git dependencies: please start publishing the crates for easier review of versions, better fetching/caching in cargo, and less audit lint failures for unreviewable sources

There are alternatives here:

• if the workspace dependencies can't be salvaged, maybe it's more feasible to provide an SDK with a smaller dependency footprint (e.g., a Rust crate that doesn't pull clap, pinned tokio, a TLS provider etc. on its own when it's meant as a library SDK)?
• split the workspace into internal development-facing and downstream-facing
• …

Actual behaviour

See above, the IOTA workspace dependencies propagate downstream, causing

• version clashes
• lockfile explosions
• new security advisory fails
• pulling of quite a few git dependencies

[thibault-martinez]

Hi @ft-filancore,

Thanks for opening this issue.
These are all valid concerns that we are fully aware of and already working on.

• conflict on tokio versions (previously also on chrono versions) because of pins in this repository

#7951

• lockfile size explosion compared to an equal setup using stardust
• audit lint failures due to dependencies and transitive dependencies, some of which unused / not needed for actual downstream consumption

This is an unfortunate situation that we inherited from upstream and there is nothing we can do about it immediately.
That being said, we are working on a completely new SDK that does not depend on the monorepo at all, meaning a minimal set of dependencies. This is expected to be beta-released towards the end of this quarter.
If you are interested, this is the branch we are working on https://github.com/iotaledger/iota-rust-sdk/tree/sdk-bindings