Feature gate for HTTP client

[gmilleramilar]

Pulling in the reqwest crate results in a huuuge number of sub-dependencies, increasing build times and complicating the software supply chain. Would you be amenable to putting it behind a feature gate? I can send a pull request.

[mmarx]

I did a quick experiment to check which would be the crates we would (currently) no longer depend on if we feature-gated reqwest. Some of those (e.g., brotli and zstd) we might want to add back in to support reading compressed files using those formats. Overall, this doesn't look like it would save that much (in particular, we're still pulling in tokio with all its dependencies, and this is really unavoidable). On the other hand, properly feature-gating this will require new error handling to indicate that the feature is not available (which needs to be maintained and tested as well). Thus, I'd say this is probably not worth it.

What use case do you have in mind where (i) you don't need support for remote resources (especially, but not limited to, SPARQL queries) and (ii) you repeatedly build Nemo from an empty cargo cache?

Feature-gating `reqwest` would currently avoid a dependency on these crates

• alloc-no-stdlib
• alloc-stdlib
• async-compression
• atomic-waker
• base64
• bitflags
• brotli
• brotli-decompressor
• cc
• encoding_rs
• fnv
• foreign-types
• foreign-types-shared
• h2
• http
• http-body
• http-body-util
• hyper
• hyper-tls
• hyper-util
• ipnet
• iri-string
• jobserver
• lsp-document
• mime
• mio
• native-tls
• openssl
• openssl-macros
• openssl-probe
• openssl-sys
• pkg-config
• reqwest
• rustls-pki-types
• shlex
• socket2
• sync_wrapper
• tokio-native-tls
• tower
• tower-http
• tower-lsp
• try-lock
• vcpkg
• want
• zeroize
• zstd
• zstd-safe
• zstd-sys

Once we add brotli und zstd decompression, this goes down to even less

• async-compression
• atomic-waker
• base64
• bitflags
• encoding_rs
• fnv
• foreign-types
• foreign-types-shared
• h2
• http
• http-body
• http-body-util
• hyper
• hyper-tls
• hyper-util
• ipnet
• iri-string
• lsp-document
• mime
• mio
• native-tls
• openssl
• openssl-macros
• openssl-probe
• openssl-sys
• reqwest
• rustls-pki-types
• socket2
• sync_wrapper
• tokio-native-tls
• tower
• tower-http
• tower-lsp
• try-lock
• vcpkg
• want
• zeroize

[gmilleramilar]

What use case do you have in mind where (i) you don't need support for remote resources (especially, but not limited to, SPARQL queries) and (ii) you repeatedly build Nemo from an empty cargo cache?

Use case is local analysis of compiled code in a verifier. CI servers tend to build from scratch every time. Also, as I mentioned, it complicates the software supply chain.

Some of those (e.g., brotli and zstd) we might want to add back in to support reading compressed files using those formats.

To me compression doesn't sound like a core responsibility of a Datalog engine (TBH, neither do SPARQL queries). But I'm sure you have a better picture of what users of Nemo want.

No worries, I can maintain a fork, thanks for your consideration.

[mmarx]

Ah, so you're using nemo as a crate, then? If this only hits the nemo crate, we could get by without any new diagnostics and just assume that the feature is always enabled.