Please reduce package size

[samuelcolvin]

Describe the bug

The botocore package is massive, from pypi:

• botocore-1.38.30.tar.gz (13.9 MB)
• botocore-1.38.30-py3-none-any.whl (13.6 MB)

Most of this size comes from the botocore/data/ directory.

How many people are actually using this data directory?

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

My suggestion would be to move that data to a separate package and make that an optional dependency.

If you really don't want to break existing behaviour, create a botocore-slim package and then make botocore include both botocore-slim and botocore-data.

Either way, botocore is one of the most commonly downloaded python packages, so it would be advantageous to PyPI and your users to make this package much smaller.

Current Behavior

download takes ages over slow connections. We'll probably remove botocore from being a default dependency of pydantic-ai if this isn't fixed.

Reproduction Steps

uv add botocore

Possible Solution

No response

Additional Information/Context

No response

SDK version used

botocore 1.38.30

Environment details (OS name and version, etc.)

NA

[nateprewitt]

Hi @samuelcolvin,

Thanks for reaching out. The SDK is actually already fairly optimized for size, it was previously ~90MB on disk, we have the majority of the data directory compressed on disk now until runtime. The data directory unfortunately is the SDK, removing it would mean you have no functional code for AWS. The data files are used at runtime to generate the client and all of its contents through metaprogramming.

We've spent a fair amount of time trying to streamline this but are always open to further refinement/suggestions. The models will need to stay in the SDK though as they're core functionality. We are currently working on a more modular SDK to avoid needing all AWS services and you'll be able to choose what you need. That will likely be the longer term solution.

[nateprewitt]

Taking a closer look at pydantic-ai, it looks like you're only using bedrock-runtime? If that's the case, the modular SDK I referenced above has an experimental client available for only bedrock-runtime. It's still going to have some potential functionality gaps in regards to credential sources, but may be worth having a look at if you need something slimmer. It's available on pypi as aws-sdk-bedrock-runtime.

[Kludex]

@nateprewitt This package only supports 3.12+...

https://github.com/awslabs/aws-sdk-python/blob/387d9c39484391354f62dfb37a37d654f68575fb/clients/aws-sdk-bedrock-runtime/pyproject.toml#L9

[nateprewitt]

That is correct, if that's a limitation, then it may not be suitable for pydantic-ai.