Skip to content

Conversation

dannycjones
Copy link
Contributor

@dannycjones dannycjones commented Jun 25, 2025

This change adds the ability to configure the prefetch benchmark with a disk cache. The configuration for the benchmark is slightly different from the main mount-s3 CLI - we allow the cache cleanup to be turned off to test warm caches. Additionally, the eviction mechanism is turned off and the cache is unbounded.

This patch includes some small refactors, to allow us to share defaults across benchmarks and the main CLI such that they are kept in sync.

It also introduces CLI headings for the help, to make it clearer to users (us) which options relate to which components: S3 client, prefetcher, etc..

Does this change impact existing behavior?

Benchmark change only.

Does this change need a changelog entry? Does it require a version change?

No, benchmark change only.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

@dannycjones dannycjones requested review from a team, mansi153 and sahityadg June 25, 2025 13:57
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 25, 2025 13:58 — with GitHub Actions Inactive
Signed-off-by: Daniel Carl Jones <djonesoa@amazon.com>
Signed-off-by: Daniel Carl Jones <djonesoa@amazon.com>
@dannycjones dannycjones force-pushed the add-cache-to-prefetch-benchmark branch from 7dbc290 to 79a29f6 Compare June 27, 2025 13:28
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones had a problem deploying to PR integration tests June 27, 2025 13:28 — with GitHub Actions Failure
@dannycjones dannycjones force-pushed the add-cache-to-prefetch-benchmark branch from 79a29f6 to 65412ad Compare June 27, 2025 14:18
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests June 27, 2025 14:18 — with GitHub Actions Inactive
@dannycjones dannycjones marked this pull request as ready for review June 27, 2025 14:21
@dannycjones
Copy link
Contributor Author

New help output for prefetch_benchmark:

Run workloads against the prefetcher component of Mountpoint. Fetched data is discarded.

Usage: prefetch_benchmark [OPTIONS] <BUCKET> <S3_KEY>

Arguments:
  <BUCKET>  S3 bucket name containing the S3 object to fetch
  <S3_KEY>  S3 object key to fetch

Options:
      --region <AWS_REGION>  AWS region of the bucket [default: us-east-1]
  -h, --help                 Print help

S3 client options:
      --maximum-throughput-gbps <Gib/s>
          Target throughput in gibibits per second
      --crt-memory-limit-gib <GiB>
          Override value for CRT memory limit in gibibytes
      --part-size <BYTES>
          Part size for multi-part GET in bytes
      --bind <NETWORK_INTERFACE>
          One or more network interfaces to use when accessing S3. Requires Linux 5.7+ or running as root.

Prefetcher options:
      --max-memory-target <MiB>    Maximum memory usage target for Mountpoint's memory limiter [default: 95% of total system memory]
      --read-size <BYTES>          Size of read requests requests to the prefetcher [default: 131072]
      --disk-cache <DIRECTORY>     Enable caching of object content to the given directory. The disk cache has no max size for this benchmark, so you must ensure there is enough available space. [aliases: --cache]
      --disk-cache-cleanup <MODE>  Configure how the benchmark cleans the data cache directory. [default: every-iteration] [possible values: never, every-iteration]

Benchmark options:
      --iterations <N>  Number of times to download the S3 object [default: 1]
      --downloads <N>   Number of concurrent downloads [default: 1]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant