OSS Android Benchmarking: minibench and microbench

[kirklandsign]

🚀 The feature, motivation and pitch

We have an open source Android benchmarking app (minibench) which creates Activities for benchmarking ET runtime/model in a real app setup.

It does well in simulating a real app user use case, because all apps need to be wrapped in an APK, not a shell binary, so it has the same permission and OOM priority as the user app. However, it doesn’t do well in generating profiling reports and flame graphs with simpleperf/perfetto, without Android Studio. We need a separate binary program, microbench.

Meanwhile, the new microbench can be used to benchmark and profile single operators, useful for fine granite tuning.

Minibench (existing)

• The first step to get the overall idea of how a model runs on a platform.
• Simulates real user workflow.
• Can't generate profiling reports on servers.
• Needs adb shell am to start, and hard to debug when there is an error.

Microbench (new)

• Get detailed profiling data.
• Native binary, similar to running a program in linux.
• simpleperf/perfetto can generate profiling reports.
• If a model runs on microbench, it may not necessarily run on minibench, or real user app, because binary may have higher OOM priority (even than System UI!).

Tasks

• Build the new microbench app (binary) for OSS.
• Obtain the simpleperf/perfetto profiling data, and write a script for it.
• Set up workflow to trigger the run.

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @guangy10 @huydhn @shoumikhin

[guangy10]

the new microbench can be used to benchmark and profile single operators, useful for fine granite tuning

How to support this? Profiling single operator means will compile a new binary with the main function just run the operator to be benchmarked right? That seems to be a different binary than microbench.

If a model runs on microbench, it may not necessarily run on minibench, or real user app, because binary may have higher OOM priority (even than System UI!).

If binary is likely to be killed due to OOM, won't be the opposite instead, i.e. if a model runs on minibench, it may not necessarily run on microbench?

[guangy10]

For Android, if the profiler can run only with binary, maybe we don't need to build the new binary, users can build whatever binary they want. Then they can push it to S3, and schedule a benchmark job via the script. The UX can look like this:

Step 1:
benchmark.py --publish <local/path/to/my/binary>, which will upload the binary to S3 and return the URL.
Step 2:
benchmark.py --model my_model, --config my_benchmark_config --binary <URL link to s#> --enable_profiling which will fetch the prebuilt binary from S3 and scheduling a benchmark job with profiling enabled. At the end of the run, the generated artifacts will be pushed to S3 for users to download and visualize.

wdyt @kirklandsign @digantdesai @kimishpatel ? Would this mimic your local dev with profilers, or dev using AIBench?

[kimishpatel]

For Android, if the profiler can run only with binary, maybe we don't need to build the new binary, users can build whatever binary they want. Then they can push it to S3, and schedule a benchmark job via the script. The UX can look like this:

Step 1: benchmark.py --publish <local/path/to/my/binary>, which will upload the binary to S3 and return the URL. Step 2: benchmark.py --model my_model, --config my_benchmark_config --binary <URL link to s#> --enable_profiling which will fetch the prebuilt binary from S3 and scheduling a benchmark job with profiling enabled. At the end of the run, the generated artifacts will be pushed to S3 for users to download and visualize.

wdyt @kirklandsign @digantdesai @kimishpatel ? Would this mimic your local dev with profilers, or dev using AIBench?

Do you I need the two step process. Sounds like implementation detail that can be hidden behind a wrapper?

[kirklandsign]

How to support this? Profiling single operator means will compile a new binary with the main function just run the operator to be benchmarked right? That seems to be a different binary than microbench.

Maybe just export a single-op model?

If binary is likely to be killed due to OOM, won't be the opposite instead, i.e. if a model runs on minibench, it may not necessarily run on microbench?

The other way. if a model runs on microbench (native binary), it may not necessarily run on minibench (APK)

users can build whatever binary they want. Then they can push it to S3, and schedule a benchmark job via the script

There is security concern. So ideally upload PTE, instead of executable. The implication is they can't benchmark arbitrary (their own) backed. But arbitrary backend also has security concern on our OSS infra.

[guangy10]

For Android, if the profiler can run only with binary, maybe we don't need to build the new binary, users can build whatever binary they want. Then they can push it to S3, and schedule a benchmark job via the script. The UX can look like this:
Step 1: benchmark.py --publish <local/path/to/my/binary>, which will upload the binary to S3 and return the URL. Step 2: benchmark.py --model my_model, --config my_benchmark_config --binary <URL link to s#> --enable_profiling which will fetch the prebuilt binary from S3 and scheduling a benchmark job with profiling enabled. At the end of the run, the generated artifacts will be pushed to S3 for users to download and visualize.
wdyt @kirklandsign @digantdesai @kimishpatel ? Would this mimic your local dev with profilers, or dev using AIBench?

Do you I need the two step process. Sounds like implementation detail that can be hidden behind a wrapper?

@kimishpatel I think you can do it in one step. Showing the two-step case just to highlight that you can reuse the pre-uploaded artifacts again in the new run. Not necessarily that you will always have to upload new artifacts for developers efficiency.

@digantdesai @kimishpatel if the infra allows you to upload any binary to S3 and use it for benchmarking on Android w/ profiling, do you think the microbench is still needed?

[guangy10]

How to support this? Profiling single operator means will compile a new binary with the main function just run the operator to be benchmarked right? That seems to be a different binary than microbench.

Maybe just export a single-op model?

If binary is likely to be killed due to OOM, won't be the opposite instead, i.e. if a model runs on minibench, it may not necessarily run on microbench?

The other way. if a model runs on microbench (native binary), it may not necessarily run on minibench (APK)

So it is the apk get higher priority to be killed due to OOM?

users can build whatever binary they want. Then they can push it to S3, and schedule a benchmark job via the script

There is security concern. So ideally upload PTE, instead of executable. The implication is they can't benchmark arbitrary (their own) backed. But arbitrary backend also has security concern on our OSS infra.

Can you be specific about the security concern?
From the infra perspective, users can upload any artifacts to be used by the infra, not only the pte file, but the customized binary/app. Of course there is an assumption that users should know what they are uploading. IIUC AIBench can support deploy customized binary to Android for benchmarking, right?

[kirklandsign]

So it is the apk get higher priority to be killed due to OOM?

Yes

Of course there is an assumption that users should know what they are uploading. IIUC AIBench can support deploy customized binary to Android for benchmarking, right

Yes for AIBench user can deploy customized binary. For OSS, we shouldn't let user do that right? I can submit CPU intensive jobs or malicious app.

[guangy10]

Yes for AIBench user can deploy customized binary. For OSS, we shouldn't let user do that right? I can submit CPU intensive jobs or malicious app.

We're referring to different users I think. Benchmarking with profiling and the ability to support benchmark using uploaded artifacts are for ET developers, utilizing the on-demand workflow. There is no plan to open this up to entire OSS users in the short-term. So we should grant the capability for ET developers (whoever has admin or write permissions) to deploy customized model and binaries.

[kimishpatel]

There is no plan to open this up to entire OSS users in the short-term. So we should grant the capability for ET developers (whoever has admin or write permissions) to deploy customized model and binaries.

Thats right.

Regarding operator benchmarking via microbench or such:

If our workflow allows benchmarking any binary then we dont have to distinguish between model and operator really. A binary that runs models deals with model benchmarking. But I can write a standalone benchmark binary using google benchmark suite and that could also work. Wrapping op in model is fine as well but binary allows us the flexiblity