Skip to content

Commit 656b77b

Browse files
cretzcretz
authored
Add simple benchmark script (#180)
1 parent b4b3f82 commit 656b77b

File tree

9 files changed

+310
-7
lines changed

9 files changed

+310
-7
lines changed

‎.github/workflows/nightly.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
name: Nightly
2+
3+
on:
4+
schedule:
5+
# (12 AM PST)
6+
- cron: "00 07 * * *"
7+
8+
jobs:
9+
nightly:
10+
uses: ./.github/workflows/run-bench.yml

‎.github/workflows/run-bench.yml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: Run Bench
2+
on:
3+
workflow_call:
4+
inputs:
5+
sandbox-arg:
6+
description: "Sandbox argument"
7+
required: false
8+
default: "--sandbox"
9+
type: string
10+
workflow_dispatch:
11+
inputs:
12+
sandbox-arg:
13+
description: "Sandbox argument"
14+
required: false
15+
default: "--sandbox"
16+
type: choice
17+
options:
18+
- "--sandbox"
19+
- "--no-sandbox"
20+
21+
jobs:
22+
run-bench:
23+
strategy:
24+
matrix:
25+
os: [ubuntu-latest-4-cores, windows-latest]
26+
runs-on: ${{ matrix.os }}
27+
steps:
28+
# Prepare
29+
- uses: actions/checkout@v2
30+
with:
31+
submodules: recursive
32+
- uses: actions-rs/toolchain@v1
33+
with:
34+
toolchain: stable
35+
- uses: Swatinem/rust-cache@v1
36+
with:
37+
working-directory: temporalio/bridge
38+
- uses: actions/setup-python@v4
39+
with:
40+
python-version: "3.11"
41+
42+
# Build
43+
- run: python -m pip install --upgrade wheel poetry poethepoet
44+
- run: poetry install --no-root -E opentelemetry
45+
- run: poe build-develop-with-release
46+
47+
# Run a bunch of bench tests. We run multiple times since results vary.
48+
49+
- run: poe run-bench --workflow-count 100 --max-cached-workflows 100 --max-concurrent 100 ${{ inputs.sandbox-arg }}
50+
- run: poe run-bench --workflow-count 100 --max-cached-workflows 100 --max-concurrent 100 ${{ inputs.sandbox-arg }}
51+
- run: poe run-bench --workflow-count 100 --max-cached-workflows 100 --max-concurrent 100 ${{ inputs.sandbox-arg }}
52+
53+
- run: poe run-bench --workflow-count 1000 --max-cached-workflows 1000 --max-concurrent 1000 ${{ inputs.sandbox-arg }}
54+
- run: poe run-bench --workflow-count 1000 --max-cached-workflows 1000 --max-concurrent 1000 ${{ inputs.sandbox-arg }}
55+
- run: poe run-bench --workflow-count 1000 --max-cached-workflows 1000 --max-concurrent 1000 ${{ inputs.sandbox-arg }}
56+
57+
- run: poe run-bench --workflow-count 1000 --max-cached-workflows 100 --max-concurrent 100 ${{ inputs.sandbox-arg }}
58+
- run: poe run-bench --workflow-count 1000 --max-cached-workflows 100 --max-concurrent 100 ${{ inputs.sandbox-arg }}
59+
- run: poe run-bench --workflow-count 1000 --max-cached-workflows 100 --max-concurrent 100 ${{ inputs.sandbox-arg }}
60+
61+
- run: poe run-bench --workflow-count 10000 --max-cached-workflows 10000 --max-concurrent 10000 ${{ inputs.sandbox-arg }}
62+
- run: poe run-bench --workflow-count 10000 --max-cached-workflows 10000 --max-concurrent 10000 ${{ inputs.sandbox-arg }}
63+
64+
- run: poe run-bench --workflow-count 10000 --max-cached-workflows 1000 --max-concurrent 1000 ${{ inputs.sandbox-arg }}
65+
- run: poe run-bench --workflow-count 10000 --max-cached-workflows 1000 --max-concurrent 1000 ${{ inputs.sandbox-arg }}

‎README.md

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -297,7 +297,8 @@ async def create_greeting_activity(info: GreetingInfo) -> str:
297297
Some things to note about the above code:
298298

299299
* Workflows run in a sandbox by default. Users are encouraged to define workflows in files with no side effects or other
300-
complicated code. See the [Workflow Sandbox](#workflow-sandbox) section for more details.
300+
complicated code or unnecessary imports to other third party libraries. See the [Workflow Sandbox](#workflow-sandbox)
301+
section for more details.
301302
* This workflow continually updates the queryable current greeting when signalled and can complete with the greeting on
302303
a different signal
303304
* Workflows are always classes and must have a single `@workflow.run` which is an `async def` function
@@ -642,7 +643,7 @@ is immutable and contains three fields that can be customized, but only two have
642643

643644
###### Passthrough Modules
644645

645-
To make the sandbox quicker when importing known third party libraries, they can be added to the
646+
To make the sandbox quicker and use less memory when importing known third party libraries, they can be added to the
646647
`SandboxRestrictions.passthrough_modules` set like so:
647648

648649
```python
@@ -708,7 +709,17 @@ The sandbox is only a helper, it does not provide full protection.
708709

709710
###### Sandbox Performance
710711

711-
TODO: This is actively being measured; results to come soon
712+
The sandbox does not add significant CPU or memory overhead for workflows that are in files which only import standard
713+
library modules. This is because they are passed through from outside of the sandbox. However, every
714+
non-standard-library import that is performed at the top of the same file the workflow is in will add CPU overhead (the
715+
module is re-imported every workflow run) and memory overhead (each module independently cached as part of the workflow
716+
run for isolation reasons). This becomes more apparent for large numbers of workflow runs.
717+
718+
To mitigate this, users should:
719+
720+
* Define workflows in files that have as few non-standard-library imports as possible
721+
* Alter the max workflow cache and/or max concurrent workflows settings if memory grows too large
722+
* Set third-party libraries as passthrough modules if they are known to be side-effect free
712723

713724
###### Extending Restricted Classes
714725

‎poetry.lock

Lines changed: 50 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ isort = "^5.10.1"
4141
mypy = "^0.971"
4242
mypy-protobuf = "^3.3.0"
4343
protoc-wheel-0 = "^21.1"
44+
psutil = "^5.9.3"
4445
pydantic = "^1.9.1"
4546
pydocstyle = "^6.1.1"
4647
# TODO(cretz): Update when https://github.com/twisted/pydoctor/pull/595 released
@@ -59,8 +60,8 @@ opentelemetry = ["opentelemetry-api", "opentelemetry-sdk"]
5960
grpc = ["grpc"]
6061

6162
[tool.poe.tasks]
62-
build-develop = ["build-bridge-develop"]
63-
build-bridge-develop = "python scripts/setup_bridge.py develop"
63+
build-develop = "python scripts/setup_bridge.py develop"
64+
build-develop-with-release = { cmd = "python scripts/setup_bridge.py develop", env = { TEMPORAL_BUILD_RELEASE = "1" }}
6465
fix-wheel = "python scripts/fix_wheel.py"
6566
format = [{cmd = "black ."}, {cmd = "isort ."}]
6667
gen-docs = "pydoctor"
@@ -75,6 +76,7 @@ lint = [
7576
# https://github.com/PyCQA/pydocstyle/pull/511?
7677
lint-docs = "pydocstyle --ignore-decorators=overload"
7778
lint-types = "mypy --namespace-packages ."
79+
run-bench = "python scripts/run_bench.py"
7880
test = "pytest"
7981

8082
# Install local, run single pytest with env var, uninstall local

‎scripts/run_bench.py

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
import argparse
2+
import asyncio
3+
import json
4+
import logging
5+
import sys
6+
import time
7+
import uuid
8+
from contextlib import asynccontextmanager
9+
from datetime import timedelta
10+
from typing import AsyncIterator
11+
12+
from temporalio import activity, workflow
13+
from temporalio.testing import WorkflowEnvironment
14+
from temporalio.worker import UnsandboxedWorkflowRunner, Worker
15+
from temporalio.worker.workflow_sandbox import SandboxedWorkflowRunner
16+
17+
18+
@workflow.defn
19+
class BenchWorkflow:
20+
@workflow.run
21+
async def run(self, name: str) -> str:
22+
return await workflow.execute_activity(
23+
bench_activity, name, start_to_close_timeout=timedelta(seconds=30)
24+
)
25+
26+
27+
@activity.defn
28+
async def bench_activity(name: str) -> str:
29+
return f"Hello, {name}!"
30+
31+
32+
async def main():
33+
logging.basicConfig(
34+
format="%(asctime)s.%(msecs)03d %(levelname)-8s %(message)s",
35+
level=logging.WARN,
36+
datefmt="%Y-%m-%d %H:%M:%S",
37+
)
38+
39+
logger = logging.getLogger(__name__)
40+
max_mem = -1
41+
42+
parser = argparse.ArgumentParser(description="Run bench")
43+
parser.add_argument("--workflow-count", type=int, required=True)
44+
parser.add_argument("--sandbox", action=argparse.BooleanOptionalAction)
45+
parser.add_argument("--max-cached-workflows", type=int, required=True)
46+
parser.add_argument("--max-concurrent", type=int, required=True)
47+
args = parser.parse_args()
48+
49+
@asynccontextmanager
50+
async def track_mem() -> AsyncIterator[None]:
51+
# We intentionally import in here so the sandbox doesn't grow huge with
52+
# this import
53+
import psutil
54+
55+
# Get mem every 800ms
56+
process = psutil.Process()
57+
58+
async def report_mem():
59+
nonlocal max_mem
60+
while True:
61+
try:
62+
await asyncio.sleep(0.8)
63+
finally:
64+
# TODO(cretz): "vms" appears more accurate on Windows, but
65+
# rss is more accurate on Linux
66+
used_mem = process.memory_info().rss
67+
if used_mem > max_mem:
68+
max_mem = used_mem
69+
70+
report_mem_task = asyncio.create_task(report_mem())
71+
try:
72+
yield None
73+
finally:
74+
report_mem_task.cancel()
75+
76+
logger.info("Running %s workflows", args.workflow_count)
77+
async with track_mem():
78+
# Run with a local workflow environment
79+
logger.debug("Starting local environment")
80+
async with await WorkflowEnvironment.start_local() as env:
81+
task_queue = f"task-queue-{uuid.uuid4()}"
82+
83+
# Create a bunch of workflows
84+
logger.debug("Starting %s workflows", args.workflow_count)
85+
pre_start_seconds = time.monotonic()
86+
handles = [
87+
await env.client.start_workflow(
88+
BenchWorkflow.run,
89+
f"user-{i}",
90+
id=f"workflow-{i}-{uuid.uuid4()}",
91+
task_queue=task_queue,
92+
)
93+
for i in range(args.workflow_count)
94+
]
95+
start_seconds = time.monotonic() - pre_start_seconds
96+
97+
# Start a worker to run them
98+
logger.debug("Starting worker")
99+
async with Worker(
100+
env.client,
101+
task_queue=task_queue,
102+
workflows=[BenchWorkflow],
103+
activities=[bench_activity],
104+
workflow_runner=SandboxedWorkflowRunner()
105+
if args.sandbox
106+
else UnsandboxedWorkflowRunner(),
107+
max_cached_workflows=args.max_cached_workflows,
108+
max_concurrent_workflow_tasks=args.max_concurrent,
109+
max_concurrent_activities=args.max_concurrent,
110+
):
111+
logger.debug("Worker started")
112+
# Wait for them all
113+
pre_result_seconds = time.monotonic()
114+
for h in handles:
115+
await h.result()
116+
result_seconds = time.monotonic() - pre_result_seconds
117+
logger.debug("All workflows complete")
118+
119+
# Print results
120+
json.dump(
121+
{
122+
"workflow_count": args.workflow_count,
123+
"sandbox": args.sandbox or False,
124+
"max_cached_workflows": args.max_cached_workflows,
125+
"max_concurrent": args.max_concurrent,
126+
"max_mem_mib": round(max_mem / 1024**2, 1),
127+
"start_seconds": round(start_seconds, 1),
128+
"result_seconds": round(result_seconds, 1),
129+
"workflows_per_second": round(args.workflow_count / result_seconds, 1),
130+
},
131+
sys.stdout,
132+
indent=2,
133+
)
134+
135+
136+
if __name__ == "__main__":
137+
asyncio.run(main())

‎scripts/setup_bridge.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import os
12
import shutil
23

34
from setuptools import setup
@@ -11,6 +12,8 @@
1112
binding=Binding.PyO3,
1213
py_limited_api=True,
1314
features=["pyo3/abi3-py37"],
15+
# Allow local release builds if requested
16+
debug=False if os.environ.get("TEMPORAL_BUILD_RELEASE") == "1" else None,
1417
)
1518
]
1619

0 commit comments

Comments
 (0)