Skip to content

Python 3.13 support: wrapping update #359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 68 commits into from
Closed
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
5b47fc8
Create new process worker script and run it on context creation
bitterpanda63 Mar 20, 2025
6aa778b
Remove INITIALIZE_ROUTE command
bitterpanda63 Mar 21, 2025
ead26f8
Cleanup of process_worker logic
bitterpanda63 Mar 21, 2025
932d840
Create a process_worker and process_worker_loader in thread
bitterpanda63 Mar 21, 2025
c10d834
make thread_cache process-llocal
bitterpanda63 Mar 21, 2025
312c115
Improve route increment stuff
bitterpanda63 Mar 21, 2025
a1f9a25
fix thread_cache_test test cases
bitterpanda63 Mar 21, 2025
a74b346
Fix all test cases that relied on internal thread cache logic
bitterpanda63 Mar 21, 2025
2f08b88
Cleanup thread_cache.py
bitterpanda63 Mar 21, 2025
950849c
make log message make some sense
bitterpanda63 Mar 21, 2025
436f630
fix thread_cache res check and only start process_worker with context
bitterpanda63 Mar 21, 2025
f6d6cfc
Fix broken routes test
bitterpanda63 Mar 21, 2025
3f570f3
Fix django mysql e2e test
bitterpanda63 Mar 25, 2025
f41512e
Also add apispec to e2e test
bitterpanda63 Mar 25, 2025
19c7b62
Total is 3 requests for django_mysql e2e
bitterpanda63 Mar 26, 2025
5b83da4
Fix request_handler
bitterpanda63 Mar 26, 2025
0c929cc
Fix test cases of request_handler
bitterpanda63 Mar 26, 2025
05a87f5
Fix request_handler and it's test cases
bitterpanda63 Mar 26, 2025
4ddd294
Fix broken test case for vulnerability scanner
bitterpanda63 Mar 26, 2025
c43ab2f
validate the firewall lists API response correctly
bitterpanda63 Mar 26, 2025
645e303
Merge remote-tracking branch 'origin/fix/validate-firewall-lists-corr…
bitterpanda63 Mar 26, 2025
61c9ae9
Merge branch 'main' into improve-thread-caching-separated
bitterpanda63 Mar 26, 2025
3d228bb
Merge branch 'main' into improve-thread-caching-separated
bitterpanda63 Apr 1, 2025
960212f
Merge remote-tracking branch 'origin/main' into improve-thread-cachin…
bitterpanda63 Apr 1, 2025
5782587
Improves heartbeat event validation
bitterpanda63 Apr 1, 2025
d3225ba
Merge remote-tracking branch 'origin/main' into improve-thread-cachin…
bitterpanda63 Apr 10, 2025
b08726d
Merge branch 'main' into improve-thread-caching-separated
bitterpanda63 Apr 10, 2025
116d7b2
Test 3.13 as well for unit tests and e2e tests
bitterpanda63 Apr 14, 2025
9d9ad25
Benchmarks for all supported python versions
bitterpanda63 Apr 14, 2025
72cd57d
init tests as a module
bitterpanda63 Apr 14, 2025
8336b97
Refactor pymysql to use wrapt
bitterpanda63 Apr 14, 2025
dde3ba9
(poetry) install wrapt
bitterpanda63 Apr 14, 2025
47058fd
Update subprocess patching
bitterpanda63 Apr 14, 2025
32034d4
pymysql patches change order
bitterpanda63 Apr 14, 2025
7dc0a6f
move back to bottom, python order for pymysql, subprocess
bitterpanda63 Apr 14, 2025
e16b8ed
convert os_system patch
bitterpanda63 Apr 14, 2025
15a28eb
pymongo convert code to wrapt code
bitterpanda63 Apr 14, 2025
23e5fcf
convert mysqlclient
bitterpanda63 Apr 14, 2025
76894de
convert io module
bitterpanda63 Apr 14, 2025
afe9b77
Convert shutil module
bitterpanda63 Apr 14, 2025
142770d
Convert asyncpg module
bitterpanda63 Apr 14, 2025
fb309e7
Convert builtins.py
bitterpanda63 Apr 14, 2025
91ca20d
define new standards for the wrapping
bitterpanda63 Apr 14, 2025
dd9b4b0
update existing modules
bitterpanda63 Apr 14, 2025
9d37dc9
Update io module to use new system
bitterpanda63 Apr 14, 2025
1dab0f2
Update both builtins.py and asyncpg.py to use the new patching system
bitterpanda63 Apr 14, 2025
cffd729
Cleanup checks for builtins and shutil
bitterpanda63 Apr 14, 2025
0196b11
linting for asyncpg.py
bitterpanda63 Apr 14, 2025
6a9711d
Cleanup the patching module
bitterpanda63 Apr 14, 2025
57faa80
Convert os.py
bitterpanda63 Apr 14, 2025
e8c50d3
Update http_client.py sink, adding the @after
bitterpanda63 Apr 14, 2025
87892fd
Update psycopg sink
bitterpanda63 Apr 14, 2025
fa9f54f
convert socket.py to new format
bitterpanda63 Apr 14, 2025
cb37a0c
Fix shutil and shutil test cases
bitterpanda63 Apr 14, 2025
06e7abd
Convert psycopg2 module
bitterpanda63 Apr 14, 2025
6af31a4
update sample app lockfiles
bitterpanda63 Apr 14, 2025
7834861
Update actions/setup-python v2 -> v5
bitterpanda63 Apr 15, 2025
70305bf
Fix @after by removing finally: which was swallowing the error
bitterpanda63 Apr 15, 2025
35c4c82
Fix broken subprocess test cases
bitterpanda63 Apr 15, 2025
3618197
Remove ms checks for starlette benchmark in favour of percentages
bitterpanda63 Apr 15, 2025
31705c1
Exclude psycopg2 testing for python 3.13
bitterpanda63 Apr 15, 2025
3aa49cc
Make sure psycopg2.py implementation is the same as prev one
bitterpanda63 Apr 15, 2025
807b981
Add extra tests for psycopg2 and fix issue for immutables
bitterpanda63 Apr 15, 2025
b8eec8f
use assert_any_call for python 3.13
bitterpanda63 Apr 15, 2025
0aabc2c
Patch os.path.realpath for python 3.13
bitterpanda63 Apr 16, 2025
b29cc3e
Patch os.path.realpath for python 3.13
bitterpanda63 Apr 16, 2025
5a2037c
Merge remote-tracking branch 'origin/python-3.13-support' into python…
bitterpanda63 Apr 16, 2025
f77ec4a
Merge branch 'main' into python-3.13-support
bitterpanda63 May 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: Benchmark
name: 📊 Benchmarks
on:
push: {}
workflow_call: {}

jobs:
benchmark_sql_algorithm:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -30,15 +31,24 @@ jobs:
run: |
poetry run python ./benchmarks/sql_benchmark/sql_benchmark_fw.py
poetry run python ./benchmarks/sql_benchmark/sql_benchmark_no_fw.py

benchmark_with_flask_mysql:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start databases
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies & build
run: |
python -m pip install --upgrade pip
Expand All @@ -51,15 +61,24 @@ jobs:
- name: Run flask-mysql k6 Benchmark
run: |
k6 run -q ./benchmarks/flask-mysql-benchmarks.js

benchmark_with_starlette_app:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start databases
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies & build
run: |
python -m pip install --upgrade pip
Expand All @@ -71,9 +90,5 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install -y wrk
- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Run benchmark
run: python ./benchmarks/starlette_benchmark.py
2 changes: 1 addition & 1 deletion .github/workflows/end2end.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- { name: flask-postgres-xml, testfile: end2end/flask_postgres_xml_lxml_test.py }
- { name: quart-postgres-uvicorn, testfile: end2end/quart_postgres_uvicorn_test.py }
- { name: starlette-postgres-uvicorn, testfile: end2end/starlette_postgres_uvicorn_test.py }
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- name: Install packages
run: sudo apt update && sudo apt install python3-dev libmysqlclient-dev
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
# Don't cancel jobs if one fails
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
Expand Down
2 changes: 0 additions & 2 deletions aikido_zen/background_process/commands/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from aikido_zen.helpers.logging import logger
from .attack import process_attack
from .read_property import process_read_property
from .initialize_route import process_initialize_route
from .user import process_user
from .should_ratelimit import process_should_ratelimit
from .kill import process_kill
Expand All @@ -16,7 +15,6 @@
# This maps to a tuple : (function, returns_data?)
# Commands that don't return data :
"ATTACK": (process_attack, False),
"INITIALIZE_ROUTE": (process_initialize_route, False),
"USER": (process_user, False),
"KILL": (process_kill, False),
"STATISTICS": (process_statistics, False),
Expand Down
11 changes: 0 additions & 11 deletions aikido_zen/background_process/commands/initialize_route.py

This file was deleted.

37 changes: 0 additions & 37 deletions aikido_zen/background_process/commands/initialize_route_test.py

This file was deleted.

11 changes: 4 additions & 7 deletions aikido_zen/background_process/routes/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,17 @@ def __init__(self, max_size=1000):

def initialize_route(self, route_metadata):
"""
Initializes a route for the first time.
Initializes a route for the first time. `hits_delta_since_sync` counts delta between syncs.
"""
self.manage_routes_size()
key = route_to_key(route_metadata)
if self.routes.get(key):
return
self.routes[key] = {
"method": route_metadata.get("method"),
"path": route_metadata.get("route"),
"hits": 0,
"hits_delta_since_sync": 0,
"apispec": {},
}
# This field counts the difference in hits in between synchronisation for threads :
self.routes[key]["hits_delta_since_sync"] = 0

def increment_route(self, route_metadata):
"""
Expand All @@ -41,8 +38,8 @@ def increment_route(self, route_metadata):
"""
key = route_to_key(route_metadata)
if not self.routes.get(key):
return
# Add a hit to the route :
self.initialize_route(route_metadata)
# Add hits to
route = self.routes.get(key)
route["hits"] += 1
route["hits_delta_since_sync"] += 1
Expand Down
7 changes: 0 additions & 7 deletions aikido_zen/background_process/routes/init_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,13 +102,6 @@ def test_increment_route_twice():
assert routes.routes["GET:/api/resource"]["hits"] == 2


def test_increment_route_that_does_not_exist():
routes = Routes(max_size=3)
routes.increment_route(gen_route_metadata(route="/api/resource"))
routes.increment_route(gen_route_metadata(route="/api/resource"))
assert len(routes.routes) == 0


def test_clear_routes():
routes = Routes(max_size=3)
routes.initialize_route(gen_route_metadata(route="/api/resource"))
Expand Down
15 changes: 6 additions & 9 deletions aikido_zen/middleware/init_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,18 @@

import pytest
from aikido_zen.context import current_context, Context, get_current_context
from aikido_zen.thread.thread_cache import ThreadCache, threadlocal_storage
from aikido_zen.thread.thread_cache import ThreadCache, get_cache
from . import should_block_request


@pytest.fixture(autouse=True)
def run_around_tests():
get_cache().reset()
yield
# Make sure to reset context and cache after every test so it does not
# interfere with other tests
current_context.set(None)
get_cache().reset()


def test_without_context():
Expand Down Expand Up @@ -39,20 +41,15 @@ def set_context(user=None, executed_middleware=False):
).set_as_current_context()


class MyThreadCache(ThreadCache):
def renew_if_ttl_expired(self):
return


def test_with_context_without_cache():
set_context()
threadlocal_storage.cache = None
get_cache().cache = None
assert should_block_request() == {"block": False}


def test_with_context_with_cache():
set_context(user={"id": "123"})
thread_cache = MyThreadCache()
thread_cache = get_cache()

thread_cache.config.blocked_uids = ["123"]
assert get_current_context().executed_middleware == False
Expand All @@ -76,7 +73,7 @@ def test_with_context_with_cache():

def test_cache_comms_with_endpoints():
set_context(user={"id": "456"})
thread_cache = MyThreadCache()
thread_cache = get_cache()
thread_cache.config.blocked_uids = ["123"]
thread_cache.config.endpoints = [
{
Expand Down
70 changes: 70 additions & 0 deletions aikido_zen/sinks/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
from wrapt import wrap_object, FunctionWrapper, when_imported
from aikido_zen.background_process.packages import ANY_VERSION, is_package_compatible
from aikido_zen.errors import AikidoException
from aikido_zen.helpers.logging import logger


def on_import(name, package="", version_requirement=ANY_VERSION):
"""
Decorator to register a function to be called when a package is imported.
It checks if the package is compatible with the specified version requirement.
"""

def decorator(func):
if package and not is_package_compatible(package, version_requirement):
return

when_imported(name)(func) # Register the function to be called on import

return decorator


def patch_function(module, name, wrapper):
"""
Patches a function in the specified module with a wrapper function.
"""
try:
wrap_object(module, name, FunctionWrapper, (wrapper,))
except Exception as e:
logger.info("Failed to wrap %s:%s, due to: %s", module, name, e)


def before(wrapper):
"""
Surrounds a patch with try-except and calls the original function at the end
"""

def decorator(func, instance, args, kwargs):
try:
wrapper(func, instance, args, kwargs) # Call the patch
except AikidoException as e:
raise e # Re-raise AikidoException
except Exception as e:
logger.debug(
"%s:%s wrapping-before error: %s", func.__module__, func.__name__, e
)

return func(*args, **kwargs) # Call the original function

return decorator


def after(wrapper):
"""
Surrounds a patch with try-except, calls the original function and gives the return value to the patch
"""

def decorator(func, instance, args, kwargs):
return_value = func(*args, **kwargs) # Call the original function
try:
wrapper(func, instance, args, kwargs, return_value) # Call the patch
except AikidoException as e:
raise e # Re-raise AikidoException
except Exception as e:
logger.debug(
"%s:%s wrapping-after error: %s", func.__module__, func.__name__, e
)
finally:
return return_value

return decorator
Loading
Loading