Skip to content

Python 3.13 support: wrapping update #359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 68 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
5b47fc8
Create new process worker script and run it on context creation
bitterpanda63 Mar 20, 2025
6aa778b
Remove INITIALIZE_ROUTE command
bitterpanda63 Mar 21, 2025
ead26f8
Cleanup of process_worker logic
bitterpanda63 Mar 21, 2025
932d840
Create a process_worker and process_worker_loader in thread
bitterpanda63 Mar 21, 2025
c10d834
make thread_cache process-llocal
bitterpanda63 Mar 21, 2025
312c115
Improve route increment stuff
bitterpanda63 Mar 21, 2025
a1f9a25
fix thread_cache_test test cases
bitterpanda63 Mar 21, 2025
a74b346
Fix all test cases that relied on internal thread cache logic
bitterpanda63 Mar 21, 2025
2f08b88
Cleanup thread_cache.py
bitterpanda63 Mar 21, 2025
950849c
make log message make some sense
bitterpanda63 Mar 21, 2025
436f630
fix thread_cache res check and only start process_worker with context
bitterpanda63 Mar 21, 2025
f6d6cfc
Fix broken routes test
bitterpanda63 Mar 21, 2025
3f570f3
Fix django mysql e2e test
bitterpanda63 Mar 25, 2025
f41512e
Also add apispec to e2e test
bitterpanda63 Mar 25, 2025
19c7b62
Total is 3 requests for django_mysql e2e
bitterpanda63 Mar 26, 2025
5b83da4
Fix request_handler
bitterpanda63 Mar 26, 2025
0c929cc
Fix test cases of request_handler
bitterpanda63 Mar 26, 2025
05a87f5
Fix request_handler and it's test cases
bitterpanda63 Mar 26, 2025
4ddd294
Fix broken test case for vulnerability scanner
bitterpanda63 Mar 26, 2025
c43ab2f
validate the firewall lists API response correctly
bitterpanda63 Mar 26, 2025
645e303
Merge remote-tracking branch 'origin/fix/validate-firewall-lists-corr…
bitterpanda63 Mar 26, 2025
61c9ae9
Merge branch 'main' into improve-thread-caching-separated
bitterpanda63 Mar 26, 2025
3d228bb
Merge branch 'main' into improve-thread-caching-separated
bitterpanda63 Apr 1, 2025
960212f
Merge remote-tracking branch 'origin/main' into improve-thread-cachin…
bitterpanda63 Apr 1, 2025
5782587
Improves heartbeat event validation
bitterpanda63 Apr 1, 2025
d3225ba
Merge remote-tracking branch 'origin/main' into improve-thread-cachin…
bitterpanda63 Apr 10, 2025
b08726d
Merge branch 'main' into improve-thread-caching-separated
bitterpanda63 Apr 10, 2025
116d7b2
Test 3.13 as well for unit tests and e2e tests
bitterpanda63 Apr 14, 2025
9d9ad25
Benchmarks for all supported python versions
bitterpanda63 Apr 14, 2025
72cd57d
init tests as a module
bitterpanda63 Apr 14, 2025
8336b97
Refactor pymysql to use wrapt
bitterpanda63 Apr 14, 2025
dde3ba9
(poetry) install wrapt
bitterpanda63 Apr 14, 2025
47058fd
Update subprocess patching
bitterpanda63 Apr 14, 2025
32034d4
pymysql patches change order
bitterpanda63 Apr 14, 2025
7dc0a6f
move back to bottom, python order for pymysql, subprocess
bitterpanda63 Apr 14, 2025
e16b8ed
convert os_system patch
bitterpanda63 Apr 14, 2025
15a28eb
pymongo convert code to wrapt code
bitterpanda63 Apr 14, 2025
23e5fcf
convert mysqlclient
bitterpanda63 Apr 14, 2025
76894de
convert io module
bitterpanda63 Apr 14, 2025
afe9b77
Convert shutil module
bitterpanda63 Apr 14, 2025
142770d
Convert asyncpg module
bitterpanda63 Apr 14, 2025
fb309e7
Convert builtins.py
bitterpanda63 Apr 14, 2025
91ca20d
define new standards for the wrapping
bitterpanda63 Apr 14, 2025
dd9b4b0
update existing modules
bitterpanda63 Apr 14, 2025
9d37dc9
Update io module to use new system
bitterpanda63 Apr 14, 2025
1dab0f2
Update both builtins.py and asyncpg.py to use the new patching system
bitterpanda63 Apr 14, 2025
cffd729
Cleanup checks for builtins and shutil
bitterpanda63 Apr 14, 2025
0196b11
linting for asyncpg.py
bitterpanda63 Apr 14, 2025
6a9711d
Cleanup the patching module
bitterpanda63 Apr 14, 2025
57faa80
Convert os.py
bitterpanda63 Apr 14, 2025
e8c50d3
Update http_client.py sink, adding the @after
bitterpanda63 Apr 14, 2025
87892fd
Update psycopg sink
bitterpanda63 Apr 14, 2025
fa9f54f
convert socket.py to new format
bitterpanda63 Apr 14, 2025
cb37a0c
Fix shutil and shutil test cases
bitterpanda63 Apr 14, 2025
06e7abd
Convert psycopg2 module
bitterpanda63 Apr 14, 2025
6af31a4
update sample app lockfiles
bitterpanda63 Apr 14, 2025
7834861
Update actions/setup-python v2 -> v5
bitterpanda63 Apr 15, 2025
70305bf
Fix @after by removing finally: which was swallowing the error
bitterpanda63 Apr 15, 2025
35c4c82
Fix broken subprocess test cases
bitterpanda63 Apr 15, 2025
3618197
Remove ms checks for starlette benchmark in favour of percentages
bitterpanda63 Apr 15, 2025
31705c1
Exclude psycopg2 testing for python 3.13
bitterpanda63 Apr 15, 2025
3aa49cc
Make sure psycopg2.py implementation is the same as prev one
bitterpanda63 Apr 15, 2025
807b981
Add extra tests for psycopg2 and fix issue for immutables
bitterpanda63 Apr 15, 2025
b8eec8f
use assert_any_call for python 3.13
bitterpanda63 Apr 15, 2025
0aabc2c
Patch os.path.realpath for python 3.13
bitterpanda63 Apr 16, 2025
b29cc3e
Patch os.path.realpath for python 3.13
bitterpanda63 Apr 16, 2025
5a2037c
Merge remote-tracking branch 'origin/python-3.13-support' into python…
bitterpanda63 Apr 16, 2025
f77ec4a
Merge branch 'main' into python-3.13-support
bitterpanda63 May 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
name: Benchmark
name: 📊 Benchmarks
on:
push: {}
workflow_call: {}

jobs:
benchmark_sql_algorithm:
runs-on: ubuntu-latest
Expand All @@ -13,7 +14,7 @@ jobs:
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python 3.9
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.9

Expand All @@ -30,15 +31,24 @@ jobs:
run: |
poetry run python ./benchmarks/sql_benchmark/sql_benchmark_fw.py
poetry run python ./benchmarks/sql_benchmark/sql_benchmark_no_fw.py

benchmark_with_flask_mysql:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start databases
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies & build
run: |
python -m pip install --upgrade pip
Expand All @@ -51,15 +61,24 @@ jobs:
- name: Run flask-mysql k6 Benchmark
run: |
k6 run -q ./benchmarks/flask-mysql-benchmarks.js

benchmark_with_starlette_app:
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Start databases
working-directory: ./sample-apps/databases
run: docker compose up --build -d
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies & build
run: |
python -m pip install --upgrade pip
Expand All @@ -71,9 +90,5 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install -y wrk
- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Run benchmark
run: python ./benchmarks/starlette_benchmark.py
4 changes: 2 additions & 2 deletions .github/workflows/end2end.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- { name: flask-postgres-xml, testfile: end2end/flask_postgres_xml_lxml_test.py }
- { name: quart-postgres-uvicorn, testfile: end2end/quart_postgres_uvicorn_test.py }
- { name: starlette-postgres-uvicorn, testfile: end2end/starlette_postgres_uvicorn_test.py }
python-version: ["3.10", "3.11", "3.12"]
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- name: Install packages
run: sudo apt update && sudo apt install python3-dev libmysqlclient-dev
Expand All @@ -53,7 +53,7 @@ jobs:
docker run --name mock_core -d -p 5000:5000 mock_core

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: '3.x'

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/smoke-test-ffi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: 3.12

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/unit-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
# Don't cancel jobs if one fails
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
steps:
- name: Checkout code
uses: actions/checkout@v4
Expand All @@ -22,7 +22,7 @@ jobs:
run: |
sudo echo "127.0.0.1 local.aikido.io" | sudo tee -a /etc/hosts
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

Expand Down
70 changes: 70 additions & 0 deletions aikido_zen/sinks/__init__.py
Original file line number Diff line number Diff line change
@@ -1,0 +1,70 @@
from wrapt import wrap_object, FunctionWrapper, when_imported
from aikido_zen.background_process.packages import ANY_VERSION, is_package_compatible
from aikido_zen.errors import AikidoException
from aikido_zen.helpers.logging import logger


def on_import(name, package="", version_requirement=ANY_VERSION):
"""
Decorator to register a function to be called when a package is imported.
It checks if the package is compatible with the specified version requirement.
"""

def decorator(func):
if package and not is_package_compatible(package, version_requirement):
return

Check warning on line 15 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L15

Added line #L15 was not covered by tests

when_imported(name)(func) # Register the function to be called on import

return decorator


def patch_function(module, name, wrapper):
"""
Patches a function in the specified module with a wrapper function.
"""
try:
wrap_object(module, name, FunctionWrapper, (wrapper,))
except Exception as e:
logger.info("Failed to wrap %s:%s, due to: %s", module, name, e)

Check warning on line 29 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L28-L29

Added lines #L28 - L29 were not covered by tests


def before(wrapper):
"""
Surrounds a patch with try-except and calls the original function at the end
"""

def decorator(func, instance, args, kwargs):
try:
wrapper(func, instance, args, kwargs) # Call the patch
except AikidoException as e:
raise e # Re-raise AikidoException
except Exception as e:
logger.debug(

Check warning on line 43 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L40-L43

Added lines #L40 - L43 were not covered by tests
"%s:%s wrapping-before error: %s", func.__module__, func.__name__, e
)

return func(*args, **kwargs) # Call the original function

return decorator


def after(wrapper):
"""
Surrounds a patch with try-except, calls the original function and gives the return value to the patch
"""

def decorator(func, instance, args, kwargs):
return_value = func(*args, **kwargs) # Call the original function
try:
wrapper(func, instance, args, kwargs, return_value) # Call the patch
except AikidoException as e:
raise e # Re-raise AikidoException
except Exception as e:
logger.debug(

Check warning on line 64 in aikido_zen/sinks/__init__.py

View check run for this annotation

Codecov / codecov/patch

aikido_zen/sinks/__init__.py#L63-L64

Added lines #L63 - L64 were not covered by tests
"%s:%s wrapping-after error: %s", func.__module__, func.__name__, e
)

return return_value

return decorator
70 changes: 17 additions & 53 deletions aikido_zen/sinks/asyncpg.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,64 +2,28 @@
Sink module for `asyncpg`
"""

import copy
import aikido_zen.importhook as importhook
from aikido_zen.background_process.packages import is_package_compatible
import aikido_zen.vulnerabilities as vulns
from aikido_zen.helpers.logging import logger
from aikido_zen.helpers.get_argument import get_argument
from aikido_zen.sinks import patch_function, before, on_import

REQUIRED_ASYNCPG_VERSION = "0.27.0"


@importhook.on_import("asyncpg.connection")
def on_asyncpg_import(asyncpg):
@on_import("asyncpg.connection", "asyncpg", version_requirement="0.27.0")
def patch(m):
"""
Hook 'n wrap on `asyncpg.connection`
* the Cursor classes in asyncpg.cursor are only used to fetch data. (Currently not supported)
* Pool class uses Connection class (Wrapping supported for Connection class)
* _execute(...) get's called by all except execute and executemany
Our goal is to wrap the _execute(), execute(), executemany() functions in Connection class :
https://github.com/MagicStack/asyncpg/blob/85d7eed40637e7cad73a44ed2439ffeb2a8dc1c2/asyncpg/connection.py#L43
Returns : Modified asyncpg.connection object
patching module asyncpg.connection
- patches Connection.execute, Connection.executemany, Connection._execute
- doesn't patch Cursor class -> are only used to fetch data.
- doesn't patch Pool class -> uses Connection class
src: https://github.com/MagicStack/asyncpg/blob/85d7eed40637e7cad73a44ed2439ffeb2a8dc1c2/asyncpg/connection.py#L43
"""
if not is_package_compatible("asyncpg", REQUIRED_ASYNCPG_VERSION):
return asyncpg
modified_asyncpg = importhook.copy_module(asyncpg)

# pylint: disable=protected-access # We need to wrap this function
former__execute = copy.deepcopy(asyncpg.Connection._execute)
former_executemany = copy.deepcopy(asyncpg.Connection.executemany)
former_execute = copy.deepcopy(asyncpg.Connection.execute)

def aikido_new__execute(_self, query, *args, **kwargs):
vulns.run_vulnerability_scan(
kind="sql_injection",
op="asyncpg.connection.Connection._execute",
args=(query, "postgres"),
)

return former__execute(_self, query, *args, **kwargs)

def aikido_new_executemany(_self, query, *args, **kwargs):
# This query is just a string, not a list, see docs.
vulns.run_vulnerability_scan(
kind="sql_injection",
op="asyncpg.connection.Connection.executemany",
args=(query, "postgres"),
)
return former_executemany(_self, query, *args, **kwargs)
patch_function(m, "Connection.execute", _execute)
patch_function(m, "Connection.executemany", _execute)
patch_function(m, "Connection._execute", _execute)

def aikido_new_execute(_self, query, *args, **kwargs):
vulns.run_vulnerability_scan(
kind="sql_injection",
op="asyncpg.connection.Connection.execute",
args=(query, "postgres"),
)
return former_execute(_self, query, *args, **kwargs)

# pylint: disable=no-member
setattr(asyncpg.Connection, "_execute", aikido_new__execute)
setattr(asyncpg.Connection, "executemany", aikido_new_executemany)
setattr(asyncpg.Connection, "execute", aikido_new_execute)
@before
def _execute(func, instance, args, kwargs):
query = get_argument(args, kwargs, 0, "query")

return modified_asyncpg
op = f"asyncpg.connection.Connection.{func.__name__}"
vulns.run_vulnerability_scan(kind="sql_injection", op=op, args=(query, "postgres"))
38 changes: 15 additions & 23 deletions aikido_zen/sinks/builtins.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,34 +3,26 @@
"""

from pathlib import PurePath
import aikido_zen.importhook as importhook
import aikido_zen.vulnerabilities as vulns
from aikido_zen.helpers.get_argument import get_argument
from aikido_zen.sinks import patch_function, on_import, before


def aikido_open_decorator(func):
"""Decorator for open(...)"""
@before
def _open(func, instance, args, kwargs):
filename = get_argument(args, kwargs, 0, "filename")
if not isinstance(filename, (str, bytes, PurePath)):
return

def wrapper(*args, **kwargs):
# args[0] is thefunc_name filename
if len(args) > 0 and isinstance(args[0], (str, bytes, PurePath)):
vulns.run_vulnerability_scan(
kind="path_traversal", op="builtins.open", args=(args[0],)
)
return func(*args, **kwargs)
vulns.run_vulnerability_scan(
kind="path_traversal", op="builtins.open", args=(filename,)
)

return wrapper


@importhook.on_import("builtins")
def on_builtins_import(builtins):
@on_import("builtins")
def patch(m):
"""
Hook 'n wrap on `builtins`, python's built-in functions
Our goal is to wrap the open() function, which you use when opening files
Returns : Modified builtins object
patching module builtins
- patches builtins.open
"""
modified_builtins = importhook.copy_module(builtins)

# pylint: disable=no-member
setattr(builtins, "open", aikido_open_decorator(builtins.open))
setattr(modified_builtins, "open", aikido_open_decorator(builtins.open))
return modified_builtins
patch_function(m, "open", _open)
Loading
Loading