Skip to content

Fix stack overflow when decoding deeply nested structures like arrays and maps #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Feb 17, 2025

Conversation

MarshalX
Copy link
Owner

@MarshalX MarshalX commented Nov 18, 2024

fixes #9

It uses 2 approaches. First for Python 3.8 and second for Python 3.9+. The first one hard limits recursion to 1k (same as Python by default). The second one respects sys.setrecursionlimit. decode_dag_cbor function could now raise RecursionError

CI fails because of temporarily disabled endpoint by bsky team to deal with network load

use of Py_EnterRecursiveCall and Py_LeaveRecursiveCall is slow; so just count depth and check it using Py_GetRecursionLimit; raise PyRecursionError

Copy link

codspeed-hq bot commented Nov 23, 2024

CodSpeed Performance Report

Merging #51 will degrade performances by 28.99%

Comparing fix-overflow-on-deep-structures (1083c37) with main (7bf9fb9)

Summary

⚡ 4 improvements
❌ 8 regressions
✅ 180 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
test_dag_cbor_decode[roundtrip01.json] 23.2 µs 16.7 µs +39.2%
test_dag_cbor_decode[roundtrip22.json] 16.8 µs 22.7 µs -26%
test_dag_cbor_encode[roundtrip05.json] 22.9 µs 16.9 µs +35.38%
test_dag_cbor_encode[roundtrip06.json] 15.1 µs 21.3 µs -28.99%
test_dag_cbor_encode[roundtrip11.json] 22.9 µs 17.6 µs +29.73%
test_dag_cbor_encode[roundtrip12.json] 16.3 µs 18.2 µs -10.83%
test_dag_cbor_encode[roundtrip16.json] 16.9 µs 22.1 µs -23.69%
test_dag_cbor_encode[roundtrip20.json] 22.6 µs 16.6 µs +35.8%
test_dag_cbor_encode[roundtrip21.json] 16.1 µs 18.2 µs -11.26%
test_dag_cbor_encode[roundtrip25.json] 16 µs 22 µs -27.44%
test_dag_cbor_encode[roundtrip26.json] 16.1 µs 18.3 µs -12.15%
test_dag_cbor_encode_real_data[canada.json] 13.6 ms 15.4 ms -11.95%

@MarshalX
Copy link
Owner Author

R.I.P

image

@MarshalX
Copy link
Owner Author

okay, now it should not affect perf

@MarshalX MarshalX merged commit 744bc2f into main Feb 17, 2025
2 of 3 checks passed
@MarshalX MarshalX deleted the fix-overflow-on-deep-structures branch February 17, 2025 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Segfault on decoding of nested lists and maps
1 participant