PyO3 Migration from v0.20.* to 0.24.* #1168

VishnuSanal · 2025-05-17T06:49:13Z

Description

This PR fixes #1167

Summary

This PR does PyO3 Migration from v0.20.* to 0.24.*

PR Checklist

Please ensure that:

The PR contains a descriptive title
The PR contains a descriptive summary of the changes
You build and test your changes before submitting a PR.
You have added relevant documentation
You have added relevant tests. We prefer integration tests wherever possible

Pre-Commit Instructions:

Ensure that you have run the pre-commit hooks on your PR.

vercel · 2025-05-17T06:49:19Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
robyn	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 5, 2025 9:44am

k4976 · 2025-05-17T18:01:15Z

Bro
This migration not gonna be an easy task
I saw many errors and warnings when i did maturin develop

i get warnings and in that i get the link to this guide https://pyo3.rs/v0.23.0/migration

this PR is to migrate from v0.20 to v0.21 but you upgrade it to v0.24, that's why so many warnings and error came, try to read all the migration guide carefully if you are trying to upgrade to latest version, i think which is 1.24

This is the reason, i got the migration guide link in warning when compiling the file

see logs also for more detailed info and the actual reason behind the errors

k4976 · 2025-05-17T18:05:35Z

⚠️  Warning: You specified maturin>=0.12,<0.13 in pyproject.toml under `build-system.requires`, but the current maturin version is 1.8.6
warning: `/workspaces/Robyn/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
📦 Including license file "/workspaces/Robyn/LICENSE"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /workspaces/Robyn/.venv/bin/python
Audited 7 packages in 1ms
warning: `/workspaces/Robyn/.cargo/config` is deprecated in favor of `config.toml`
note: if you need to support cargo 1.38 or earlier, you can symlink `config` to `config.toml`
   Compiling robyn v0.66.2 (/workspaces/Robyn)
warning: unused import: `types::PyAny`
  --> src/routers/const_router.rs:13:12
   |
13 | use pyo3::{types::PyAny, Bound};
   |            ^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: use of deprecated trait `pyo3::ToPyObject`: `ToPyObject` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23.0/migration) for more information.
  --> src/types/request.rs:33:6
   |
33 | impl ToPyObject for Request {
   |      ^^^^^^^^^^

warning: use of deprecated method `pyo3::ToPyObject::to_object`: `ToPyObject` is going to be replaced by `IntoPyObject`. See the migration guide (https://pyo3.rs/v0.23.0/migration) for more information.
  --> src/executors/mod.rs:28:39
   |
28 |     let function_args = function_args.to_object(py);
   |                                       ^^^^^^^^^

error[E0277]: `*mut pyo3::Python<'static>` cannot be sent between threads safely
   --> src/server.rs:136:17
    |
136 |                 HttpServer::new(move || {
    |                 ^^^^^^^^^^      ------- within this `{closure@src/server.rs:136:33: 136:40}`
    |                 |
    |                 `*mut pyo3::Python<'static>` cannot be sent between threads safely
    |
    = help: within `{closure@src/server.rs:136:33: 136:40}`, the trait `std::marker::Send` is not implemented for `*mut pyo3::Python<'static>`
note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>`
   --> /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/marker.rs:777:12
note: required because it appears within the type `impl_::not_send::NotSend`
   --> /home/codespace/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.24.2/src/impl_/not_send.rs:7:19
    |

error[E0277]: the trait bound `TaskLocals: Clone` is not satisfied
  --> src/websockets/mod.rs:28:5
   |
23 | #[derive(Clone)]
   |          ----- in this derive macro expansion
...
28 |     pub task_locals: TaskLocals,
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `Clone` is not implemented for `TaskLocals`

For more information about this error, try `rustc --explain E0277`.
warning: `robyn` (lib) generated 23 warnings
error: could not compile `robyn` (lib) due to 3 previous errors; 23 warnings emitted
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.9-64bit" PYO3_PYTHON="/workspaces/Robyn/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/workspaces/Robyn/.venv/bin/python" "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/workspaces/Robyn/Cargo.toml" "--lib" "--crate-type" "cdylib"`

there are many errors but i removed same types of error occuring again & again, for more info see the log file i am attaching

logs.txt

k4976 · 2025-05-17T18:37:01Z

If we are upgrading PyO3 only to provide support for python3.13 then uvloop will not install as 1.19.0, it needs 1.21.0 with python3.13, else we might need to try upgrading dependencies
I getting same above error in python3.13 as well as in python3.9, shared the logs previously in above messages

VishnuSanal · 2025-05-23T20:29:01Z

okay, so the code compiles now & can handle a request successfully. but, a lot of stuff are still broken & the tests don't pass.

one of the main issues currently is the inability to clone FunctionInfo without the GIL being held. for that I tried using Python::with_gil(|_| { function_info.clone() }); wherever needed & that works. but the current issue is the need to apply the same for all the structs that use FunctionInfo. we have 720 implementations for the same!

I am searching for a better way to do this; my intuition tells that there must be a better way to do this, rather than to introduce this extra overhead. if you are reading this & have a better solution, please LMK in a comment below! :)

cc @k4976

recurseml · 2025-05-27T19:22:28Z

⚠️ Only 5 files will be analyzed due to processing limits.

recurseml · 2025-05-27T19:22:39Z

⚠️ Only 5 files will be analyzed due to processing limits.

src/executors/mod.rs

recurseml · 2025-05-27T19:23:38Z

src/executors/mod.rs

 where
    T: ToPyObject,
 {
-    let handler = function.handler.as_ref(py);
-    let kwargs = function.kwargs.as_ref(py);
+    let handler = function.handler.bind(py).downcast()?;


Potential unhandled error case with downcast. The original code using as_ref() didn't require type checking, but the new code assumes the type can be downcast. If the type doesn't match expectations, it will fail at runtime. Need to add proper error handling or type verification.

React with 👍 to tell me that this comment was useful, or 👎 if not (and I'll stop posting more comments like this in the future)

@VishnuSanal , what do you think here?

this is still PyAny -- we can do the following if we want. but I feel like it would be a redundant & introduce unnecessary complexity. what do you think @sansyrox?

let handler = function.handler.bind(py).downcast::<pyo3::types::PyAny>() .map_err(|_| PyErr::new::<pyo3::exceptions::PyTypeError, _>( "Wrong type found for function.handler", ))?;

this is still handled gracefully anyways

Err(e) => { error!( "Error while executing route function for endpoint `{}`: {}", req.uri().path(), get_traceback(&e) ); Response::internal_server_error(None) }

src/executors/web_socket_executors.rs

src/routers/const_router.rs

recurseml · 2025-05-27T19:23:44Z

😱 Found 4 issues. Time to roll up your sleeves! 😱

codspeed-hq · 2025-05-28T07:53:40Z

CodSpeed Performance Report

Merging #1168 will not alter performance

_{Comparing VishnuSanal:pyo3-migration (56e9d34) with main (3b85e62)}

Summary

✅ 138 untouched benchmarks

sansyrox

@VishnuSanal , all looks good. Have some questions

sansyrox · 2025-06-05T10:41:45Z

Looks good to me! Waiting for recurse to update.

sansyrox · 2025-06-06T05:03:25Z

@recurseml re-review

recurseml · 2025-06-06T21:48:30Z

⚠️ Only 5 files will be analyzed due to processing limits.

recurseml · 2025-06-06T21:48:42Z

⚠️ Only 5 files will be analyzed due to processing limits.

recurseml · 2025-06-06T21:49:28Z

✨ No issues found! Your code is sparkling clean! ✨

sansyrox

LGTM! Love it

VishnuSanal added 2 commits May 17, 2025 12:14

chore: update versions

290c392

wip fix syntax from migration

db9fc48

vercel bot deployed to Preview May 17, 2025 06:49 View deployment

VishnuSanal mentioned this pull request May 17, 2025

fix: robyn builds failing for python v3.13 #1006

Merged

6 tasks

vercel bot deployed to Preview May 19, 2025 15:38 View deployment

VishnuSanal force-pushed the pyo3-migration branch from 88c9e20 to 392c2f3 Compare May 21, 2025 19:02

vercel bot deployed to Preview May 21, 2025 19:03 View deployment

fixes task_locals clone issue on server.rs ;___;

7681ae5

VishnuSanal force-pushed the pyo3-migration branch from 392c2f3 to 7681ae5 Compare May 22, 2025 07:24

vercel bot deployed to Preview May 22, 2025 07:25 View deployment

VishnuSanal added 2 commits May 23, 2025 00:06

fix function_info clone issue

2c9e9cd

?is_none to is_ok

443a2c8

VishnuSanal changed the title [WIP] PyO3 Migration from v0.20.* to 0.21.* [WIP] PyO3 Migration from v0.20.* to 0.24.* May 22, 2025

vercel bot deployed to Preview May 22, 2025 18:51 View deployment

impl Clone for FunctionInfo

6b4efac

VishnuSanal force-pushed the pyo3-migration branch from 73cb007 to 6b4efac Compare May 24, 2025 06:20

vercel bot deployed to Preview May 24, 2025 06:21 View deployment

VishnuSanal added 2 commits May 24, 2025 11:59

fix typo

4e74284

getattr => get_item

0f9816e

vercel bot deployed to Preview May 24, 2025 06:33 View deployment

chore: apply rust-fix suggestions

3e808ee

vercel bot deployed to Preview May 26, 2025 18:54 View deployment

VishnuSanal added 2 commits May 27, 2025 19:29

add default value to pydict#get

f040bf9

manual clone impl for WebSocketConnector

c758ba9

vercel bot deployed to Preview May 27, 2025 14:56 View deployment

recurseml bot reviewed May 27, 2025

View reviewed changes

src/executors/mod.rs Outdated Show resolved Hide resolved

recurseml bot reviewed May 27, 2025

View reviewed changes

src/executors/web_socket_executors.rs Outdated Show resolved Hide resolved

recurseml bot reviewed May 27, 2025

View reviewed changes

src/routers/const_router.rs Outdated Show resolved Hide resolved

is_ok to is_ok_and to handle None

0d3bc4a

vercel bot deployed to Preview May 28, 2025 07:46 View deployment

chore: cargo fmt

772d014

vercel bot deployed to Preview May 28, 2025 07:47 View deployment

Merge branch 'refs/heads/main' into pyo3-migration

cdaf93f

vercel bot deployed to Preview May 28, 2025 13:40 View deployment

VishnuSanal marked this pull request as ready for review May 28, 2025 13:41

VishnuSanal changed the title [WIP] PyO3 Migration from v0.20.* to 0.24.* PyO3 Migration from v0.20.* to 0.24.* May 28, 2025

sansyrox reviewed Jun 4, 2025

View reviewed changes

add explicit GIL lifetime management for add_route function

56e9d34

vercel bot deployed to Preview June 5, 2025 09:44 View deployment

sansyrox approved these changes Jun 7, 2025

View reviewed changes

sansyrox merged commit 608fefd into sparckles:main Jun 7, 2025
51 checks passed

VishnuSanal deleted the pyo3-migration branch June 7, 2025 07:08

Uh oh!

PyO3 Migration from v0.20.* to 0.24.* #1168

PyO3 Migration from v0.20.* to 0.24.* #1168

Uh oh!

Conversation

VishnuSanal commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Summary

PR Checklist

Pre-Commit Instructions:

Uh oh!

vercel bot commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k4976 commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k4976 commented May 17, 2025

Uh oh!

k4976 commented May 17, 2025

Uh oh!

VishnuSanal commented May 23, 2025

Uh oh!

recurseml bot commented May 27, 2025

Uh oh!

recurseml bot commented May 27, 2025

Uh oh!

Uh oh!

recurseml bot May 27, 2025

Choose a reason for hiding this comment

Uh oh!

sansyrox Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

VishnuSanal Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

recurseml bot commented May 27, 2025

Uh oh!

codspeed-hq bot commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #1168 will not alter performance

Summary

Uh oh!

sansyrox left a comment

Choose a reason for hiding this comment

Uh oh!

sansyrox commented Jun 5, 2025

Uh oh!

sansyrox commented Jun 6, 2025

Uh oh!

recurseml bot commented Jun 6, 2025

Uh oh!

recurseml bot commented Jun 6, 2025

Uh oh!

recurseml bot commented Jun 6, 2025

Uh oh!

sansyrox left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

VishnuSanal commented May 17, 2025 •

edited

Loading

vercel bot commented May 17, 2025 •

edited

Loading

k4976 commented May 17, 2025 •

edited

Loading

codspeed-hq bot commented May 28, 2025 •

edited

Loading