|
| 1 | ++++ |
| 2 | +# Copyright (c) godot-rust; Bromeon and contributors. |
| 3 | +# This Source Code Form is subject to the terms of the Mozilla Public |
| 4 | +# License, v. 2.0. If a copy of the MPL was not distributed with this |
| 5 | +# file, You can obtain one at https://mozilla.org/MPL/2.0/. |
| 6 | + |
| 7 | +title = "2023 in Review:~Establishing Rust as a Godot 4 language" |
| 8 | +authors = ["Bromeon"] |
| 9 | + |
| 10 | +[extra] |
| 11 | +summary = "A retrospective on the achievements of the godot-rust community in 2023." |
| 12 | +tags = ["retrospective", "dev-update", "2023"] |
| 13 | ++++ |
| 14 | + |
| 15 | +2023 was a crucial year for the **godot-rust** project. Not only did Rust gain traction as a language in the Godot ecosystem, but we also saw the release of Godot 4.0 and with it an important milestone towards production-readiness. |
| 16 | + |
| 17 | +This article serves as a retrospective on the past year and highlights the achievements of the godot-rust community. |
| 18 | + |
| 19 | + |
| 20 | +## Where it started |
| 21 | + |
| 22 | +2022 was a year of extensive tinkering in a small setting, accompanied by highly appreciated input from [cuddlefishie][user-cuddlefishie], [chitoyuu][user-chitoyuu] and [Waridley][user-waridley], who were already involved with the **gdnative** library (the Rust bindings for Godot 3). |
| 23 | +In November 2022, I open-sourced **gdext** as the experimental Godot 4 bindings. |
| 24 | + |
| 25 | +While the library was able to run the dodge-the-creeps example, a lot of central features were missing at the time. Arrays and dictionaries were not implemented, and neither were property exports. The project featured the absolute minimum to run the example. There were loads of bugs and safety issues. Nevertheless, 2022 resulted in a working prototype and an open-source foundation for the community to build upon. |
| 26 | + |
| 27 | +January and February 2023 marked the point where several early adopters joined and contributed missing builtins: [arrays][#85], [packed arrays][#91], [dictionaries][#92], [vectors][#71], [quaternions][#68], [matrices][#124] and [colors][#123]. We also cultivated [issue #24][#24] as a central point for an up-to-date overview of the implemented features. Furthermore, [utility functions][#61], property exports ([#45], [#177]) and a [GDScript test runner][#103] were implemented. Most of the contributions at the time came from [ttencate][user-ttencate], [mhoff12358][user-mhoff12358] and [lilizoey][user-lilizoey] (who has also helped maintain the library since). Thanks a lot for jumping into the cold water! |
| 28 | + |
| 29 | + |
| 30 | +## Godot 4.0 and the road to stability |
| 31 | + |
| 32 | +After years of ambitious work, Godot developers [released major version 4.0][godot-4.0] on March 1st, effectively turning Godot 4 from an in-dev prototype into a tangible release. According to official plans, it would take 1-2 minor versions to get a lot of the initial work ironed out (due to new user feedback), but the foundation was definitely there. |
| 33 | + |
| 34 | +For binding maintainers, 4.0 was a great relief as well, since it meant that we'd now have official versions to target and that breaking changes would not occur every other week anymore -- as was the case during Godot 4 alpha and beta stages. |
| 35 | + |
| 36 | +However, the GDExtension API was not fully ready at the time; its design didn't allow for long-term binary backward compatibility. This was fixed for Godot 4.1, in large parts due to the great efforts of [dsnopek][user-dsnopek] and the [GDExtension team][godot-gdextension-team], who came up with thorough proposals toward stability and extensibility. |
| 37 | + |
| 38 | +The significance of backward compatibility from 4.1 onwards cannot be overstated: it meant that users developing an extension in 4.1 can still run it with Godot 4.9. What sounds like a "nice-to-have" is absolutely essential for building an ecosystem: if you want to combine a handful GDExtension plugins (e.g. some written in C++, some in Swift, some in Rust), it's very likely that they target different Godot versions and are maintained with varying activity. Not having compatibility would force you to recompile every single plugin for the Godot version you target. |
| 39 | + |
| 40 | + |
| 41 | +## Features upon features |
| 42 | + |
| 43 | +Spring and summer were defined by a continued mapping of GDExtension APIs to Rust. |
| 44 | +Some noteworthy highlights: |
| 45 | + |
| 46 | +1. [Virtual function traits][#136] extended virtual methods far beyond the hardcoded `ready()`, `process()` and `physics_process()`. A dedicated trait for each class allowed to very specifically select which functionality to override. |
| 47 | +1. [Default parameters][#322] allowed to optionally provide arguments to engine APIs. Since Rust doesn't natively offer default parameters, we use the builder pattern. |
| 48 | + * Example: `object.connect_ex(signal, callable).flags(flags).done()` |
| 49 | +1. Improved code generation for engine API. |
| 50 | + * [Native structures][#272] |
| 51 | + * [Constants][#219] |
| 52 | + * [Bitfields][#524] |
| 53 | +1. Extended builtin support and integration of Godot data structures with Rust idioms. |
| 54 | + * [`Callable`][#231] |
| 55 | + * [`GodotString::chars_checked()`][#161] to access UTF-32 characters. |
| 56 | + * [`PackedArray::as_slice()`][#307] to access elements via slice. |
| 57 | +1. Registration of Rust symbols. |
| 58 | + * [`#[derive(FromVariant, ToVariant)]`][#246] swiftly maps user types to and from `Variant`. |
| 59 | + * [User-defined constants][#321] |
| 60 | + * [User-defined enums][#371] |
| 61 | + * [Notification API][#223]: type-safe rather than integer constants. |
| 62 | + * [Export parameters][#309]: GDScript's `@export_range` etc. |
| 63 | + * [`OnReady<T>`][#534] to enable late initialization of properties. |
| 64 | +1. [Hot reloading][#437] was one of the most anticipated Godot 4.2 features. Its integration into gdext marked a tremendous improvement for dev cycles, no longer needing to restart the editor. |
| 65 | +1. Higher-level abstractions: in cases where a "raw" Godot API is prone to memory/type unsafety or other errors, higher-level interfaces can provide a nicer user experience. Examples include: |
| 66 | + * [Script extensions][#492], allowing the user to implement `GDExtensionScriptInstance`. |
| 67 | + * [The `GFile` struct][#473], which wraps `FileAccess` and integrates it with Rust's `std::io` traits. |
| 68 | + * `load::<T>(file_path)` to load Godot resources from the `res://` filesystem. |
| 69 | + * `get_node_as::<T>(node_path)` to get a node, given a path and type. |
| 70 | + * `InstanceId` as a non-null object ID. |
| 71 | + * `godot_print!` that prints to the Godot console and works like `println!`. |
| 72 | +1. Support for more build configurations. |
| 73 | + * [Double-precision builds][#149] enabled using Godot's `precision=double` build flag. |
| 74 | + * [An early approach to threading][#212] opened the door to further discussion on `Sync` types. |
| 75 | + * [The `serde` feature][#237] enabled serialization with the `serde` crate (later extended in [#508]). |
| 76 | + |
| 77 | + |
| 78 | + |
| 79 | +## WebAssembly support |
| 80 | + |
| 81 | +Initial support for Web exports has been added in pull request [#493]. While still experimental, this addition is a proof-of-concept that Rust can be used for Godot web games and marks a huge step forward in our platform support. There is also a [tutorial in the book][book-wasm] explaining the setup. |
| 82 | + |
| 83 | +Thanks a lot to [zecozephyr][user-zecozephyr], [Esption][user-esption] and [PgBiel][user-pgbiel] for their great collaboration towards this achievement. |
| 84 | + |
| 85 | + |
| 86 | +## Large-scale refactors and FFI hell |
| 87 | + |
| 88 | +Now that the shiny parts are out of the way, let's descend into the mines. |
| 89 | + |
| 90 | +The library underwent _a lot_ of bugfixes and refactorings during 2023, many of which are related to the interaction with the GDExtension C API via FFI. These are typically not witnessed immediately, but contribute to an overall more robust and streamlined experience. FFI with Godot is a topic of significant complexity, especially because there are so many different integration points, all of which work in subtly different ways.[^gdextension-dimensions] |
| 91 | + |
| 92 | +It would be too much to list all the changes, but here is a sample of larger refactorings: |
| 93 | + |
| 94 | +1. [`ToGodot`/`FromGodot` traits][#421] unified two previously competing marshalling mechanisms into a single API. This reduces the burden for users to make their types interoperate with Godot, while at the same time benefiting from optimizations (FFI ptrcalls) where possible. |
| 95 | +1. [Lazy function pointers][#387] significantly reduce both runtime overhead to perform FFI calls and the amount of code that needs to be compiled in `godot-core`. An extended discussion of the improvements is available in the blog post _[FFI Optimizations and Benchmarking][devlog-ffi]_. |
| 96 | +1. [Pointer call refactoring][#204] changed the way how values are passed to and from Godot, fixing several use-after-free bugs. |
| 97 | + |
| 98 | +A more detailed insight can be obtained by looking at [merged pull requests with the `ub` label][prs-ub] (15 at the time of writing). These include fixes related to undefined behavior, and thus improved robustness of the library. Almost all of them are covered by regression tests, ensuring that we won't encounter them again in the future. |
| 99 | + |
| 100 | + |
| 101 | +## Tooling and infrastructure |
| 102 | + |
| 103 | +The godot-rust project involves much more than just code. We have strived to improve the contribution experience for newcomers, providing short and helpful feedback loops. Efforts that have gone into infrastructure span different areas: |
| 104 | + |
| 105 | +1. [Ever-increasing CI coverage][#38] boosts confidence that new contributions are held to a high standard and that existing functionality keeps working. Cargo's toolchain lets us run unit tests, clippy and rustfmt. Additionally, we built a custom integration test suite that runs gdext code against multiple versions of Godot in CI. A minimalist local version of these checks additionally exists as the `check.sh` script. |
| 106 | +1. [Address and leak sanitizers][#325] are tools from the C++ ecosystem, but equally useful in Rust. We cannot generally use `miri` due to FFI; however, address sanitizers can detect undefined behavior such as use-after-free or out-of-bound access. Leak sanitizers can detect memory leaks. |
| 107 | +1. [Prebuilt artifacts][#211] shrink the dependency tree to around 30% of its previous size, avoiding heavy crates like `bindgen` and `syn`. By pre-compiling `bindgen` artifacts for different platforms, initial builds are sped up considerably and an external LLVM installation is no longer needed. This in turn benefits CI times, further improving integration cycles. |
| 108 | +1. [The book][github-book] provides tutorials to get started with gdext and is also maintained on GitHub. |
| 109 | + It was split from the [gdnative book][github-gdnative-book] for clarity. |
| 110 | +1. [The website][github-website] was fundamentally rewritten with a fresh look. It links to all the relevant places, hosts API docs and this devlog. |
| 111 | +1. [Automated doc generation in PRs][#259] runs `cargo doc` for every single pull request opened, publishes the results to a temporary page on our website, and lets a bot comment on the PR with a link to the docs. This allows both contributors and reviewers to see how the resulting API looks. |
| 112 | +1. [Custom formatting][#303] replaces the annoyingly slow `rustfmt` for generated code, while still preserving human readability. |
| 113 | +1. [GitHub merge queues][#337] are the successor to the `bors` bot. They enforce that checks are always run against latest `master`, independent of any other changes that have landed since a branch was opened. |
| 114 | + |
| 115 | +See also [list of merged tooling-related PRs][prs-tooling] (54 at the time of writing). While these only include pull requests on the main repo, they give a good insight into all the smaller tweaks behind the scenes. |
| 116 | + |
| 117 | +The [gdnative][github-gdnative] library continues to be maintained -- it is mostly feature-complete and lays a focus on stability and minimizing breaking changes. There is likely going to be a Godot 3.6 LTS release in the coming year. Thanks to [chitoyuu][user-chitoyuu] for ongoing work on the gdnative front! |
| 118 | + |
| 119 | + |
| 120 | +## Outlook |
| 121 | + |
| 122 | +In numbers, the [GitHub project for gdext][github-gdext] today has: |
| 123 | +* 1239 commits, by 55 contributors |
| 124 | +* 2050 stars and 131 forks |
| 125 | +* 139 issues closed as completed, 91 open |
| 126 | +* 262 pull requests merged, 2 open |
| 127 | + |
| 128 | +2023 was a year of great progress for godot-rust. What started as a small proof-of-concept, has matured a lot in terms of features and robustness. This would not have been remotely possible without the help of the generous open-source community. Thanks a lot to everyone for the countless hours spent -- whether it's [as a contributor][github-contributors] navigating through FFI mineshafts in search of light, helping others on Discord, or simply trying out the library and providing feedback -- every effort is highly appreciated! |
| 129 | + |
| 130 | +For the coming year, the focus lies on making the core systems even more stable, as well as promoting an ergonomic, pragmatic gamedev workflow. There will likely still be some breaking changes to unify the overall experience, but we try to work with deprecations to ease migrations. |
| 131 | + |
| 132 | +Some of the larger things planned for 2024 are: |
| 133 | +* [Re-entrant calls for object references][#501] (in progress). |
| 134 | +* [Publish to crates.io.][#2] |
| 135 | +* [Better threading support.][#18] |
| 136 | +* [Builder API for registration.][#4] |
| 137 | +* Initial support for [Android][#470] and [iOS][#498] -- would however need some volunteers (also due to separate devices/toolchains). |
| 138 | + |
| 139 | +The majority of the scaffolding and infrastructure is now there, so future work can focus more directly on innovation. With that and a growing user base, 2024 looks very exciting! |
| 140 | + |
| 141 | +A Happy New Year to everyone! |
| 142 | + |
| 143 | +<br> |
| 144 | + |
| 145 | +--- |
| 146 | + |
| 147 | +### Footnotes |
| 148 | + |
| 149 | +[^gdextension-dimensions]: |
| 150 | +GDExtension offers different dimensions to interop with Godot, all of which need to be honored for safe FFI: |
| 151 | +* Calls to engine: builtin methods, class methods, utility functions. |
| 152 | +* Callbacks from engine: virtual functions, separate callbacks for `to_string`, `notification`, etc. |
| 153 | +* Calling convention: _varcall_ vs. _ptrcall_. |
| 154 | +* Memory management: value semantics, manually managed, ref-counted, copy-on-write. |
| 155 | + |
| 156 | +[#508]: https://github.com/godot-rust/gdext/pull/508 |
| 157 | +[#437]: https://github.com/godot-rust/gdext/pull/437 |
| 158 | +[#237]: https://github.com/godot-rust/gdext/pull/237 |
| 159 | +[#103]: https://github.com/godot-rust/gdext/pull/103 |
| 160 | +[#123]: https://github.com/godot-rust/gdext/pull/123 |
| 161 | +[#124]: https://github.com/godot-rust/gdext/pull/124 |
| 162 | +[#136]: https://github.com/godot-rust/gdext/pull/136 |
| 163 | +[#149]: https://github.com/godot-rust/gdext/pull/149 |
| 164 | +[#161]: https://github.com/godot-rust/gdext/pull/161 |
| 165 | +[#177]: https://github.com/godot-rust/gdext/pull/177 |
| 166 | +[#18]: https://github.com/godot-rust/gdext/issues/18 |
| 167 | +[#204]: https://github.com/godot-rust/gdext/pull/204 |
| 168 | +[#211]: https://github.com/godot-rust/gdext/pull/211 |
| 169 | +[#212]: https://github.com/godot-rust/gdext/pull/212 |
| 170 | +[#219]: https://github.com/godot-rust/gdext/pull/219 |
| 171 | +[#223]: https://github.com/godot-rust/gdext/pull/223 |
| 172 | +[#231]: https://github.com/godot-rust/gdext/pull/231 |
| 173 | +[#246]: https://github.com/godot-rust/gdext/pull/246 |
| 174 | +[#24]: https://github.com/godot-rust/gdext/issues/24 |
| 175 | +[#259]: https://github.com/godot-rust/gdext/pull/259 |
| 176 | +[#272]: https://github.com/godot-rust/gdext/pull/272 |
| 177 | +[#2]: https://github.com/godot-rust/gdext/issues/2 |
| 178 | +[#303]: https://github.com/godot-rust/gdext/pull/303 |
| 179 | +[#307]: https://github.com/godot-rust/gdext/pull/307 |
| 180 | +[#309]: https://github.com/godot-rust/gdext/pull/309 |
| 181 | +[#321]: https://github.com/godot-rust/gdext/pull/321 |
| 182 | +[#322]: https://github.com/godot-rust/gdext/pull/322 |
| 183 | +[#325]: https://github.com/godot-rust/gdext/pull/325 |
| 184 | +[#337]: https://github.com/godot-rust/gdext/pull/337 |
| 185 | +[#371]: https://github.com/godot-rust/gdext/pull/371 |
| 186 | +[#387]: https://github.com/godot-rust/gdext/pull/387 |
| 187 | +[#38]: https://github.com/godot-rust/gdext/issues/38 |
| 188 | +[#421]: https://github.com/godot-rust/gdext/pull/421 |
| 189 | +[#45]: https://github.com/godot-rust/gdext/pull/45 |
| 190 | +[#470]: https://github.com/godot-rust/gdext/issues/470 |
| 191 | +[#473]: https://github.com/godot-rust/gdext/pull/473 |
| 192 | +[#492]: https://github.com/godot-rust/gdext/pull/492 |
| 193 | +[#493]: https://github.com/godot-rust/gdext/pull/493 |
| 194 | +[#498]: https://github.com/godot-rust/gdext/issues/498 |
| 195 | +[#4]: https://github.com/godot-rust/gdext/issues/4 |
| 196 | +[#501]: https://github.com/godot-rust/gdext/pull/501 |
| 197 | +[#524]: https://github.com/godot-rust/gdext/pull/524 |
| 198 | +[#534]: https://github.com/godot-rust/gdext/pull/534 |
| 199 | +[#61]: https://github.com/godot-rust/gdext/pull/61 |
| 200 | +[#68]: https://github.com/godot-rust/gdext/pull/68 |
| 201 | +[#71]: https://github.com/godot-rust/gdext/pull/71 |
| 202 | +[#85]: https://github.com/godot-rust/gdext/pull/85 |
| 203 | +[#91]: https://github.com/godot-rust/gdext/pull/91 |
| 204 | +[#92]: https://github.com/godot-rust/gdext/pull/92 |
| 205 | +[book-wasm]: https://godot-rust.github.io/book/toolchain/export-web.html |
| 206 | +[devlog-ffi]: https://godot-rust.github.io/dev/ffi-optimizations-benchmarking |
| 207 | +[github-book]: https://github.com/godot-rust/book |
| 208 | +[github-contributors]: https://github.com/godot-rust/gdext/graphs/contributors |
| 209 | +[github-gdext]: https://github.com/godot-rust/gdext |
| 210 | +[github-gdnative]: https://github.com/godot-rust/gdnative |
| 211 | +[github-gdnative-book]: https://github.com/godot-rust/gdnative-book |
| 212 | +[github-website]: https://github.com/godot-rust/godot-rust.github.io |
| 213 | +[godot-4.0]: https://godotengine.org/article/godot-4-0-sets-sail |
| 214 | +[godot-gdextension-team]: https://godotengine.org/teams |
| 215 | +[prs-tooling]: https://github.com/godot-rust/gdext/pulls?q=is%3Apr+is%3Amerged+label%3A%22c%3A+tooling%22+sort%3Acreated-asc+ |
| 216 | +[prs-ub]: https://github.com/godot-rust/gdext/pulls?q=label%3Aub+is%3Amerged+sort%3Acreated-asc |
| 217 | +[user-chitoyuu]: https://github.com/chitoyuu |
| 218 | +[user-cuddlefishie]: https://github.com/cuddlefishie |
| 219 | +[user-dsnopek]: https://github.com/dsnopek |
| 220 | +[user-esption]: https://github.com/Esption |
| 221 | +[user-lilizoey]: https://github.com/lilizoey |
| 222 | +[user-mhoff12358]: https://github.com/mhoff12358 |
| 223 | +[user-pgbiel]: https://github.com/PgBiel |
| 224 | +[user-ttencate]: https://github.com/ttencate |
| 225 | +[user-waridley]: https://github.com/Waridley |
| 226 | +[user-zecozephyr]: https://github.com/zecozephyr |
0 commit comments