Support for zig-0.13 and forthcoming zig-0.14

[jburgy]

(continued from #420)

@gatesn how do you propose we "engage on some proposals for how to bring the project up to date with latest Zig"? I have some spare cycles and am happy to get my hands dirty. Would love pointers on "features of comptime that are no longer supported >0.12.0"

[gatesn]

Speficially, we have mutable comptime state in

ziggy-pydust/pydust/src/discovery.zig

Lines 36 to 43 in 5407e11

	comptime var privateMethods: [1000]*anyopaque = undefined;
	comptime var privateMethodsSize: usize = 0;
	comptime var definitions: [1000]Definition = undefined;
	comptime var definitionsSize: usize = 0;
	comptime var identifiers: [1000]Identifier = undefined;
	comptime var identifiersSize: usize = 0;

which is no longer supported (rightfully!).

So we're really looking for folks to experiment with different ways of designing a new API that doesn't use discovery in this way. We can then try to reach consensus on one that works well.

This isn't an easy thing to do. I don't think it makes sense to hand off to a maintainer that doesn't have a good grasp or good ideas on how to redevelop this part of the project.

[jburgy]

Helpful, thank you @gatesn. The first version of zig I used is 0.13.x so I never knew comptime vars were once mutable. Here's how I handled something slightly similar (see this blog post for context):

fn defword(
    comptime last: ?[]const Instr,
    comptime flag: Flag,
    comptime name: []const u8,
    comptime data: []const []const Instr,
) [offset + data.len + 1]Instr {
    const code = [_]Instr{.{ .code = docol_ }} ++ comptime init: {
        var code: [data.len]Instr = undefined;
        for (&code, data) |*d, s| {
            d.* = .{ .word = codeFieldAddress(s.ptr) };
        }
        break :init &code;
    };
    return defword_(last, flag, name, code);
}

namely mutating a "regular" var inside a comptime initializer block. Do you think this approach might work, at least in theory?

[jburgy]

I took the first step to restore zig build for 0.13. Curiously enough, the command succeeds without warning. How do I run tests?

[jburgy]

So I'm pawing in the dark but zig build test doesn't produce a compile error on fe96a2f! It does, however, fail at runtime with

test
└─ run test stderr
Exception ignored on threading shutdown:
OverflowError: can't convert negative int to unsigned

Investigating further

[jburgy]

Uh, not sure what happened but

 zig build test; echo $?
0

What else should I be testing?

[robert3005]

There's a weird behaviour that we sometimes observed where you will build a new python library but because that library existed when the program started you might have already loaded the old one thus you're effectively testing the old binary and running the second time makes things succeed. I never got to the bottom of why things we actually being loaded eagerly

[slonik-az]

comptime var are certainly mutable at comptime at least in zig-0.13. I use this feature daily. What i am missing here?

UPDATE: I just realized that I have been succefully using comptime var on local scope. On global scope it does NOT work.

[jburgy]

Over a month and many false starts later, I finally got something that works with zig 0.13 and python 3.12. I did have to tweak the API:

const py = @import("pydust");

const root = @This();

pub fn hello() !py.PyString(root) {
    return try py.PyString(root).create("Hello!");
}

comptime {
    py.rootmodule(root);
}

which gives me

~/ziggy-pydust$ PYTHONPATH=example python -c "from hello import hello; print(hello())"
Hello!

So instead of py.class and friends operating via side effect on mutable global comptime state (which is no longer supported), I leverage comptime memoization. State members like definitions and identifiers become comptime functions.

@gatesn, @robert3005, @slonik-az, please review #429

[slonik-az]

Thanks @jburgy!!! I like your tricks in src/discovery.zig. Very clever!
I am curious, would it be possible to get rid of poetry and rely solely on zig's own build system to build python module. Poetry dependency is a hard no-go for me at the moment.

[jburgy]

My pleasure (trust me)!

While I agree with the ambition of getting rid of poetry, I have the following reservations:

1. Plenty of scope for Restore zig build for 0.13 #429 as it is, let's limit creep
2. I have barely scratched the surface of zig's build system so not in best position to leverage it
3. From what I see, ziggy-pydust doesn't really use poetry to build but rather custom python scripts like buildzig.py

[jburgy]

@gatesn good news: #429 is in reasonable shape and all python tests pass! example/pytest.zig::pytest.test.pydust-expected-failure does fail but that failure appears expected:

test "pydust-expected-failure" {
    py.initialize();
    defer py.finalize();

    const str = try py.PyString(root).create("hello");
    defer str.decref();

    try std.testing.expectEqualStrings("world", try str.asSlice());
}

[jburgy]

#429 was merged which renders this discussion moot