libc++abi: terminating due to uncaught exception of type duckdb::InternalException: Attempted to dereference shared_ptr that is NULL!

[derekperkins]

Under high concurrency, creating a new arrow connection per request eventually and reliably throws this error. The original connector is created once in main.go as connector, err := duckdb.NewConnector(), then this is the per request connection code.

conn, err := connector.Connect(ctx)
if err != nil {
	return nil
}

arrow, err := duckdb.NewArrowFromConn(conn)
if err != nil {
	return nil
}

Eventually, within ~100 requests in 1s, this error is thrown and ends the program.

libc++abi: terminating due to uncaught exception of type duckdb::InternalException: {"exception_type":"INTERNAL","exception_message":"Attempted to dereference shared_ptr that is NULL!","stack_trace_pointers":"4390184520;4390122860;4381557828;4393827324;4386593228;4393147196;4381101280;4376908732"}
SIGABRT: abort
PC=0x1910b6388 m=12 sigcode=0
signal arrived during cgo execution

goroutine 3100 gp=0x14000550e00 m=12 mp=0x14000101808 [syscall]:
runtime.cgocall(0x105224cbc, 0x1400039b648)
	/opt/homebrew/opt/go/libexec/src/runtime/cgocall.go:167 +0x44 fp=0x1400039b610 sp=0x1400039b5d0 pc=0x104e1a9a4
github.com/duckdb/duckdb-go-bindings/darwin-arm64._Cfunc_duckdb_query_arrow_array(0x600002f6c200, 0x14000300000)
	_cgo_gotypes.go:4064 +0x30 fp=0x1400039b640 sp=0x1400039b610 pc=0x104ff6a80
github.com/duckdb/duckdb-go-bindings/darwin-arm64.QueryArrowArray.func1(...)
	/Users/derek/go/pkg/mod/github.com/duckdb/duckdb-go-bindings/darwin-arm64@v0.1.12/bindings.go:3440
github.com/duckdb/duckdb-go-bindings/darwin-arm64.QueryArrowArray({0x600002f6c200}, 0x50?)
	/Users/derek/go/pkg/mod/github.com/duckdb/duckdb-go-bindings/darwin-arm64@v0.1.12/bindings.go:3440 +0x6c fp=0x1400039b680 sp=0x1400039b640 pc=0x104ffec8c
github.com/marcboeker/go-duckdb/v2.queryArrowArray(0x140004061e8, 0x140002826c0)
	/Users/derek/go/pkg/mod/github.com/marcboeker/go-duckdb/v2@v2.3.5/arrow.go:283 +0xec fp=0x1400039b710 sp=0x1400039b680 pc=0x10519b6ac
github.com/marcboeker/go-duckdb/v2.(*arrowStreamReader).Next(0x14000282720)
	/Users/derek/go/pkg/mod/github.com/marcboeker/go-duckdb/v2@v2.3.5/arrow.go:164 +0x114 fp=0x1400039b750 sp=0x1400039b710 pc=0x10519a5a4
github.com/uwdata/mosaic/packages/server/duckdb-server-go/internal/query.(*DB).WriteArrow(0x140001d5480, {0x107404e30, 0x140000e0910}, {0x140003b61a0, 0xc1}, {0x0?, 0x1400039b988?, 0x104ea2a00?}, {0x1074000a0, 0x14000583200})
	/Users/derek/Code/github.com/uwdata/mosaic/packages/server/duckdb-server-go/internal/query/query.go:312 +0x278 fp=0x1400039b8e0 sp=0x1400039b750 pc=0x1051b1448

go-duckdb v2.3.5
MBP M3 Max - Sequoia 15.5

[mlafeldt]

Thanks for the report. For reproduction, could you please share more details, such as the framework used, the shape of requests, or even the actual request handling code?

[derekperkins]

I'm using https://github.com/uwdata/mosaic/tree/main/packages/server/duckdb-server-go. I was in the middle of adding a connection pool in this PR (uwdata/mosaic#877), and I was able to reproduce this crash 10-15 times. I did some additional cleanup to get it merged, and even now where I'm trying to actively reproduce the error, I can't.

Here's a quick screencap of what I was doing that triggered the bug. Each request was creating a new DuckDB connection.
https://screen.studio/share/GEv4n65j

Full crash dump: https://gist.github.com/derekperkins/189aafaef6ae1f1efe8c37fa1b83d10a

It's possible that I had some sort of race condition that was causing it. If this isn't enough detail, especially where I can't reproduce it, I'm happy to go ahead and close the issue.

[mlafeldt]

Awesome - thank you!

#528 should fix the crash (while changing nothing about the concurrency model).