Panic/Err when reading certain NDJSON with Bodkin + Arrow JSON reader

[caldempsey]

While integrating Bodkin in one of our pipelines we bumped into a panic that we could narrow down to a very specific piece of shaped data. I’m opening this issue in case it’s helpful for you to reproduce and (if appropriate) guard against.

When the reader hits the record it panics somewhere inside Arrow’s JSON reader, presumably because the inferred schema can’t accommodate the second record’s shape. Our wrapper catches the panic, but because Arrow panics before returning an error, the resulting message is fairly generic.

I think ideally if Arrow can't handle the downstream data Bodkin would surface a clear validation error (schema mismatch, nested map too deep, etc.) instead of Arrow panicking. If that’s outside Bodkin’s remit, totally understood because Arrow shouldn't be panicking in the first place but I can see warnings being surfaced from Bodkin in the console logs. Would be helpful to understand why this happens if you're familiar.

Thanks for your amazing repo! Here's a reproduction script:

package main

import (
	"bytes"
	"fmt"
	"github.com/loicalleyne/bodkin"
	"log"
	"strings"

	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/arrow-go/v18/arrow/memory"
)

func main() {
	fmt.Println("json 1")
	jsonPass := []byte(`{"books": [{ "id": 1, "title": "To Kill a Mockingbird", "author": "Harper Lee" }, { "id": 2, "title": "1984", "author": "George Orwell" }, { "id": 3, "title": "The Great Gatsby", "author": "F. Scott Fitzgerald" }]}
{"books": [{ "id": 1, "title": "To Kill a Mockingbird", "author": "Harper Lee" }, { "newIDColumn": 2, "title": "1984", "author": "George Orwell" }, { "id": 3, "title": "The Great Gatsby", "author": "F. Scott Fitzgerald" }]}`)
	buildTableFromJSON(jsonPass)
	fmt.Println("successfully created arrow table from json 1")

	fmt.Println("json 2")
	// Deeply nested JSON data (NDJSON format - each record on its own line)
	jsonFail := []byte(`{"file_path":"s3://test-bucket/batch-gnarly/file-0.jpg","some_label":"weird_label_#0","confidence":0.99,"model_name":"🤖_model_v9","processed_at":"2025-07-23T14:00:00Z","batch_id":"gnarly-batch-0001","metadata":{"notes":"This is a \"very\" strange record with \n newlines, tabs\t and unicode: 🌐🚀","edge_cases":{"null_value":null,"bool_true":true,"bool_false":false,"nested":{"inner":{"value":123,"list":[1,2,3,{"deep":"value"}]}}},"timestamps":["2020-01-01T00:00:00Z","2050-12-31T23:59:59Z"],"weird_keys":{"1key":"numeric start","spaces in key":"hello world","symbols":"!@#$%^&*()"},"json_string":{"embedded_json":"yes"}}}
{"file_path":"s3://test-bucket/batch-gnarly/file-1.jpg","some_label":"$$$-odd_label","confidence":0.42,"model_name":"model_<script>alert(1)</script>","processed_at":"2025-07-23T14:00:00Z","batch_id":"gnarly-batch-0001","metadata":{"long_text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit.","deeply_nested":{"a":{"b":{"c":{"d":{"e":{"f":{"g":"too deep?"}}}}}}},"mixed_list":[true,42,"string",null,{"inner":"thing"}],"random_data":"<0001f9ec>🔣🌈"}}`)
	buildTableFromJSON(jsonFail)
}

func buildTableFromJSON(json []byte) {
	u := bodkin.NewBodkin(
		bodkin.WithInferTimeUnits(),
		bodkin.WithTypeConversion(),
	)

	lines := strings.Split(strings.TrimSpace(string(json)), "\n")
	// unify the schema
	var cleanedLines []string

	for _, l := range lines {
		l = strings.TrimSpace(l)
		if l == "" {
			continue
		}
		if err := u.Unify(l); err != nil {
			log.Fatal(err)
		}
		cleanedLines = append(cleanedLines, l)
	}

	schema, err := u.Schema()
	if err != nil {
		log.Fatal(err)
	}

	// Debug: print schema info
	log.Printf("Schema has %d fields", len(schema.Fields()))
	for i, field := range schema.Fields() {
		log.Printf("Field %d: %s (%s)", i, field.Name, field.Type)
	}

	// Create clean NDJSON with proper newlines
	cleanJSON := strings.Join(cleanedLines, "\n")

	// Try with a smaller chunk size to avoid memory issues
	rdr := array.NewJSONReader(bytes.NewReader([]byte(cleanJSON)),
		schema,
		array.WithAllocator(memory.DefaultAllocator),
		array.WithChunk(-1), // set to 1 reveals 2025/07/24 11:33:12 Reader error: json: cannot unmarshal { into Go value of type int64
	)
	defer rdr.Release()

	var records []arrow.Record
	for rdr.Next() {
		rec := rdr.Record()
		rec.Retain()
		records = append(records, rec)
		log.Printf("Successfully read record with %d columns", rec.NumCols())
	}
	if err := rdr.Err(); err != nil {
		log.Fatalf("Reader error: %v", err)
	}

	if len(records) == 0 {
		log.Fatal("No records were read")
	}

	table := array.NewTableFromRecords(schema, records)
	defer table.Release()

	log.Printf("Successfully created table with %d records and %d columns",
		table.NumRows(), table.NumCols())

	// Clean up
	for _, rec := range records {
		rec.Release()
	}
}

[loicalleyne]

Hi @caldempsey, thanks for reaching out, I'm so glad someone found a thing I made useful and is actually using it. (^_^).
Bodkin can deal with very deeply nested types, we're using it to handle some very gnarly JSON in our data pipelines.
In the repro above I've determined that this bit is tripping up the decode:

"mixed_list":[true,42,"string",null,{"inner":"thing"}]

Bodkin doesn't yet support union types (wasn't in my use case at all), sliceElemType() only looks at the first element in the array and expects all elements to be of the same type. arrow-go's UnmarshalJSON supports JSON with union types but the data has to be laid out in a specific way which it probably wouldn't be if you're inferring its schema.

WRT improving the situation in Bodkin, perhaps an improvement could be adding an option to check that LIST elements all have the same type and either surfacing it in bodkin.Err() or setting that list element's type as VARIANT.

[caldempsey]

@loicalleyne hey thanks! no this is probably one of the best schema inference libraries ive ever used. im genuinely so impressed by your work. i really hope you contribute this as a PR to the core arrow codebase. i started working on the spark connect go codebase, and submitting prs is pretty easy to do: because the community is open source you just gotta raise a ticket in the jira on why you need it after you make an account, and then roll up your sleeves for someone to take a look.

since schema inference is supported in pyarrow, i think this is something we should have too, but you've done such a good job with this, i really don't see why anyone wouldn't want it.

[caldempsey]

to answer your question, raising an error would be so much better here. if we can't infer a schema then fair enough i think in most cases its reasonable to ask people to supply one. this already does a good enough job across consistent data types, and since most langs are statically typed this union type situation should be exceedingly rare, and worth just erroring gracefully (so we don't have to eat the runtime implications of a panic) until more people (when you finally contribute this magnum opus) ask for explicit support.

[loicalleyne]

@caldempsey I just ran a quick experiment with the new default variant type and the arrow json unmarshaler had no trouble at all with the mixed_type field that tripped it up in your example. So just for usability's sake, I think I'll add options to check all list elements' type, optionally "upgrade" to variant if the elements are not all of the same type, otherwise return an error.

I've already submitted a PR in arrow-go to add schema.Unify that uses bodkin's schema tree-building logic to unify arrow schemas, reviewer suggested imitating another language's implementation using the already existing schema.Builder( or record. Builder, can't remember which); working on all the recursive schema logic takes a huge amount of concentration (and also motivation to replace something I already got working).
I'll revisit it to see if it's as hard as I imagined it was or not, the logic for schema inference and schema unification are quite closely related.

[caldempsey]

@loicalleyne Glad the variant type is working, and thank you so much for finding the issue here! I really really appreciate it !

Would love to see the PR thread to understand what the reviewer's actually proposing? If they just mean lodging Bodkin behind a common interface which already exists, then that would be more understandable. That's pretty common in Apache projects, they try to use the same interfaces across langs but that doesn't mean the actual implementation details are the same.

I have a bit of empathy here. You have working, tested code solving a genuinely hard problem. If the reviewer is asking someone to rewrite it just to match another language's pattern (in Go of all languages, where type constraints are completely different) feels like bikeshedding, because ultimately Golang is such a unique language nothing is really going to look like it, and the work you've done is quite clearly as good as it's going to get.

The schema unification gap is real and your solution fills it. Consider writing it up - would help others facing the same problem. I suspect a massive part of the issue of Arrow adoption in Go is a lack of this feature, and I think it should be celebrated.

[loicalleyne]

@caldempsey to be fair and transparent, the PR's reviewer (zeroshade) is a much more experienced Go developer than I, has contributed a large part of the arrow-go implementation, has had a hand in just about every major vendor's Arrow-related implementation in Go that I've seen to date (not to mention in several other languages) and literally wrote the book on Arrow - his feedback is usually very much on point; was just explaining why the inference stuff hasn't been contributed yet. I tend to build tools that address my specific use case, sometimes the implementation is alright for me but needs adjusting to be suitable to be included in a widely used Apache library.

If you have any feedback/suggestions on improving bodkin's API to make it more idiomatic or easier to use, I'd love to hear about your use cases.

[caldempsey]

@loicalleyne I see the comment, and yes zeroshade is a top bloke. I think its valid feedback. The re-implementation might not be a big lift. You already have the logic here to do the hard bit - the type resolution, and then you have the tree construction. If I'm not misreading, I think the proposal here is the whole process can be made more succinct by reframing it as a recursion problem (rather than a tree problem); because ultimately the schema resolution is additive and can be re-constructed on a per-node basis. I might be able to find time to pick this up if you're okay with me lifting bits of Bodkin into the PR. Would you be comfortable with me picking this up?

[caldempsey]

@loicalleyne Think the biggest gripe I have had with Bodkin is having to do the Unification across NDJSON myself, and I presume this would also be an issue for an array of JSON inputs. That was a bit painful. I think instead of taking []byte you could expose UnifyAll with []json.RawMessage as the input parameter, which iterates JSON payloads of [{}] or {} depending on your With options. I think WithNewLineDelimited would be a good naming convention to pick up from in an IDE.

Specifically having to do this was certainly surprising:

	u := bodkin.NewBodkin(
		bodkin.WithInferTimeUnits(),
		bodkin.WithTypeConversion(),
	)

	for i, l := range strings.Split(strings.TrimSpace(string(jsonData.Bytes())), "\n") {
		l = strings.TrimSpace(l)
		if l == "" {
			continue
		}
		if err := u.Unify(l); err != nil {
			return nil, fmt.Errorf("schema inference: line %d: %w", i+1, err)
		}
	}

I had to dig into your source code to really understand that Unify had to be called many times across different lines, and I couldn't just send my whole NDJSON formatted strings in. I expected Bodkin to recognise the \n split or have an option to out the box.

[loicalleyne]

@caldempsey
for blob of NDJSON I think you should be able to create a bodkin with the option WithIOReader where you can pass an io.Reader and define the delimiter (default is \n) and then use bodkin.UnifyScan(). I realize now I probably could have done that in bodkin/json2parquet.FromReader().

[loicalleyne]

@loicalleyne I see the comment, and yes zeroshade is a top bloke. I think its valid feedback. The re-implementation might not be a big lift. You already have the logic here to do the hard bit - the type resolution, and then you have the tree construction. If I'm not misreading, I think the proposal here is the whole process can be made more succinct by reframing it as a recursion problem (rather than a tree problem); because ultimately the schema resolution is additive and can be re-constructed on a per-node basis. I might be able to find time to pick this up if you're okay with me lifting bits of Bodkin into the PR. Would you be comfortable with me picking this up?

@caldempsey absolutely, that's what open source is for, innit? 😁

[loicalleyne]

@caldempsey for blob of NDJSON I think you should be able to create a bodkin with the option WithIOReader where you can pass an io.Reader and define the delimiter (default is \n) and then use bodkin.UnifyScan(). I realize now I probably could have done that in bodkin/json2parquet.FromReader().

And just like that whilst writing a test for this I realized this is broken [facepalm]. Working on a fix - bear with...

[loicalleyne]

@caldempsey for blob of NDJSON I think you should be able to create a bodkin with the option WithIOReader where you can pass an io.Reader and define the delimiter (default is \n) and then use bodkin.UnifyScan(). I realize now I probably could have done that in bodkin/json2parquet.FromReader().

And just like that whilst writing a test for this I realized this is broken [facepalm]. Working on a fix - bear with...

WithIOReader fixed.

[loicalleyne]

Note: Added the following options to bodkin v0.3.11

1. WithCheckForUnion()
2. WithUseVariantForUnions()
   For now, only LIST elements can be checked for union types, with the option to define the element type as VARIANT.
   Checking non-list fields for union types would make the schema merging and type upgrades more complex, besides, if the input data is JSON with union types in non-array fields you might as well put the whole thing in a VARIANT field and forgo the inference.