High Mem & GC when compiling large cyclic data structure

[kaijietti]

go.mod:

module gojsontest

go 1.21

require (
	github.com/goccy/go-json v0.10.3
	github.com/stripe/stripe-go/v79 v79.2.0
)

main.go:

package main

import (
	"fmt"
	"log"
	"net/http"
	_ "net/http/pprof"

	"github.com/goccy/go-json"
	"github.com/stripe/stripe-go/v79"
)

func main() {
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()

	bytes, err := json.Marshal(&stripe.CheckoutSession{})
	if err != nil {
		panic(err)
	}
	fmt.Println(string(bytes))
}

The above code stuck in func (c *Compiler) compile(typeptr uintptr) phase. Plenty of memory was allocated and then GC contributed most of the CPU time.

A pprof sample is here.
pprof.samples.cpu.001.pb.gz

And I used the following script to check how many cycle definition:

package main

import (
	"fmt"
	"reflect"

	"github.com/stripe/stripe-go/v79"
)

func findTypeCycle(t reflect.Type, visited map[reflect.Type]bool, cycleCount *int) {
	if visited[t] {
		*cycleCount++
		return
	}

	visited[t] = true
	defer delete(visited, t)

	for i := 0; i < t.NumField(); i++ {
		field := t.Field(i)
		fieldType := field.Type

		if fieldType.Kind() == reflect.Ptr {
			fieldType = fieldType.Elem()
		}

		if fieldType.Kind() == reflect.Struct {
			findTypeCycle(fieldType, visited, cycleCount)
		}
	}
}

func main() {
	cycleCount := 0
	findTypeCycle(reflect.TypeOf(stripe.CheckoutSession{}), make(map[reflect.Type]bool), &cycleCount)
	fmt.Printf("Total number of cycles detected: %d\n", cycleCount)
}

And the output is:

Total number of cycles detected: 1083720

[allaboutstrategy]

I can say the same.

I build a trading bot, getting JSON data around 1 MB per second,
The cpu time is a bit lower but my Memory goes up and up and up over time.

After 1 day I am at 200mb compared to 24mb with standard encoding/json

[zkqiang]

same, it soared from 400MB to 1500MB in 15mins, but this is Unmarshal

(pprof) top
Showing nodes accounting for 1932.59MB, 96.26% of 2007.71MB total
Dropped 222 nodes (cum <= 10.04MB)
Showing top 10 nodes out of 63
      flat  flat%   sum%        cum   cum%
 1511.21MB 75.27% 75.27%  1591.93MB 79.29%  github.com/goccy/go-json.unmarshal

[pingvincible]

Hi I also have huge memory consumption when unmarshalling slice of structs and saving them to the map. If I use encoding/json, than memory footprint of the same number of elements in the map after unmarshaling three times lower.
And if I strings.clone elements of the struct to new struct and than save to the map then it also takes three times less memory than saving unmarshalled struct to the map.