Use ordered map to store key-value pairs?

[dclong]

When parsing a TOML file (of unknown schema) into a generic object, can we use ordered maps to store key-value pairs? This helps preserve the order keys.

[pelletier]

Sorry, I haven't forgotten about this question, but I don't have a good answer at the moment.

[dmcgowan]

Is there any other way to get the key ordering in v2? I'm trying to update to v2 in containerd and figuring out a replacement for how we get an ordered configuration value here.

[pelletier]

As of go-toml v2.0.6 you can use the go-toml/unstable package that exposes the AST generated by the parser. This gives you access to the data exactly in the order read by the parser. It's barebones but it should help!

I grabbed one of the tests from containerd to support an example of how to use the API to extract the hosts in order: https://go.dev/play/p/vbFdajcQJtT

package main

import (
	"fmt"

	tomlu "github.com/pelletier/go-toml/v2/unstable"
)

func main() {
	var hostsInOrder []string

	p := tomlu.Parser{}
	p.Reset([]byte(testtoml))

	// iterate over all top level expressions
	for p.NextExpression() {
		e := p.Expression()

		if e.Kind != tomlu.Table {
			continue
		}

		// Let's look at the key. It's an iterator over the multiple dotted parts of the key.
		it := e.Key()
		parts := keyAsStrings(it)

		// we're only considering keys that look like `hosts.XXX`
		if len(parts) != 2 {
			continue
		}
		if parts[0] != "host" {
			continue
		}

		hostsInOrder = append(hostsInOrder, string(parts[1]))
	}

	for i, host := range hostsInOrder {
		fmt.Printf("#%d: %s\n", i, host)
	}
}

// helper to transfor a key iterator to a slice of strings
func keyAsStrings(it tomlu.Iterator) []string {
	var parts []string
	for it.Next() {
		n := it.Node()
		parts = append(parts, string(n.Data))
	}
	return parts
}

const testtoml = `
server = "https://test-default.registry"
ca = "/etc/path/default"
[header]
  x-custom-1 = "custom header"

[host."https://mirror.registry"]
  capabilities = ["pull"]
  ca = "/etc/certs/mirror.pem"
  skip_verify = false
  [host."https://mirror.registry".header]
    x-custom-2 = ["value1", "value2"]

[host."https://mirror-bak.registry/us"]
  capabilities = ["pull"]
  skip_verify = true

[host."http://mirror.registry"]
  capabilities = ["pull"]

[host."https://test-1.registry"]
  capabilities = ["pull", "resolve", "push"]
  ca = ["/etc/certs/test-1-ca.pem", "/etc/certs/special.pem"]
  client = [["/etc/certs/client.cert", "/etc/certs/client.key"],["/etc/certs/client.pem", ""]]

[host."https://test-2.registry"]
  client = "/etc/certs/client.pem"

[host."https://test-3.registry"]
  client = ["/etc/certs/client-1.pem", "/etc/certs/client-2.pem"]

[host."https://noncompliantmirror.registry/v2/namespaceprefix"]
  capabilities = ["pull"]
  override_path = true

[host."https://noprefixnoncompliant.registry"]
  override_path = true
`

This would need a bit more work to be more resistant to other shapes of inputs if that matters to your use-case, but that should be doable. Practically speaking even though this package is named unstable, it hasn't changed much since it was released.

Hopefully that helps!

[dmcgowan]

That works perfectly, you're amazing! I understand introducing an interface there at the top level is tricky and wish I had a good interface to suggest.

[pelletier]

Glad to hear! Let me know if you end up writing some code that you think would be generic and useful to integrate in that package.