Add container metadata collection using NRI

[yonch]

Background

Our collector currently measures CPU cycles, instructions, and LLC misses at 1ms frequency per process. While this provides valuable data, in cloud-native Kubernetes environments, process-level metrics have limited usefulness because many containers run identical processes (e.g., Java, Node.js). Process-level metrics without container context make it difficult to identify which workloads are causing memory interference.

Node Resource Interface (NRI) provides a standard way for plugins to receive real-time container lifecycle events and metadata from container runtimes. By integrating with NRI, we can enrich our metrics with container context including pod name, namespace, and other Kubernetes metadata.

Requirements

We need to implement an NRI plugin that:

1. Registers with the container runtime via the NRI socket
2. Synchronizes with existing container state on startup
3. Listens for container lifecycle events (start/stop)
4. Maps cgroup inode numbers to container metadata
5. Integrates this metadata with our eBPF-collected metrics
6. Outputs enriched metrics with container context to Parquet files

Implementation Plan

This work will be split into several sub-issues for manageability:

1. De-risk eBPF cgroup inode collection (highest priority)

• Add eBPF code to collect cgroup inode numbers for processes
• Validate we can reliably map processes to their cgroups
• This should be done first to confirm technical feasibility

2. Generate Rust bindings for NRI protocol

• Use the NRI protobuf definition to generate Rust code
• Leverage the TTRPC-Rust library for RPC communication
• Create data structures to represent container metadata

3. Implement NRI socket multiplexer in Rust

• Port the Go multiplexer implementation to Rust
• Support bidirectional communication over a single socket
• Handle framing protocol as specified by NRI

4. Create NRI plugin for metadata collection

• Implement plugin registration with container runtime
• Handle synchronization for existing containers
• Process container lifecycle events (start/stop)
• Build and maintain a mapping of cgroup inodes to container metadata

5. Integrate with existing metrics pipeline

• Extend the metrics schema to include container metadata
• Join process metrics with container context using cgroup inode
• Update Parquet output to include container fields

Technical Considerations

• We'll need to use the stat syscall to obtain inode numbers for cgroups
• The container runtime must be configured to enable the NRI socket (issue Enable NRI in containerd configuration via an init container in collector Helm chart #151 is not a prerequisite as we have manually configured a test cluster)
• Container metadata priority:
  • High priority: Pod name and Namespace
  • Nice to have: Container name (when pod has multiple containers)
• Note: For containers where NRI information isn't available, we will not provide container metadata in this implementation. The collector already outputs process names as a fallback. Future work (to be tracked in a separate issue) will explore parsing systemd directory inside cgroups to get systemd service names.

Success Criteria

• All metrics include container context when available
• Plugin correctly handles container lifecycle events
• Metrics can be aggregated and analyzed by pod/namespace/container

Testing Approach

We will implement two levels of testing:

1. Integration tests for metadata collection

• Spin up a lightweight Kubernetes environment (K3s or KinD)
• Test synchronization with pre-existing pods:
  • Deploy pods before starting the collector
  • Verify metadata collection correctly identifies all existing containers
• Test handling of new pods:
  • Start collector first
  • Deploy new pods after collector is running
  • Verify metadata is correctly collected for these new containers
• Test cleanup of metadata:
  • Verify metadata is properly removed when containers terminate
  • Test cleanup for both pre-existing containers and containers created after synchronization
  • Ensure the metadata data structure doesn't grow unbounded over time

2. End-to-end tests

• Test the complete collector pipeline including eBPF collection with container metadata
• Verify with both pre-existing pods and pods created after collector starts
• Confirm metrics in Parquet files have correct container metadata associated with the right processes

References

• NRI Protocol: https://github.com/containerd/nri/blob/main/pkg/api/api.proto
• Logger Plugin Example: https://github.com/containerd/nri/blob/main/plugins/logger/nri-logger.go
• Multiplexer Implementation: https://github.com/containerd/nri/blob/main/pkg/net/multiplex/mux.go
• TTRPC-Rust: https://github.com/containerd/ttrpc-rust

[yonch]

@KarstenB ttrpc-rust#292 mentions you're working on an NRI plugin. We're also working on an NRI plugin; do you have any code or notes we could consult, or alternatively are you interested in working on an implementation with us?

[KarstenB]

While I find your project really interesting, the NRI plugin is something that I work on for my company, so unfortunately I can not share code. I think you already found the most frustrating bit: #158 took me way too long to find that out.

[yonch]

@KarstenB thanks for the feedback. Do you have any advice on enabling NRI on pre-2.0 versions? e.g., what is described in #151 ?