Detectors #5003
Draft
Detectors #5003
+4,837
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Create public DataStore API for unified access to system state: New files: - api/v1beta1/datastores/types.go: Core data types * ProcessInfo, ContainerInfo, SymbolInfo, DNSResponse * HealthInfo, HealthStatus, DataStoreMetrics * DataStoreMetadata, error types - api/v1beta1/datastores/interfaces.go: Store interfaces * DataStore: Base interface with health & metrics * ProcessStore: Process tree access * ContainerStore: Container information * KernelSymbolStore: Symbol resolution * DNSCacheStore: DNS query cache - api/v1beta1/datastores/registry.go: Registry interface * Unified access to all datastores * Custom store registration support * Metadata and availability queries
- Add store.go files implementing DataStore and specific store interfaces - Process: GetProcess, GetChildProcesses - Container: GetContainer, GetContainerByName - Symbol: ResolveSymbolByAddress (returns all aliases), GetSymbolAddress, ResolveSymbolsBatch - DNS: GetDNSResponse - Track lastAccess timestamp for operational metrics - Rename Manager.GetContainer -> LookupContainer to avoid method collision All datastores implement Name(), GetHealth(), GetMetrics() for monitoring.
Add tests for DataStore interface implementations in process, container, symbol, and dns datastores, validating Name(), GetHealth(), GetMetrics(), store-specific methods, and LastAccess tracking.
Implement Registry providing centralized, thread-safe access to all registered datastores: - Exported Registry struct with RegisterStore() for Tracee engine - Implements datastores.Registry interface for detector access - Type-safe accessor methods for core stores - Generic GetCustom() for extension datastores - Store listing, availability checks, and metadata retrieval - Comprehensive tests covering all functionality Implements single concrete type usable both for registration (Tracee engine) and read-only access (detectors via interface).
Initialize the datastore registry in Tracee.Init() and register core datastores (process, container, symbol, dns) for access by detectors and extensions. Kernel symbol store uses an adapter to support runtime hot-reload without stale references.
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (31.84%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #5003 +/- ##
=======================================
Coverage ? 29.53%
=======================================
Files ? 246
Lines ? 27988
Branches ? 0
=======================================
Hits ? 8267
Misses ? 19172
Partials ? 549
🚀 New features to boost your workflow:
|
Implement detector API foundation with EventDetector interface, DetectorDefinition, EventRequirement with dependency types, and DetectorConfig for type-safe configuration access.
Implement generic event data extraction and creation helpers: - GetData[T] and GetDataSafe[T] with type constraints - Null-safe context helpers (process, container) - CreateEventFromBase for unified event creation Clear separation of responsibilities: - Engine: assigns Id/Name, auto-populates Threat/DetectedFrom - Detector: populates Data with runtime results
Add detector management infrastructure in pkg/detectors/: Engine (engine.go): - Orchestrates registry and dispatcher - Public API: RegisterDetector, UnregisterDetector, ListDetectors, GetDetector, EnableDetector, DisableDetector, DispatchToDetectors Registry (registry.go): - Manages detector lifecycle and enforces event name uniqueness - Two-phase event ID allocation: predefined events or dynamic (7000+) - Uniqueness ensures event identity, detector chains, and security Dispatcher (dispatch.go): - Manages event routing map (event ID → detector IDs) - Dispatches events to detectors, assigns IDs, returns outputs Integration: - Added detectorEngine to Tracee struct (pkg/ebpf/tracee.go) - Defined detector ID range 7000-7999 (pkg/events/core.go) - Added LookupPredefinedEventID() helper All internal types unexported for proper encapsulation.
Add declarative field auto-population in dispatcher: - Threat: Deep clone ThreatMetadata (including MITRE, properties) - DetectedFrom: Reference input event for audit trail - ProcessAncestry: TODO (requires ProcessStore.GetAncestry) Detectors opt-in via AutoPopulateFields in their definition. All fields default to false for minimal overhead.
Add bidirectional conversion between internal trace.Event and protobuf v1beta1.Event for the detector API. ConvertToProto: Maps all workload fields (process, thread, container, k8s, ancestors) ConvertFromProto: Converts detector outputs back to trace.Event for pipeline
Integrate detector engine into event processing pipeline with full detector chain support: - Add detectEvents stage between derive and engine stages - Convert events using pkg/events conversion helpers - Implement breadth-first detector chain traversal (max depth 5) - Apply policy filtering to detector outputs - Send original event first, then detector outputs Pipeline order: decode → sort → process → enrich → derive → detect → engine → sink
Create detectors module with init-based auto-registration: **detectors/registry.go:** - Global registry for detector auto-registration - register() called by detector init() functions - GetAllDetectors() exports all registered detectors - Thread-safe with RWMutex **detectors/go.mod:** - Separate Go module for detector implementations - Depends only on api module (clean separation) **Pattern:** Detectors register themselves via init() - no manual list maintenance. Simply add a detector file with init() calling register() and it's automatically discovered. Benefits: - Zero maintenance (no export lists) - Self-contained (each detector manages registration) - No forgotten detectors (compile = registered) - Standard Go pattern (familiar from database drivers)
Pre-register detector events in events.Core before policy init, allowing policy manager to select detector events like regular events. Only initialize detectors whose events are selected by policy to avoid resource waste. **Key Changes:** - DetectorConfig stores all discovered detectors - CollectAllDetectors/CreateEventsFromDetectors pre-register events - Registry checks IsEventSelected() before calling detector Init() - Unselected detectors: registered but not initialized (enabled=false) - Tracee's dependency manager handles detector event requirements **Benefits:** - Detector events selectable via policy - Resource efficient: no Init() for unselected detectors - Dependency resolution handled by existing infrastructure
Enable/Disable now properly manage detector lifecycle: - EnableDetector: Calls Init() and sets enabled=true - DisableDetector: Calls Close() and sets enabled=false - UnregisterDetector: Calls Close() only if enabled Store DetectorParams in entry to support re-initialization on enable.
Add ExampleDetector demonstrating the detector API: - Subscribes to execve events - Produces example_detection event with 3 fields - Shows field definitions, threat metadata, and auto-population - Implements full detector lifecycle (Init/OnEvent/Close) This detector serves as: - Reference implementation for new detectors - Test case for detector infrastructure - Live documentation of API patterns
- Fix detector output events to inherit MatchedPoliciesKernel from source event - Simplify detectEvents to save policy context once per chain instead of per-event - Add IsDetector() method supporting both predefined (3000-3999) and dynamic (7000-7999) ranges - Add runtime validation for predefined detector event IDs - Update policy manager to auto-select dependencies for detector events - Add debug logging to dispatcher rebuild for troubleshooting
- Collect and register detector events in list command (similar to signatures) - Add 'Detector Events' section to PrintEventList output - Exclude detector events from 'Other Events' section - Makes detector events discoverable via 'tracee list'
Issue: DataStore registry was being initialized and populated before the datastores themselves were created, resulting in nil pointers being registered. This caused crashes when detectors tried to access datastores. Root cause: ProcessTree initialization happens on line 506, but registry registration was happening on line 438 (before ProcessTree creation). Fix: - Move datastore registry initialization to after ProcessTree is created - Move detector registration to after datastores are registered - Mark ProcessStore as optional (required=false) since ProcessTree can be disabled via config (Source=SourceNone is the default) This ensures: 1. Datastores are fully initialized before being registered 2. Detectors have access to valid datastore instances during Init() 3. System gracefully handles disabled datastores (like ProcessTree) Related: ProcessTree is disabled by default for performance reasons. Users can enable it with --proctree source=events|signals|both
- Use ProcessStore to get parent process info (ppid) - Use ContainerStore to get container name - Add fields: parent_process, container_id, container_name - Add nil checks for optional/disabled datastores - Demonstrates best practices for DataStore access in detectors
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Detectors draft