Add status conditions to Function CRD for improved observability

[Copilot]

Overview

This PR enhances the Function CRD with robust status conditions to improve error visibility and troubleshooting. Users can now run kubectl describe function <name> and see clear, actionable information about reconciliation state, database matching, criteria expression evaluation, and Kusto execution status.

Problem Statement

Currently, when Functions fail to reconcile, errors are only visible in ingestor logs, making troubleshooting difficult. Common issues like case-sensitivity in database names, criteria expression evaluation failures, and Kusto execution errors are not exposed to users through standard Kubernetes tooling.

Solution

Following the established SummaryRule pattern, this PR adds three condition types to the Function CRD:

1. function.adx-mon.azure.com/DatabaseMatch

Indicates whether the Function's database matches an ingestor endpoint (case-insensitive comparison).

Example - Mismatch:

conditions:
- type: function.adx-mon.azure.com/DatabaseMatch
  status: "False"
  reason: DatabaseMismatch
  message: Function database 'AKSProd' does not match ingestor endpoint database 'AKSprod' (case-insensitive comparison)

2. function.adx-mon.azure.com/CriteriaMatch

Reflects criteria expression evaluation results.

Example - Expression Error:

conditions:
- type: function.adx-mon.azure.com/CriteriaMatch
  status: "False"
  reason: CriteriaExpressionError
  message: 'CriteriaExpression evaluation failed: parse error: invalid operator'

Example - Silent Skip:

conditions:
- type: function.adx-mon.azure.com/CriteriaMatch
  status: "False"
  reason: CriteriaNotMatched
  message: CriteriaExpression 'region == "westus"' evaluated to false, skipping function

3. function.adx-mon.azure.com/Reconciled

Primary reconciliation status indicating overall Function health.

Example - Success:

conditions:
- type: function.adx-mon.azure.com/Reconciled
  status: "True"
  reason: KustoExecutionSucceeded
  message: Function created at https://example.kusto.windows.net

Example - Transient Failure:

conditions:
- type: function.adx-mon.azure.com/Reconciled
  status: "False"
  reason: KustoExecutionRetrying
  message: 'Transient Kusto failure: network timeout'

Implementation Details

API Changes (api/v1/function_types.go)

• Added three condition type constants following domain naming convention
• Implemented helper methods:
  • GetCondition() - Retrieves the primary reconciliation condition
  • SetCondition(status, reason, message) - Sets reconciliation status
  • SetDatabaseMatchCondition(matched, configuredDB, availableDB) - Sets database matching status
  • SetCriteriaMatchCondition(matched, expression, err) - Sets criteria evaluation status
• All methods automatically set ObservedGeneration and LastTransitionTime

Ingestor Updates (ingestor/adx/tasks.go)

Updated SyncFunctionsTask to set conditions at every decision point:

• Database matching/mismatching
• Criteria expression evaluation (success, silent skip, error)
• Kusto execution (success, transient failure, permanent failure)
• Function deletion workflow

Testing

• Added 12 comprehensive unit tests covering all helper methods
• Updated integration tests to verify condition behavior
• All tests passing

User Experience Improvement

Before: Errors only visible in logs

$ kubectl get function my-func -o yaml
status:
  status: Failed
  error: "criteriaExpression evaluation failed"

After: Clear, actionable information

$ kubectl describe function my-func
Status:
  Conditions:
    Type:     function.adx-mon.azure.com/CriteriaMatch
    Status:   False
    Reason:   CriteriaExpressionError
    Message:  CriteriaExpression evaluation failed: cel typecheck: undeclared reference to 'region'
    Last Transition Time:  2025-10-16T20:05:00Z
    Observed Generation:   2

Backward Compatibility

• Existing FunctionStatus fields (Status, Error, Message, Reason) are preserved
• The Conditions field was already present in the CRD schema
• No breaking changes to existing Functions
• All existing tests continue to pass

Design Decisions

1. Followed SummaryRule Pattern - Maintains consistency with existing CRD patterns in the codebase
2. Domain Naming Convention - Uses function.adx-mon.azure.com/* prefix per project standards
3. Visibility on Silent Skips - Status updates even when skipping due to criteria mismatch, providing visibility without noise
4. Comprehensive Messages - Condition messages include actionable details (e.g., showing both expected and actual database names)
5. Generation Tracking - All conditions include ObservedGeneration for accurate reconciliation state tracking

Testing

# API tests
$ go test ./api/v1/... -v -run TestFunction
✅ All 12 test cases passing

# Integration tests  
$ go test ./ingestor/adx/... -short
✅ All tests passing

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add subresources to Functions for greater visibility</issue_title>
<issue_description># Function Status Conditions Enhancement Plan

Overview

This plan outlines the implementation of robust status conditions for the Function CRD to improve error visibility and user experience. The current case-insensitive database matching bug highlighted the need for better observability into Function reconciliation state.

Problem Statement

Users cannot easily diagnose why their Function CRDs are not being reconciled. Errors like:

• Database name mismatch (case-sensitivity issues)
• CriteriaExpression evaluation failures
• Kusto execution errors
• Endpoint mismatches

These errors are currently only visible in ingestor logs, making troubleshooting difficult.

Success Criteria

• Users can run kubectl describe function <name> and see clear status information
• All reconciliation errors are captured in status conditions
• Status conditions follow Kubernetes best practices (similar to SummaryRule)
• Backward compatibility maintained with existing FunctionStatus fields
• Integration tests validate status condition behavior

---

Phase 1: API Type Enhancements

1.1 Define Function Condition Types

File: api/v1/function_types.go

• Add condition type constants following SummaryRule pattern:

  const (
      // FunctionReconciled indicates the function has been successfully reconciled
      FunctionReconciled = "function.adx-mon.azure.com/Reconciled"
      // FunctionDatabaseMatch indicates if the function's database matches an ingestor endpoint
      FunctionDatabaseMatch = "function.adx-mon.azure.com/DatabaseMatch"
      // FunctionCriteriaMatch indicates if the function's criteria expression matches cluster labels
      FunctionCriteriaMatch = "function.adx-mon.azure.com/CriteriaMatch"
  )

• Document condition semantics in comments

• Ensure constants follow domain naming convention (function.adx-mon.azure.com)

1.2 Add Helper Methods to Function Type

File: api/v1/function_types.go

• Add GetCondition() method:

  func (f *Function) GetCondition() *metav1.Condition {
      return meta.FindStatusCondition(f.Status.Conditions, FunctionReconciled)
  }

• Add SetCondition() method for reconciliation status:

  func (f *Function) SetCondition(status metav1.ConditionStatus, reason, message string)

• Add SetDatabaseMatchCondition() for database matching status:

  func (f *Function) SetDatabaseMatchCondition(matched bool, configuredDB, availableDBs string)

• Add SetCriteriaMatchCondition() for criteria evaluation:

  func (f *Function) SetCriteriaMatchCondition(matched bool, expression string, err error)

• Ensure all methods properly set:

  • ObservedGeneration
  • LastTransitionTime
  • Type
  • Status (True/False/Unknown)
  • Reason (CamelCase string)
  • Message (human-readable details)

1.3 Update FunctionStatus Structure

File: api/v1/function_types.go

• Verify Conditions []metav1.Condition field exists in FunctionStatus
• Add godoc comments explaining condition usage
• Add examples in comments showing expected condition states

---

Phase 2: Storage Layer Updates

2.1 Update Functions Storage Interface

File: ingestor/storage/kql_functions.go

• Review UpdateStatus() method to ensure it updates conditions
• Verify status subresource updates work correctly
• Add logging for condition updates
• Ensure atomic status updates (use status subresource)

2.2 Add Condition Update Helpers

File: ingestor/storage/kql_functions.go

• Create helper to update function condition:

  func (f *functions) UpdateCondition(ctx context.Context, fn *Function, condition metav1.Condition) error

• Ensure proper error handling

• Add retry logic if needed

• Log condition changes for debugging

---

Phase 3: Ingestor Task Updates

3.1 Update SyncFunctionsTask - Database Matching

File: ingestor/adx/tasks.go (lines 130-145)

• When skipping function due to database mismatch:

  if function.Spec.Database != v1.AllDatabases && !strings.EqualFold(function.Spec.Database, t.kustoCli.Database()) {
      // Set condition indicating database mismatch
      function.SetDatabaseMatchCondition(false, function.Spec.Database, t.kustoCli.Database())
      _ = t.store.UpdateStatus(ctx, function)
      continue
  }

• Add success condition when database matches

• Include available databases in message for user clarity

• Log condition updates at debug level

3.2 Update SyncFunctionsTask - Criteria Expression Evaluation

File: ingestor/adx/tasks.go (lines 145-160)

• When CriteriaExpression evaluation fails:

  if err != nil {
      err = fmt.Errorf("criteriaExpres...

Fixes #975

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[jessejlt]

I'm not sure if it's able to run our manifest generation since it's a docker-in-docker thing.