Config Validation for Helm Charts

[jblang]

Too many times, I've ended up with vector pods in CrashLoopBackoff when I make a mistake in my helm values file. I'm trying to use vector validate to validate my configuration locally before deploying the helm chart, but validation fails when it tries to test the Splunk sink.

~ Unknown env var in config. name = "SPLUNK_HEC_TOKEN"
~ Unknown env var in config. name = "SPLUNK_HEC_ENDPOINT"
~ Unknown env var in config. name = "SPLUNK_HEC_INDEX"
~ Unknown env var in config. name = "DATADOG_API_KEY"

√ Component configuration
2022-07-26T21:32:09.502593Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Unexpected status: 403 Forbidden component_kind="sink" component_type="datadog_metrics" component_id=vector_metrics_out component_name=vector_metrics_out
x Health check for "vector_metrics_out" failed
2022-07-26T21:32:09.502905Z ERROR vector::internal_events::http_client: HTTP error. error=client requires absolute-form URIs error_type="request_failed" stage="processing"
2022-07-26T21:32:09.502957Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Failed to make HTTP(S) request: client requires absolute-form URIs component_kind="sink" component_type="splunk_hec_logs" component_id=splunk_logs_out component_name=splunk_logs_out
x Health check for "splunk_logs_out" failed

I would have assumed that this scenario is what the --no-environment flag was for, but when I pass that, it doesn't seem to validate the configuration at all. Validation seems to always pass without catching fallible functions that need error handling or even blatant syntax errors. Am I missing something?

[jblang]

Here's the workaround I came up with:

# locally render the template to the temp directory
dump_template() {
  helm template \
    --output-dir $tmpdir \
    $component $chart \
    --create-namespace \
    --namespace $namespace \
    $sets \
    $values \
    > /dev/null
}

# extract the specified key from the configmap in the temp directory
extract_configmap() {
  local key=$1  
  cat $tmpdir/$component/templates/configmap.yaml | 
    grep -v '^---$' | 
    yq -r ".data.\"$key\"" > "/$tmpdir/$key"
}

# convert all vector sinks to blackholes
blackhole_sinks() {
  local blackhole='"type": "blackhole", "inputs":'
  local sinks=$(yq -r ".sinks | keys | .[]" $tmpdir/vector.yaml)
  for sink in $sinks; do
    local inputs=$(yq -c ".sinks.$sink.inputs" $tmpdir/vector.yaml)
    yq -y ".sinks.$sink = {$blackhole $inputs}" $tmpdir/vector.yaml > $tmpdir/temp.yaml
    mv $tmpdir/temp.yaml $tmpdir/vector.yaml
  done
}

# validate the vector configuration
validate_vector() {
  if ! which vector yq > /dev/null; then
    echo "WARNING: vector or yq executable is not in your path. Unable to validate config."
  else
    dump_template
    extract_configmap vector.yaml
    blackhole_sinks
    vector validate $tmpdir/vector.yaml
  fi
}

As a side note, using yq to manipulate the helm template output is not ideal. It would be nice if vector validate had an option to accept a ConfigMap file directly and then validate the vector.yaml configuration embedded in it for syntax errors, etc. That way I wouldn't have to rely on additional tools like jq and yq being installed. It would be even nicer if it could accept the full helm template output on stdin and extract and validate the configuration from it, ignoring the rest. That way I could skip the temporary directory and just pipe the helm output directly into vector validate.

@jszwedko It would really help out if Vector offered some direct support for validating helm template output, and a flag to automatically converting all sinks to blackholes that still accepted the original inputs. Is this an enhancement you would consider?