Skip to content

Commit abad255

Browse files
authored
feat: Implement SARIF output (#1042)
* feat: add SARIF output format support Add Static Analysis Results Interchange Format (SARIF) v2.1.0 output support to conftest. SARIF is a standard JSON format for static analysis tools. - SARIF v2.1.0 schema compliance - Includes file locations and rule metadata - Tracks execution timing and status - Test coverage - Documentation Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> * feat(output): implement SARIF output using go-sarif library Add Static Analysis Results Interchange Format (SARIF) v2.1.0 output support using the go-sarif library. This provides a standard JSON format for static analysis results with proper schema compliance. Key changes: - Use go-sarif/v2 library instead of custom implementation - Support all result types (failures, warnings, exceptions, successes) - Add comprehensive test coverage with JSON comparison - Document new output format in options.md The SARIF output includes: - File locations and rule metadata - Proper result levels (error/warning/note/none) - Execution status and exit codes - Rule properties from result metadata Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> * refactor: address pr comments - refactor: remove getRuleIndex Use direct map lookups instead - refactor: succinct map lookups Map lookup with a fallback - refactor: move result type logic to addResult func Cleaner code, while not really idiomatic due to go-sarif library design. - fix: treat exceptions as success A file with only exceptions will be treated as a success. Exceptions will still be logged (with level "note") for visibility. The exit code will be 0 (success) when there are only exceptions. - refactor: simplify hasFailures and hasWarnings Risk of typo is too high - refactor: treat exceptions as successes in SARIF output Exceptions are now treated as successes in the SARIF output, removing the separate exception handling. - test: type safe test input for SARIF Probably helps writing further test cases, instead of bare JSON - refactor: use google/go-cmp for json diff Based on PR comment Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> --------- Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
1 parent 9efcd87 commit abad255

File tree

7 files changed

+832
-0
lines changed

7 files changed

+832
-0
lines changed

docs/options.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ As of today Conftest supports the following output types:
173173
- JUnit `--output=junit`
174174
- GitHub `--output=github`
175175
- AzureDevOps `--output=azuredevops`
176+
- SARIF `--output=sarif`
176177

177178
### Plaintext
178179

@@ -322,6 +323,13 @@ success file=examples/kubernetes/deployment.yaml 1
322323
5 tests, 1 passed, 0 warnings, 4 failures, 0 exceptions
323324
```
324325

326+
### SARIF
327+
328+
```console
329+
$ conftest test --proto-file-dirs examples/textproto/protos -p examples/textproto/policy examples/textproto/fail.textproto -o sarif
330+
{"version":"2.1.0","$schema":"https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json","runs":[{"tool":{"driver":{"informationUri":"https://github.com/open-policy-agent/conftest","name":"conftest","rules":[{"id":"main/deny","shortDescription":{"text":"Policy violation"}}]}},"invocations":[{"executionSuccessful":true,"exitCode":1,"exitCodeDescription":"Policy violations found"}],"results":[{"ruleId":"main/deny","ruleIndex":0,"level":"error","message":{"text":"fail: Power level must be over 9000"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"examples/textproto/fail.textproto"}}}]}]}]}
331+
```
332+
325333
## `--parser`
326334

327335
Conftest normally detects which parser to used based on the file extension of the file, even when multiple input files are passed in. However, it is possible force a specific parser to be used with the `--parser` flag.

go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ require (
8585
github.com/moby/docker-image-spec v1.3.1 // indirect
8686
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
8787
github.com/opencontainers/go-digest v1.0.0 // indirect
88+
github.com/owenrumney/go-sarif/v2 v2.3.3 // indirect
8889
github.com/pelletier/go-toml/v2 v2.2.3 // indirect
8990
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
9091
github.com/prometheus/client_golang v1.20.5 // indirect

go.sum

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1007,6 +1007,10 @@ github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8
10071007
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
10081008
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
10091009
github.com/opencontainers/image-spec v1.1.0/go.mod h1:W4s4sFTMaBeK1BQLXbG4AdM2szdn85PY75RI83NrTrM=
1010+
github.com/owenrumney/go-sarif v1.1.1 h1:QNObu6YX1igyFKhdzd7vgzmw7XsWN3/6NMGuDzBgXmE=
1011+
github.com/owenrumney/go-sarif v1.1.1/go.mod h1:dNDiPlF04ESR/6fHlPyq7gHKmrM0sHUvAGjsoh8ZH0U=
1012+
github.com/owenrumney/go-sarif/v2 v2.3.3 h1:ubWDJcF5i3L/EIOER+ZyQ03IfplbSU1BLOE26uKQIIU=
1013+
github.com/owenrumney/go-sarif/v2 v2.3.3/go.mod h1:MSqMMx9WqlBSY7pXoOZWgEsVB4FDNfhcaXDA1j6Sr+w=
10101014
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
10111015
github.com/pelletier/go-toml/v2 v2.2.3 h1:YmeHyLY8mFWbdkNWwpr+qIL2bEqT0o95WSdkNHvL12M=
10121016
github.com/pelletier/go-toml/v2 v2.2.3/go.mod h1:MfCQTFTvCcUyyvvwm1+G6H/jORL20Xlb6rzQu9GuUkc=
@@ -1146,6 +1150,7 @@ github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1
11461150
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
11471151
github.com/zclconf/go-cty v1.2.0/go.mod h1:hOPWgoHbaTUnI5k4D2ld+GRpFJSCe6bCM7m1q/N4PQ8=
11481152
github.com/zclconf/go-cty v1.6.1/go.mod h1:VDR4+I79ubFBGm1uJac1226K5yANQFHeauxPBoP54+o=
1153+
github.com/zclconf/go-cty v1.10.0/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
11491154
github.com/zclconf/go-cty v1.13.2 h1:4GvrUxe/QUDYuJKAav4EYqdM47/kZa672LwmXFmEKT0=
11501155
github.com/zclconf/go-cty v1.13.2/go.mod h1:YKQzy/7pZ7iq2jNFzy5go57xdxdWoLLpaEp4u238AE0=
11511156
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
@@ -1871,6 +1876,7 @@ google.golang.org/protobuf v1.36.3 h1:82DV7MYdb8anAVi3qge1wSnMDrnKK7ebr+I0hHRN1B
18711876
google.golang.org/protobuf v1.36.3/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
18721877
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
18731878
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
1879+
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
18741880
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
18751881
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
18761882
gopkg.in/cheggaaa/pb.v1 v1.0.27/go.mod h1:V/YB90LKu/1FcN3WVnfiiE5oMCibMjukxqG/qStrOgw=

output/output.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ const (
3434
OutputJUnit = "junit"
3535
OutputGitHub = "github"
3636
OutputAzureDevOps = "azuredevops"
37+
OutputSARIF = "sarif"
3738
)
3839

3940
// Get returns a type that can render output in the given format.
@@ -57,6 +58,8 @@ func Get(format string, options Options) Outputter {
5758
return NewGitHub(options.File)
5859
case OutputAzureDevOps:
5960
return NewAzureDevOps(options.File)
61+
case OutputSARIF:
62+
return NewSARIF(options.File)
6063
default:
6164
return NewStandard(options.File)
6265
}
@@ -72,5 +75,6 @@ func Outputs() []string {
7275
OutputJUnit,
7376
OutputGitHub,
7477
OutputAzureDevOps,
78+
OutputSARIF,
7579
}
7680
}

output/output_test.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ func TestGetOutputter(t *testing.T) {
3939
input: OutputAzureDevOps,
4040
expected: NewAzureDevOps(os.Stdout),
4141
},
42+
{
43+
input: OutputSARIF,
44+
expected: NewSARIF(os.Stdout),
45+
},
4246
{
4347
input: "unknown_format",
4448
expected: NewStandard(os.Stdout),

output/sarif.go

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
package output
2+
3+
import (
4+
"fmt"
5+
"io"
6+
"path/filepath"
7+
"strings"
8+
9+
"github.com/open-policy-agent/opa/tester"
10+
"github.com/owenrumney/go-sarif/v2/sarif"
11+
"golang.org/x/exp/slices"
12+
)
13+
14+
const (
15+
// Tool information
16+
toolName = "conftest"
17+
toolURI = "https://github.com/open-policy-agent/conftest"
18+
sarifVersion = sarif.Version210
19+
20+
// Result descriptions
21+
successDesc = "Policy was satisfied successfully"
22+
skippedDesc = "Policy check was skipped"
23+
failureDesc = "Policy violation"
24+
warningDesc = "Policy warning"
25+
exceptionDesc = "Policy exception"
26+
27+
// Exit code descriptions
28+
exitNoViolations = "No policy violations found"
29+
exitViolations = "Policy violations found"
30+
exitWarnings = "Policy warnings found"
31+
)
32+
33+
// SARIF represents an Outputter that outputs results in SARIF format.
34+
type SARIF struct {
35+
writer io.Writer
36+
}
37+
38+
// NewSARIF creates a new SARIF with the given writer.
39+
func NewSARIF(w io.Writer) *SARIF {
40+
return &SARIF{
41+
writer: w,
42+
}
43+
}
44+
45+
// getRuleID generates a stable rule ID based on namespace and rule type
46+
func getRuleID(namespace string, ruleType string) string {
47+
return fmt.Sprintf("%s/%s", namespace, ruleType)
48+
}
49+
50+
// getRuleDescription returns the appropriate description based on the rule type
51+
func getRuleDescription(ruleID string) string {
52+
switch {
53+
case strings.HasSuffix(ruleID, "/success"):
54+
return successDesc
55+
case strings.HasSuffix(ruleID, "/skip"):
56+
return skippedDesc
57+
case strings.HasSuffix(ruleID, "/allow"):
58+
return exceptionDesc
59+
case strings.HasSuffix(ruleID, "/warn"):
60+
return warningDesc
61+
default:
62+
return failureDesc
63+
}
64+
}
65+
66+
// addRuleIndex adds a new rule to the SARIF run and returns its index.
67+
func addRuleIndex(run *sarif.Run, ruleID string, result Result, indices map[string]int) int {
68+
addRule(run, ruleID, result)
69+
idx := len(run.Tool.Driver.Rules) - 1
70+
indices[ruleID] = idx
71+
return idx
72+
}
73+
74+
// addRule adds a new rule to the SARIF run with the given ID and result metadata.
75+
func addRule(run *sarif.Run, ruleID string, result Result) {
76+
desc := getRuleDescription(ruleID)
77+
rule := run.AddRule(ruleID).
78+
WithDescription(desc).
79+
WithShortDescription(&sarif.MultiformatMessageString{
80+
Text: &desc,
81+
})
82+
83+
if result.Metadata != nil {
84+
props := sarif.NewPropertyBag()
85+
for k, v := range result.Metadata {
86+
props.Add(k, v)
87+
}
88+
rule.WithProperties(props.Properties)
89+
}
90+
}
91+
92+
// addResult adds a result to the SARIF run
93+
func addResult(run *sarif.Run, result Result, namespace, ruleType, level, fileName string, indices map[string]int) {
94+
ruleID := getRuleID(namespace, ruleType)
95+
idx, ok := indices[ruleID]
96+
if !ok {
97+
idx = addRuleIndex(run, ruleID, result, indices)
98+
}
99+
100+
run.CreateResultForRule(ruleID).
101+
WithRuleIndex(idx).
102+
WithLevel(level).
103+
WithMessage(sarif.NewTextMessage(result.Message)).
104+
AddLocation(
105+
sarif.NewLocationWithPhysicalLocation(
106+
sarif.NewPhysicalLocation().
107+
WithArtifactLocation(
108+
sarif.NewSimpleArtifactLocation(filepath.ToSlash(fileName)),
109+
),
110+
),
111+
)
112+
}
113+
114+
// Output outputs the results in SARIF format.
115+
func (s *SARIF) Output(results []CheckResult) error {
116+
report, err := sarif.New(sarifVersion)
117+
if err != nil {
118+
return fmt.Errorf("create sarif report: %w", err)
119+
}
120+
121+
run := sarif.NewRunWithInformationURI(toolName, toolURI)
122+
indices := make(map[string]int)
123+
124+
for _, result := range results {
125+
// Process failures
126+
for _, failure := range result.Failures {
127+
addResult(run, failure, result.Namespace, "deny", "error", result.FileName, indices)
128+
}
129+
130+
// Process warnings
131+
for _, warning := range result.Warnings {
132+
addResult(run, warning, result.Namespace, "warn", "warning", result.FileName, indices)
133+
}
134+
135+
// Process exceptions (treated as successes)
136+
hasSuccesses := result.Successes > 0
137+
for _, exception := range result.Exceptions {
138+
addResult(run, exception, result.Namespace, "allow", "note", result.FileName, indices)
139+
hasSuccesses = true
140+
}
141+
142+
// Don't add success/skip results if there are failures or warnings
143+
hasErrors := len(result.Failures) > 0 || len(result.Warnings) > 0
144+
if hasErrors {
145+
continue
146+
}
147+
148+
// Add success/exception results if there are no failures or warnings
149+
if hasSuccesses {
150+
statusResult := Result{
151+
Message: successDesc,
152+
Metadata: map[string]interface{}{
153+
"description": successDesc,
154+
},
155+
}
156+
addResult(run, statusResult, result.Namespace, "success", "none", result.FileName, indices)
157+
} else {
158+
statusResult := Result{
159+
Message: skippedDesc,
160+
Metadata: map[string]interface{}{
161+
"description": skippedDesc,
162+
},
163+
}
164+
addResult(run, statusResult, result.Namespace, "skip", "none", result.FileName, indices)
165+
}
166+
}
167+
168+
// Add run metadata
169+
exitCode := 0
170+
exitDesc := exitNoViolations
171+
if hasFailures(results) {
172+
exitCode = 1
173+
exitDesc = exitViolations
174+
} else if hasWarnings(results) {
175+
exitDesc = exitWarnings
176+
}
177+
178+
successful := true
179+
invocation := sarif.NewInvocation()
180+
invocation.ExecutionSuccessful = &successful
181+
invocation.ExitCode = &exitCode
182+
invocation.ExitCodeDescription = &exitDesc
183+
184+
run.Invocations = []*sarif.Invocation{invocation}
185+
186+
// Add the run to the report
187+
report.AddRun(run)
188+
189+
// Write the report
190+
return report.Write(s.writer)
191+
}
192+
193+
// Report is not supported in SARIF output
194+
func (s *SARIF) Report(_ []*tester.Result, _ string) error {
195+
return fmt.Errorf("report is not supported in SARIF output")
196+
}
197+
198+
// hasFailures returns true if any of the results contain failures
199+
func hasFailures(results []CheckResult) bool {
200+
return slices.ContainsFunc(results, func(r CheckResult) bool {
201+
return len(r.Failures) > 0
202+
})
203+
}
204+
205+
// hasWarnings returns true if any of the results contain warnings
206+
func hasWarnings(results []CheckResult) bool {
207+
return slices.ContainsFunc(results, func(r CheckResult) bool {
208+
return len(r.Warnings) > 0
209+
})
210+
}

0 commit comments

Comments
 (0)