Skip to content

Provide -filter and -override with command line generator #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,9 @@ func main() {
```

# Generate support for third-party modules
This is an example that takes the path of a third-party module source code to generate support for it. Assume the source code path of that module is `./src`. You can call `go run cmd/generate/main.go ./src`. The stdout will be like
This is a simple example that takes the path of a third-party module source code to generate support for it. For detailed usage of the tool, please run
`go run ./cmd/generate/ --help`
Assuming the source code path of that module is `./src`, you can call `go run ./cmd/generate/ --src-path=./src`. The output will be similar to:

```go
/**
Expand Down
99 changes: 99 additions & 0 deletions cmd/generate/cmd_util.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
/**
* Copyright (c) F5, Inc.
*
* This source code is licensed under the Apache License, Version 2.0 license found in the
* LICENSE file in the root directory of this source tree.
*/

package main

import (
"errors"
"fmt"
"strings"

"github.com/nginxinc/nginx-go-crossplane/internal/generator"
)

type filterFlag struct {
filter map[string]struct{}
}

func (f *filterFlag) Set(value string) error {
if f.filter == nil {
f.filter = make(map[string]struct{})
}
f.filter[value] = struct{}{}
return nil
}

func (f *filterFlag) String() string {
return fmt.Sprintf("%v", *f)
}

type overrideItem struct {
directive string
masks []generator.Mask
}

func (item *overrideItem) UnmarshalText(text []byte) error {
rawOverride := string(text)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The bytes package may offer all of the functions from the strings package that you may be able to use to avoid the (very small) cost of casting []byte to string

Copy link
Contributor Author

@dengsh12 dengsh12 Jul 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, really surprised to see bytes provide such strong capabilities. Thanks for telling me this. But for here I think the cost is acceptable? Transfer it into string seems more make sense(since override is a string originally).


// rawStr should follow the format: directive:bitmask00|bitmask01|...,bitmask10|bitmask11|...
directive, definition, found := strings.Cut(rawOverride, ":")
if !found {
return errors.New("colon not found")
}
directive = strings.TrimSpace(directive)

item.directive = directive
if directive == "" {
return errors.New("directive name is empty")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include the override to communicate which one is being rejected?

return fmt.Errorf("directive name is empty: override=%s", rawOverride)

I think I would do this for most errors where it would be helpful to see the specific input being rejected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We moved it to Set to simplify the errors in item.UnmarshalText
#106 (comment)

func (ov *override) Set(value string) error {
	if *ov == nil {
		*ov = override{}
	}
	var item overrideItem
	err := item.UnmarshalText([]byte(value))
	if err != nil {
		return fmt.Errorf("invalid override %s:%w", value, err)
	}

	(*ov)[item.directive] = item.masks
	return nil
}

}

definition = strings.TrimSpace(definition)
if definition == "" {
return errors.New("directive definition is empty")
}

for _, varNamesStr := range strings.Split(definition, ",") {
varNamesList := strings.Split(varNamesStr, "|")
varNamesNum := len(varNamesList)
directiveMask := make(generator.Mask, varNamesNum)

for idx, varName := range varNamesList {
trimmedName := strings.TrimSpace(varName)
if trimmedName == "" {
return errors.New("one directive bitmask is empty, check if there are unnecessary |")
}

directiveMask[idx] = trimmedName
}
item.masks = append(item.masks, directiveMask)
}

return nil
}

type override map[string][]generator.Mask

func (ov *override) String() string {
if ov == nil {
return "nil"
}
return fmt.Sprintf("%v", *ov)
}

func (ov *override) Set(value string) error {
if *ov == nil {
*ov = override{}
}
Comment on lines +88 to +90
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: These patterns always give me pause. I think calling a method on a nil value is generally a programming error where a panic is warranted, but there are reasons to do this. I'm not sure that it makes sense here though. I think you want the caller to initialize the value.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went in this way because most of the situations user doesn't provide a -override, lazy initialization can avoid unnecessary allocation(and Set is not run in parallel so lazy initialization is safe here).

var item overrideItem
err := item.UnmarshalText([]byte(value))
if err != nil {
return fmt.Errorf("invalid override %s:%w", value, err)
}

(*ov)[item.directive] = item.masks
return nil
}
95 changes: 95 additions & 0 deletions cmd/generate/cmd_util_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
/**
* Copyright (c) F5, Inc.
*
* This source code is licensed under the Apache License, Version 2.0 license found in the
* LICENSE file in the root directory of this source tree.
*/

package main

import (
"testing"

"github.com/nginxinc/nginx-go-crossplane/internal/generator"
"github.com/stretchr/testify/require"
)

//nolint:funlen
func TestOverrideParser(t *testing.T) {
t.Parallel()
tests := []struct {
name string
input string
expected overrideItem
wantErr bool
}{
{
Comment on lines +20 to +26
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there's another idiom for table-driven tests that uses a map to specify the name of the test case, I think that reads a little better:

	testcases := map[string]struct{
		input    string
		expected overrideItem
		wantErr  bool
	}{
		"normalFormat_pass": {
			input: "location:ngxHTTPMainConf|ngxConfTake12,ngxStreamMainConf",
			expected: overrideItem{
				directive: "location",
				masks: []generator.Mask{
					{"ngxHTTPMainConf", "ngxConfTake12"},
					{"ngxStreamMainConf"},
				},
			},
		},
		// <other test cases omitted for brevity>
	}
	for name, tc := range testcases{
		tc := tc
		t.Run(name, func(t *testing.T) {
			t.Parallel()
			
			// <test omitted for brevity>
		})
	}}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, this one is better and align with other ut in crossplane. Applied.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ryepup Many crossplane tests use a slice, but I agree that a map is better. You start to see hints of this when it becomes necessary to add a "name" or other key to the testcase struct.

@dengsh12 A more important benefit to using a map vs slice is that Go randomizes the order of keys when ranging over the map. I believe-- I do not have documentation that states this as fact on hand at the moment-- this is to dissuade anyone from getting used to maps being ordered in the event the internal details are changed. You can generally use this to your benefit as it will randomize the order the tests are run which can help to detect any unintentional shared memory that leaks through tests in the form of test flake.

name: "normalFormat_pass",
input: "location:ngxHTTPMainConf|ngxConfTake12,ngxStreamMainConf",

expected: overrideItem{
directive: "location",
masks: []generator.Mask{
{"ngxHTTPMainConf", "ngxConfTake12"},
{"ngxStreamMainConf"},
},
},
wantErr: false,
},
{
name: "withSpaces_pass",
input: "hash:ngxHTTPUpsConf | ngxConfTake12, ngxStreamUpsConf | ngxConfTake12",
expected: overrideItem{
directive: "hash",
masks: []generator.Mask{
{"ngxHTTPUpsConf", "ngxConfTake12"},
{"ngxStreamUpsConf", "ngxConfTake12"},
},
},
wantErr: false,
},
{
name: "withoutColon_fail",
input: "hashngxHTTPUpsConf | ngxConfTake12,ngxStreamUpsConf | ngxConfTake12",
wantErr: true,
},
{
name: "colonLeftsideEmpty_fail",
input: " :ngxHTTPUpsConf | ngxConfTake12,ngxStreamUpsConf | ngxConfTake12",
wantErr: true,
},
{
name: "colonRightsideEmpty_fail",
input: "hash: ",
wantErr: true,
},
{
name: "emptyBitmask_fail",
input: "hash: ngxHTTPUpsConf| ",
wantErr: true,
},
}

for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
var got overrideItem
err := got.UnmarshalText([]byte(tc.input))

if tc.wantErr {
require.Error(t, err)
} else {
require.NoError(t, err)
}

// If the testcase wants an error and there is an error, skip the output file validation.
// Output makes no sense when there is an error.
if err != nil {
return
}

require.Equal(t, tc.expected, got)
})
}
}
16 changes: 14 additions & 2 deletions cmd/generate/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,22 @@ import (
func main() {
var (
sourceCodePath = flag.String("src-path", "",
"the path of source code your want to generate support from, it can be either a file or a directory. (required)")
"The path of source code your want to generate support from, it can be either a file or a directory. (required)")
filterflags filterFlag
directiveOverride override
)
flag.Var(&filterflags, "filter",
"A list of strings specifying the directives to exclude from the output. "+
"An example is: -filter directive1 -filter directive2...(optional)")
flag.Var(&directiveOverride, "override",
"A list of strings, used to override the output. "+
"It should follow the format:{directive:bitmask00|bitmask01...,bitmask10|bitmask11...}"+"\n"+
"An example is -override=log_format:ngxHTTPMainConf|ngxConf2More,ngxStreamMainConf|ngxConf2More"+"\n"+
`To use | and , in command line, you may need to enclose your input in quotes, i.e. -override="directive:mask1,mask2,...". (optional)`)

flag.Parse()
err := generator.Generate(*sourceCodePath, os.Stdout)

err := generator.Generate(*sourceCodePath, os.Stdout, filterflags.filter, directiveOverride)
if err != nil {
log.Fatal(err)
}
Expand Down
4 changes: 2 additions & 2 deletions internal/generator/generator.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ import (
// extract all the directives definitions from the .c and .cpp files in
// sourcePath and its subdirectories, then output the corresponding directive
// masks map named "directives" and matchFunc named "Match" via writer.
func Generate(sourcePath string, writer io.Writer) error {
return genFromSrcCode(sourcePath, "directives", "Match", writer)
func Generate(sourcePath string, writer io.Writer, filter map[string]struct{}, override map[string][]Mask) error {
return genFromSrcCode(sourcePath, "directives", "Match", writer, filter, override)
}
33 changes: 25 additions & 8 deletions internal/generator/generator_util.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,14 @@ import (
"strings"
)

// A mask is a list of string, includes several variable names,
// A Mask is a list of string, includes several variable names,
// which specify a behavior of a directive.
// An example is []string{"ngxHTTPMainConf", "ngxConfFlag",}.
// A directive can have several masks.
type mask []string
type Mask []string

type supportFileTmplStruct struct {
Directive2Masks map[string][]mask
Directive2Masks map[string][]Mask
MapVariableName string
MatchFnName string
}
Expand Down Expand Up @@ -99,8 +99,8 @@ var ngxVarNameToGo = map[string]string{
}

//nolint:nonamedreturns
func masksFromFile(path string) (directive2Masks map[string][]mask, err error) {
directive2Masks = make(map[string][]mask, 0)
func masksFromFile(path string) (directive2Masks map[string][]Mask, err error) {
directive2Masks = make(map[string][]Mask, 0)
byteContent, err := os.ReadFile(path)
if err != nil {
return nil, err
Expand Down Expand Up @@ -143,8 +143,8 @@ func masksFromFile(path string) (directive2Masks map[string][]mask, err error) {
}

//nolint:nonamedreturns
func getMasksFromPath(path string) (directive2Masks map[string][]mask, err error) {
directive2Masks = make(map[string][]mask, 0)
func getMasksFromPath(path string) (directive2Masks map[string][]Mask, err error) {
directive2Masks = make(map[string][]Mask, 0)

err = filepath.WalkDir(path, func(path string, d fs.DirEntry, err error) error {
if err != nil {
Expand Down Expand Up @@ -184,12 +184,29 @@ func getMasksFromPath(path string) (directive2Masks map[string][]mask, err error
return directive2Masks, nil
}

func genFromSrcCode(codePath string, mapVariableName string, matchFnName string, writer io.Writer) error {
func genFromSrcCode(codePath string, mapVariableName string, matchFnName string, writer io.Writer,
filter map[string]struct{}, override map[string][]Mask) error {
directive2Masks, err := getMasksFromPath(codePath)
if err != nil {
return err
}

if len(filter) > 0 {
for d := range directive2Masks {
if _, found := filter[d]; found {
delete(directive2Masks, d)
}
}
}

if override != nil {
for d := range directive2Masks {
if newMasks, found := override[d]; found {
directive2Masks[d] = newMasks
}
}
}

err = supportFileTmpl.Execute(writer, supportFileTmplStruct{
Directive2Masks: directive2Masks,
MapVariableName: mapVariableName,
Expand Down
Loading
Loading