Struggling to use nuclei as embedded SDK

[mikerott]

I'm trying to use Nuclei as embedded SDK, for use in an AWS lambda. Basically, I expected my output.Writer implementation (which is passed to protocols.ExecutorOptions.Output) to have its WriteFailure function called when the template execution succeeds to reach the target, but fails to match. Is that not how it works?

I've made a repo with the reproduction scenario, with as minimal usage of nuclei as SDK as possible. I could really use some help! We're using nuclei with custom templates to perform detections on hundreds of thousands of ip addresses and hostnames daily. It's become a critical tool!

Repo with README here: https://github.com/mikerott/nuclei-wrapped

[mikerott]

Looks like this line is preventing the call to my WriteFailure function due to false matcherStatus: https://github.com/projectdiscovery/nuclei/blob/v3.1.3/pkg/tmplexec/exec.go#L107

In older version, no "matcher status" check is performed, so the WriteFailure function is called: https://github.com/projectdiscovery/nuclei/blob/v2.7.5/v2/pkg/protocols/common/executer/executer.go#L83

I'd like to differentiate between the following conditions:

• target was searched and there is a match (this works fine in latest version; my Write function gets called)
• target was searched and there is no match (I expected WriteFailure to be called, which does happen in older versions, but not in 3.1.3)
• target was unreachable (in older versions if I get no call to either Write or WriteFailure, I assumed it means the target was not reachable or timed out - which would also be a nice thing to differentiate between)

[mikerott]

Ooooooh I got it. I was right there in the right spot with my last comment. I just needed to map in my brain that the matcherStatus was an option I needed to add!

So I've added defaultOpts.MatcherStatus = true to my test and it now calls my WriteFailure function.

All that said, may I ask that a test similar to the one I have in the linked repo be added, and referred as the "how to use nuclei as embedded SDK" example in the docs? I can make a PR if you'd prefer a community contribution.

[ehsandeep]

@mikerott, while Nuclei initially focused on CLI usage, we're expanding to include more SDK documentation. Recent updates offer SDK usage examples, but we're aware there's room for improvement. If you can contribute additional documentation that could benefit others, we'd greatly appreciate it. You can submit your documents here: Nuclei Documentation GitHub

[mikerott]

Thanks for the reply. That page is a good reference. I made a fork and tried some minor surgery to add something like this to lib/config.go:

// WithCatalog sets the Catalog implementation to use for loading templates.
// The catalog implementation will receive the excludedTemplates and includedTemplates
// as arguments in the GetTemplatesPath function call.
func WithCatalog(catalog catalog.Catalog, includedTemplates []string, excludedTemplates []string) NucleiSDKOptions {
	return func(e *NucleiEngine) error {
		e.catalog = catalog
		e.executerOpts.Options.IncludeTemplates = includedTemplates
		e.executerOpts.Options.ExcludedTemplates = excludedTemplates
		return nil
	}
}

Then a test in sdk_test.go to use it:

func TestSimpleNucleiWithCustomCatalog(t *testing.T) {
	myCatalog := myCatalog{}
	ne, err := nuclei.NewNucleiEngine(
		nuclei.WithTemplateFilters(nuclei.TemplateFilters{ProtocolTypes: "dns"}),
		nuclei.WithCatalog(&myCatalog, []string{"myIncludedTemplate"}, []string{"myExcludedTemplate"}),
	)
	require.Nil(t, err)
	ne.LoadTargets([]string{"scanme.sh"}, false) // probe non http/https target is set to false here
	// when callback is nil it nuclei will print JSON output to stdout
	err = ne.ExecuteWithCallback(nil)
	require.Nil(t, err)
	require.True(t, myCatalog.wasCalledTracker)
	defer ne.Close()
}

You can guess what myCatalog looks like - it just satisfies the catalog.Catalog interface. I set my debugger to watch when it's called, with what arguments, and through which call stacks. Turns out my idea would take major surgery, as the nuclei codebase is very much a filesystem-focused codebase, as you said. It has lots of assumptions about filesystem paths. Unfortunately I don't have time to contribute such a major change, but ultimately I'd like a really simple catalog interface signature that is less filesystem focused, then all of the filesystem accessor work would be encapsulated within disk.DiskCatalog while all the S3 accessor work would be encapsulated within aws.Catalog. Ideally, then, all template filtering would happen in the catalog implementation.