Skip to content

Update FOFA API Request、Configurable SDK Integration For FOFA #687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: dev
Choose a base branch
from
Open

Update FOFA API Request、Configurable SDK Integration For FOFA #687

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8c84518
Update FOFA API Request、Configurable SDK Integration For FOFA
TonyD0g Jul 8, 2025
88c89fb
bug fix
TonyD0g Jul 8, 2025
ea7e0bf
[bug fix] Update FOFA API Request、Configurable SDK Integration For FOFA
TonyD0g Jul 8, 2025
c9d10fe
Update agent.go
TonyD0g Jul 10, 2025
d1fdd91
Update fofa.go
TonyD0g Jul 10, 2025
bbdf9d8
Update hunter.go
TonyD0g Jul 10, 2025
d5ef418
Update request.go
TonyD0g Jul 10, 2025
2b02ad5
Update uncover.go
TonyD0g Jul 10, 2025
f21170c
fix + lint
Mzack9999 Oct 6, 2025
4629ac2
osx bump
Mzack9999 Oct 6, 2025
3695413
go bump
Mzack9999 Oct 6, 2025
3acea57
Merge branch 'dev' into pr/687
Mzack9999 Oct 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 23 additions & 6 deletions sources/agent/fofa/fofa.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import (
"encoding/base64"
"encoding/json"
"fmt"
"github.com/projectdiscovery/gologger"
"io"
"net/http"
"strconv"

Expand All @@ -13,11 +15,13 @@ import (
)

const (
URL = "https://fofa.info/api/v1/search/all?email=%s&key=%s&qbase64=%s&fields=%s&page=%d&size=%d"
Fields = "ip,port,host"
Size = 100
URL = "https://fofa.info/api/v1/search/all?key=%s&qbase64=%s&fields=%s&page=%d&size=%d&full=%t"
)

var Size = 100

var Fields = "ip,port,host"

type Agent struct{}

func (agent *Agent) Name() string {
Expand Down Expand Up @@ -61,7 +65,7 @@ func (agent *Agent) Query(session *sources.Session, query *sources.Query) (chan

func (agent *Agent) queryURL(session *sources.Session, URL string, fofaRequest *FofaRequest) (*http.Response, error) {
base64Query := base64.StdEncoding.EncodeToString([]byte(fofaRequest.Query))
fofaURL := fmt.Sprintf(URL, session.Keys.FofaEmail, session.Keys.FofaKey, base64Query, Fields, fofaRequest.Page, fofaRequest.Size)
fofaURL := fmt.Sprintf(URL, session.Keys.FofaKey, base64Query, Fields, fofaRequest.Page, fofaRequest.Size, fofaRequest.Full)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Missing integration with Options.Full field.

The URL construction uses fofaRequest.Full but this field is never populated from the Options.Full field added in uncover.go. The integration between the two is missing.

The FOFA agent needs to access the Options.Full value. Consider updating the agent interface to pass options or modify the session to include this configuration:

func (agent *Agent) Query(session *sources.Session, query *sources.Query) (chan sources.Result, error) {
	// ... existing code ...
	
	fofaRequest := &FofaRequest{
		Query:  query.Query,
		Fields: Fields,
		Size:   Size,
		Page:   page,
+		Full:   session.Options.Full, // Assuming session has access to options
	}
🤖 Prompt for AI Agents 
In sources/agent/fofa/fofa.go at line 68, the URL uses fofaRequest.Full which is
not set from the Options.Full field introduced in uncover.go. To fix this,
update the FOFA agent interface or session initialization to pass the
Options.Full value into fofaRequest.Full before constructing the URL, ensuring
the full query option is correctly integrated and used in the request.

request, err := sources.NewHTTPRequest(http.MethodGet, fofaURL, nil)
if err != nil {
return nil, err
Expand All @@ -79,7 +83,20 @@ func (agent *Agent) query(URL string, session *sources.Session, fofaRequest *Fof
fofaResponse := &FofaResponse{}

if err := json.NewDecoder(resp.Body).Decode(fofaResponse); err != nil {
results <- sources.Result{Source: agent.Name(), Error: err}
result := sources.Result{Source: agent.Name()}
defer func(Body io.ReadCloser) {
bodyCloseErr := Body.Close()
if bodyCloseErr != nil {
gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
}
}(resp.Body)
respBodyData, err := io.ReadAll(resp.Body)
if err != nil {
return nil
}
raw, _ := json.Marshal(respBodyData)
result.Raw = raw
results <- result
return nil
Comment on lines 93 to 102
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improve error handling resource management.

The enhanced error handling approach is good as it provides raw response data for debugging. However, the defer statement for closing the response body should be placed immediately after checking for request errors to ensure proper resource cleanup.

Consider restructuring the error handling:

func (agent *Agent) query(URL string, session *sources.Session, fofaRequest *FofaRequest, results chan sources.Result) *FofaResponse {
	resp, err := agent.queryURL(session, URL, fofaRequest)
	if err != nil {
		results <- sources.Result{Source: agent.Name(), Error: err}
		return nil
	}
+	defer func(Body io.ReadCloser) {
+		if bodyCloseErr := Body.Close(); bodyCloseErr != nil {
+			gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
+		}
+	}(resp.Body)
	
	fofaResponse := &FofaResponse{}
	if err := json.NewDecoder(resp.Body).Decode(fofaResponse); err != nil {
		result := sources.Result{Source: agent.Name()}
-		defer func(Body io.ReadCloser) {
-			bodyCloseErr := Body.Close()
-			if bodyCloseErr != nil {
-				gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
-			}
-		}(resp.Body)
		respBodyData, err := io.ReadAll(resp.Body)
		if err != nil {
			return nil
		}
		raw, _ := json.Marshal(respBodyData)
		result.Raw = raw
		results <- result
		return nil
	}
	// ... rest of the method
}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
result := sources.Result{Source: agent.Name()}
defer func(Body io.ReadCloser) {
bodyCloseErr := Body.Close()
if bodyCloseErr != nil {
gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
}
}(resp.Body)
respBodyData, err := io.ReadAll(resp.Body)
if err != nil {
return nil
}
raw, _ := json.Marshal(respBodyData)
result.Raw = raw
results <- result
return nil
func (agent *Agent) query(URL string, session *sources.Session, fofaRequest *FofaRequest, results chan sources.Result) *FofaResponse {
resp, err := agent.queryURL(session, URL, fofaRequest)
if err != nil {
results <- sources.Result{Source: agent.Name(), Error: err}
return nil
}
defer func(Body io.ReadCloser) {
if bodyCloseErr := Body.Close(); bodyCloseErr != nil {
gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
}
}(resp.Body)
fofaResponse := &FofaResponse{}
if err := json.NewDecoder(resp.Body).Decode(fofaResponse); err != nil {
result := sources.Result{Source: agent.Name()}
respBodyData, err := io.ReadAll(resp.Body)
if err != nil {
return nil
}
raw, _ := json.Marshal(respBodyData)
result.Raw = raw
results <- result
return nil
}
// ... rest of the method
}
🤖 Prompt for AI Agents 
In sources/agent/fofa/fofa.go around lines 86 to 100, move the defer statement
that closes resp.Body to immediately after confirming that the HTTP request did
not return an error and resp is not nil. This ensures the response body is
always closed properly even if subsequent operations fail. First check for
request errors, then defer closing resp.Body, and only then proceed to read and
process the response body.

}
if fofaResponse.Error {
Expand All @@ -104,5 +121,5 @@ type FofaRequest struct {
Fields string
Page int
Size int
Full string
Full bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify FofaRequest.Full field integration.

The Full field is properly defined but ensure it's being populated from the Options.Full field to complete the integration.

Let me check if there are any other places where the Full field should be set:

🏁 Script executed:

#!/bin/bash
# Description: Search for FofaRequest struct usage to verify Full field population

# Search for FofaRequest struct instantiation
ast-grep --pattern 'FofaRequest{$$$}'

# Search for Full field usage
rg -A 3 -B 3 "\.Full"

Length of output: 978

🏁 Script executed:

#!/bin/bash
# Retrieve context around the FofaRequest instantiation
rg -n "fofaRequest := &FofaRequest" -A5 -B5 sources/agent/fofa/fofa.go

Length of output: 293

Populate the Full field in FofaRequest

The Full flag is consumed when constructing the URL (see fofaRequest.Full in the fmt.Sprintf), but it isn’t being set on the struct. Add it to the literal at lines 44–49 in sources/agent/fofa/fofa.go:

• File: sources/agent/fofa/fofa.go
Location: around line 44

Suggested diff:

  fofaRequest := &FofaRequest{
      Query:  query.Query,
      Fields: Fields,
      Size:   Size,
      Page:   page,
+     Full:   options.Full,
  }
🤖 Prompt for AI Agents 
In sources/agent/fofa/fofa.go around line 44, the Full field of the FofaRequest
struct is not being set when the struct is instantiated, but it is used later
when constructing the URL. Update the struct literal at lines 44–49 to include
the Full field and assign it the appropriate boolean value so that the URL
construction uses the correct flag.

}
1 change: 1 addition & 0 deletions uncover.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ type Options struct {
RateLimit uint // default 30 req
RateLimitUnit time.Duration // default unit
Proxy string // http proxy to use with uncover
Full bool // (fofa use) default search for data within one year, specify true to search all data
}

// Service handler of all uncover Agents
Expand Down