Update FOFA API Request、Configurable SDK Integration For FOFA #687
Conversation
According to the latest documentation on the FOFA official website, update the API request construction,and convert some const variables used by FOFA into regular variables to allow customizable configuration when used as an SDK.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThe changes update the FOFA agent to modify its API request signature, replacing the email parameter with a boolean Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Uncover
participant FOFA Agent
participant FOFA API
User->>Uncover: Set Options (Full: true/false)
Uncover->>FOFA Agent: Create FofaRequest (Full: true/false)
FOFA Agent->>FOFA API: Send API request (includes &full=true/false)
FOFA API-->>FOFA Agent: Return response
FOFA Agent->>Uncover: Return results or raw response on decode error
Poem
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
sources/agent/fofa/fofa.go(5 hunks)uncover.go(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
sources/agent/fofa/fofa.go (2)
sources/keys.go (1)
Keys(3-22)sources/result.go (1)
Result(8-17)
🔇 Additional comments (4)
uncover.go (1)
44-44: LGTM! Clean addition of configurable FOFA option.The
Fullfield addition is well-documented and aligns with the PR objectives for making FOFA configurable when used as an SDK.sources/agent/fofa/fofa.go (3)
7-8: LGTM! Necessary imports for enhanced functionality.The added imports support the improved error handling and logging features introduced in this update.
18-18: LGTM! URL format updated to align with latest FOFA API.The URL format change removes the email parameter and adds the
fullboolean parameter, which aligns with the PR objectives to update the FOFA API request construction.
21-23: LGTM! Constants converted to variables for SDK configurability.Converting
SizeandFieldsfrom constants to variables enables customizable configuration when used as an SDK, which matches the PR objectives.
| func (agent *Agent) queryURL(session *sources.Session, URL string, fofaRequest *FofaRequest) (*http.Response, error) { | ||
| base64Query := base64.StdEncoding.EncodeToString([]byte(fofaRequest.Query)) | ||
| fofaURL := fmt.Sprintf(URL, session.Keys.FofaEmail, session.Keys.FofaKey, base64Query, Fields, fofaRequest.Page, fofaRequest.Size) | ||
| fofaURL := fmt.Sprintf(URL, session.Keys.FofaKey, base64Query, Fields, fofaRequest.Page, fofaRequest.Size, fofaRequest.Full) |
There was a problem hiding this comment.
Missing integration with Options.Full field.
The URL construction uses fofaRequest.Full but this field is never populated from the Options.Full field added in uncover.go. The integration between the two is missing.
The FOFA agent needs to access the Options.Full value. Consider updating the agent interface to pass options or modify the session to include this configuration:
func (agent *Agent) Query(session *sources.Session, query *sources.Query) (chan sources.Result, error) {
// ... existing code ...
fofaRequest := &FofaRequest{
Query: query.Query,
Fields: Fields,
Size: Size,
Page: page,
+ Full: session.Options.Full, // Assuming session has access to options
}🤖 Prompt for AI Agents
In sources/agent/fofa/fofa.go at line 68, the URL uses fofaRequest.Full which is
not set from the Options.Full field introduced in uncover.go. To fix this,
update the FOFA agent interface or session initialization to pass the
Options.Full value into fofaRequest.Full before constructing the URL, ensuring
the full query option is correctly integrated and used in the request.
| result := sources.Result{Source: agent.Name()} | ||
| defer func(Body io.ReadCloser) { | ||
| bodyCloseErr := Body.Close() | ||
| if bodyCloseErr != nil { | ||
| gologger.Info().Msgf("response body close error : %v", bodyCloseErr) | ||
| } | ||
| }(resp.Body) | ||
| respBodyData, err := io.ReadAll(resp.Body) | ||
| if err != nil { | ||
| return nil | ||
| } | ||
| raw, _ := json.Marshal(respBodyData) | ||
| result.Raw = raw | ||
| results <- result | ||
| return nil |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Improve error handling resource management.
The enhanced error handling approach is good as it provides raw response data for debugging. However, the defer statement for closing the response body should be placed immediately after checking for request errors to ensure proper resource cleanup.
Consider restructuring the error handling:
func (agent *Agent) query(URL string, session *sources.Session, fofaRequest *FofaRequest, results chan sources.Result) *FofaResponse {
resp, err := agent.queryURL(session, URL, fofaRequest)
if err != nil {
results <- sources.Result{Source: agent.Name(), Error: err}
return nil
}
+ defer func(Body io.ReadCloser) {
+ if bodyCloseErr := Body.Close(); bodyCloseErr != nil {
+ gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
+ }
+ }(resp.Body)
fofaResponse := &FofaResponse{}
if err := json.NewDecoder(resp.Body).Decode(fofaResponse); err != nil {
result := sources.Result{Source: agent.Name()}
- defer func(Body io.ReadCloser) {
- bodyCloseErr := Body.Close()
- if bodyCloseErr != nil {
- gologger.Info().Msgf("response body close error : %v", bodyCloseErr)
- }
- }(resp.Body)
respBodyData, err := io.ReadAll(resp.Body)
if err != nil {
return nil
}
raw, _ := json.Marshal(respBodyData)
result.Raw = raw
results <- result
return nil
}
// ... rest of the method
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| result := sources.Result{Source: agent.Name()} | |
| defer func(Body io.ReadCloser) { | |
| bodyCloseErr := Body.Close() | |
| if bodyCloseErr != nil { | |
| gologger.Info().Msgf("response body close error : %v", bodyCloseErr) | |
| } | |
| }(resp.Body) | |
| respBodyData, err := io.ReadAll(resp.Body) | |
| if err != nil { | |
| return nil | |
| } | |
| raw, _ := json.Marshal(respBodyData) | |
| result.Raw = raw | |
| results <- result | |
| return nil | |
| func (agent *Agent) query(URL string, session *sources.Session, fofaRequest *FofaRequest, results chan sources.Result) *FofaResponse { | |
| resp, err := agent.queryURL(session, URL, fofaRequest) | |
| if err != nil { | |
| results <- sources.Result{Source: agent.Name(), Error: err} | |
| return nil | |
| } | |
| defer func(Body io.ReadCloser) { | |
| if bodyCloseErr := Body.Close(); bodyCloseErr != nil { | |
| gologger.Info().Msgf("response body close error : %v", bodyCloseErr) | |
| } | |
| }(resp.Body) | |
| fofaResponse := &FofaResponse{} | |
| if err := json.NewDecoder(resp.Body).Decode(fofaResponse); err != nil { | |
| result := sources.Result{Source: agent.Name()} | |
| respBodyData, err := io.ReadAll(resp.Body) | |
| if err != nil { | |
| return nil | |
| } | |
| raw, _ := json.Marshal(respBodyData) | |
| result.Raw = raw | |
| results <- result | |
| return nil | |
| } | |
| // ... rest of the method | |
| } |
🤖 Prompt for AI Agents
In sources/agent/fofa/fofa.go around lines 86 to 100, move the defer statement
that closes resp.Body to immediately after confirming that the HTTP request did
not return an error and resp is not nil. This ensures the response body is
always closed properly even if subsequent operations fail. First check for
request errors, then defer closing resp.Body, and only then proceed to read and
process the response body.
| Page int | ||
| Size int | ||
| Full string | ||
| Full bool |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Verify FofaRequest.Full field integration.
The Full field is properly defined but ensure it's being populated from the Options.Full field to complete the integration.
Let me check if there are any other places where the Full field should be set:
🏁 Script executed:
#!/bin/bash
# Description: Search for FofaRequest struct usage to verify Full field population
# Search for FofaRequest struct instantiation
ast-grep --pattern 'FofaRequest{$$$}'
# Search for Full field usage
rg -A 3 -B 3 "\.Full"Length of output: 978
🏁 Script executed:
#!/bin/bash
# Retrieve context around the FofaRequest instantiation
rg -n "fofaRequest := &FofaRequest" -A5 -B5 sources/agent/fofa/fofa.goLength of output: 293
Populate the Full field in FofaRequest
The Full flag is consumed when constructing the URL (see fofaRequest.Full in the fmt.Sprintf), but it isn’t being set on the struct. Add it to the literal at lines 44–49 in sources/agent/fofa/fofa.go:
• File: sources/agent/fofa/fofa.go
Location: around line 44
Suggested diff:
fofaRequest := &FofaRequest{
Query: query.Query,
Fields: Fields,
Size: Size,
Page: page,
+ Full: options.Full,
}🤖 Prompt for AI Agents
In sources/agent/fofa/fofa.go around line 44, the Full field of the FofaRequest
struct is not being set when the struct is instantiated, but it is used later
when constructing the URL. Update the struct literal at lines 44–49 to include
the Full field and assign it the appropriate boolean value so that the URL
construction uses the correct flag.
dogancanbakir
requested review from
Mzack9999 and
dwisiswant0
and removed request for
dogancanbakir
|
@CodeRabbit create an issue for this pull request |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain@Mzack9999: I'll create an issue to track this pull request. 🏁 Script executed: Length of output: 54 @Mzack9999: I've created issue #700 to track this pull request: #700 The issue has been assigned to you and includes a summary of the changes proposed in PR #687. |
3 participants
According to the latest documentation on the FOFA official website, update the API request construction,and convert some const variables used by FOFA into regular variables to allow customizable configuration when used as an SDK.
Summary by CodeRabbit
New Features
Improvements