Performance issue

[4durban]

Hello,

We have deployed the Vuls application following the client database architecture. I will try to explain myself:

In our architecture the clients connect to the Vuls-Server through HTTP sending the properly curl:

device:/tmp$ curl --max-time 900 --connect-timeout 900 -X POST -H "Content-Type: text/plain" -H "X-Vuls-OS-Family: `lsb_release -si | awk '{print tolower($1)}'`" -H "X-Vuls-OS-Release: `lsb_release -sr | awk '{print $1}'`" -H "X-Vuls-Kernel-Release: `uname -r`" -H "X-Vuls-Server-Name: `hostname`" --data-binary @packages.txt https://vuls-server.domain.com/vuls

We see this error in the vuls-server

user/vuls-server-787cb77b88-r4l95[vuls-server]: time="Jun 26 13:27:48" level=debug msg="HTTP Request to http://vuls-gost:1325/ubuntu/2204/pkgs/libfastjson/fixed-cves" 
user/vuls-server-787cb77b88-r4l95[vuls-server]: time="Jun 26 13:27:48" level=warning msg="Failed to HTTP GET. retrying in 512.518063ms seconds. err: HTTP GET error, url: http://vuls-gost:1325/ubuntu/2204/pkgs/firefox/fixed-cves, resp: <nil>, err: [Get \"http://vuls-gost:1325/ubuntu/2204/pkgs/firefox/fixed-cves\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        /go/src/github.com/future-architect/vuls/gost/util.go:164" 
user/vuls-server-787cb77b88-r4l95[vuls-server]: time="Jun 26 13:27:48" level=debug msg="HTTP Request to http://vuls-gost:1325/ubuntu/2204/pkgs/linux-signed-hwe-5.19/fixed-cves" 

And after 3 appearings of that message (MAX_RETRIES I guess) we see this error in the vuls-server

vuls/vuls-server-76774d9b7f-lcrtt[vuls-server]: time="Jun 27 14:41:00" level=error msg="Failed to detect Pkg CVE: Failed to detect CVE with gost:\n    github.com/future-architect/vuls/detector.DetectPkgCves\n        /go/src/github.com/future-architect/vuls/detector/detector.go:228\n  - Failed to detect CVEs with gost:\n    github.com/future-architect/vuls/detector.detectPkgsCvesWithGost\n        /go/src/github.com/future-architect/vuls/detector/detector.go:477\n  - Failed to detect fixed CVEs. err:\n    github.com/future-architect/vuls/gost.Ubuntu.DetectCVEs\n        /go/src/github.com/future-architect/vuls/gost/ubuntu.go:88\n  - Failed to get fixed CVEs via HTTP. err:\n    github.com/future-architect/vuls/gost.Ubuntu.detectCVEsWithFixState\n        /go/src/github.com/future-architect/vuls/gost/ubuntu.go:112\n  - Failed to fetch Gost. err: %!w([]error=[0xc002d341b0 0xc002209ad0 0xc002d534a0 0xc0034800f0 0xc000a96060 0xc000f3a240 0xc002c03140 0xc003c19500 0xc002682600 0xc0027d7b60 0xc0027d7bc0]):\n    github.com/future-architect/vuls/gost.getCvesWithFixStateViaHTTP\n        /go/src/github.com/future-architect/vuls/gost/util.go:146" 

And if we check the curl output we see the following error:

device:/tmp$ curl --max-time 900 --connect-timeout 900 -X POST -H "Content-Type: text/plain" -H "X-Vuls-OS-Family: `lsb_release -si | awk '{print tolower($1)}'`" -H "X-Vuls-OS-Release: `lsb_release -sr | awk '{print $1}'`" -H "X-Vuls-Kernel-Release: `uname -r`" -H "X-Vuls-Server-Name: `hostname`" --data-binary @packages.txt https://${VULS_SERVER}/vuls
Failed to detect CVE with gost: Failed to detect CVEs with gost: Failed to detect unfixed CVEs. err: Failed to get fixed CVEs via HTTP. err: Failed to fetch Gost. err: %!w([]error=[0xc000aaa00

The architecture that we are following is that the vuls server is in one k8s pod and each database in different pods.

The user connects to the server with the curl in HTTP and the server connects to databases also through HTTP.

We think that it is a performance issue, we have tried increasing resources to the pods and it seems to solve the problem partially for few endpoints, but as soon as we increase to multiple endpoints (more than 3) we seem to hit that performance ceiling, and the issues start to appear again. If we only get the CVEs for one endpoint everything works fine, but the bigger the amount, the more issues appear.

Do you know what could be causing the problem? Is there a parameter that we need to fix in order to improve the performance? Something like generate child threads or something like that?

Thank you for your time!

[MaineK00n]

I am writing this on the assumption that the GET requests between Vuls and DB(In this case, gost DB) is failing due to timeout.

It may be solved by changing the gost timeout so that it can be adjusted.

vuls/gost/util.go

Line 133 in 4253550

timeout := time.After(2 * 60 * time.Second)

vuls/gost/util.go

Line 158 in 4253550

resp, body, errs = gorequest.New().Timeout(10 * time.Second).Get(url).End()

If you don't mind, could you please set the hard-coded timeout to a longer time and verify if the error does not occur?

Alternatively, it may be possible to select a DB Type that responds a little faster than the DB currently used.
What DB Type do you use?

If timeout is not the cause, a different countermeasure must be considered.

[GEownt]

For everyone with the same/similar error, the fix provided by @MaineK00n is working. We had the following errors:

time="Apr 23 07:06:30" level=warning msg="Failed to HTTP GET. retrying in 402.148508ms seconds. err: HTTP GET error, url: http://gost:1325/ubuntu/2404/pkgs/linux/fixed-cves, resp: <nil>, err: [Get \"http://gost:1325/ubuntu/2404/pkgs/linux/fixed-cves\": read tcp 10.50.1.8:39418->10.50.1.2:1325: i/o timeout]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        github.com/future-architect/vuls/gost/util.go:158" 
time="Apr 23 07:06:30" level=warning msg="Failed to HTTP GET. retrying in 700.930375ms seconds. err: HTTP GET error, url: http://gost:1325/ubuntu/2404/pkgs/linux/fixed-cves, resp: <nil>, err: [Get \"http://gost:1325/ubuntu/2404/pkgs/linux/fixed-cves\": read tcp 10.50.1.8:40106->10.50.1.2:1325: i/o timeout]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        github.com/future-architect/vuls/gost/util.go:158" 
time="Apr 23 07:06:51" level=warning msg="Failed to HTTP GET. retrying in 486.746686ms seconds. err: HTTP GET error, url: http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves, resp: <nil>, err: [Get \"http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves\": read tcp 10.50.1.8:33264->10.50.1.2:1325: i/o timeout]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        github.com/future-architect/vuls/gost/util.go:158" 
time="Apr 23 07:06:51" level=warning msg="Failed to HTTP GET. retrying in 438.144815ms seconds. err: HTTP GET error, url: http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves, resp: <nil>, err: [Get \"http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves\": read tcp 10.50.1.8:33864->10.50.1.2:1325: i/o timeout]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        github.com/future-architect/vuls/gost/util.go:158" 
time="Apr 23 07:07:01" level=warning msg="Failed to HTTP GET. retrying in 1.054661792s seconds. err: HTTP GET error, url: http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves, resp: <nil>, err: [Get \"http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves\": read tcp 10.50.1.8:40090->10.50.1.2:1325: i/o timeout]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        github.com/future-architect/vuls/gost/util.go:158" 
time="Apr 23 07:07:02" level=warning msg="Failed to HTTP GET. retrying in 631.399136ms seconds. err: HTTP GET error, url: http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves, resp: <nil>, err: [Get \"http://gost:1325/ubuntu/2404/pkgs/linux/unfixed-cves\": read tcp 10.50.1.8:40106->10.50.1.2:1325: i/o timeout]:\n    github.com/future-architect/vuls/gost.httpGet.func1\n        github.com/future-architect/vuls/gost/util.go:158" 
time="Apr 23 07:07:12" level=error msg="Failed to detect Pkg CVE:\n    github.com/future-architect/vuls/detector.Detect\n        github.com/future-architect/vuls/detector/detector.go:54\n  - Failed to detect CVE with gost:\n    github.com/future-architect/vuls/detector.DetectPkgCves\n        github.com/future-architect/vuls/detector/detector.go:341\n  - Failed to detect CVEs with gost:\n    github.com/future-architect/vuls/detector.detectPkgsCvesWithGost\n        github.com/future-architect/vuls/detector/detector.go:593\n  - Failed to detect CVEs. err:\n    github.com/future-architect/vuls/gost.Ubuntu.DetectCVEs\n        github.com/future-architect/vuls/gost/ubuntu.go:91\n  - Failed to get unfixed CVEs via HTTP. err:\n    github.com/future-architect/vuls/gost.Ubuntu.detectCVEs\n        github.com/future-architect/vuls/gost/ubuntu.go:110\n  - Failed to fetch Gost. err: %!w([]error=[0xc0046d0000 0xc0046d0060]):\n    github.com/future-architect/vuls/gost.getCvesWithFixStateViaHTTP\n        github.com/future-architect/vuls/gost/util.go:140" 

After increasing the timeouts in vuls/gost/util.go everything works as expected.

@MaineK00n Is there a possibility to get some new configuration parameters to set the timeouts in config.toml?

[MaineK00n]

@GEownt
I would like to implement it in a way that allows timeout to be used in other things besides gost as well.