Issue with POST request which contain binary files in body with Content-Type as "application/x-www-form-urlencoded"

[gullu44]

We are facing issue with POST request which contain binary files in body with Content-Type as "application/x-www-form-urlencoded"

we are using below versions

Apache :- apache-2.4.41
mod_security :- 2.9.3
CRS :- 3.0

ModSec Directive setting at Vhost level

SecRuleEngine On

SecRequestBodyAccess On
SecRequestBodyLimitAction ProcessPartial

SecRequestBodyNoFilesLimit 90000000
SecRequestBodyLimit 90000000
SecRequestBodyInMemoryLimit 5000000

Our is hosting Environment and many application send big files in the body(using curl or Client java code) with Content-Type: application/x-www-form-urlencoded. So following i have noticed

1. Sent a 83864080 bytes file with

   curl $CURL_OPTS -H "Expect: 100-continue" --data-binary @test.txt https://example.com

where $CURL_OPTS contain TLS, cert etc

2. As binary file is embedded in body, seems mod_sec apply SecRequestBodyNoFilesLimit not SecRequestBodyLimit. If we set SecRequestBodyNoFilesLimit limit anything smaller than the file limit e.g 20 MB, then only 20 MB file is being transferred to backend server ( we use mod proxy balancer) and rest body request is truncated. Is this expected behavior with content-Type: application/x-www-form-urlencoded ? Though DOS is another concern with such higher values.

3.Setting SecRequestBodyNoFilesLimit to such higherlimit adding high latency for upload request. If i set modsec in detection mode , this request just take 13-14 seconed for its completion ( end to end), but turning ON mod_sec with SecRequestBodyNoFilesLimit (90000000) taking 9-11 Minutes .

4. No improvement noticed by Increasing SecRequestBodyInMemoryLimit to 5 MB . I think this limit is applicable only for Content-Type: application/multipart/form-data

5. Debug logs show that during REQUEST_BODY phase, input filter always read data in 8192 bytes

    +++++++++++++++
   

29/Apr/2020:22:26:54.317475 +0200] [test1.example.com/sid#3680840][rid#f76e890][/endpoints/web][4] Second phase starting (dcfg 22ec680).
[29/Apr/2020:22:26:54.317492 +0200] [test1.example.com/sid#3680840][rid#f76e890][/endpoints/web][4] Input filter: Reading request body.
[29/Apr/2020:22:26:54.324645 +0200] [test1.example.com/sid#3680840][rid#f76e890][/endpoints/web][9] Input filter: Bucket type TRANSIENT contains 8192 bytes.
[29/Apr/2020:22:26:54.324673 +0200] [test1.example.com/sid#3680840][rid#f76e890][/endpoints/web][9] Input filter: Bucket type TRANSIENT contains 8192 bytes.

........Above lines are repeated 10868 time

I think its taking long time to read body and is cause of high latency. How Can we increase this 8192 limit? In our apache we are not using any mod_buffer though we have set RLimitMEM which anyway is on higher side.

[martinhsv]

Hi @gullu44 ,

For the truncated body aspect, there was a similar-sounding issue fixed post-2.9.3 in May, 2019 (#2092).

If you're using 2.9.3 from a repo, or built from the 2.9.3-tagged commit, you might want to try building using the most recent commit in the v2/master branch.

[gullu44]

Hi @martinhsv ,
Thanks for the update. Yes we already using the fixed version, though we will verify again ( MarkBlackman was part of our team and We already using the updated repo which contain @rainerjung fix), we have not faced that issue so far.
Issue i highlighted seems different, current issue is limited to POST request which contain binary files in body with Content-Type as "application/x-www-form-urlencoded" and we have to increase SecRequestBodyNoFilesLimit limit same or more than the binary file size that was send in the request. Once that higher limit is set, full request is reaching at backend server .
But now increasing SecRequestBodyNoFilesLimit limit to that high value lead to performance issue. It taking 9-11 minutes at client site to sent POST request (mod_sec in blocking mode) as compare to 13-15 second when mod_sec is in detection mode.
Please let me know if you need any more information or detail debug logs

[gullu44]

Hi @martinhsv , confirming you that we are using the #2092 patched version.

[martinhsv]

Hi @gullu44 ,

I haven't had a chance to look at this in detail, but I can offer a couple of things:

First, regarding your item 4 in your original posting, I can confirm that SecRequestBodyInMemoryLimit has no effect if the Content-Type is application/x-www-form-urlencoded.

Second, if you are looking for a short-term workaround, the only thing I can suggest off hand is to consider creating a phase:1 rule that tests for the expected url, Content-Type=application/x-www-form-urlencoded, and (probably) Content-Length header exceeding some large integer. This rule could then have "ctl:requestBodyAccess=Off" as one of its actions.

[gullu44]

Hi @martinhsv
As we are hosting environment, list of sites is too big and making our job little hard to place requestBodyAccess=Off rule. Anyway we will rethink on that again ( for sort term)
But appreciate if you can look my issue in detail.

[martinhsv]

Hi @gullu44 ,

While I have not been able to precisely reproduce your main use case (truncated 20MB sent to backend), I do believe I have identified a commit that may have broken some expected behaviour around the limits-functionality.

I am suspicious of a particular v2/master commit on Sept. 5, 2018.

I'll follow up on this further on my end, but if you wanted to test this against your problem, you could try going to a github revision prior to that date. Or, as an alternative, try experimentally going down to v2.9.2

[gullu44]

Hi @martinhsv
Ya in parallel, i will try to test this with v2.9.2.

[martinhsv]

Hi @gullu44 ,

Just to clarify one thing, the combination of your points 1 and 2 means you have these settings:

SecRuleEngine On
SecRequestBodyAccess On
SecRequestBodyLimitAction ProcessPartial
SecRequestBodyNoFilesLimit 20000000
SecRequestBodyLimit 90000000

Ordinarily I would expect this to result in an application/x-www-form-urlencoded request with body size 83864080 being denied by ModSecurity. This would be due to the default SecRule 200002 in modsecurity.conf (the rule includes the action 'deny' if REQBODY_ERROR is set ... and current v2/master code should set this).

Do you have that REQBODY_ERROR rule (200002) disabled? If not, do you have some other custom setting that might be relevant?

[gullu44]

Hi @martinhsv
I do not see any SecRule 200002 in our setup. Also we have not done much customization apart from adding additional 1-2 sanitise rule (preventing password etc to get logged) , but those does not seems relevant
As you also pointed, i also expect a blocked request (denied) if its content type is application/x-www-form-urlencoded and breaching SecRequestBodyNoFilesLimit limit

[martinhsv]

Hi @gullu44 ,

ModSecurity comes with a file called modsecurity.conf-recommended which can be renamed modsecurity.conf and used as part of your configuration. It is only a recommended starting point of course, but it does include the REQBODY_ERROR check that I mentioned.

I am able to reproduce ModSecurity (in reverse-proxy mode) sending on only the truncated body to the endpoint server. As it turns out, going back before the 2018 commit that I highlighted does not directly help. Indeed, I tried going back to Apache 2.4.18 with ModSecurity 2.9.0, and the truncation use case was unchanged.

Unfortunately, I don't have the time right now to code a proper fix for this functionality.

In the meantime ...

If denying requests whose body exceeds your limit is an acceptable outcome, then that would be the safest way to proceed. E.g. Use SecRequestBodyLimitAction Reject instead of ProcessPartial. (Enabling REQBODY_ERROR rule 200002 in modsecurity.conf will accomplish the same thing in current code, albeit a little less obviously.)

If you cannot deny these requests because it's a requirement that the endpoint servers must receive them, you could either:

1. continue to use the mitigation strategy discussed earlier (a phase 1 rule that sets ctl:requestBodyAccess=Off)
2. pursue an alternative strategy and try to reduce the very high latency with a 90M limit that you mentioned in your original item 3. The 9-11 minutes that you cited is shockingly high. Make sure that your SecDebugLogLevel isn't set too high as that can noticeably impact performance.

Along these lines you could also create a phase 1 rule with conditions similar to the one discussed earlier, only this time don't have the actions omit body access entirely, but instead disable some of the most costly and least useful rules. A very broad action against CRS rules could be something like 'ctl:ruleRemoveById=920250-944300' -- although in real life you'd probably at least want to be somewhat more discriminating (e.g. for your use case, there isn't much advantage to removing rules that don't include ARGS_NAMES as a target).

[gullu44]

Hi @martinhsv
Thanks for the detail explanation. As of now now we do not want to block such request (they are genuine production workload) and we have many such sites.
For the time being we can turn off RequestBodyAccess for such sites ( as also recommended by you). So ya we have workaround with but with some compromise.
Will be good if we have fix for this in coming months ... will wait .

[victorhora]

I recall SecStreamInBodyInspection being a common cause for slow uploads (see #1366).

@gullu44 I'd also suggest checking if you have that configuration enabled and also consider enabling the optional --enable-large-stream-input compilation time flag. 6406e21