Nouveau sometimes returns retry_later

[jonasplaum]

Description

We are using Nouveau for some queries, and on multiple systems we encountered failing nouveau queries with the following response from CouchDB:
500 Internal Server Error {"error":"unknown_error","reason":"retry_later"}

We increased max_sessions to 400 and later to 500 in our local.ini:

[nouveau]
enable = true
url = http://127.0.0.1:5987
max_sessions = 400 

The failures got less frequent, but are still there. Also on this systems there are only a small number of concurrent nouveau queries, so i would not expect to come near the limit of 500 or even 200. (Even if multiplied by our shard count)
We will increase the limit for now to a much higher value.

Steps to Reproduce

Query Nouveau with a few concurrent request, but this may not the only thing needed, or someone else would have encountered this.

Expected Behaviour

Nouveau request complete succesfully.

Your Environment

• CouchDB version used: 3.5.0
• Operating system and version: Debian 12

[rnewson]

thanks for filing.

@nickva and I have looked into ibrowse internals for this. Nick points to this piece of code in particular: https://github.com/cmullaparthi/ibrowse/blob/22d6fd6baa6e83633aa2923f41589945c1d2dc2f/src/ibrowse_lb.erl#L127

that is, ibrowse will always start a new connection for each request until max_sessions is reached. Only at that point will it try to reuse an existing connection.

not sure yet why you'd get retry_later if your concurrency is lower than max_sessions but the above behaviour, still looking.

[rnewson]

#5598 should help with this.

[rnewson]

#5598 is merged