Error running query when using /db/_find

[irinhwng]

Background information:

• single node couchDB on docker container where 1 db has ~500k records
• I built a python class that makes continuous queries (using the previous bookmark for the next query) until 0 queries are found. My queries became more complex, so I was testing them on a sample db of 500 records (and it worked perfectly fine).

The issue:

• As I'm trying to run the main class on the full db, I'm getting the following 504 error when using requests.post:

ChunkedEncodingError(ProtocolError("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)))

Note: it can run the query successfully until the very end (meaning: status code is 200 when bookmark = None and bookmark = xyz) but fails at the last bookmark.

So I thought to myself: there's probably a bug in my class. As a sanity check, I decided to use the UI from _utils and use the Mango Query dropdown. Here's the error I got from mango query:

I further analyzed the pod logs and found these errors:

[error] 2023-03-28T20:22:19.810779Z nonode@nohost emulator -------- Error in process <0.10272.644> with exit value:
[error] 2023-03-28T20:22:19.810850Z nonode@nohost emulator -------- Error in process <0.10272.644> with exit value:

What is going on here? Any pointers would be greatly appreciated. I'm scouring the internet for solutions and I have no idea what's going on.

PS:
The query is using the date field to only query records based on 2 date ranges. Sample query is:

{
   "selector": {
      "$or": [
         {
            "date": {
               "$gte": "2022-11-01 00:01:09",
               "$lt": "2022-11-01 00:03:15"
            }
         },
         {
            "date": {
               "$gt": "2022-11-01 00:07:57",
               "$lte": "2022-11-01 00:08:11"
            }
         }
      ]
   }
}

[willholley]

@irinhwng there's not really anything to go on from the log snipped you posted. Are you able to reproduce the issue using a script that you can share and/or share the full command and log? Your query is going to read every record in the database and apply the filter (since $or predicates cannot be accelerated using an index) so I wonder whether your container is hitting its memory limit.

[irinhwng]

yeah - I think I'm hitting a memory limit. I'm following the pod logs real-time whenever I send the query and noticed there are 0 error logs. What's interesting is there's a huge lag (ie: if I send the query at the 50th minute, the pod logs show a status code of 200 when looking at the request. Then a few seconds pass and I get the ChunkedEncodingError from my jupyter notebook (shown above). The reason I'm saying there's a lag is that the pod logs are surfacing a status code of 200 at the 58th minute which infers my couchdb is reading the request successfully (I think).

SOLUTION: I added use_index and specified the ddoc id along with the index name.

HOW: after reading the docs for the _find post request, I enabled execution stats which returned the following warning message

The number of documents examined is high in proportion to the number of results returned. Consider adding a more specific index to improve this

After adding a use_index array to my original request body for _find my query was able to run successfully :)

Question: what's the difference between ["<design_document>", "<index_name>"] and "<index_name>" for the use_index param?

@willholley since all fields are mentioned on both sides of the $or statement, using an index name is appropriate, right? reference: #1852