Memory leak when saving and indexing (clouseau) a big document (10MiB)

[rmartinez-dasnano]

Description

Using ibm docker (couchdb+lucene index) based on couchdb 3.1.1 (ibmcom/couchdb3:3.1.1), we are storing a big document into couchdb.
One of the fields in the document contains an array of arrays created from a 10MiB csv. When this document is created container memory usage starts growing until it reaches memory limit for the container (8Gib), or host limit (running in a 16GiB machine), or pocess limit

• If machine limit is reached, oom-killer kills beam.smp
• If container limit is reached, container is restarted.
• When none of these limits is reached, then the issue seems to be in dreyfus index updater. It's seems there is an OOM in OS process

out of memory
[info] 2021-08-01T10:09:57.661891Z couchdb@127.0.0.1 <0.236.0> -------- couch_proc_manager <0.18737.1> died normal
[error] 2021-08-01T10:09:57.661891Z couchdb@127.0.0.1 <0.22307.1> -------- OS Process Error <0.18737.1> :: {os_process_error,{exit_status,1}}
[error] 2021-08-01T10:09:57.662507Z couchdb@127.0.0.1 emulator -------- Error in process <0.22307.1> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,proc_prompt,2,[{file,"src/couch_query_servers.erl"},{line,520}]},{dreyfus_index_updater,update_or_delete_index,4,[{file,"src/dreyfus_index_updater.erl"},{line,141}]},{dreyfus_index_updater,load_docs,2,[{file,"src/dreyfus_index_updater.erl"},{line,80}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,1177}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,851}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,778}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,224}]}]}

Javascript fragment in the desgin document indexing this field: joins position 1 of arrays in a variable and then we index it in a single call to index function (built string size is ~ 11MiB). It's the same if we call index function as many times as rows in the array, same error.

if (doc.content && typeof(doc.content) !== 'undefined') {
           var stringToIndex="";
           for (i = 0; i < doc.content.length; i++) {
                 stringToIndex= stringToIndex+ doc.content[i][1]+" ";
            } 
            index("doc_number", stringToIndex, {"boost": 1, "facet":false, "index": true, "store": false});
}

To avoid os_process_error issue, we tried to increase max memory of couchjs processes with COUCHDB_QUERY_SERVER_JAVASCRIPT

environment:
  - COUCHDB_USER=****
  - COUCHDB_PASSWORD=****
  - COUCHDB_QUERY_SERVER_JAVASCRIPT="/opt/couchdb/bin/couchjs -S 536870912 /opt/couchdb/share/server/main.js"

But it seems it has no effect

docker container top 5bb095ba7a85
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
1001                333932              333911              0                   09:21               ?                   00:00:00            runsvdir -P -H /etc/service log: ...........................................................................................................................................................................................................................................................................................................................................................................................................
1001                334124              333932              0                   09:21               ?                   00:00:00            runsv couchdb
1001                334125              333932              0                   09:21               ?                   00:00:00            runsv couchdb-search
1001                334127              334125              0                   09:21               ?                   00:00:56            java -server -Xmx2G -Dsun.net.inetaddr.ttl=30 -Dsun.net.inetaddr.negative.ttl=30 -Dlog4j.configuration=file:/opt/couchdb-search/etc/log4j.properties -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -classpath /opt/couchdb-search/lib/* com.cloudant.clouseau.Main /opt/couchdb-search/etc/clouseau.ini
1001                334141              333932              0                   09:21               ?                   00:00:00            /opt/couchdb/bin/../erts-9.3.3.14/bin/epmd -daemon
1001                366563              334124              13                  11:41               ?                   00:09:12            /opt/couchdb/bin/../erts-9.3.3.14/bin/beam.smp -K true -A 16 -Bd -- -root /opt/couchdb/bin/.. -progname couchdb -- -home /opt/couchdb -- -boot /opt/couchdb/bin/../releases/3.1.1/couchdb -kernel inet_dist_listen_min 9100 -kernel inet_dist_listen_max 9100 -crypto fips_mode true -kernel error_logger silent -sasl sasl_error_logger false -noshell -noinput -name couchdb@127.0.0.1 -config /opt/couchdb/bin/../releases/3.1.1/sys.config -setcookie monster
1001                366594              366563              0                   11:41               ?                   00:00:00            erl_child_setup 1048576
1001                366647              366594              0                   11:41               ?                   00:00:00            inet_gethost 4
1001                366648              366647              0                   11:41               ?                   00:00:00            inet_gethost 4
1001                367618              366594              0                   11:45               ?                   00:00:30            ./bin/couchjs ./share/server/main.js
1001                367619              366594              0                   11:45               ?                   00:00:30            ./bin/couchjs ./share/server/main.js
1001                367659              366594              0                   11:45               ?                   00:00:29            ./bin/couchjs ./share/server/main.js
1001                369407              366594              0                   11:55               ?                   00:00:16            ./bin/couchjs ./share/server/main.js
1001                369417              366594              0                   11:55               ?                   00:00:17            ./bin/couchjs ./share/server/main.js
1001                369427              366594              0                   11:55               ?                   00:00:17            ./bin/couchjs ./share/server/main.js
1001                370634              366594              0                   12:07               ?                   00:00:12            ./bin/couchjs ./share/server/main.js
1001                370808              366594              0                   12:09               ?                   00:00:06            ./bin/couchjs ./share/server/main.js

    PID USUARIO   PR  NI    VIRT    RES    SHR S  %CPU  %MEM     HORA+ ORDEN                                                                                                                                                                                                      
 334127 1001      20   0 7741732 218544      0 S   0,3   1,4   0:58.75 java                                                                                                                                                                                                       
 366563 1001      20   0 5061660 195320  11820 S   4,6   1,2   9:52.21 beam.smp                                                                                                                                                                                                   
 367659 1001      20   0 1770568 140796  10688 S   0,0   0,9   0:29.68 couchjs                                                                                                                                                                                                    
 369427 1001      20   0 1770548 140676  10632 S   0,0   0,9   0:17.14 couchjs                                                                                                                                                                                                    
 369417 1001      20   0 1770504 140052  10368 S   0,0   0,9   0:17.03 couchjs                                                                                                                                                                                                    
 367618 1001      20   0 1769084 139404  10608 S   0,0   0,9   0:30.41 couchjs                                                                                                                                                                                                    
 370634 1001      20   0 1769480 139092  10464 S   0,0   0,9   0:12.20 couchjs                                                                                                                                                                                                    
 367619 1001      20   0 1768648 138664  10252 S   0,0   0,9   0:30.17 couchjs                                                                                                                                                                                                    
 369407 1001      20   0 1767160 136556  10536 S   0,0   0,9   0:16.80 couchjs                                                                                                                                                                                                    
 370808 1001      20   0 1727328  96888  10536 S   0,0   0,6   0:06.60 couchjs                                                                                                                                                                                                    
 366648 1001      20   0   14248   1636   1496 S   0,0   0,0   0:00.00 inet_gethost                                                                                                                                                                                               
 366594 1001      20   0    4368   1316   1232 S   0,0   0,0   0:00.33 erl_child_setup                                                                                                                                                                                            
 366647 1001      20   0   12124   1276   1172 S   0,0   0,0   0:00.00 inet_gethost                                                                                                                                                                                               
 334141 1001      20   0   12128     88      0 S   0,0   0,0   0:00.21 epmd                                                                                                                                                                                                       
 334124 1001      20   0    4244     44      0 S   0,0   0,0   0:00.01 runsv                                                                                                                                                                                                      
 333932 1001      20   0    4396     28      0 S   0,0   0,0   0:00.46 runsvdir                                                                                                                                                                                                   
 334125 1001      20   0    4244      4      0 S   0,0   0,0   0:00.00 runsv                                                                                                                                                                                                      

Steps to Reproduce

• Start docker image ibmcom/couchdb3:3.1.1
• Create the document
• Wait 5 seconds and memory will grow from 350Mib aprox to 6GiB

Expected Behaviour

• Memory consumption does not grow so much. 10MyB document -> 6Gib of memory consumotion looks like a memory leak.
• COUCHDB_QUERY_SERVER_JAVASCRIPT env variable should work as expected

Your Environment

• CouchDB version used: 3.1.1

• Docker version: Docker version 20.10.7, build f0df350

• Container limits:

       deploy:
      resources:
        limits:
          cpus: "4"
          memory: 8192M
        reservations:
          cpus: "4"
          memory: 4096M
  

• Couchdb configuration:

    [attachments] compressible_types="text/*, application/javascript, application/json, application/xml"
  [attachments] compression_level="8"
  [chttpd] backlog="512"
  [chttpd] bind_address="any"
  [chttpd] max_db_number_for_dbs_info_req="100"
  [chttpd] port="5984"
  [chttpd] prefer_minimal="Cache-Control, Content-Length, Content-Range, Content-Type, ETag, Server, Transfer-Encoding, Vary"
  [chttpd] require_valid_user="false"
  [chttpd] server_options="[{backlog, 512}, {acceptor_pool_size, 64}, {max, 4096}]"
  [chttpd] socket_options="[{sndbuf, 262144}, {nodelay, true}]"
  [cluster] n="3"
  [cluster] q="2"
  [cors] credentials="false"
  [couch_httpd_auth] allow_persistent_cookies="true"
  [couch_httpd_auth] auth_cache_size="50"
  [couch_httpd_auth] authentication_db="_users"
  [couch_httpd_auth] authentication_redirect="/_utils/session.html"
  [couch_httpd_auth] iterations="10"
  [couch_httpd_auth] require_valid_user="false"
  [couch_httpd_auth] secret="aaaaaa"
  [couch_httpd_auth] timeout="600"
  [couch_peruser] database_prefix="userdb-"
  [couch_peruser] delete_dbs="false"
  [couch_peruser] enable="false"
  [couchdb] attachment_stream_buffer_size="4096"
  [couchdb] changes_doc_ids_optimization_threshold="100"
  [couchdb] database_dir="./data"
  [couchdb] default_engine="couch"
  [couchdb] default_security="admin_only"
  [couchdb] file_compression="snappy"
  [couchdb] max_dbs_open="10000"
  [couchdb] max_document_size="4294967296"
  [couchdb] os_process_timeout="120000"
  [couchdb] single_node="true"
  [couchdb] users_db_security_editable="false"
  [couchdb] uuid="aaaa"
  [couchdb] view_index_dir="./data"
  [couchdb_engines] couch="couch_bt_engine"
  [csp] enable="true"
  [dreyfus] name="clouseau@127.0.0.1"
  [fabric] request_timeout="infinity"
  [feature_flags] partitioned||*="true"
  [httpd] allow_jsonp="false"
  [httpd] authentication_handlers="{couch_httpd_auth, cookie_authentication_handler}, {couch_httpd_auth, default_authentication_handler}"
  [httpd] bind_address="any"
  [httpd] enable_cors="false"
  [httpd] enable_xframe_options="false"
  [httpd] max_http_request_size="4294967296"
  [httpd] port="5986"
  [httpd] secure_rewrites="true"
  [httpd] socket_options="[{sndbuf, 262144}]"
  [indexers] couch_mrview="true"
  [ioq] concurrency="10"
  [ioq] ratio="0.01"
  [ioq.bypass] compaction="false"
  [ioq.bypass] os_process="true"
  [ioq.bypass] read="true"
  [ioq.bypass] shard_sync="false"
  [ioq.bypass] view_update="true"
  [ioq.bypass] write="true"
  [log] level="debug"
  [log] writer="stderr"
  [query_server_config] os_process_limit="2000"
  [query_server_config] os_process_soft_limit="1000"
  [query_server_config] reduce_limit="true"
  [replicator] connection_timeout="30000"
  [replicator] http_connections="20"
  [replicator] interval="60000"
  [replicator] max_churn="20"
  [replicator] max_jobs="500"
  [replicator] retries_per_request="5"
  [replicator] socket_options="[{keepalive, true}, {nodelay, false}]"
  [replicator] ssl_certificate_max_depth="3"
  [replicator] startup_jitter="5000"
  [replicator] verify_ssl_certificates="false"
  [replicator] worker_batch_size="500"
  [replicator] worker_processes="4"
  [ssl] port="6984"
  [uuids] algorithm="sequential"
  [uuids] max_count="1000"
  [vendor] name="The Apache Software Foundation"
  

• Operating system and version: Ubuntu 20.04

Additional Context

[wohali]

@rmartinez-dasnano We aren't responsible for the ibmcom image and don't directly support Dreyfus indexing, sorry - that code has not been adopted by Apache CouchDB. As such I'm moving this to our Discussions area.

You'll need to work with IBM on this one. You may have luck on our Slack channel at https://couchdb.apache.org/#chat , where some of its maintainers are.

[rmartinez-dasnano]

Ok. I'll try with dreyfus guys, as the index part is mentioned in the doc I supposed it was a good starting point. Thanks! Raúl El dom, 1 ago 2021 a las 19:38, Joan Touzet ***@***.***>) escribió:

…

@rmartinez-dasnano <https://github.com/rmartinez-dasnano> We aren't responsible for the ibmcom image and don't directly support Dreyfus indexing, sorry - that code has not been adopted by Apache CouchDB. As such I'm moving this to our Discussions area. You'll need to work with IBM on this one. You may have luck on our Slack channel at https://couchdb.apache.org/#chat , where some of its maintainers are. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3689 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUPS53NRMDWE2FWZSUCALDLT2WBAPANCNFSM5BLLANOQ> .

-- Raúl Martínez Maestre das-Gate | Engineering R&D <https://www.linkedin.com/company/das-nano/> <https://twitter.com/dasnano> das-nano.com <https://www.das-nano.com/> das-Nano, S.L. ***@***.*** Mobile: +34 697 88 74 69 | Phone: +34 948 24 62 95 Pamplona | Madrid | Ciudad de México | Bogotá | San Francisco