Database failed to load

[aboegge]

Description

A database failed to load and I got errors in the log file.

• Content of LogFile:

`[info] 2025-06-25T09:14:15.597000Z couchdb@localhost <0.253.0> -------- Preflight check: Checking For Monsters

[info] 2025-06-25T09:14:15.597000Z couchdb@localhost <0.253.0> -------- Preflight check: Asserting Admin Account

[info] 2025-06-25T09:14:15.600000Z couchdb@localhost <0.253.0> -------- Apache CouchDB 3.3.2 is starting.

[info] 2025-06-25T09:14:15.600000Z couchdb@localhost <0.254.0> -------- Starting couch_sup
[info] 2025-06-25T09:14:15.650000Z couchdb@localhost <0.253.0> -------- Apache CouchDB has started. Time to relax.

[notice] 2025-06-25T09:14:15.659000Z couchdb@localhost <0.328.0> -------- rexi_server : started servers
[notice] 2025-06-25T09:14:15.661000Z couchdb@localhost <0.332.0> -------- rexi_buffer : started servers
[notice] 2025-06-25T09:14:15.751000Z couchdb@localhost <0.363.0> -------- mem3_reshard_dbdoc start init()
[notice] 2025-06-25T09:14:15.756000Z couchdb@localhost <0.365.0> -------- mem3_reshard start init()
[notice] 2025-06-25T09:14:15.756000Z couchdb@localhost <0.366.0> -------- mem3_reshard db monitor <0.366.0> starting
[notice] 2025-06-25T09:14:15.758000Z couchdb@localhost <0.365.0> -------- mem3_reshard starting reloading jobs
[notice] 2025-06-25T09:14:15.758000Z couchdb@localhost <0.365.0> -------- mem3_reshard finished reloading jobs
[info] 2025-06-25T09:14:15.797000Z couchdb@localhost <0.372.0> -------- Apache CouchDB has started. Time to relax.

[info] 2025-06-25T09:14:15.797000Z couchdb@localhost <0.372.0> -------- Apache CouchDB has started on http://undefined:5656/
[notice] 2025-06-25T09:14:15.888000Z couchdb@localhost <0.462.0> -------- All system databases exist.
[notice] 2025-06-25T09:14:20.834000Z couchdb@localhost <0.424.0> -------- couch_replicator_clustering : cluster stable
[notice] 2025-06-25T09:14:20.840000Z couchdb@localhost <0.446.0> -------- Started replicator db changes listener <0.577.0>
[info] 2025-06-25T09:14:20.841000Z couchdb@localhost <0.579.0> -------- open_result error {not_found,no_db_file} for _replicator
[notice] 2025-06-25T09:14:30.661000Z couchdb@localhost <0.328.0> -------- rexi_server : cluster stable
[notice] 2025-06-25T09:14:30.661000Z couchdb@localhost <0.332.0> -------- rexi_buffer : cluster stable
[notice] 2025-06-25T09:15:20.420000Z couchdb@localhost <0.955.0> ca42b47afd localhost:5656 127.0.0.1 undefined GET /_utils/ 304 ok 31
[notice] 2025-06-25T09:15:20.524000Z couchdb@localhost <0.955.0> ea1fe5e9ec localhost:5656 127.0.0.1 undefined GET / 200 ok 20
[notice] 2025-06-25T09:15:20.531000Z couchdb@localhost <0.955.0> ff36c3c7be localhost:5656 127.0.0.1 undefined GET / 200 ok 3
[notice] 2025-06-25T09:15:20.534000Z couchdb@localhost <0.955.0> 2efbc207d2 localhost:5656 127.0.0.1 undefined GET / 200 ok 2
[notice] 2025-06-25T09:15:20.536000Z couchdb@localhost <0.955.0> 28213ccda3 localhost:5656 127.0.0.1 undefined GET /_session 200 ok 1
[notice] 2025-06-25T09:15:20.544000Z couchdb@localhost <0.955.0> 5d5172e1cb localhost:5656 127.0.0.1 undefined GET /_session 200 ok 1
[notice] 2025-06-25T09:15:20.545000Z couchdb@localhost <0.955.0> c8daae21df localhost:5656 127.0.0.1 undefined GET /_session 200 ok 0
[notice] 2025-06-25T09:15:20.552000Z couchdb@localhost <0.955.0> 6b6cd6161e localhost:5656 127.0.0.1 undefined GET /_utils/dashboard.assets/img/couchdb-logo.png 304 ok 1
[notice] 2025-06-25T09:15:20.569000Z couchdb@localhost <0.955.0> b266432cea localhost:5656 127.0.0.1 undefined GET /_utils/dashboard.assets/img/couchdb-logo.png 304 ok 0
[notice] 2025-06-25T09:15:22.500000Z couchdb@localhost <0.955.0> f49257f092 localhost:5656 127.0.0.1 undefined POST /_session 200 ok 15
[notice] 2025-06-25T09:15:22.514000Z couchdb@localhost <0.955.0> 7f107a6457 localhost:5656 127.0.0.1 admin GET /_session 200 ok 1
[notice] 2025-06-25T09:15:22.573000Z couchdb@localhost <0.955.0> 00a80012e6 localhost:5656 127.0.0.1 admin GET /_all_dbs 200 ok 46
[notice] 2025-06-25T09:15:22.612000Z couchdb@localhost <0.1056.0> 10cf8fca0d localhost:5656 127.0.0.1 admin GET /_users 200 ok 20
[notice] 2025-06-25T09:15:22.613000Z couchdb@localhost <0.1055.0> 8ab82c1923 localhost:5656 127.0.0.1 admin GET /_replicator 200 ok 23
[notice] 2025-06-25T09:15:22.627000Z couchdb@localhost <0.1058.0> 703935d8fc localhost:5656 127.0.0.1 admin GET /purchase_order 200 ok 36
[notice] 2025-06-25T09:15:22.632000Z couchdb@localhost <0.955.0> d4654d7f99 localhost:5656 127.0.0.1 admin GET /_global_changes 200 ok 43
[notice] 2025-06-25T09:15:22.637000Z couchdb@localhost <0.1055.0> cc57eca2bc localhost:5656 127.0.0.1 admin GET /supplier 200 ok 18
[notice] 2025-06-25T09:15:22.640000Z couchdb@localhost <0.1056.0> c41d450375 localhost:5656 127.0.0.1 admin GET /receiving 200 ok 21
[notice] 2025-06-25T09:15:22.641000Z couchdb@localhost <0.1058.0> ba1964aeff localhost:5656 127.0.0.1 admin GET /template 200 ok 9
[error] 2025-06-25T09:15:22.654000Z couchdb@localhost <0.1148.0> -------- gen_server <0.1148.0> terminated with reason: no match of right hand value {error,eio} at couch_file:find_header/3(line:689) <= couch_file:handle_call/3(line:616) <= gen_server:try_handle_call/4(line:721) <= gen_server:handle_msg/6(line:750) <= proc_lib:init_p_do_apply/3(line:226)
last msg: redacted
state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2253318731.1437990936.112090>,owner => <0.1148.0>,r_ahead_size => 0,r_buffer => #Ref<0.2253318731.1437990913.112248>}},false,14034047218,undefined,infinity}},{"InitialFilePath","./data/shards/80000000-ffffffff/invoice.1701791151.couch"}]}]
extra: [<0.1147.0>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,read_header,1,[{file,"src/couch_file.erl"},{line,393}]},{couch_bt_engine,init,2,[{file,"src/couch_bt_engine.erl"},{line,163}]},{couch_db_engine,init,3,[{file,"src/couch_db_engine.erl"},{line,744}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,42}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]]
[error] 2025-06-25T09:15:22.655000Z couchdb@localhost <0.1148.0> -------- gen_server <0.1148.0> terminated with reason: no match of right hand value {error,eio} at couch_file:find_header/3(line:689) <= couch_file:handle_call/3(line:616) <= gen_server:try_handle_call/4(line:721) <= gen_server:handle_msg/6(line:750) <= proc_lib:init_p_do_apply/3(line:226)
last msg: redacted
state: [{data,[{"State",{file,{file_descriptor,prim_file,#{handle => #Ref<0.2253318731.1437990936.112090>,owner => <0.1148.0>,r_ahead_size => 0,r_buffer => #Ref<0.2253318731.1437990913.112248>}},false,14034047218,undefined,infinity}},{"InitialFilePath","./data/shards/80000000-ffffffff/invoice.1701791151.couch"}]}]
extra: [<0.1147.0>,[{gen,do_call,4,[{file,"gen.erl"},{line,214}]},{gen_server,call,3,[{file,"gen_server.erl"},{line,243}]},{couch_file,read_header,1,[{file,"src/couch_file.erl"},{line,393}]},{couch_bt_engine,init,2,[{file,"src/couch_bt_engine.erl"},{line,163}]},{couch_db_engine,init,3,[{file,"src/couch_db_engine.erl"},{line,744}]},{couch_db_updater,init,1,[{file,"src/couch_db_updater.erl"},{line,42}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]]
[error] 2025-06-25T09:15:22.655000Z couchdb@localhost <0.1148.0> -------- CRASH REPORT Process (<0.1148.0>) with 2 neighbors crashed with reason: no match of right hand value {error,eio} at couch_file:find_header/3(line:689) <= couch_file:handle_call/3(line:616) <= gen_server:try_handle_call/4(line:721) <= gen_server:handle_msg/6(line:750) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.1147.0>,<0.1143.0>], message_queue_len: 0, links: [<0.1147.0>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2253318731.1437990936.112090>,...}},...}},...], trap_exit: false, status: running, heap_size: 6772, stack_size: 29, reductions: 11157
[error] 2025-06-25T09:15:22.655000Z couchdb@localhost <0.1148.0> -------- CRASH REPORT Process (<0.1148.0>) with 2 neighbors crashed with reason: no match of right hand value {error,eio} at couch_file:find_header/3(line:689) <= couch_file:handle_call/3(line:616) <= gen_server:try_handle_call/4(line:721) <= gen_server:handle_msg/6(line:750) <= proc_lib:init_p_do_apply/3(line:226); initial_call: {couch_file,init,['Argument__1']}, ancestors: [<0.1147.0>,<0.1143.0>], message_queue_len: 0, links: [<0.1147.0>], dictionary: [{couch_file_fd,{{file_descriptor,prim_file,#{handle => #Ref<0.2253318731.1437990936.112090>,...}},...}},...], trap_exit: false, status: running, heap_size: 6772, stack_size: 29, reductions: 11157
[info] 2025-06-25T09:15:22.655000Z couchdb@localhost <0.274.0> -------- db shards/80000000-ffffffff/invoice.1701791151 died with reason {{badmatch,{error,eio}},[{couch_file,find_header,3,[{file,"src/couch_file.erl"},{line,689}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,616}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
[notice] 2025-06-25T09:15:22.656000Z couchdb@localhost <0.1057.0> b40d59cc63 localhost:5656 127.0.0.1 admin GET /invoice 500 ok 67
[notice] 2025-06-25T09:15:22.929000Z couchdb@localhost <0.1057.0> 4c580b9601 localhost:5656 127.0.0.1 admin GET /_all_dbs?startkey=%22%22&endkey=%22%E9%A6%99%22&limit=30 200 ok 3
`

Steps to Reproduce

Expected Behaviour

Your Environment

• CouchDB version used: CouchDB 3.3.2
• Browser name and version:
• Operating system and version: Windows Server 2022.

Additional Context

[nickva]

Error eio indicates a general file I/O error. Noticed it in Terminated with reason: no match of right hand value {error,eio} at couch_file:find_header/3(line:689)

According to https://man7.org/linux/man-pages/man2/read.2.html

It may also occur when there is a low-level I/O error while reading
from a disk or tape. A further possible cause of EIO on
networked filesystems is when an advisory lock had been
taken out on the file descriptor and this lock has been
lost.

In other words, it seems like a low level disk error that erlang VM and couchdb is passing through. It's general enough that's hard to say what exactly it is, but hopefully the above hint from the man page provides a clue for you, depending on your particular setup.