Shard splitting job failed

[csemio]

Hi,
I have a database that contains around 100,000 docs, with 2 shards
I want to split one shards in two.

I do that :
`
range="80000000-ffffffff"

curl -u$credentials -s -H "Content-type: application/json" -XPOST
$rootUrl"_reshard/jobs"
-d '{"type": "split", "db":"'$dbName'", "range":"'$range'"}'
`

But the job failed with this error:

{
  "total_rows": 1,
  "offset": 0,
  "jobs": [
    {
      "id": "001-c4c4d0cdde77c061d6c3fbb4183a5894a0675059a1acf9da34e7b965be8fdc34",
      "type": "split",
      "source": "shards/80000000-ffffffff/timezones.1631172775",
      "target": [
        "shards/80000000-bfffffff/timezones.1631172775",
        "shards/c0000000-ffffffff/timezones.1631172775"
      ],
      "job_state": "running",
      "split_state": "initial_copy",
      "state_info": {
        "error": "{{invalid_start_purge_seq,0},\n [{couch_bt_engine,fold_purge_infos,5,\n                   [{file,\"src/couch_bt_engine.erl\"},{line,630}]},\n  {couch_db_split,copy_purge_info,1,\n                  [{file,\"src/couch_db_split.erl\"},{line,304}]},\n  {couch_db_split,split,6,[{file,\"src/couch_db_split.erl\"},{line,164}]},\n  {couch_db_split,split,3,[{file,\"src/couch_db_split.erl\"},{line,65}]},\n  {mem3_reshard_job,initial_copy_impl,1,\n                    [{file,\"src/mem3_reshard_job.erl\"},{line,400}]}]}"
      },
      "node": "nonode@nohost",
      "start_time": "2021-09-09T07:38:29Z",
      "update_time": "2021-09-09T07:38:59Z",
      "history": [
        {
          "timestamp": "2021-09-09T07:38:29Z",
          "type": "new",
          "detail": null
        },
        {
          "timestamp": "2021-09-09T07:38:29Z",
          "type": "running",
          "detail": "initial_copy"
        }
      ]
    }
  ]
}

I tries to compact the db, but no change
Have you ever had this problem ?

[nickva]

Thanks for reaching out. A few question to help debug it:

• What is the CouchDB version?

• Can you show the db info $host/$db output to see what the purge sequence looks like

• Was this db used on an older CouchDB version 2.x then upgraded to 3.x?

• Was _purge endpoint used on it?

• Any errors in the logs while the db is compacting / during shard splitting?

It would be idea if we could have a script or a few steps to reproduce the issue.

[csemio]

We are using the 3.0.0, on a 3 nodes cluster, but i'have tested on the lastest docker image, and i get the same result

I'have tested to play with shards for recreating all shards and in this case it's ok (remove shards from node 0 and 2 then re-add them, remove shards form node 1 and recreate them), but it is not an option for us on our production envirronment due to the time taken for shards recreation

Db info :

  "db_name": "timezones2",
  "purge_seq": "47077-g1AAAADZeJzLYWBgYMpgTmHwT84vTc5ISXIIy9UN8QzWDago0IUKGeolluaVGpcYGKdmlaSWJRsUpRmnGxYZJxulpeglVuhl5pWkFuUl5ugl5-SXpiQWFOjlpZbkgAzOYwGSDA1A6j8QZCUxMMTMJmCZEdmWJTIk1UNtifbKAgBXPUmX",
  "update_seq": "1576421-g1AAAADZeJzLYWBgYMpgTmHwT84vTc5ISXIIy9UN8QzWDago0IUKGeolluaVGpcYGKdmlaSWJRsUpRmnGxYZJxulpeglVuhl5pWkFuUl5ugl5-SXpiQWFOjlpZbkgAzOYwGSDA1A6j8QZCUx8CjOJWCZEdmWJTIk1UNs4X7jkQUARq1KBA",
  "sizes": {
    "file": 90432288,
    "external": 14056475,
    "active": 90136552
  },
  "props": {},
  "doc_del_count": 0,
  "doc_count": 100131,
  "disk_format_version": 8,
  "compact_running": false,
  "cluster": {
    "q": 2,
    "n": 1,
    "w": 1,
    "r": 1
  },
  "instance_start_time": "0"
}

it is possible that the database come from a migration of a 2.x version, I do not have the history

there's some warning and errors in logs (logs come from another db with the same problems :

[warning] 2021-09-09T16:47:54.886695Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.14098.141> -------- mem3_reshard_job : "#job{001-72851542c1f3d71cd155bc529efe12375bfa2c5e37446c7d542c6b281c6fbd5d shards/00000000-7fffffff/chrono-logs.1631205690 /2 job_state:running split_state:initial_copy pid:<0.11797.141>}" resetting <<"shards/00000000-3fffffff/chrono-logs.1631205690">> target
[warning] 2021-09-09T16:47:54.888934Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.14098.141> -------- mem3_reshard_job : "#job{001-72851542c1f3d71cd155bc529efe12375bfa2c5e37446c7d542c6b281c6fbd5d shards/00000000-7fffffff/chrono-logs.1631205690 /2 job_state:running split_state:initial_copy pid:<0.11797.141>}" resetting <<"shards/40000000-7fffffff/chrono-logs.1631205690">> target


[error] 2021-09-09T16:47:58.998672Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.11797.141> -------- mem3_reshard_job worker error "#job{001-72851542c1f3d71cd155bc529efe12375bfa2c5e37446c7d542c6b281c6fbd5d shards/00000000-7fffffff/chrono-logs.1631205690 /2 job_state:running split_state:initial_copy pid:<0.11797.141>}" {{invalid_start_purge_seq,0},[{couch_bt_engine,fold_purge_infos,5,[{file,"src/couch_bt_engine.erl"},{line,630}]},{couch_db_split,copy_purge_info,1,[{file,"src/couch_db_split.erl"},{line,304}]},{couch_db_split,split,6,[{file,"src/couch_db_split.erl"},{line,164}]},{couch_db_split,split,3,[{file,"src/couch_db_split.erl"},{line,65}]},{mem3_reshard_job,initial_copy_impl,1,[{file,"src/mem3_reshard_job.erl"},{line,400}]}]}
[error] 2021-09-09T16:47:58.998844Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net emulator -------- Error in process <0.14098.141> on node 'couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net' with exit value:
{{invalid_start_purge_seq,0},[{couch_bt_engine,fold_purge_infos,5,[{file,"src/couch_bt_engine.erl"},{line,630}]},{couch_db_split,copy_purge_info,1,[{file,"src/couch_db_split.erl"},{line,304}]},{couch_db_split,split,6,[{file,"src/couch_db_split.erl"},{line,164}]},{couch_db_split,split,3,[{file,"src/couch_db_split.erl"},{line,65}]},{mem3_reshard_job,initial_copy_impl,1,[{file,"src/mem3_reshard_job.erl"},{line,400}]}]}

[notice] 2021-09-09T16:47:58.999063Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.301.0> -------- mem3_reshard job 001-72851542c1f3d71cd155bc529efe12375bfa2c5e37446c7d542c6b281c6fbd5d exit {{invalid_start_purge_seq,0},[{couch_bt_engine,fold_purge_infos,5,[{file,"src/couch_bt_engine.erl"},{line,630}]},{couch_db_split,copy_purge_info,1,[{file,"src/couch_db_split.erl"},{line,304}]},{couch_db_split,split,6,[{file,"src/couch_db_split.erl"},{line,164}]},{couch_db_split,split,3,[{file,"src/couch_db_split.erl"},{line,65}]},{mem3_reshard_job,initial_copy_impl,1,[{file,"src/mem3_reshard_job.erl"},{line,400}]}]}
[notice] 2021-09-09T16:47:58.999170Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.301.0> -------- mem3_reshard job 001-72851542c1f3d71cd155bc529efe12375bfa2c5e37446c7d542c6b281c6fbd5d failed {{invalid_start_purge_seq,0},[{couch_bt_engine,fold_purge_infos,5,[{file,"src/couch_bt_engine.erl"},{line,630}]},{couch_db_split,copy_purge_info,1,[{file,"src/couch_db_split.erl"},{line,304}]},{couch_db_split,split,6,[{file,"src/couch_db_split.erl"},{line,164}]},{couch_db_split,split,3,[{file,"src/couch_db_split.erl"},{line,65}]},{mem3_reshard_job,initial_copy_impl,1,[{file,"src/mem3_reshard_job.erl"},{line,400}]}]}
[error] 2021-09-09T16:47:58.999172Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.11797.141> -------- CRASH REPORT Process  (<0.11797.141>) with 0 neighbors exited with reason: {invalid_start_purge_seq,0} at couch_bt_engine:fold_purge_infos/5(line:630) <= couch_db_split:copy_purge_info/1(line:304) <= couch_db_split:split/6(line:164) <= couch_db_split:split/3(line:65) <= mem3_reshard_job:initial_copy_impl/1(line:400) at mem3_reshard_job:handle_worker_exit/3(line:297) <= mem3_reshard_job:wait_for_workers/1(line:256) <= mem3_reshard_job:run/1(line:175) <= proc_lib:init_p_do_apply/3(line:247); initial_call: {mem3_reshard_job,init,['Argument__1']}, ancestors: [mem3_reshard_job_sup,mem3_reshard_sup,mem3_sup,...], message_queue_len: 0, messages: [], links: [<0.300.0>], dictionary: [], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 29670
[error] 2021-09-09T16:47:58.999389Z couchdb@Vm-TIS-Pxp-couchdb1.aunu3t03ejtevc0rf3g1r3c2fd.ax.internal.cloudapp.net <0.300.0> -------- Supervisor mem3_reshard_job_sup had child mem3_reshard_job started with {mem3_reshard_job,start_link,undefined} at <0.11797.141> exit with reason {invalid_start_purge_seq,0} at couch_bt_engine:fold_purge_infos/5(line:630) <= couch_db_split:copy_purge_info/1(line:304) <= couch_db_split:split/6(line:164) <= couch_db_split:split/3(line:65) <= mem3_reshard_job:initial_copy_impl/1(line:400) in context child_terminated

Another point _local_docs give me this :

{"total_rows":null,"offset":null,"rows":[
{"id":"_local/purge-mrview-6e2cce38bb97ddd1adee63a12da3a08d","key":"_local/purge-mrview-6e2cce38bb97ddd1adee63a12da3a08d","value":{"rev":"0-11410"}},
{"id":"_local/shard-sync-1WNITWXFRNV-0FYfvOGPIQ-6M8LTGTHCpSG4HZSRzGdVg","key":"_local/shard-sync-1WNITWXFRNV-0FYfvOGPIQ-6M8LTGTHCpSG4HZSRzGdVg","value":{"rev":"0-1"}},
{"id":"_local/shard-sync-6M8LTGTHCpSG4HZSRzGdVg-1WNITWXFRNV-0FYfvOGPIQ","key":"_local/shard-sync-6M8LTGTHCpSG4HZSRzGdVg-1WNITWXFRNV-0FYfvOGPIQ","value":{"rev":"0-1"}},
{"id":"_local/shard-sync-6M8LTGTHCpSG4HZSRzGdVg-jsMNOl1pH8jTLIdedriKPg","key":"_local/shard-sync-6M8LTGTHCpSG4HZSRzGdVg-jsMNOl1pH8jTLIdedriKPg","value":{"rev":"0-1"}},
{"id":"_local/shard-sync-IChlhMqRjTF4zO7-KP9-RA-Vvtt5j4065_qYR1wOz8Opw","key":"_local/shard-sync-IChlhMqRjTF4zO7-KP9-RA-Vvtt5j4065_qYR1wOz8Opw","value":{"rev":"0-1"}},
{"id":"_local/shard-sync-Vvtt5j4065_qYR1wOz8Opw-IChlhMqRjTF4zO7-KP9-RA","key":"_local/shard-sync-Vvtt5j4065_qYR1wOz8Opw-IChlhMqRjTF4zO7-KP9-RA","value":{"rev":"0-1"}},
{"id":"_local/shard-sync-Vvtt5j4065_qYR1wOz8Opw-tDivphvKbQfuIRX-9scWMQ","key":"_local/shard-sync-Vvtt5j4065_qYR1wOz8Opw-tDivphvKbQfuIRX-9scWMQ","value":{"rev":"0-1"}},
{"id":"_local/shard-sync-jsMNOl1pH8jTLIdedriKPg-6M8LTGTHCpSG4HZSRzGdVg","key":"_local/shard-sync-jsMNOl1pH8jTLIdedriKPg-6M8LTGTHCpSG4HZSRzGdVg","value":{"rev":"0-1"}}
]}

I can't delete the first item couchdb tell me that the document is missing

Steps to reproduce are very simple :
Create a new db with q=2 n=3
Set maintenance mode
Replace all shards and design with a backup version
Unset maintenance mode
Make a query to check if data are restored
try to split a shard

[nickva]

Thank you for the detailed information. I think this is a bug in how purge sequences are iterated over during shard splitting. It would happen in a database where purges were used extensively, as it seems to be this case, where the number of purges is higher than the max purge limit (1000) per shard. In that case, iterating over the purge sequence cannot happen from 0 and instead the minimal purge sequence has to be used, or it would crash.

[nickva]

I'll create an issue from this discussion. In the meantime I think, you should be able to replicate to a new database (timezones3) with a higher Q value.

Alternatively, replicate to one with the same Q/N values, then perform shard splitting (as purge sequences will be reset to 0, but once replication happens, might as well perform the shard expansion at that point).

If you don't mind building couchdb from scratch, could try to apply the patch where instead of 0 in here we pass in the oldest known sequence which can be obtained as:

StartSeq = couch_db_engine:get_oldest_purge_seq(Db) - 1,

https://github.com/apache/couchdb/blob/3.x/src/couch/src/couch_db_split.erl#L304

[nickva]

Issued a PR with the fix #3739