Batch commit throw DEADLINE_EXCEEDED but it actually create the documents

[MrLibya]

Environment

• Operating System version: Mac M1 arm64
• Firebase SDK version: 13.0.1
• Firebase Product: firestore
• Node.js version: v20.18.0
• NPM version: 10.9.0

The problem

I've class to handle more then 500 operation by simply creating new batch when reach 500, And when commit I'll run commit on each of the batches, And whenever any commit is failing it go to re-try commit that failed batch.

I had the batch commit fail with error of Deadline exceeded after 60.001s,metadata filters: 0.085s,LB pick: 0.001s,remote_addr=142.250.181.234:443 then after few re-try it was failing for new reason ALREADY_EXISTS: Document already exists and the data was created in the firestore !, If batch commit throw error it shouldn't write in firestore, but it seems commit throw error of the 60s timeout limits but data was actually successfully committed,

full log

commiting batch index 0 , ID: 0_3b6ca345a4a22d6f4d0a2f38eb5507bb
index  0 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.998s,metadata filters: 0.086s,remote_addr=142.250.181.234:443
commiting batch index 1 , ID: 1_a20059deface571877dbc25093dee067
index  1 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.889s,metadata filters: 0.041s,remote_addr=142.250.181.234:443
commiting batch index 2 , ID: 2_de705d40bf041659aa787c5e749ea4d3
index  2 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.999s,metadata filters: 0.025s,remote_addr=142.250.181.234:443
re-commit batchs count  3
total batchs 3 Retry count 2
start commit in sequentially mode
commiting batch index 0 , ID: 0_3b6ca345a4a22d6f4d0a2f38eb5507bb
index  0 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.999s,metadata filters: 0.087s,remote_addr=142.250.181.234:443
commiting batch index 1 , ID: 1_a20059deface571877dbc25093dee067
index  1 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 60.000s,metadata filters: 0.045s,remote_addr=142.250.181.234:443
commiting batch index 2 , ID: 2_de705d40bf041659aa787c5e749ea4d3
index  2 failed 6 ALREADY_EXISTS: Document already exists: projects/PROJECT_ID/databases/(default)/documents/COLLECTION_NAME/KSYlyNKEwJ4tkKGyZ0WR
re-commit batchs count  3
total batchs 3 Retry count 3
start commit in sequentially mode
commiting batch index 0 , ID: 0_3b6ca345a4a22d6f4d0a2f38eb5507bb
index  0 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.999s,metadata filters: 0.086s,remote_addr=142.250.181.234:443
commiting batch index 1 , ID: 1_a20059deface571877dbc25093dee067
index  1 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.999s,metadata filters: 0.043s,LB pick: 0.001s,remote_addr=142.250.181.234:443
commiting batch index 2 , ID: 2_de705d40bf041659aa787c5e749ea4d3
index  2 failed 6 ALREADY_EXISTS: Document already exists: projects/PROJECT_ID/databases/(default)/documents/COLLECTION_NAME/KSYlyNKEwJ4tkKGyZ0WR
re-commit batchs count  3
total batchs 3 Retry count 4
start commit in sequentially mode
commiting batch index 0 , ID: 0_3b6ca345a4a22d6f4d0a2f38eb5507bb
index  0 failed 4 DEADLINE_EXCEEDED: Deadline exceeded after 59.999s,metadata filters: 0.086s,remote_addr=142.250.181.234:443
commiting batch index 1 , ID: 1_a20059deface571877dbc25093dee067
index  1 failed 6 ALREADY_EXISTS: Document already exists: projects/PROJECT_ID/databases/(default)/documents/COLLECTION_NAME/fMQvNaE8y6m8Gx7hUwdb
commiting batch index 2 , ID: 2_de705d40bf041659aa787c5e749ea4d3
index  2 failed 6 ALREADY_EXISTS: Document already exists: projects/PROJECT_ID/databases/(default)/documents/COLLECTION_NAME/KSYlyNKEwJ4tkKGyZ0WR
Reached max try limit,  4 aborting commits.

Steps to reproduce:

I had bad internet and I was writing the 1300 documents with create operation

Relevant Code:

Code of my class

	_createBatch(index) {
		const batch = {
			id: index + '_' + crypto.randomBytes(16).toString('hex'),
			batch: this.firestore.batch(),
		}
		this.batches.push(batch)
		return batch
	}

	async commit() {
			for (let i = 0; i < originalBatchsCount; i++) {
				const item = this.batches[i]
				if (showBatchCounterLog) {
					console.log('commiting batch index', i, ', ID:', item.id)
				}
				try {
					await item.batch.commit()
					console.log('index is finished', i, 'ID', item.id)
					fulfilledIDs.push(item.id)
				} catch (e) {
					console.log('index ', i, 'failed', e.message)
				}
			}
}

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[milaGGL]

Hi @MrLibya, thank you for raising this issue. Quick confirmation, any batch that has been successful, will be removed from the re-try logic (this.batches), is it?

[MrLibya]

Hello @milaGGL , Yes it will be auto removed, you also can see that when commit I log the batch id, so its same id each time and also in that case I only had 3 batchs and all 3 were failing, So there wasn't any batch got removed.

Note that I only encountered this with slow internet connection, I still didn't dive deep into the firestore code but is there client side limit of 60s ? and it throw error ? or the error did actually came back from the server ?

[milaGGL]

A timeout doesn't necessarily mean the operation failed, so the operation might be still running on the server side when the client timed out. I assume you can see the docs successfully added if you check the database.

A workaround could be implement exponential backoff for retries, like increase the delay between retries exponentially, giving server more time to complete the batch operation.

Or if it is acceptable, instead of creating documents, use set() with merge: true. If the document does not exist, it will be created. If the document exists, the data will be merged. This could avoid the "Document already exists" error.

Or, you might want to consider making a smaller batch, so that it could be done within the timeout limit.

[milaGGL]

Have you considered using BulkWriter instead? It should handle errors and retries on behalf.

[MrLibya]

@milaGGL In the firestore documents it state that remote server will terminate request after 60s, So it shouldn't keep going.

I can't use set with merge because its batch so it might have different operation, update, delete or create, Only in this test all documents was create.

There's many workarounds can be done, like minimize operation per batch from 500 to 300 or something when having slow internet connection, But this doesn't solve the original issue which is if remote server return timeout it shouldn't persist the data and abort.

About bulkWriter is not what im looking for, As I need those data to be persist as transaction ( data linked to each other, all or nothing )