onRelationshipsFlushed, onEntitiesFlushed possible unnecessarily locking?

[zemberdotnet]

sdk/packages/integration-sdk-runtime/src/storage/FileSystemGraphObjectStore/FileSystemGraphObjectStore.ts

Lines 286 to 320 in 2139b03

	await this.lockOperation(() =>
	pMap(
	this.localGraphObjectStore.collectRelationshipsByStep(),
	async ([stepId, relationships]) => {
	const indexable = relationships.filter((r) => {
	const indexMetadata = this.getIndexMetadataForGraphObjectType({
	stepId,
	_type: r._type,
	graphObjectCollectionType: 'relationships',
	});
	if (typeof indexMetadata === 'undefined') {
	return true;
	}
	return indexMetadata.enabled === true;
	});
	if (indexable.length) {
	await flushDataToDisk({
	storageDirectoryPath: stepId,
	collectionType: 'relationships',
	data: indexable,
	pretty: this.prettifyFiles,
	});
	}
	this.localGraphObjectStore.flushRelationships(relationships);
	if (onRelationshipsFlushed) {
	await onRelationshipsFlushed(relationships);
	}
	},
	),
	);

Does the onRelationshipsFlushed hook also need to be locked in the code above? My impression is that after the relationships/entities have been flushed, the hook should be free to run concurrently. If separately a step is running and able to fill the buffer, it must wait until the onRelationshipsFlushed/onEntitiesFlushed hook has finished. If the hook is an upload, that may unnecessarily slow down filling up and flushing the buffer again while uploads are happening. The hook seems like it might be unnecessarily blocking.

[zemberdotnet]

Thinking about this further, I think that onRelationshipsFlushed/onEntitiesFlushed does not need to be locked. The order of uploads shouldn't matter as long as the uploads are not duplicated. The locking only needs to ensure that the upload isn't duplicated. It can do that by ensuring the buffer is cleared without needing to ensure the upload is finished.

[zemberdotnet]

However, this function would need to be refactored to support that as the reference to the entities or relationships is created during the lock operation itself, but it doesn't escape the block currently.