When deleting dataset on disk, clear its caches #8638
Conversation
📝 WalkthroughWalkthroughThis change introduces a dedicated method to clear all relevant caches when a dataset is deleted, ensuring that any subsequent dataset with the same name does not use stale cached data. Related methods and repository calls are updated to use this cache-clearing logic and to synchronize dataset removal both locally and remotely. Changes
Assessment against linked issues
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
fm3
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceRepository.scala (1)
49-54: Good method rename with improved implementation, but consider error handling.The rename from
cleanUpDataSourcetoremoveDataSourceprovides better semantic clarity. The implementation now properly sequences local and remote operations within the Fox monad.However, consider the failure scenario: if the remote deletion fails after local removal succeeds, the data source will be removed locally but still exist remotely, potentially causing inconsistency.
Consider adding error recovery or changing the order of operations:
def removeDataSource(dataSourceId: DataSourceId): Fox[Unit] = for { - _ <- Fox.successful(remove(dataSourceId)) _ <- remoteWebknossosClient.deleteDataSource(dataSourceId) + _ <- Fox.successful(remove(dataSourceId)) } yield ()This would ensure remote deletion succeeds before local cleanup, maintaining consistency.
CHANGELOG.unreleased.md (1)
52-52: Minor style improvement for American English.The changelog entry accurately describes the cache clearing fix. Consider using "afterward" instead of "afterwards" for consistency with American English conventions.
-When deleting a dataset, its caches are cleared, so that if a new dataset by the same name is uploaded afterwards, only new data is loaded. [#8638](https://github.com/scalableminds/webknossos/pull/8638) +When deleting a dataset, its caches are cleared, so that if a new dataset by the same name is uploaded afterward, only new data is loaded. [#8638](https://github.com/scalableminds/webknossos/pull/8638)🧰 Tools
🪛 LanguageTool
[locale-violation] ~52-~52: In American English, ‘afterward’ is the preferred variant. ‘Afterwards’ is more commonly used in British English and other dialects.
Context: ...ew dataset by the same name is uploaded afterwards, only new data is loaded. [#8638](https...(AFTERWARDS_US)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
CHANGELOG.unreleased.md(1 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/controllers/DataSourceController.scala(3 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceRepository.scala(1 hunks)webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/uploading/UploadService.scala(2 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/uploading/UploadService.scala (2)
webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/DataSourceRepository.scala (1)
removeDataSource(49-54)app/models/dataset/Dataset.scala (1)
dataSourceId(83-83)
🪛 LanguageTool
CHANGELOG.unreleased.md
[locale-violation] ~52-~52: In American English, ‘afterward’ is the preferred variant. ‘Afterwards’ is more commonly used in British English and other dialects.
Context: ...ew dataset by the same name is uploaded afterwards, only new data is loaded. [#8638](https...
(AFTERWARDS_US)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: build-smoketest-push
🔇 Additional comments (7)
webknossos-datastore/app/com/scalableminds/webknossos/datastore/services/uploading/UploadService.scala (2)
455-455: LGTM: Consistent method call update.The update from
cleanUpDataSourcetoremoveDataSourcemaintains consistency with the method rename inDataSourceRepository.
676-676: LGTM: Consistent method call update.The update from
cleanUpDataSourcetoremoveDataSourcemaintains consistency with the method rename inDataSourceRepository.webknossos-datastore/app/com/scalableminds/webknossos/datastore/controllers/DataSourceController.scala (5)
25-25: LGTM: Import added for safe operation wrapper.The
tryoimport is appropriately added to support safe cache clearing in thedeleteOnDiskmethod.
471-487: Excellent refactoring: Cache clearing logic consolidated.The new
clearCachesOfDataSourcemethod effectively consolidates cache clearing logic that was previously scattered. Key benefits:
- Comprehensive cache coverage: Clears binary data service cache, mesh file cache, dataset error logs, and vault cache
- Detailed logging: Provides clear information about cleared cache entries
- Reusability: Can be used by both
reloadanddeleteOnDiskactions- Data source reloading: Returns a fresh
InboxDataSourceafter cache clearingThe implementation correctly handles different cache types and provides good observability through logging.
494-494: LGTM: Refactored to use consolidated cache clearing.The
reloadmethod is cleanly refactored to use the newclearCachesOfDataSourcemethod, eliminating code duplication and improving maintainability.
505-505: Good defensive programming: Safe cache clearing before deletion.Using
tryoto wrapclearCachesOfDataSourceis appropriate here since:
- Cache clearing failures shouldn't prevent dataset deletion
- The dataset is about to be deleted anyway, so stale cache entries will become irrelevant
- It ensures the deletion process is robust against cache-related errors
511-511: LGTM: Consistent with repository method rename.The update from
cleanUpDataSourcetoremoveDataSourcemaintains consistency with the method rename inDataSourceRepositoryand ensures both local and remote cleanup occur.
Steps to test:
Issues: