Transaction file cleanup

[Flo9520]

I recently tried to reduce the size of our storage by compacting the transaction files since they are already 50MB each with 8 channels. To do so I have set the max transaction file size to 10MB with the following config:

private StorageDataFileEvaluator createStorageDataFileEvaluator() { return StorageDataFileEvaluator.New(StorageDataFileEvaluator.Defaults.defaultFileMinimumSize(), StorageDataFileEvaluator.Defaults.defaultFileMaximumSize(), StorageDataFileEvaluator.Defaults.defaultMinimumUseRatio(), StorageDataFileEvaluator.Defaults.defaultResolveHeadfile(), 10 * 1024 * 1024); }

When I restart my application I see log messages that the transaction files get compacted and after the application started they are each around 3KB only - great. Everything works fine until I restart again. The application crashes due to wrong timestamps in the data files.

Caused by: org.eclipse.store.storage.exceptions.StorageException: null at org.eclipse.store.storage.types.StorageFileManager$Default.readTransactionsFile(StorageFileManager.java:730) at org.eclipse.store.storage.types.StorageFileManager$Default.readStorage(StorageFileManager.java:695) at org.eclipse.store.storage.types.StorageChannel$Default.readStorage(StorageChannel.java:685) at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.internalProcessBy(StorageChannelTaskInitialize.java:192) at org.eclipse.store.storage.types.StorageChannelTaskInitialize$Default.internalProcessBy(StorageChannelTaskInitialize.java:34) at org.eclipse.store.storage.types.StorageChannelTask$Abstract.processBy(StorageChannelTask.java:244) at org.eclipse.store.storage.types.StorageChannel$Default.work(StorageChannel.java:414) at org.eclipse.store.storage.types.StorageChannel$Default.run(StorageChannel.java:487) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: org.eclipse.store.storage.exceptions.StorageExceptionConsistency: 2 Inconsistent timestamp of file 1356: 1748010710422000000 <= 1751932704992000000 at org.eclipse.store.storage.types.StorageTransactionsAnalysis$EntryAggregator.handleEntryStore(StorageTransactionsAnalysis.java:785) at org.eclipse.store.storage.types.StorageTransactionsAnalysis$EntryAggregator.accept(StorageTransactionsAnalysis.java:701) at org.eclipse.store.storage.types.StorageTransactionsAnalysis$Logic.processBufferedEntities(StorageTransactionsAnalysis.java:378) at org.eclipse.store.storage.types.StorageTransactionsAnalysis$Logic.processInputFile(StorageTransactionsAnalysis.java:340) at org.eclipse.store.storage.types.StorageTransactionsAnalysis$Logic.processInputFile(StorageTransactionsAnalysis.java:298) at org.eclipse.store.storage.types.StorageLiveTransactionsFile$Default.processBy(StorageLiveTransactionsFile.java:86) at org.eclipse.store.storage.types.StorageFileManager$Default.readTransactionsFile(StorageFileManager.java:724) ... 8 common frames omitted

I have tried it with different storage dumps from different days - same issue.
I am pretty afraid of my system reaching the default 100MB max transaction size which is default and then also performing the compacting.

Thank you in advance for any help/suggestions.

[hg-ms]

I’d been interested in those transaction log files to analyze the problem. Do have any option to provide those Files to us?

As your log files are 50 MB it will take some time until the default limit is reached:
The Default transaction log file size limit before any truncation is done is 1GB. The technical limit is 2GB, but you need to consider some MB tolerance so that the storage can compact the files before they reach that limit.

If you are interested in looking at the log files you can convert them to more human readable format. To do so you need to build the converter tool yourself:

mvn -Ptransaction-file-converter clean package

The tool will be in:
\storage\storage\target

[Flo9520]

I have created a github repo and invited you to it. There you will find the data before and after the compacting. Please let me know if you need and further information. Thank you very much in advance

[hg-ms]

Many thanks for the logs.

[hg-ms]

Was able to reproduce the issue => bug.
StorageTransactionsFileCleaner needs to take care of creation and store entries order in the transaction log.

[Flo9520]

Thanks a lot! Do you have any idea about the timeframe when this bug will be solved?

[hg-ms]

fix is available in pr #424 and will be released with the next release.

[hg-ms]

Here is an other option to clean the transaction log, i forgot to mention earlier:
You can delete the log files (transactions_x.sft) in all channels manually when the storage has been shut down.
On next startup the storage will start new to write new transaction logs.

[Flo9520]

This solution works fine for now - thank you!