Disk data usage doesn't decrease when I delete some records. Is it possible?

[epikgit]

As I reported in the title, when I delete a record in the IndexedDB, the previously reserved disk space is not released.

I discovered this behaviour using JsStore in a PWA app for Android. When I check the app data disk space usage it doesn't decrease even when I delete some records.
I can't understand if this behaviour is due to the Android System WebView normal functioning or to JsStore itself. When I check the same app with the browser dev tools, this particular value is not available so I can't compare.

I need help to understand two things.

• Why is this happening? Who is responsible? JsStore or Android System WebView?
• Are there any techniques to force the DB to shrink to the minimum space really needed for active data?
  I don't want my app to be charged as the delinquent that fills up the device space... let's leave this dirty job to WhatsApp and its acolytes.

[ujjwalguptaofficial]

Have you tried removing entire database or clearing all tables data. Few rows of data is not that much to reflect the changes.

[epikgit]

Well if you remember my previous issue with the connection.update() command (#388) every row is big and heavy in my app. So the deletion of a single row does make the difference.

Anyway, in the application there's an option to clear all the tables and I just use it.... well the storage occupation has increased... a lot (???).

I'm coming from relational DB servers like Oracle and SQL Server and in those worlds the disk occupation is actually used in an exclusive way by the DB engine. When a DB reserves a disk space you can delete whatever you want but that disk remains reserved until you explicitly shrink the DB. You can even retrieve deleted data because the deletion of a record is only a logical operation.

Instead I'm quite new to indexed DB but seeing the same behaviour as a relational DB server in a noSQL DB is a non sense for me. That's the reason for which I asked if the responsible is indexed DB/JsStore or Android System WebView. When I know who is responsible for this waste of occupation, I can do something to fix it.

[ujjwalguptaofficial]

I believe its webview - they might be keeping data in cache and doing garbage collection later.

From code perspective - when you are clearning data, and it's not present which means things are done from our side. So most proobably its webview.

I will recommend to check the data size in browser app first - what's happening when you are deleting data : how size are changing and then you can deep dive into webview.

[epikgit]

Ok Ujjwal, I successfully tested the data size through a browser. I used Google Chrome because Android WebView is based on Chromium too.
Since the browser dev tools don't offer anything useful to check the data size, I thought I could test the disk occupation by directly checking the folders size (really? Genius!).

So I realized that Chromium reserves an entire folder to IndexedDB that is called IndexedDB located in %LocalAppData%\Google\Chrome\User Data\Default.
Inside this folder you'll find two folders for every visited website that uses IndexedDB.
The two folders are called:

• https_[website]_0.indexeddb.blob
• https_[website]_0.indexeddb.leveldb

Regardless of what I really do with the data managed with IndexedDB, the second folder, https_[website]_0.indexeddb.leveldb, grows and grows.
Even a complete emptying of all the used tables with the connection.clear('table name') command, makes the folder grow.
Gotcha!

What is growing inside the folder is precisely a file called MANIFEST-xxxxxx where xxxxxx is an increasing number.
I suspect that this file is a sort of transaction log as you find in the traditional transactional DB engine like SQL Server. The transaction log file is used to go back in time if a transaction fails. In SQL Server this file must be regularly managed, emptying it with manual or automatic tasks, when the old data becomes useless otherwise your disk space occupation skyrockets.

With this test, that you can easily reproduce, the responsibility is clear now.
What I don't know is how can I reset, shrink, delete, or not use this "transactional log". Seeing the other files (the *.ldb) growing dynamics in fact, I'm pretty sure the data resides there and not in the MANIFEST-xxxxxx file. Unfortunately I can't be 100% sure about that because all the data are binary files and not plain text ones so I can't read them.

What do you think about that?

[epikgit]

Hi @ujjwalguptaofficial,
Happy new year!
Have you got any news about this problem?

Thank you

[ujjwalguptaofficial]

Hey @epikgit nope, didn't find anything around this. But i was thinking may be its because of large string size of your image, so one suggestion - could you try storing image as blob.

Let's see what happens.

I feel storing blob is binary file, so if writing data in logs they will write just blob, or binary or object.

[epikgit]

Hi @ujjwalguptaofficial,
From my first tests the trick seems to work. As you correctly thought, using objects instead of strings doesn't inflate the log file. This way the IndexedDB folder size increases proportionally on what it really contains. Problem solved.

IMHO the doubt on what the heck is happening using text fields remains.
It's true, I was in a particular condition with few big svg images stored as strings, but the problem probably remains with thousands or more small string records.
The log file should be recycled in some way.
Anyway if you want to try something else to go deep with this issue I'm available even if my personal problem is now fixed.

I'm surprised how much IndexedDB is similar to a server DB. I always underestimated it and I was wrong.

Bye and thank you.

[ujjwalguptaofficial]

Nice nice, i am glad to know this. This is definitely going to help other users.

Regarding still logs file -

I feel they will be garbage collected after sometime, not instantly but after sometime.

But can you confirm if you are still seeing those logs file increasing ? I feel it might be due to larger string - there might be requirement to add logs, you know more for debugging when something breaks , so this is like a special condition where they are logging more and more in case if something breaks they can use log files to debug.

[epikgit]

I let it grow for a while but now I'm sure the log file is not inflating anymore or at least it inflates with cause.
Since now my use of string records is minimal, I can't evaluate if the log file recycles or not but I'm not agreeing with you about the previous reason for the inflation (need for debugging).

Since I've started to consider IndexedDB as a real database I think the log file has similar functions as that of a server DB. And a server DB log file is not filled with events like a code log (typically info, warnings and errors) but it contains data.
To be clear if you have a record and you modify it, the transaction log is filled with the previous recorded data.
What's the sense of this?
It's a simple way to restore previous data just in case the writing transaction fails in some way.

So you can ask why doesn't the transactional log recycle every time the transaction closes?
It's because of the other uses it's subjected to for example the statistics extraction.
The statistics for a server DB are fundamental to choose how to extract data (if using an index seek or a scan, how big a table is, and so on). In other words its meaning is to help the DB engine to create the execution plans.
Well the other utilities of a transaction log are for DB snapshot, data recovery, DB backup, and so on but I doubt IndexedDB allows such advanced features even if it amazed me once already so who knows...?
Anyway I'm quite sure the inflating ratio of the log file when I was using long strings and the lack of recycling show that it works exactly as a server DB transaction log.

The sad consequence of this speculation is that the recycling never happens because it is on the DB administrator (the app using it in my case) but I haven't discovered any documentation about any command to do that.

What do you think?
Is it a believable scenario in your opinion?