Is it possible to have multiple dynamic TSDBs on the same flash partition?

[ValterMinute]

I describe my usage scenario.
I have a device that captures time series data in different "sessions" and I need to differentiate captured data for each session.
The sessions can overlap in terms of time (ex: I can acquire some data for "session A", then capture some in "session B" and then switch back to "session A" to do some more captures.
I can have a single TSDB with a session ID field, but this will force me to iterate through all sessions when doing data transfers etc. while for my purposes, it would be more efficient to be able to iterate each session separately.
My idea is to use fdb_tsdb_init to create multiple tsdbs (one per session) in each partition, add records etc.
If this works, would it be possible to do TSL adds in multiple sessions?
And what if I need to remove a tsdb/session completely (ex: when I uploaded it to the cloud or when the user ask to delete it)? Should I use fdb_tsl_clean or fdb_tsl_deinit? Would that free all the storage used by the session or there will be some flash space still in use?

[armink]

Hi ValterMinute, I tried the AI ​​reply and it worked pretty well. You can take a look.

Hello ValterMinute,

Thank you for your inquiry regarding the possibility of creating multiple dynamic Time Series Databases (TSDBs) within the same Flash partition using FlashDB. This is a common question, and I'd like to provide some insights and recommendations based on the current design and capabilities of FlashDB.

---

🧠 Understanding FlashDB's Design

FlashDB is designed to be lightweight and efficient, particularly for embedded systems with limited resources. The fdb_tsdb_init function initializes a TSDB instance within a specified Flash partition. However, initializing multiple TSDBs within the same partition can lead to several challenges:

• Metadata Overlap: Each TSDB instance requires a certain amount of metadata to manage its internal structures. Placing multiple TSDBs in the same partition can cause their metadata to overlap, leading to data corruption or unexpected behavior.

• Garbage Collection (GC) Interference: FlashDB employs a garbage collection mechanism to manage free space within the Flash. Multiple TSDBs sharing the same partition can cause GC operations to interfere with each other, potentially leading to performance degradation or data integrity issues.

• Resource Contention: Flash memory has limited write endurance. Sharing a single partition among multiple TSDBs can lead to uneven wear and potential premature failure of the Flash memory.

---

✅ Recommended Approaches

To achieve the desired functionality of differentiating data for each session while maintaining data integrity and performance, consider the following approaches:

1. Use Separate Flash Partitions for Each TSDB

Allocate a separate Flash partition for each session's TSDB. This approach ensures that each TSDB has its own dedicated space, preventing metadata overlap and GC interference.

• Configuration: Define multiple partitions in the Flash abstraction layer (FAL) configuration.

  #define FAL_PART_TABLE { \
      {FAL_PART_MAGIC_WROD, "session_A", "norflash0", 0L, 2L*1024L*1024L}, \
      {FAL_PART_MAGIC_WROD, "session_B", "norflash0", 2L*1024L*1024L, 2L*1024L*1024L}, \
      // Add more partitions as needed \
  }

• Initialization: Initialize each TSDB instance with its corresponding partition.

  fdb_tsdb_t tsdb_A, tsdb_B;
  fdb_tsdb_init(&tsdb_A, "session_A", NULL, get_time, 1024, NULL);
  fdb_tsdb_init(&tsdb_B, "session_B", NULL, get_time, 1024, NULL);

• Data Operations: Perform data operations on each TSDB instance independently.

  fdb_tsl_add(&tsdb_A, &tsl_A);
  fdb_tsl_add(&tsdb_B, &tsl_B);

This method provides clear separation between sessions and ensures optimal performance and data integrity.

2. Dynamic Partition Allocation (If Supported)

If your platform supports dynamic partition allocation, you can create new partitions at runtime to accommodate additional TSDBs.

• Dynamic Partition Creation: Use the appropriate API to create a new partition.

  struct fal_partition new_partition = {
      .name = "session_C",
      .flash_dev_name = "norflash0",
      .offset = 4L*1024L*1024L,
      .len = 2L*1024L*1024L
  };
  fal_set_partition_table_temp(&new_partition, 1);

• Initialization: Initialize a new TSDB instance in the newly created partition.

  fdb_tsdb_t tsdb_C;
  fdb_tsdb_init(&tsdb_C, "session_C", NULL, get_time, 1024, NULL);

• Data Operations: Perform data operations on the new TSDB instance.

  fdb_tsl_add(&tsdb_C, &tsl_C);

This approach provides flexibility in managing TSDBs dynamically, but it requires that your platform supports dynamic partition management.

---

⚠️ Important Considerations

• Flash Wear Leveling: Ensure that your Flash memory has wear leveling capabilities to distribute write operations evenly across the memory cells.

• Partition Size: Allocate sufficient space for each TSDB to accommodate the expected data volume and growth over time.

• Performance Testing: Conduct thorough performance testing to ensure that the chosen approach meets your application's requirements.

---

If you need further assistance or have additional questions, feel free to ask. I'm here to help you implement the most suitable solution for your application.

Best regards,
armink

[ValterMinute]

Thank you for the quick and detailed answer.
For me the issue about partioning is that I don't know in advance how many TSL I'll have in each session (is user-decision dependent), so it would be hard to statically define a size. Using a smaller size can impact maximum number of TSL per session, using a large one will reduce the number of sessions...
But I understand the technical constraints you expand an will try to figure out a good trade off.