Efficient logging of selective data changes with FlatBuffers and multiple channels

[ClimentJackey]

I am developing a logging system for my project using MCAP for data persistence. My goal is to optimize the logging by writing to the file only when a specific value changes.

My current approach involves using FlatBuffers to define the message schemas. Since I need to write different types of data (e.g., speed, position) independently based on their change, I have split my data into separate FlatBuffer tables:

namespace VehicleSim;

table VehicleStateSpeed{
speed: float;
}

table VehicleStatePos{
x: float;
y: float;
}

root_type VehicleStateSpeed;
root_type VehicleStatePos;

n my MCAP writer implementation, I have registered a single FlatBuffer schema with the name "vehicle_sim". To handle the separate writing of speed and position, I am currently using multiple channels associated with this single schema:

mcap::Schema MySchema("vehicle_sim", "flatbuffer","binary");
writer.addSchema(MySchema);
mcap::Channel SpeedChannel("speed", "flatbuffer", MySchema.id);
writer.addChannel(SpeedChannel);
mcap::Channel PosChannel("pos", "flatbuffer", MySchema.id);
writer.addChannel(PosChannel);

My questions are:

Is this the most efficient or recommended way to handle selective logging of different data types that share a common serialization format (FlatBuffers) but are written independently based on value changes? Using multiple channels seems to introduce some overhead and might complicate later If i want to add more data's.

In the mcap::Schema constructor, how do I correctly provide the binary representation of my FlatBuffer schema (binary in my current code)? Since I have two root types in separate .fbs files, what binary data should I be providing here to represent a schema that can encompass both VehicleStateSpeed and VehicleStatePos (even though I'm writing them separately)?

Is there a more streamlined approach using a single channel and a single FlatBuffer schema to achieve the goal of writing only changed values for different data fields? For example, could I have a single schema with optional fields for speed and position, and then only populate and write the fields that have changed in a given message? If so, how would the FlatBuffer schema and the MCAP writing process look in this scenario?

I am looking for guidance on best practices for this kind of selective logging with MCAP and FlatBuffers to ensure efficiency and maintainability.

[jtbandes]

Using multiple channels seems to introduce some overhead

Which overhead are you concerned about? There is some setup code required, yes, but it does not add per-message overhead, since each message is logged with a channel id (an integer) and that would be true even if they shared the same channel.

how do I correctly provide the binary representation of my FlatBuffer schema (binary in my current code)? Since I have two root types in separate .fbs files, what binary data should I be providing here to represent a schema that can encompass both VehicleStateSpeed and VehicleStatePos (even though I'm writing them separately)?

The requirements are documented here: https://mcap.dev/spec/registry#flatbuffer-1
Since the name is part of the mcap::Schema record, you cannot reuse the same Schema for your two different Channels if you want to reference a different type. You could however create two Schemas that use the same binary data, and only change the name. Whether you want to do that or create completely separate bfbs data for the two schemas is up to you.

By the way, it's invalid to have two root_types in one fbs file.

could I have a single schema with optional fields for speed and position, and then only populate and write the fields that have changed in a given message?

I believe Foxglove will currently fill in the default values for missing fields in flatbuffer messages, so this might not work the way you want. Although it may work with protobuf -- haven't tried recently. Generally though, this might give you problems if you are trying to visualize certain data, e.g. if you try to plot VehicleState.speed and VehicleState.pos over time, then you may see strange gaps around the null values when the vehicle state was logged with a missing value for one of the fields.