Data loss with persistent QoS

[lobmoo]

Is there an already existing issue for this?

• I have searched the existing issues

Expected behavior

When I use Fast DDS, I run into a problem.
I send 1000 uint16 data points per write, and I write 100 times — that’s a total of 100,000 data points. I use persistence storage and I want the reader to be able to receive the data even if it starts later. However, I’m seeing that when the reader starts after the writer, it only receives the first 1,000–10,000 data points — the rest never arrive.

If I change it to send 1,000 writes instead, the reader can receive more historical data — around 10,000 points — but still not all of them. I’ve also tested with SQLite persistence, and the result is the same, which confuses me.

Here is my QoS configuration:

void DDSDataWriterQosHanler::setDurability(u_int32_t depth)
{
m_dataWriterQos.reliability().kind = RELIABLE_RELIABILITY_QOS;
m_dataWriterQos.reliability().max_blocking_time = {1, 0};
m_dataWriterQos.durability().kind = TRANSIENT_DURABILITY_QOS;
m_dataWriterQos.history().kind = KEEP_ALL_HISTORY_QOS;
// m_dataWriterQos.properties().properties().emplace_back("dds.persistence.guid", generate_ascii_guid());
m_dataWriterQos.history().depth = static_cast<int32_t>(depth);
}

void DDSDataWriterQosHanler::setResourceLimits(uint32_t max_samples = 0,
uint32_t max_instances = 0,
uint32_t max_samples_per_instance = 0)
{
m_dataWriterQos.resource_limits().max_samples = static_cast<int32_t>(max_samples);
m_dataWriterQos.resource_limits().max_instances = static_cast<int32_t>(max_instances);
m_dataWriterQos.resource_limits().max_samples_per_instance =
static_cast<int32_t>(max_samples_per_instance);
}
handler.setDurability(10000);
handler.setResourceLimits(-1, -1, -1);
The following figure: originally 100,000 points were sent, but only the first 8,000 were received.”

Current behavior

no

Steps to reproduce

no

Fast DDS version/commit

3.2.2

Platform/Architecture

Ubuntu Focal 20.04 amd64

Transport layer

Default configuration, UDPv4 & SHM

Additional context

No response

XML configuration file

Relevant log output

Network traffic capture

No response

[MiguelCompany]

@lobmoo What about the DataReader QoS?

[lobmoo]

DataReader QoS 怎么样？

void DDSDataReaderQosHandler::setDurability(u_int32_t depth)
{
m_dataReaderQos.reliability().kind = RELIABLE_RELIABILITY_QOS;
m_dataReaderQos.reliability().max_blocking_time = {1, 0};
m_dataReaderQos.durability().kind = TRANSIENT_DURABILITY_QOS;
m_dataReaderQos.history().kind = KEEP_ALL_HISTORY_QOS;

m_dataReaderQos.properties().properties().emplace_back("dds.persistence.guid",
                                                        generate_ascii_guid());
m_dataReaderQos.history().depth = static_cast<int32_t>(depth);

}

/**

• @brief 配置dds写入器的资源限制
• @param max_samples 所有实例加起来最大能存多少样本总数
• @param max_instances DataWriter 或 DataReader 能管理的最大实例
• @param max_samples_per_instanceMyParamDoc 单个实例最多能存多少样本
  */
  void DDSDataReaderQosHandler::setResourceLimits(uint32_t max_samples = 0,
  uint32_t max_instances = 0,
  uint32_t max_samples_per_instance = 0)
  {
  m_dataReaderQos.resource_limits().max_samples = static_cast<int32_t>(max_samples);
  m_dataReaderQos.resource_limits().max_instances = static_cast<int32_t>(max_instances);
  m_dataReaderQos.resource_limits().max_samples_per_instance =
  static_cast<int32_t>(max_samples_per_instance);
  }

DDSDataReaderQosHandler qos_configurator_data_reader()
{
DDSDataReaderQosHandler handler;
handler.setDurability(10000);
handler.setResourceLimits(-1, -1, -1);
handler.setCloseDataSharing();
return handler;
}

Thank you for your response. The above is my QoS configuration, where the reader and writer settings are basically the same.

[lobmoo]

@lobmoo What about the DataReader QoS?

https://github.com/[lobmoo/test_demo](https://github.com/lobmoo/test_demo) This is my test case, and the branch is demo-branch

[cferreiragonz]

Hi @lobmoo, the link you provided is not accessible. Could you provide us with the info about the DataReader QoS as @MiguelCompany requested in any other way?

[MiguelCompany]

@lobmoo Please be patient. Don't start opening new issues to try obtaining an answer faster.

Please give us some time to take a look at your code.

Hi @lobmoo, the link you provided is not accessible. Could you provide us with the info about the DataReader QoS as @MiguelCompany requested in any other way?

@cferreiragonz Don't worry, I have access to the repository linked by @lobmoo

[lobmoo]

mkdir build
cd build
cmake ..
make -j$(nproc)

Source the environment variables:
source ../env.sh
Then open two terminals to run the example nodes:

# Terminal 1: Subscriber
./node_example_exec sub

# Terminal 2: Publisher
./node_example_exec pub

The main function is located at: sample/node_example/example.cpp.

Observed Behavior

Sending 1000 uint16 values, 100 times:

If the subscriber starts first, then the publisher, most messages are received, but some packets are lost.

If using persistence, and the publisher starts first before the subscriber, the subscriber only receives about 10% of the messages.

The larger the total message volume, the more messages the subscriber can receive after starting, but it never receives all messages.

demo.tar.gz

[lobmoo]

请耐心等待。不要为了更快地获得答案而开始打开新问题。

请给我们一些时间来查看您的代码。

嗨，@lobmoo，您提供的链接无法访问。您能否根据@MiguelCompany任何其他方式向我们提供有关 DataReader QoS 的信息？

别担心，我可以访问由

I have already uploaded my minimal demo and would greatly appreciate any timely assistance. Thank you very much.

[lobmoo]

Hi @lobmoo, the link you provided is not accessible. Could you provide us with the info about the DataReader QoS as @MiguelCompany requested in any other way?

I have already uploaded my minimal demo and would greatly appreciate any timely assistance. Thank you very much.