Write speed is very slow and CPU very hight

[geiliwanghaichao]

Hi: I'm using an API interface to test on an x86 server. After adding a dozen or so custom auxiliary fields, I wrote 10,000 records into a slow5 file, with each record's len_raw_signal set to 100,000. Neither zlib nor svb compression was enabled. The resulting file size is 4.7 GB, the process took 30 seconds(my x86 server is very strong), and the CPU utilization reached 99.9% (single core %Cpu1 : 94.1 us, 4.3 sy, 0.0 ni, 3.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st). This write speed is too slow. How can I optimize it? Thank you. 1.3.1 version

demo:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <slow5/slow5.h>
#include <time.h>

// #define FILE_PATH "output.blow5"
#define FILE_PATH "output.slow5"
#define NUM_RECORDS 10000
#define SIGNAL_LEN 100000
#define READ_ID_LEN 64

int16_t *shared_signal_buffer = NULL;

void set_header_aux_fields(slow5_file_t *sp)
{
const char *aux_fields[] = {
"CycleId", "ChannelId", "SensorNum", "ReadNum", "StartIndex",
"SignalLen", "ReadScale", "ReadOffset", "offset", "scale",
"K", "B", "Version", "Platform"};
const enum slow5_aux_type types[] = {
SLOW5_UINT16_T, SLOW5_UINT16_T, SLOW5_UINT16_T,
SLOW5_INT32_T, SLOW5_INT32_T, SLOW5_INT32_T,
SLOW5_DOUBLE, SLOW5_DOUBLE, SLOW5_FLOAT, SLOW5_FLOAT,
SLOW5_FLOAT, SLOW5_FLOAT, SLOW5_STRING, SLOW5_STRING};

for (int i = 0; i < 14; i++)
{
    if (slow5_aux_add(aux_fields[i], types[i], sp->header) < 0)
    {
        fprintf(stderr, "Error adding field %s\n", aux_fields[i]);
        exit(EXIT_FAILURE);
    }
}

}

int init_shared_signal_buffer()
{
shared_signal_buffer = (int16_t *)malloc(SIGNAL_LEN * sizeof(int16_t));
if (!shared_signal_buffer)
return -1;

for (uint64_t j = 0; j < SIGNAL_LEN; j++)
{
    shared_signal_buffer[j] = rand() & 4095;
}
return 0;

}

void free_shared_signal_buffer()
{
if (shared_signal_buffer)
{
free(shared_signal_buffer);
shared_signal_buffer = NULL;
}
}

slow5_rec_t *create_record(int record_num, slow5_file_t *sp)
{
slow5_rec_t *rec = slow5_rec_init();
if (!rec)
return NULL;

char *read_id = (char *)malloc(READ_ID_LEN);
if (!read_id)
{
    slow5_rec_free(rec);
    return NULL;
}
snprintf(read_id, READ_ID_LEN, "2507500506021_1_12_230_10209834_%d", record_num);
rec->read_id = read_id;
rec->read_id_len = strlen(read_id);

rec->read_group = 0;
rec->digitisation = 0;
rec->offset = 0;
rec->range = 0;
rec->sampling_rate = 0;
rec->len_raw_signal = SIGNAL_LEN;

rec->raw_signal = (int16_t *)malloc(SIGNAL_LEN * sizeof(int16_t));
if (!rec->raw_signal)
{
    slow5_rec_free(rec);
    return NULL;
}
memcpy(rec->raw_signal, shared_signal_buffer, SIGNAL_LEN * sizeof(int16_t));

uint16_t CycleId = record_num % 10;
uint16_t ChannelId = (record_num % 2048);
uint16_t SensorNum = (record_num % 4);
int32_t ReadNum = record_num;
int32_t StartIndex = record_num % 1000;
int32_t SignalLen = SIGNAL_LEN;
double ReadScale = (record_num % 100) / 100.0;
double ReadOffset = (record_num % 500) / 100.0;
float offset = 1.23232f, scale = 2.434f, K = 1.322345f, B = 3.1314f;
const char *version = "v1.0";
const char *platform = "testbench";

slow5_aux_set(rec, "CycleId", &CycleId, sp->header);
slow5_aux_set(rec, "ChannelId", &ChannelId, sp->header);
slow5_aux_set(rec, "SensorNum", &SensorNum, sp->header);
slow5_aux_set(rec, "ReadNum", &ReadNum, sp->header);
slow5_aux_set(rec, "StartIndex", &StartIndex, sp->header);
slow5_aux_set(rec, "SignalLen", &SignalLen, sp->header);
slow5_aux_set(rec, "ReadScale", &ReadScale, sp->header);
slow5_aux_set(rec, "ReadOffset", &ReadOffset, sp->header);
slow5_aux_set(rec, "offset", &offset, sp->header);
slow5_aux_set(rec, "scale", &scale, sp->header);
slow5_aux_set(rec, "K", &K, sp->header);
slow5_aux_set(rec, "B", &B, sp->header);
slow5_aux_set_string(rec, "Version", version, sp->header);
slow5_aux_set_string(rec, "Platform", platform, sp->header);

return rec;

}

int main()
{
srand(time(NULL));

if (init_shared_signal_buffer() < 0)
{
    fprintf(stderr, "Failed to allocate shared signal buffer\n");
    exit(EXIT_FAILURE);
}

slow5_file_t *sp = slow5_open(FILE_PATH, "w");
if (!sp)
{
    fprintf(stderr, "Error opening file %s\n", FILE_PATH);
    free_shared_signal_buffer();
    exit(EXIT_FAILURE);
}

// // 设置压缩
// if (slow5_set_press(sp, SLOW5_COMPRESS_ZLIB, SLOW5_COMPRESS_SVB_ZD) < 0) {
//     fprintf(stderr, "Warning: Failed to set compression\n");
// }

set_header_aux_fields(sp);
if (slow5_hdr_write(sp) < 0)
{
    fprintf(stderr, "Error writing header\n");
    free_shared_signal_buffer();
    slow5_close(sp);
    exit(EXIT_FAILURE);
}

clock_t start = clock();

for (int i = 0; i < NUM_RECORDS; i++)
{
    slow5_rec_t *rec = create_record(i, sp);
    if (!rec)
    {
        fprintf(stderr, "Failed to create record %d\n", i);
        continue;
    }

    if (slow5_write(rec, sp) < 0)
    {
        fprintf(stderr, "Error writing record %d\n", i);
        slow5_rec_free(rec);
        break;
    }

    slow5_rec_free(rec);

    if ((i + 1) % 1000 == 0)
    {
        printf("Written %d records...\n", i + 1);
        fflush(stdout);
    }
}

clock_t end = clock();
double elapsed = (double)(end - start) / CLOCKS_PER_SEC;

printf("Successfully written %d records in %.2f seconds\n", NUM_RECORDS, elapsed);
printf("Record rate: %.2f records/sec\n", NUM_RECORDS / elapsed);

free_shared_signal_buffer();
slow5_close(sp);

return 0;

}

[hasindu2008]

Hi there,

I would need a bit more details get a bit of an idea of what you are trying to achieve. A few questions:

1. Do you want ASCII SLOW5 or binary SLOW5 (BLOW5) output?
2. What do you mean by "x86 server is very strong"?
3. What is your underlying disk system?
4. Are you willing to use multiple threads on your CPU?
5. How quickly would you like your 10,000 records to be written in?

[geiliwanghaichao]

@hasindu2008 I'm glad to receive your reply.

Regarding your questions:

1. I want binary SLOW5 (BLOW5) output. you know, BLOW5 files are much smaller than SLOW5, which is exactly the result I want.
2. My x86 server has over twenty cores with high frequency, and 64GB or more memory space.
3. The disk consists of 6 mechanical hard drives in a RAID0 array (actual tested read/write performance is 6 times that of a single drive), with a speed of 1600 MB/s.
4. In actual use, I will definitely use multiple threads to write to different files (as you know, the data volume is very large). For example, one thread writes data from 256 channels to a file. I know the API supports mt(single file written by multiple threads), and I can accept this approach too.
5. The writing speed should be as fast as possible, without occupying too much CPU.

Additionally, during testing I found that the CPU's us (user space usage) is very high. How can this be optimized?

Thank you.

[hasindu2008]

Thanks, now I have the context.

In your example, you have #define FILE_PATH "output.slow5" which means you are writing to ASCII SLOW5. Converting to a string is expensive, so it would be slow. I used your program on server a bit close to yours (36 core Xeon, 12x spinning disks on RAID)
Successfully written 10000 records in 67.73 seconds

Now I changed it to #define FILE_PATH "output.blow5". If we do not set the compression, it will default to zlib+svb_zd. Zlib compression is expensive, so more than ASCII:
written 10000 records in 90.50 seconds

Now I set the compression to efficient zstd by:

if (slow5_set_press(sp, SLOW5_COMPRESS_ZSTD, SLOW5_COMPRESS_SVB_ZD) < 0) {
     fprintf(stderr, "Warning: Failed to set compression\n");
 }

and then compile as make zstd=1 && gcc -Wall -O2 a.c -I include/ lib/libslow5.a -lz -lzstd.

written 10000 records in 11.32 seconds

See if changing to zstd improves time on yours. See the attached file: a.c

---

If you are planning to write a lot of multiple files, then using one thread per file could be adequate, as multiple threads are going to be utilised based on the files. In fact one thread per file is what I used in my slowION program as well which simulates nanopore data rates.

CPU's us (user space usage) is very high - how are you measuring this?

[hasindu2008]

To add further, if you want to compromise speed for file size, try writing to pure binary without any compression as:

if (slow5_set_press(sp, SLOW5_COMPRESS_NONE, SLOW5_COMPRESS_NONE) < 0) {
     fprintf(stderr, "Warning: Failed to set compression\n");
}

I get Successfully written 10000 records in 1.64 seconds

But be careful, as when you write multiple files in parallel, the I/O could become the bottleneck in this uncompressed binary writing approach due to the large file sizes.

[geiliwanghaichao]

@hasindu2008 Hi,Thanks for your advice.
Following your suggestion, I changed it to #define FILE_PATH "output.blow5" and set the compression to zstd using:
if (slow5_set_press(sp, SLOW5_COMPRESS_ZSTD, SLOW5_COMPRESS_SVB_ZD) < 0) {
fprintf(stderr, "Warning: Failed to set compression\n");
}
I created 22 threads to simultaneously write to 22 different files. After writing, each file was approximately 1.7 GB in size, and the total time to complete all files was around 35 seconds. I'm absolutely amazed!!! I didn't expect zstd to be so much more efficient than zlib.
The CPU usage during execution was as follows:
top - 17:20:37 up 1 day, 1:32, 6 users, load average: 10.40, 3.38, 1.47
Threads: 826 total, 23 running, 803 sleeping, 0 stopped, 0 zombie
%Cpu0 : 57.2 us, 41.1 sy, 0.0 ni, 1.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 60.2 us, 39.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 59.9 us, 40.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 46.9 us, 47.2 sy, 0.0 ni, 5.6 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu4 : 26.3 us, 60.5 sy, 0.0 ni, 3.6 id, 9.5 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 55.8 us, 43.2 sy, 0.0 ni, 1.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 55.6 us, 42.1 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 57.8 us, 42.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 57.2 us, 42.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 56.7 us, 42.0 sy, 0.0 ni, 1.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 55.3 us, 44.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 39.1 us, 51.6 sy, 0.0 ni, 0.3 id, 8.9 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 : 50.3 us, 49.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 52.5 us, 47.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 51.0 us, 49.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 51.3 us, 48.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu16 : 51.6 us, 48.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu17 : 52.0 us, 48.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu18 : 51.6 us, 48.4 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu19 : 48.2 us, 51.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu20 : 48.5 us, 50.8 sy, 0.0 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu21 : 48.7 us, 51.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15458.3 total, 2231.7 free, 1609.1 used, 12097.4 buff/cache
MiB Swap: 4096.0 total, 4095.2 free, 0.8 used. 13849.1 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

173786 aa 20 0 1626020 25180 1764 R 99.9 0.2 0:07.96 mutil_thread_wr
173790 aa 20 0 1626020 23596 1764 R 99.9 0.1 0:08.12 mutil_thread_wr
173781 aa 20 0 1626020 25004 1764 R 99.9 0.2 0:07.82 mutil_thread_wr
173785 aa 20 0 1626020 25180 1764 R 99.3 0.2 0:07.89 mutil_thread_wr
173792 aa 20 0 1626020 23772 1764 R 99.3 0.2 0:07.65 mutil_thread_wr
173793 aa 20 0 1626020 23948 1764 R 99.0 0.2 0:07.90 mutil_thread_wr
173796 aa 20 0 1626020 25004 1764 R 98.7 0.2 0:07.89 mutil_thread_wr
173797 aa 20 0 1626020 23948 1764 R 97.7 0.2 0:07.74 mutil_thread_wr
173788 aa 20 0 1626020 25356 1764 R 96.4 0.2 0:07.99 mutil_thread_wr
173791 aa 20 0 1626020 23772 1764 R 95.7 0.2 0:07.82 mutil_thread_wr
173779 aa 20 0 1626020 27996 1764 R 95.0 0.2 0:07.45 mutil_thread_wr
173780 aa 20 0 1626020 28172 1764 R 94.7 0.2 0:07.45 mutil_thread_wr
173782 aa 20 0 1626020 25180 1764 R 94.7 0.2 0:07.86 mutil_thread_wr
173798 aa 20 0 1626020 24124 1764 R 94.7 0.2 0:07.58 mutil_thread_wr
173799 aa 20 0 1626020 23068 1764 R 94.4 0.1 0:07.75 mutil_thread_wr
173784 aa 20 0 1626020 25180 1764 R 94.0 0.2 0:07.69 mutil_thread_wr
173783 aa 20 0 1626020 25180 1764 R 93.4 0.2 0:07.58 mutil_thread_wr
173787 aa 20 0 1626020 25180 1764 R 92.1 0.2 0:07.41 mutil_thread_wr
173778 aa 20 0 1626020 29052 1764 R 89.1 0.2 0:07.73 mutil_thread_wr
173794 aa 20 0 1626020 23948 1764 R 87.4 0.2 0:07.53 mutil_thread_wr
As you can see, the CPU user space (us) utilization is no longer above 90%; instead, the system space (sy) utilization has increased significantly. This indicates that the proportion of CPU time spent in user space has decreased, while the time spent in kernel space has increased.

Are there any other optimization methods to further reduce CPU usage? Because i have other tasks running concurrently, such as sending data over the network, which also require CPU resources. Any suggestions would be greatly appreciated.
Thank you!

[hasindu2008]

If the reduction of the CPU utilisation is the key goal, can you try using the no compression method I suggested here: #99 (comment).
But the overall time could increase due to file sizes being larger, but worth giving a go.

There are a few ways to optimise the zstd-based methods a bit more. I will note them down later in the next couple of days.

[geiliwanghaichao]

@hasindu2008 Hi:

Reducing CPU usage (either us or sy) is one of my goals. However, as observed in practice, zstd+svb compression significantly reduces file size and greatly improves write speed. Therefore, I definitely want to stick with zstd+svb compression.

Memory consumption is also an important consideration . I believe there are two areas with substantial room for optimization: read_id and raw_signal. These fields require malloc for every record, and frequent malloc calls can lead to memory fragmentation and significant performance overhead.

Additionally, as you mentioned, there are other ways to further optimize the process—methods that I'm currently unaware of.

Thank you!

[geiliwanghaichao]

@hasindu2008 Hi, I have another question: in the slow5lib source code, fwrite is used for writing files. Why not use write instead? The write system call allows setting options such as open("output.blow5", O_WRONLY | O_CREAT | O_DIRECT | O_SYNC, 0644), which could enable bypassing the kernel page cache and achieve direct I/O.
Thanks

[hasindu2008]

• Use newer zstd. As most CPU cycles are spent on compression, any improvements in the zstd library will help. For example:

#new zstd
ZSTD_VERSION=1.5.5
wget https://github.com/facebook/zstd/archive/refs/tags/v${ZSTD_VERSION}.tar.gz -O zstd-v${ZSTD_VERSION}.tar.gz
tar -xzf zstd-v${ZSTD_VERSION}.tar.gz
mv zstd-${ZSTD_VERSION} zstd
cd zstd
make -j8
cd ..

# now slow5lib with that zstd
make clean
make zstd_local=./zstd/lib

# now your program
gcc -Wall -O2 a.c -I include/ lib/libslow5.a zstd/lib/libzstd.a -lz 

• Change the optimisation level of the compiler from -O2 to -O3 in the slow5lib makefile and when compiling the test program

• Use a newer compiler. Newer compilers tend to have slightly better optimisations. For example:

export CC=gcc-12
make CC=$CC zstd_local=./zstd/lib
$CC -Wall -O3 a.c -I include/ lib/libslow5.a zstd/lib/libzstd.a -lz

• use the SIMD-optimised svb16-zd for signal compression in the bench branch. But please be mindful that this svb16-zd is not in the main branch of slow5lib, instead only in the bench branch for benchmarking purposes. I haven't got any use case that warrants having this (some performance improvement, but at the cost of being too coupled with the architecture)

First, change your compression to:

if (slow5_set_press(sp, SLOW5_COMPRESS_ZSTD, SLOW5_COMPRESS_SVB16_ZD) < 0) {
     fprintf(stderr, "Warning: Failed to set compression\n");
}

Then use the experimental bench branch:

git checkout bench
make CC=$CC zstd_local=./zstd/lib
$CC -Wall -O3 a.c -I include/ lib/libslow5.a zstd/lib/libzstd.a -lz

For further optimisations, it is possible to profile the slow5lib, identify hotspots and then optimise the necessary components in the slow5lib. But currently, do we have a reason that requires that level of optimisation that would be of major practical benefit?

---

About memory, are you referring to the mallocs you do in your example programme?

---

About the use of fwrite vs write. For small writes, doing a write syscall can be expensive and blocking. fwrite suits a wide variety of cases in our experience and is more portable (and less surprises). Direct write can be useful if you know the exact write sizes beforehand (that may depend on read lengths) and you are optimising for a very specific system. But such specific overoptimisations can lead to non-optimal performance in other cases. Do you see any significant improvements on your system when fwrites are replaced with writes?

[geiliwanghaichao]

@hasindu2008 Hi：
The version of the zstd library I'm currently using is 1.5.5.
Regarding memory allocation and deallocation, in my provided code:
rec->raw_signal = (int16_t *)malloc(SIGNAL_LEN * sizeof(int16_t));
if (!rec->raw_signal) {
slow5_rec_free(rec);
return NULL;
}
For each record, a separate block of memory is allocated via malloc for the raw_signal field (the same applies to read_id). This memory is freed after use (just as implemented in the example/write.c). This means that for 10,000 records, malloc is called 10,000 times. Such frequent memory allocation and deallocation can lead to significant memory fragmentation and negatively impact performance. For handling raw_signal and read_id, do you have any recommended practices to minimize or avoid frequent malloc calls?
Additionally, regarding fwrite vs write: I understand that write is lower-level than fwrite. But is write truly more efficient than fwrite? I haven't had the chance to test this yet.

Thank you!

[hasindu2008]

It is possible to reuse those memory allocations, at the cost of complicating the code, thus compromising maintainability. I can give an example when I get some time next week. But I doubt if there will be any benefit to overall performance. Your server has plenty of RAM, so these memory allocations are not that big compared to them. Also, modern glibc memory allocators are quite good. I find it unlikely that executing tens of thousands of malloc operations within a few seconds on a system like this would result in significant memory fragmentation. Do you have any numbers that show such fragmentation effect for mallocs at this scale?

[geiliwanghaichao]

@hasindu2008 Hi:
I don't have concrete data showing exactly how much impact frequent malloc/free calls have on CPU performance, but subjectively, there must be some negative effect. Similarly,
I don't have solid measurements proving that frequent malloc/free significantly increases memory fragmentation, but based on various online resources, it's widely stated that it does contribute to increased memory fragmentation.
Additionally, I've identified a critical issue when compression is enabled. In the code, I noticed the following: In slow5.c → slow5_rec_to_mem() →
if (compress->signal_press->method != SLOW5_COMPRESS_NONE) { /* signal compression */
uint8_t *raw_sig_svb = slow5_ptr_compress_solo(compress->signal_press->method, read->raw_signal, read->len_raw_signal * sizeof *read->raw_signal, &bytes_raw_sig);
if (!raw_sig_svb) {
SLOW5_ERROR("%s", "Signal compression failed.");
free(mem);
if (compress_to_free) {
slow5_press_free(compress);
}
return NULL;
}
free(read->raw_signal);
read->len_raw_signal = bytes_raw_sig;
read->raw_signal = (int16_t *) raw_sig_svb;
}
Here, free(read->raw_signal); free the buffer that was originally malloc'ed by the application when assigning data to raw_signal. Then, read->raw_signal = (int16_t *) raw_sig_svb; reassigns the read->raw_signal pointer to point to the compressed data (which was malloc'ed inside the slow5_ptr_compress_solo function).
Later, when slow5_rec_free(rec) is called:
void slow5_rec_free(struct slow5_rec *read) {
if (read != NULL) {
free(read->read_id);
free(read->raw_signal);
slow5_rec_aux_free(read->aux_map);
free(read);
}
}
The free(read->raw_signal); in slow5_rec_free() free the buffer containing the compressed data.
Now, here's why I have concerns: When I learned about malloc/free, my instructor emphasized two important principles:
1.Memory should be freed by the same component that allocated it.
2. When processing data in a buffer, avoid modifying the original source data.
According to these principles, the buffer allocated by the user application (for raw_signal) should be explicitly freed by the user, not automatically freed inside slow5_rec_free. Moreover, modifying the pointer's target within the library code violates the principle of preserving the original data.
The reason I raise this issue is that I'm considering using a memory pool for raw_signal. You know, a memory pool typically pre-allocates a large block of memory at program startup and only releases it when the program exits. However, the current code in slow5lib frees the buffer from the memory pool, rendering it unusable and preventing the user from reusing the original raw data for other purposes. This is exactly the situation I'm currently facing.
Thanks!

[Psy-Fer]

Something to note, that using pre-allocated memory buffers for variable read data can lead to tricky outcomes that are difficult to debug. This limits how the software is run depending on the situation and input data. When developing software, we take the apporach that the most portable, stable, and user friendly method wins out over other considerations like performance. We could hand write some of the methods in assembly to get better performance, but the cost of doing so is too high.

It may be helpful for us to have more information about what you are trying to do/achieve so we can better advise you and your use of slow5lib.

I'll let @hasindu2008 respond specifically to the question about slow5_rec_free

Cheers,
James