Example for ESP

[perotom]

I currently have a ESP project where I write constantly to a file with the following function:

bool startTracking() {
    file = LittleFS.open("Test.bin", "w");
    if (!file) {
        return false;
    }
    return true;
}
bool writeTrackingFile(const uint8_t *buffer, size_t length) {
    return file.write(buffer, length) == length;
}
bool stopTracking() {
    file.flush();
    file.close();
    return true;
}

The function writeTrackingFile will be called on different length data. Now I want to add the tamp compressor before writing to the file. My approach would be:

const uint8_t compressorWindowBits = 10;
uint8_t compressorWindowBuffer[1 << compressorWindowBits];
uint8_t compressorOutputBuffer[64];
TampCompressor compressor;
TampConf compressorConf = {
    .window = compressorWindowBits,
    .literal = 8,
    .use_custom_dictionary = false
};

bool startTracking() {
    file = LittleFS.open("Test.bin", "w");
    if (!file) {
        return false;
    }
    if (tamp_compressor_init(&compressor, &compressorConf, compressorWindowBuffer) != TAMP_OK) {
        ESP_LOGE(TAG, "Failed to initialize compressor");
        return false;
    }
    return true;
}
bool writeTrackingFile(const uint8_t *buffer, size_t length) {
    // TODO: Lost on how to write what to the buffer. Should I use tamp_compressor_compress_and_flush? or non flush version with sink and poll and only flush in stopTracking()?
    return file.write(buffer, length) == length;
}
bool stopTracking() {
    // TODO: Need to flush the compressor to get internal bytes?
    file.flush();
    file.close();
    return true;
}

Thank you for your help! This library looks quite promising. Later on I want to test the high speed decompression with Python C bindings.

[BrianPugh]

Hi @perotom!

At a high level, i think this looks good. Inside your writeTrackingFile, the most efficient (smallest ram usage) will look similar to the tamp_compressor_compress_cb implementation. Below is my first draft (totally untested code!) at what it should look like.

#define COMPRESSOR_OUTPUT_BUFFER_SIZE 4 // technically this could be as low as 3, but we like our powers of 2.
uint8_t compressorOutputBuffer[COMPRESSOR_OUTPUT_BUFFER_SIZE];

bool writeTrackingFile(const uint8_t *buffer, size_t length) {
  while (length > 0) {
    {
      // Sink Data into input buffer.
      size_t consumed;
      tamp_compressor_sink(compressor, buffer, length, &consumed);
      // no error checking required here; this always succeeds.
      buffer += consumed;
      length -= consumed;
    }
    // TODO: I should add a "tamp_compressor_full" function to check the
    // compressor state
    if (compressor->input_size == sizeof(compressor->input)) {
      // Input buffer is full and ready to start compressing.
      size_t chunk_output_written_size;
      tamp_res_t res;
      res = tamp_compressor_poll(compressor, compressorOutputBuffer,
                                 COMPRESSOR_OUTPUT_BUFFER_SIZE,
                                 &chunk_output_written_size);
      if (res != TAMP_OK) {
        return false;
      }

      // Write the few bytes in the output buffer to disk.
      file.write(compressorOutputBuffer, chunk_output_written_size);
      // TODO: error check file writing.
    }
  }
}

bool stopTracking() {
  size_t chunk_output_written_size;
  do{
    // flush the input and output buffers until data stops coming out.
    tamp_res_t res;
    tamp_compressor_flush(compressor, compressorOutputBuffer, COMPRESSOR_OUTPUT_BUFFER_SIZE, &chunk_output_written_size, false);
    file.write(compressorOutputBuffer, chunk_output_written_size);
    // TODO: error check file writing.
  }while(chunk_output_written_size);

  file.flush();
  file.close();
  return true;
}

semi-related; i've been wanting to add a tamp-interface to littlefs, but just haven't had time. Many applications have completely linear reads/writes, which would be perfect for tamp. Let me know if you run into any issues (or successes)!

[BrianPugh]

with the newly released v1.7.0 version, you can replace

if (compressor->input_size == sizeof(compressor->input)) {

with

if (tamp_compressor_full(compressor) {

[BrianPugh]

you know what, this has inspired me enough to write a file-api to simplify this fairly common use-case. So expect that in a few days.

[perotom]

This sounds great! Looking forward to try it 🥳

[BrianPugh]

My personal schedule has actually gotten a little hectic; i'll still work on this but i'll say it's "within a month" thing rather than "few days"