persistent storage

Doxyfile

@@ -9,6 +9,7 @@ EXTRACT_STATIC         = YES
 CITE_BIB_FILES         = doc/bibliography
 INPUT                  = .
 RECURSIVE              = YES
+EXCLUDE                = test/
 EXAMPLE_PATH           = doc/debouncing/
 IMAGE_PATH             = doc/
 USE_MDFILE_AS_MAINPAGE = ./README.md

data/usb_options.md

@@ -0,0 +1,41 @@
+# Overview of Build Options for ESP32-S3
+
+## 1. USB Mode (`ARDUINO_USB_MODE`)
+
+| Value | Label |
+|------|----------------------------|
+| `0`  | USB-OTG (TinyUSB) (Default) |
+| `1`  | Hardware CDC and JTAG |
+
+**Default value:** `0`  
+**Macro:** `-DARDUINO_USB_MODE={build.usb_mode}`
+
+## 2. USB CDC On Boot (`ARDUINO_USB_CDC_ON_BOOT`)
+
+| Value | Label |
+|------|------------|
+| `0`  | Disabled (Default) |
+| `1`  | Enabled |
+
+**Default value:** `0`  
+**Macro:** `-DARDUINO_USB_CDC_ON_BOOT={build.cdc_on_boot}`
+
+## 3. USB Firmware MSC On Boot (`ARDUINO_USB_MSC_ON_BOOT`)
+
+| Value | Label |
+|------|--------------------------------|
+| `0`  | Disabled (Default) |
+| `1`  | Enabled (Requires USB-OTG Mode) |
+
+**Default value:** `0`  
+**Macro:** `-DARDUINO_USB_MSC_ON_BOOT={build.msc_on_boot}`
+
+## 4. USB DFU On Boot (`ARDUINO_USB_DFU_ON_BOOT`)
+
+| Value | Label |
+|------|--------------------------------|
+| `0`  | Disabled (Default) |
+| `1`  | Enabled (Requires USB-OTG Mode) |
+
+**Default value:** `0`  
+**Macro:** `-DARDUINO_USB_DFU_ON_BOOT={build.dfu_on_boot}`

doc/decisions/dr-005.md

@@ -0,0 +1,100 @@
+# DR005 USB MSC Integration with FFat on ESP32-S3
+
+## Context
+
+This ADR documents the analysis, findings, and decisions regarding the implementation of **USB Mass Storage Class (MSC)** support on an **ESP32-S3** device using **FFat (ESP32 FAT on Flash File System)**. The objective is to enable the ESP32 to act as a mass storage device, allowing:
+
+- **USB host (e.g., a computer) to access stored files** as if the ESP32 were a USB flash drive.
+- **The ESP32 application to access and modify files via FFat.**
+- **Synchronization between USB MSC access and internal application access.**
+
+The project is built with **PlatformIO** and uses the **Arduino-ESP32** framework.
+
+## Considered Approaches
+
+### Existing Examples & Research
+
+Several existing projects and examples were analyzed for feasibility:
+
+1. [Espressif's "TinyUSB Mass Storage Device Example" (ESP-IDF)](https://github.com/espressif/esp-idf/tree/v5.3.2/examples/peripherals/usb/device/tusb_msc#tinyusb-mass-storage-device-example)
+   - Uses `esp_tinyusb`, which integrates with ESP-IDF.
+   - Relies on `esp_partition_write()` via ESP-IDF’s Wear Leveling API.
+   - **Issue:** Requires ESP-IDF; not directly compatible with Arduino-ESP32.
+
+2. [chegewara/EspTinyUSB "flashdisk" Example](https://github.com/chegewara/EspTinyUSB/blob/f4d63153c0417922398de2fe43114b5224ff6def/examples/device/msc/flashdisk/flashdisk.ino)
+   - Implements USB MSC with `disk_read()` and `disk_write()` from ESP-IDF FatFS.
+   - **Issue:** Library is not well-maintained and might be unstable.
+
+3. [Espressif's "SD2USBMSC" Example](https://raw.githubusercontent.com/espressif/arduino-esp32/a7907cd07e605e4e9e37f0ec862e7da7a145fa38/libraries/SD_MMC/examples/SD2USBMSC/SD2USBMSC.ino)
+   - Uses SD card storage with raw block access.
+   - **Issue:** Designed for SD cards, not for internal flash.
+
+4. [Adafruit's "msc_esp32_file_browser" Example](https://raw.githubusercontent.com/adafruit/Adafruit_TinyUSB_Arduino/refs/tags/3.4.2/examples/MassStorage/msc_esp32_file_browser/msc_esp32_file_browser.ino)
+   - Exposes flash storage via USB MSC and a web server.
+   - Uses `Adafruit_SPIFlash`, which relies on TinyUSB.
+   - **Issue:** Introduces additional dependencies; Arduino-ESP32 already includes TinyUSB, causing conflicts.
+
+### Observations & Problems Identified
+
+- Arduino-ESP32 framework includes TinyUSB, but PlatformIO does not allow excluding components.
+- FFat does not expose `disk_read()` or `disk_write()` for direct block-level access.
+- Direct partition access (`esp_partition_read()` and `esp_partition_write()`) is needed for USB MSC.
+- Concurrency between USB MSC and FFat must be managed to avoid filesystem corruption.
+- ESP-IDF solutions are not directly compatible with Arduino-ESP32.
+
+## Decision
+
+**Chosen Approach:** Direct Partition Access with FFat Synchronization
+
+- USB MSC reads/writes directly to the flash partition using `esp_partition_write()` and `esp_partition_read()`.
+- Application accesses files via FFat, using `FFat.begin()` and `FFat.end()` for synchronization.
+- Synchronization via unmount/mount cycle (`FFat.end()` before USB access, `FFat.begin()` after).
+- FreeRTOS Task for File System Operations
+  - A dedicated FreeRTOS task processes USB events.
+  - USB events do not perform filesystem operations directly.
+
+## Implementation Details
+
+### Core Components
+
+- USB MSC Read/Write Callbacks
+- File System Synchronization Task
+
+## Alternative Considered but Rejected
+
+| Approach | Reason for Rejection |
+|----------|----------------------|
+| Using `esp_tinyusb` | Requires ESP-IDF, not compatible with Arduino-ESP32 |
+| Adafruit TinyUSB | Introduces conflicts with built-in TinyUSB in Arduino-ESP32 |
+| Direct FFat Access via `fread()` | FFat does not support block-level read/write |
+
+## Consequences
+
+### Positive Outcomes
+
+- ✅ Works within **Arduino-ESP32 and PlatformIO**.
+- ✅ Uses **built-in ESP32 partition management**.
+- ✅ Keeps **dependencies minimal**.
+- ✅ Provides **clean USB MSC integration**.
+
+### Potential Issues
+
+- ⚠ **FFat remounting may cause latency** when switching access modes.
+- ⚠ **USB MSC cannot modify FFat metadata dynamically**, requiring unmount/remount.
+- ⚠ **Large writes via USB MSC could impact flash lifespan** as no wear leveling is used.
+
+## References
+
+- [Task-Tracker-Device Issue #6](https://github.com/Task-Tracker-Systems/Task-Tracker-Device/issues/6)
+- [Espressif TinyUSB MSC Example](https://github.com/espressif/esp-idf/tree/v5.3.2/examples/peripherals/usb/device/tusb_msc)
+- [Adafruit TinyUSB Arduino](https://github.com/adafruit/Adafruit_TinyUSB_Arduino)
+- [ESP32 FatFS](https://github.com/espressif/esp-idf/tree/c71d74e2f853b1135c63f47e349f2a08f63f3e01/components/fatfs)
+- [USB MSC example from Espressif for Arduino-ESP32](https://github.com/espressif/arduino-esp32/blob/988dbe29731e2a2d09db2ed642c06271afa93705/libraries/USB/examples/USBMSC/USBMSC.ino)
+- [issue in Adafruit_TinyUSB_Arduino with macro redefinitions for ESP32S2 and ESP32S3 (#484)](https://github.com/adafruit/Adafruit_TinyUSB_Arduino/issues/484)
+- [issue in Adafruit_TinyUSB_Arduino with Linker Error on esp32s3 with platformIO (#473)](https://github.com/adafruit/Adafruit_TinyUSB_Arduino/issues/473)
+- [example for using USB MSC for firmware updates](https://github.com/espressif/arduino-esp32/blob/988dbe29731e2a2d09db2ed642c06271afa93705/libraries/USB/examples/FirmwareMSC/FirmwareMSC.ino)
+- [example for using LittleFS with PlatformIO and Arduino-ESP32](https://github.com/espressif/arduino-esp32/tree/6ce43695d2591e600dc906eeb9e30a9ca3c71921/libraries/LittleFS/examples/LITTLEFS_PlatformIO)
+- ["FSBrowser" example providing SPIFFS via Web](https://raw.githubusercontent.com/espressif/arduino-esp32/a7907cd07e605e4e9e37f0ec862e7da7a145fa38/libraries/WebServer/examples/FSBrowser/FSBrowser.ino)
+- [Espressif's "TinyUSB Mass Storage Device Example" für ESP-IDF](https://github.com/espressif/esp-idf/tree/v5.3.2/examples/peripherals/usb/device/tusb_msc#tinyusb-mass-storage-device-example)
+- [Espressif's additions to TinyUSB "esp_tinyusb"](https://components.espressif.com/components/espressif/esp_tinyusb) extends [espressif/tinyusb](https://github.com/espressif/tinyusb)
+- [Wear Levelling API](https://github.com/espressif/esp-idf/tree/1160a86ba0b87b0ea68ad6b91eb10fe9c34ad692/components/wear_levelling)

lib/board_adapters/storage/storage.cpp

@@ -0,0 +1,190 @@
+#include "storage.hpp"
+#include <FFat.h>
+#include <USB.h>
+#include <USBMSC.h>
+#include <atomic>
+#include <cassert>
+#include <condition_variable>
+#include <cstdint>
+#include <esp32-hal-log.h>
+#include <esp_err.h>
+#include <esp_partition.h>
+#include <iostream>
+#include <mutex>
+#include <thread>
+
+#if defined(ARDUINO_USB_MODE)
+static_assert(ARDUINO_USB_MODE == 0, "USB must be in OTG mode");
+#endif
+
+const esp_partition_t *check_ffat_partition(const char *label); // defined in FFat.cpp
+
+static USBMSC usbMsc;
+
+static constexpr std::uint16_t blockSize = 512; // Should be 512
+
+static const esp_partition_t *partition = nullptr;
+static const char *const TAG = "STORAGE";
+
+static std::condition_variable stateChangeRequested;
+static std::condition_variable stateChanged;
+static std::atomic<bool> requestFileSystemActive;
+static std::thread stateMachine;
+static std::mutex fileSystemState_mutex; //!< used to lock the actual state
+static bool fileSystemIsActive;          //!< describes the current state
+
+std::shared_ptr<fs::FS> Storage::getFileSystem_locking()
+{
+    std::unique_lock fs_state_lock{fileSystemState_mutex};
+    if (!fileSystemIsActive)
+    {
+        stateChanged.wait(fs_state_lock, []() { return fileSystemIsActive; });
+    }
+    fs_state_lock.release();
+    return {&FFat, [](fs::FS *) { fileSystemState_mutex.unlock(); }};
+}
+
+static void requestState(const bool fileSystemActive)
+{
+    requestFileSystemActive = fileSystemActive;
+    ESP_LOGD(TAG, "request new state: %s", fileSystemActive ? "true" : "false");
+    stateChangeRequested.notify_all();
+}
+
+static void processStateRequests()
+{
+    while (true)
+    {
+        std::unique_lock fs_state_lock{fileSystemState_mutex};
+        ESP_LOGD(TAG, "waiting for state change request");
+        stateChangeRequested.wait(fs_state_lock, []() { return requestFileSystemActive != fileSystemIsActive; });
+        if (fileSystemIsActive = requestFileSystemActive)
+        {
+            ESP_LOGI(TAG, "mount FS");
+            usbMsc.mediaPresent(false);
+            FFat.end();           // invalidate cache
+            assert(FFat.begin()); // update data
+        }
+        else
+        {
+            ESP_LOGI(TAG, "unmount FS");
+            FFat.end(); // flush and unmount
+            usbMsc.mediaPresent(true);
+        }
+        stateChanged.notify_all();
+    }
+}
+
+/**
+ * Callback invoked when received WRITE10 command.
+ *
+ * Process data in buffer to disk's storage.
+ *
+ * @param lba logical block address
+ * @returns the number of written bytes (must be multiple of block size)
+ */
+static std::int32_t usbMsc_onWrite(const std::uint32_t lba, const std::uint32_t offset, std::uint8_t *const buffer,
+                                   const uint32_t bufsize)
+{
+    ESP_LOGV(TAG, "MSC WRITE: lba: %u, offset: %u, bufsize: %u\n", lba, offset, bufsize);
+    const std::uint32_t byteOffset = lba * blockSize + offset;
+    ESP_ERROR_CHECK(esp_partition_erase_range(partition, byteOffset, bufsize)); // erase must be called before write
+    ESP_ERROR_CHECK(esp_partition_write(partition, byteOffset, buffer, bufsize));
+    return bufsize;
+}
+
+/**
+ * Callback invoked when received READ10 command.
+ *
+ * Copy disk's data to buffer (up to bufsize).
+ *
+ * @param lba logical block address
+ * @returns the number of copied bytes (must be multiple of block size)
+ */
+static std::int32_t usbMsc_onRead(const std::uint32_t lba, const std::uint32_t offset, void *const buffer,
+                                  const std::uint32_t bufsize)
+{
+    ESP_LOGV(TAG, "MSC READ: lba: %u, offset: %u, bufsize: %u\n", lba, offset, bufsize);
+    const std::uint32_t byteOffset = lba * blockSize + offset;
+    ESP_ERROR_CHECK(esp_partition_read(partition, byteOffset, buffer, bufsize));
+    return bufsize;
+}
+
+static bool usbMsc_onStartStop(const std::uint8_t power_condition, const bool start, const bool load_eject)
+{
+    ESP_LOGV(TAG, "MSC START/STOP: power: %u, start: %u, eject: %u\n", power_condition, start, load_eject);
+    return true;
+}
+
+std::size_t Storage::size()
+{
+    return FFat.totalBytes();
+}
+
+static std::atomic<bool> usbIsRunning = false;
+static void usbStoppedCallback(void *, esp_event_base_t, int32_t, void *)
+{
+    if (!usbIsRunning)
+    {
+        return;
+    }
+    usbIsRunning = false;
+    requestState(true);
+}
+static void usbStartedCallback(void *, esp_event_base_t, int32_t, void *)
+{
+    usbIsRunning = true;
+    requestState(false);
+}
+
+void Storage::begin()
+{
+
+    if (!FFat.begin(true))
+    {
+        ESP_LOGE(TAG, "Failed to init files system, flash may not be formatted");
+        return;
+    }
+    ESP_LOGI(TAG, "file system initialized");
+
+    partition = check_ffat_partition(FFAT_PARTITION_LABEL);
+    if (!partition)
+    {
+        ESP_LOGE(TAG, "Error with FAT partition");
+        return;
+    }
+    ESP_LOGI(TAG, "Flash has a size of %u bytes\n", FFat.totalBytes());
+
+    // define state before callbacks are activated
+    fileSystemIsActive = true;
+    requestFileSystemActive = true;
+    stateMachine = std::thread(processStateRequests);
+
+    usbMsc.vendorID("ESP32");      // max 8 chars
+    usbMsc.productID("USB_MSC");   // max 16 chars
+    usbMsc.productRevision("1.0"); // max 4 chars
+    usbMsc.onStartStop(usbMsc_onStartStop);
+    usbMsc.mediaPresent(false);
+    // Set callback
+    usbMsc.onRead(usbMsc_onRead);
+    usbMsc.onWrite(usbMsc_onWrite);
+    USB.onEvent(ARDUINO_USB_STARTED_EVENT, usbStartedCallback);
+    USB.onEvent(ARDUINO_USB_STOPPED_EVENT, usbStoppedCallback);
+
+    // Set disk size, block size should be 512 regardless of spi flash page size
+    if (!usbMsc.begin(FFat.totalBytes() / blockSize, blockSize))
+    {
+        ESP_LOGE(TAG, "starting USB MSC failed");
+    }
+    if (!USB.begin())
+    {
+        ESP_LOGE(TAG, "starting USB failed");
+    }
+}
+
+void Storage::end()
+{
+    usbMsc.end();
+    usbIsRunning = false;
+    FFat.end();
+}

lib/board_adapters/storage/storage.hpp

@@ -0,0 +1,16 @@
+#pragma once
+#include <cstddef>
+#include <memory>
+
+namespace fs
+{
+class FS;
+}
+
+struct Storage
+{
+    static void begin();
+    static void end();
+    static std::shared_ptr<fs::FS> getFileSystem_locking();
+    static std::size_t size();
+};

platformio.ini

@@ -20,6 +20,7 @@ build_flags =
 	${env.build_flags}
 	-DLV_CONF_PATH="${PROJECT_DIR}/lib/3rd_party_adapters/LVGL/lv_conf.h" ; lvgl: use this config file
 	-DBAUD_RATE=${this.monitor_speed}
+board_build.filesystem = fatfs
 monitor_speed = 115200
 
 [env:native]
@@ -38,10 +39,26 @@ build_flags =
 	-fprofile-abs-path
 	-O0
 	-ggdb3
+	-Itest/doubles
 
 [env:esp32-s3-devkitc-1]
-platform = espressif32
+platform = espressif32@6.9.0
 board = esp32-s3-devkitc-1
 framework = arduino
 extends = production
+board_build.partitions = default_ffat_8MB.csv
 monitor_filters = esp32_exception_decoder
+build_unflags =
+	${production.build_unflags}
+    -DARDUINO_USB_MODE=1
+    -DARDUINO_USB_CDC_ON_BOOT=0
+    -DARDUINO_USB_MSC_ON_BOOT=1
+    -DARDUINO_USB_DFU_ON_BOOT=1
+build_flags = 
+	${production.build_flags}
+    -DARDUINO_USB_MODE=0
+    -DARDUINO_USB_CDC_ON_BOOT=1
+    -DARDUINO_USB_MSC_ON_BOOT=0
+    -DARDUINO_USB_DFU_ON_BOOT=0
+    -DCONFIG_ARDUHAL_LOG_COLORS=1
+    -DCORE_DEBUG_LEVEL=ARDUHAL_LOG_LEVEL_INFO