Skip to content

Simplify API and optimize bulk data copies #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 60 commits into from
Oct 20, 2025
Merged

Simplify API and optimize bulk data copies #154

merged 60 commits into from
Oct 20, 2025

Conversation

@cboulay
Copy link
Collaborator

@cboulay cboulay commented Oct 16, 2025

This is a major API-breaking change to the C-API, the Python API, and how things work internally.

Some highlights:

  • Continuous data are now in a single contiguous buffer, shape [samples][channels] -- this enables much more efficient bulk copy of the data
  • All trial_ data structures are now populated with the timestamp at the beginning of the trial, which might have been a while ago
  • The timestamps associated with the data are now the absolute timestamps that came from the device
  • You can now only pull data from one sample group per API call
  • Channels can belong to more than one sample group (e.g., 3 and 6)
  • There is no longer an option to return the data as double. This wasn't working properly anyway.
  • In the Python API, the user can now supply their own data and timestamp buffers to reduce the number of allocations.

There's still some work to be done and mostly some testing to do.

cboulay added 26 commits October 15, 2025 14:55
@cboulay
Move from "per-channel arrays shared across groups" to "per-group str…
6d02e8f 
…uctures containing channel arrays".

This uses less memory, _should_ have better cache performance, and supports multiple groups (e.g., group 3 and group 6).
@cboulay
Move new continuous data structures into their own header (for easier…
5fd400d 
… testing).
@cboulay
Add test for continuous data structure.
a281063
@cboulay
GroupContinuousData - transpose channel_data.
13be22d
@cboulay
cpp-ify ContinuousData
86b7f87
@cboulay
clang-tidy fixes
4846554
@cboulay
API-breaking refactor!
f936e5e 
* data are now contiguous - bulk copy
* no more bAbsolute or bDouble
* full timestamp vector for continuous data
@cboulay
Port recent API-breaking change to Python / Cython binding.
a05aecd
@cboulay
Fix unit test after refactor.
3d802e5
@cboulay
Try to fix tests on MSVC
40b105e
@cboulay
Fix tests on Windows
a209e11
@cboulay
Fixing logic in simple_io.cpp and found off-by-one error.
6abe55e
@cboulay
Increase test timing tolerance.
220c1cc
@cboulay
Add an application to test closed loop data generation and reading.
ce1bcd4
@cboulay
Move continuous data buffer management logic to ContinuousData.cpp an…
5f5ce04 
…d add more tests.
@cboulay
Eliminate storage of redundant sample rate information.
4826a3c
@cboulay
Refactored from single global mutex to per-group mutexes for better p…
fb5beac 
…arallelism
@cboulay
Attempt to find source of bad packet timestamps
63c05e9
@cboulay
Fix a couple include paths
23f4425
@cboulay
Eliminate concept of bLoopbackPacket from packet-processing thread.…
88ac06f 
… It was definitely incorrect so we shouldn't miss anything.
@cboulay
clang-tidy
4ff7219
@cboulay
Simpler cbsdk_sawtooth tester; only works with legacy / nPlay.
c55f050
@cboulay
Default to not building cbmex / cboct
3b87703
@cboulay
Disable Tools build on GHA
81628a6
@cboulay
tools - sawtooth tester - includes
c75d674
@cboulay
Fix test errors on Windows
053820e
@cboulay cboulay force-pushed the trial_cont_timestamp branch from 44d3941 to 053820e Compare October 17, 2025 22:50
@cboulay cboulay marked this pull request as ready for review October 18, 2025 16:58
cboulay added 15 commits October 18, 2025 13:40
@cboulay
Stricter typing of timestamps
df0b917
@cboulay
Better event array management in simple_io.cpp
8c2f47f
@cboulay
Tidy up examples
85134eb
@cboulay
Match API types to structure types
b3fbd08
@cboulay
Lots of tidying up
9198efa
@cboulay
Clarify reset/active behaviour in trial data API
fb26aa5
@cboulay
Move EventData structure from SdkApp.h to its own class.
5d9e8e5
@cboulay
clang-tidy
45e1b8c
@cboulay
Continue moving functionality from cbsdk.cpp into EventData.cpp
903804a
@cboulay
refactoring of EventData from per-channel to flat time-series layout
eb38a0a
@cboulay
Improve intuition of read/write management in EventData.
68db9f3
@cboulay
Added spikes_lsl.py to generate spiking data and spikes_cbsdk.cpp to …
446f70c 
…validate the data. There are still some bugs to work out in the pattern but overall the API is working.
@cboulay
Link LSL statically.
eb6897d
@cboulay
Fix tests' lambda captures in Windows
1b501fc
@cboulay
EventData - bulk memcpy
9a49f22
@cboulay
Copy link
Collaborator Author

cboulay commented Oct 20, 2025

Here goes nothing...

@cboulay cboulay merged commit b819c49 into master Oct 20, 2025
10 checks passed
@cboulay cboulay deleted the trial_cont_timestamp branch October 20, 2025 05:35
@cboulay cboulay mentioned this pull request Oct 20, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cboulay