Add ReadAnalogWaveforms to NiDAQmxService #1203

mikeprosserni · 2025-10-16T18:39:32Z

What does this Pull Request accomplish?

Added ReadAnalogWaveforms method to the NiDAQmxService.
Updated code generation with support for CustomCodeNoLibrary. This is needed for ReadAnalogWaveforms because it doesn't have a matching library function. (Instead, it calls library_->InternalReadAnalogWaveformPerChan(), which has very different parameters)
Introduced new waveform attributes and functions in metadata.
Updated CMake build process to include new waveform protobuf files.
Added Ninja build instructions to CONTRIBUTING.md.

Why should this Pull Request be merged?

What testing has been done?

PS C:\dev\fireserp\grpc-device> .\build\Debug\SystemTestsRunner.exe --gtest_filter="*ReadAnalogWaveforms*"                                                             
Note: Google Test filter = *ReadAnalogWaveforms*
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from NiDAQmxDriverApiTests
[ RUN      ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithNoAttributeMode_ReturnsWaveformData
[       OK ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithNoAttributeMode_ReturnsWaveformData (4493 ms)
[ RUN      ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithTimingMode_ReturnsWaveformDataWithTimingInfo
[       OK ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithTimingMode_ReturnsWaveformDataWithTimingInfo (128 ms)
[ RUN      ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithExtendedPropertiesMode_ReturnsWaveformDataWithAttributes
[       OK ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithExtendedPropertiesMode_ReturnsWaveformDataWithAttributes (15 ms)
[ RUN      ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithTimingAndExtendedPropertiesMode_ReturnsWaveformDataWithTimingAndAttributes
[       OK ] NiDAQmxDriverApiTests.ReadAnalogWaveforms_WithTimingAndExtendedPropertiesMode_ReturnsWaveformDataWithTimingAndAttributes (81 ms)
[----------] 4 tests from NiDAQmxDriverApiTests (4746 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (4754 ms total)
[  PASSED  ] 4 tests.

- Added ReadAnalogWaveforms method stub to NiDAQmxService for reading analog waveforms. - Updated metadata validation to include CustomCodeNoLibrary. - Introduced new waveform attributes and functions in metadata. - Enhanced CMake configuration for new protobuf files. - Improved CONTRIBUTING.md with Ninja build instructions.

mikeprosserni · 2025-10-21T17:22:48Z

.github/workflows/build_cmake.yml

            glibc_version: "2_31",
            cmake_generator: '-G "Ninja"',
-            build_type: RelWithDebInfo
+            build_type: RelWithDebInfo,


I made these build config changes in order to get nidaqmx_library.cpp to successfully compile on Ubuntu. As far as I can tell, the problem is that after I added the new library functions in functions.py, the nidaqmx_library.cpp became too large to compile with the original settings. It was either timing out or running out of memory or something.

I'm not sure this is the best way to fix the issue, however. Changing the optimization from O2 to O1 means the grpc device server will be less optimized for everyone on linux, right? Another approach that might work is splitting nidaqmx_library.cpp into multiple smaller .cpp files somehow. What do the reviewers think of that?

When you build it locally, does it print any more output, such as

note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without

In the past I've sometimes had to use -fno-var-tracking-assignments to compile large code-generated files. I don't know if this option is still relevant or there is a different option nowadays.

Also, how long does it take to build locally with -O2? Does it eventually succeed?

I don't have a linux machine to build locally on, I've been relying on the CI build for this. All I can see is this, after 33 minutes:

[2601/3669] Building CXX object CMakeFiles/IntegrationTestsRunner.dir/generated/nidaqmx/nidaqmx_library.cpp.o ninja: build stopped: interrupted by user. Error: Process completed with exit code 143.

https://github.com/ni/grpc-device/actions/runs/18659129011/job/53195445860

or, when I added --verbose:

[2601/3669] /usr/bin/g++-9 -DCARES_STATICLIB -DTEST_API_BUILDING -DTestApi -D_CRT_SECURE_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS -Iproto -I.././generated -I.././imports/include -I.././source -I.././third_party/grpc-sideband/src -I.././third_party/grpc-sideband/moniker_src -I../third_party/grpc/include -I../third_party/grpc/third_party/abseil-cpp -I../third_party/grpc/third_party/re2 -Igrpc/third_party/cares/cares -I../third_party/grpc/third_party/cares/cares -I../third_party/grpc/third_party/cares/cares/include -I../third_party/grpc/third_party/boringssl-with-bazel/src/include -I../third_party/grpc/third_party/protobuf/src -I../third_party/utfcpp/source -I../third_party/grpc-sideband/src -I../third_party/grpc-sideband/moniker_src -I../third_party/json/include -isystem ../third_party/googletest/googlemock/include -isystem ../third_party/googletest/googlemock -isystem ../third_party/googletest/googletest/include -isystem ../third_party/googletest/googletest -O2 -g -DNDEBUG -std=c++17 -MD -MT CMakeFiles/IntegrationTestsRunner.dir/generated/nidaqmx/nidaqmx_library.cpp.o -MF CMakeFiles/IntegrationTestsRunner.dir/generated/nidaqmx/nidaqmx_library.cpp.o.d -o CMakeFiles/IntegrationTestsRunner.dir/generated/nidaqmx/nidaqmx_library.cpp.o -c ../generated/nidaqmx/nidaqmx_library.cpp ninja: build stopped: interrupted by user. Error: Process completed with exit code 143.

https://github.com/ni/grpc-device/actions/runs/18663525332/job/53209586512

I did some googling and some guessing, and figured resource exhaustion was the most likely explanation for exit code 143. So I changed the build parameters to see if that would get the CI to build successfully, and it did. That's all I know at this point.

VTA is a potential cause of resource exhaustion. This feature tracks variable locations for each line of code in order to produce debug info that accurately shows the variable values when you step through the function, even if it's inlined. However, when your code has hundreds of variable assignments in the same function, this feature can take a lot of CPU and memory. I don't know exactly why.

VTA only affects the debugging experience, not execution speed, so if turning it off solves your problem, then it's a better solution than using -O1 and turning off a bunch of optimizations.

Please try updating https://github.com/ni/grpc-device/blob/main/source/codegen/templates/service.cpp.mako to use #pragma GCC optimize to scope the -fno-var-tracking-assignments or -O1 to *_library.cpp. You will need to guard this pragma with #ifdef __GNUC__ or similar.

In this PR, I've reverted the changes to build_cmake and implemented your suggested changes to service.cpp.mako, and nidaqmx_library.cpp is still compiling without crashing.

As another data point, I tried a different strategy in a different PR, where I reverted the other changes in build_cmake, and just used -j ${{ runner.os == 'Linux' && '2' || steps.cpu-cores.outputs.count }}, which also allwed nidaqmx_library.cpp to compile without crashing.

I checked out your branch on my laptop, commented out the -fno-var-tracking-assignments, and ran time cmake --build . -j 16 --config RelWithDebInfo on WSL:

real 21m28.891s user 298m4.721s sys 30m30.097s

Running .ninja_log through https://github.com/nico/ninjatracing shows

It seems like all of the DAQmx files are problem files, taking >3 minutes to compile.

…s instead

source/codegen/templates/service.cpp.mako

bkeryan · 2025-10-22T18:14:16Z

source/codegen/templates/service.cpp.mako

Wait, which file is causing the problem? service.cpp or library.cpp? I thought you said it was library.cpp.

nidaqmx_library.cpp was the problem. Although the changes made in service.cpp.mako seems to have fixed it: https://github.com/ni/grpc-device/actions/runs/18724889717/job/53406944072?pr=1203

[2600/3669] Building CXX object CMakeFiles/IntegrationTestsRunner.dir/proto/data_moniker.grpc.pb.cc.o [2601/3669] Building CXX object CMakeFiles/IntegrationTestsRunner.dir/generated/nidaqmx/nidaqmx_library.cpp.o [2602/3669] Building CXX object CMakeFiles/IntegrationTestsRunner.dir/generated/nidaqmx/nidaqmx_service_registrar.cpp.o

In my other PR, I'm trying out those changes in library.cpp.mako instead of service.cpp.mako: https://github.com/ni/grpc-device/pull/1205/files#diff-0cb12eeb765640c93fc98cdc6777d834a23c8042f953a4e41186bc87b72fa252

https://github.com/ni/grpc-device/actions/runs/18726362277/job/53412058096?pr=1205

(note, this did not work, the build failed)

That's weird. NiDAQmxLibrary::NiDAQmxLibrary(std::shared_ptr<nidevice_grpc::SharedLibraryInterface> shared_library) is only called from NiDAQmxLibrary::NiDAQmxLibrary() and doesn't seem to be inlined in the v2.13.0 release binaries.

That said, the build time increased by 20 minutes, so I think there may be multiple problem files. nidaqmx_library.cpp may have been the one that pushed the GitHub-hosted runner over the edge and made it run out of memory, but other files may be taking several more minutes than before.

If disabling VTA fixes the build, then I think it's worth seeing what happens when you disable VTA globally instead of per-file.

That said, the build time increased by 20 minutes, so I think there may be multiple problem files. nidaqmx_library.cpp may have been the one that pushed the GitHub-hosted runner over the edge and made it run out of memory, but other files may be taking several more minutes than before.

Oops, I was looking at the -j 2 change. Yeah, that's going to slow down the build.

Should I still try disabling VTA globally?

Ok, I've updated this PR to disable VTA globally. (since the changes with push/pop didn't work)

I reverted all other build changes, and added this to CMakeLists.txt:

# GCC-specific flags if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-var-tracking-assignments") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-var-tracking-assignments") endif()

(Note: This worked, the Ubuntu build succeeded after 66 minutes)

Mike Prosser added 5 commits October 16, 2025 13:32

fix lint

fff7769

fix validate_examples

01ea5e9

ReadAnalogWaveforms implementation (untested, but compiles)

1270fd7

add tests

836c408

mikeprosserni changed the title ~~[Under Construction] Add ReadAnalogWaveforms to NiDAQmxService~~ Add ReadAnalogWaveforms to NiDAQmxService Oct 16, 2025

Mike Prosser added 3 commits October 20, 2025 10:54

Add support for creating two AI voltage channels in tests

b9cf67a

cleanup

12ca5e5

cleanup CMakeLists.txt

8e5eb77

mikeprosserni marked this pull request as ready for review October 20, 2025 17:07

mikeprosserni requested review from astarche, maxxboehme and reckenro as code owners October 20, 2025 17:07

Mike Prosser added 4 commits October 20, 2025 14:12

attempt fix for ubuntu build

7b50a84

more verbose build output

3d29cbc

Try some different build settings

9b26670

add quotes to fix cmake flags

d117ec2

mikeprosserni commented Oct 21, 2025

View reviewed changes

Mike Prosser added 2 commits October 22, 2025 10:39

add sampsPerChanRead to ReadAnalogWaveforms response

23a6e98

revert changes in build_cmake.yml and do -fno-var-tracking-assignment…

1577116

…s instead

bkeryan reviewed Oct 22, 2025

View reviewed changes

source/codegen/templates/service.cpp.mako Outdated Show resolved Hide resolved

source/codegen/templates/service.cpp.mako Outdated Show resolved Hide resolved

bkeryan reviewed Oct 22, 2025

View reviewed changes

Mike Prosser added 3 commits October 22, 2025 13:19

use GCC push_options

c03a50d

use -fno-var-tracking-assignments globally for gcc

4c36ac8

fix timing conversion

e7055d1

Add ReadAnalogWaveforms to NiDAQmxService #1203

Are you sure you want to change the base?

Add ReadAnalogWaveforms to NiDAQmxService #1203

Uh oh!

Conversation

mikeprosserni commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this Pull Request accomplish?

Why should this Pull Request be merged?

What testing has been done?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikeprosserni Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikeprosserni Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikeprosserni Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bkeryan Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikeprosserni Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mikeprosserni commented Oct 16, 2025 •

edited

Loading

mikeprosserni Oct 21, 2025 •

edited

Loading

mikeprosserni Oct 22, 2025 •

edited

Loading

mikeprosserni Oct 22, 2025 •

edited

Loading

bkeryan Oct 22, 2025 •

edited

Loading

mikeprosserni Oct 22, 2025 •

edited

Loading