Multiple grblHAL boards in sync, add an external host in the equation

[mfp20]

Is there any existing code to make multiple boards working as one?

I've been playing a few months with Klipper and I really enjoyed the way it was doing this. It is very simple and effective. There's no strict active syncing involved (ex: heartbeat wires); the python code running on the host receives gcodes and computes raw steps using its own time, it acquires the monotonic time of each board at start, then compute the deltas for every command sent to the boards, using their time for computing the deltas. Then it loosely sync from time to time, just to be sure to not accumulate too much jitter. On the mcus it's all a matter of setting timers to trigger at some point in the future.

For my project I need multiple boards working together so I'd be happy to develop the needed code. The problem is ... well ... grbl is made to be standalone (ie: compute everything on its mcu).

I was wondering if there is a chance to:

• introduce this concept of "future-time" in current logic;
• move the gcode-steps computations into a plugin, so that it can be easily bypassed in the case we have a more powerful host to compute and we want to send raw steps to grblHAL. It should be something similar to the work done to make an HAL; so basically identify the needed methods and make them be function pointers.

Or whatever other approach can be done using grblHAL. As an example I've spotted a grblHAL simulator; some of that code might be reused to make the host low level component (for speed reasons; while the high level one would be in python for rapid development).

Any thoughts?

[terjeio]

Is there any existing code to make multiple boards working as one?

Not that I am aware of.

The problem is ... well ... grbl is made to be standalone (ie: compute everything on its mcu).

IMO not so, a sender is required to bring it to life. It could become standalone if code and/or hardware is added to select and run gcode from local storage.

move the gcode-steps computations into a plugin, so that it can be easily bypassed in the case we have a more powerful host to compute and we want to send raw steps to grblHAL.

Then you can grab a grblHAL driver and add a host interface on top of that? Pulling in the core in such a scenario does not make any sense to me.

Any thoughts?

I am thinking of putting forward a challenge to use a second core (e.g in the RP2040) for step generation and output. Or a FPGA for step generation and output... It would be nice to have the parser (and motion control?) separated from step generation (planner.c and stepper.c) in a clean way.

[mfp20]

move the gcode-steps computations into a plugin, so that it can be easily bypassed in the case we have a more powerful host to compute and we want to send raw steps to grblHAL.

Then you can grab a grblHAL driver and add a host interface on top of that? Pulling in the core in such a scenario does not make any sense to me.

Given my ignorance about grblHAL code I might be wrong but it makes sense to me. There are protocols involved that require the core anyway. It's a matter to write blocks that can be combined at will instead of writing tiny apps for single use cases.

Any thoughts?

I am thinking of putting forward a challenge to use a second core (e.g in the RP2040) for step generation and output. Or a FPGA for step generation and output... It would be nice to have the parser (and motion control?) separated from step generation (planner.c and stepper.c) in a clean way.

I had already started to write a Pico firmware before stepping on grblHAL pico port; last time I looked at grbl code a couple years ago, it was the old avr version and the high level of integration pushed me away (because too hard to mod, but I liked the code style).
About my pico code, in my mind I wanted to have the step generation only on core1 and everything else on core0, in order to enforce precise timing. Using the second core on Pico isn't hard at all; the hard part (for me) is to figure out how to properly split current grblHAL code.
I'm staring at the article you linked in the other thread in order to figure out what I need in terms of grbl-ish... so what I can do for you as well.

It seems to me there are 4 blocks only that need to be packed into a plugin: gcode parser, motion control calculator, bolts&nuts, planner. And looking at the little number of arrows connecting those blocks, it might be easier than I thought at the beginning. It is a simple pipeline.
Basically, in my personal use case those 4 blocks would be removed as the rx dispatcher would write the incoming data straight to the planner buffer, as the incoming data would be the output of the host's planner. If the communication interface is fast enough it can be done, and its buffer is already coded there!
The same buffer could be the right tool to make 2 cores communicate together (as it is just a matter of using a mutex on buffer's indexes updates; but haven't seen the buffer code yet), as the writer of that buffer is on core0 and the consumer of that buffer is the stepper that should be offloaded to core1.
I was looking at void st_prep_buffer() as consumer to be offloaded on the second core, but I see it is huge (ie: it may be using too many vars, making the concurrency pretty hard to get), and there's some math in there as well. So it doesn't really fit; but it must be some of the functions called in there.
Finally hooking the Probe should be trivial.

At this point the only piece missing for me to jump on board is to understand how the time is handled. From the pictures I see on that article looks like the only (real) time critical blocks are the same I highlighted above. All the others are async-ish.
And once those blocks are properly separated from the others (1 user because is using a dual core board, another user like me is having those blocks supplied by the host instead), it may eventually be written as a sequence of timer triggers... so multiple boards can be put in sync...

Sounds like a plan?

[terjeio]

Sounds like a plan?

I don't know...
plan_buffer_line in the planner is perhaps a better place to offload processing? Simpler API with less communication back and forth? st_prep_buffer is called far too often to be a good place? Hard to tell where it is best to do it - but IMO less traffic accross is better. It could

Note that I will be busy the coming week so I will not be able comment much.

[mfp20]

Sounds like a plan?

I don't know...
plan_buffer_line in the planner is perhaps a better place to offload processing? Simpler API with less communication back and forth? st_prep_buffer is called far too often to be a good place? Hard to tell where it is best to do it - but IMO less traffic accross is better. It could

I was looking at the wrong buffer, mis leaded by the picture in the article. Looks like there are 2 buffers:

• block_buffer in planner.c
• segment_buffer in stepper.c

I wrongly pointed you at the first one. In theory we should act on segment_buffer instead, maybe by introducing a third buffer; just a simple fifo with ready steps to execute, in order to dampen "transmission" glitches that may occur in time.
Moreover, ready steps should be smaller in size than plan_block_t and st_block_t, making those the ideal data for transmission.

It looks also the easier mod to do, as the existing code blocks would stay as is. Basically I should write a couple of ifdef code blocks in stepper_driver_interrupt_handler() and in the Stepper Port Reset Interrupt (named in the docs but I can't find the isr in stepper.c) as they work in combo. The "else" block would be much easier as it would just fetch and execute, no calc at all.

Then everything between gcode rx and step exec would be separated and can be run on second core, on host, on fpga... wherever, according to the ifdefs set in config.h.

Note that I will be busy the coming week so I will not be able comment much.

It's ok. Once we agreed on a plan I can go on my own. I just fetch, branch and make my own version hoping that will be good enough for upstream merge later.

EDIT1: it's even simpler than that, as the current interrupt handler can be replaced as a whole

int grbl_enter (void)
...
    hal.stepper.interrupt_callback = stepper_driver_interrupt_handler;
...

So there's no need to mod every board driver.

EDIT2: Added a picture in my development repo

EDIT3: I created motion_simple.{h|c} having two stub methods

status_code_t simple_execute_block (char *block, char *message);
ISR_CODE void simple_driver_interrupt_handler (void);

then made protocol.c call a function ptr execute_block instead of gc_execute_block(). Then in grbllib.c a couple of #if select the fptr on init. In the case of offloaded setup it will call simple_execute_block(), otherwise gc_execute_block(). The simple version bypasses all the steps generation; it simply stores the raw steps received into a local buffer. On the other side of the pipe we have the interrupt handler reading the buffer and executing the steps. Currently I'm trying to identify what's the format of raw steps so that I can write the specialized buffer as well as understand what must be sent on the serial stream as "raw step" ...

EDIT4: I'm writing a description in the README.md of my repo. I use it to help myself in understanding grblHAL code but might be useful to someone else.

[mfp20]

I'm trying to figure out the timing now.

   NOTE: This interrupt must be as efficient as possible and complete before the next ISR tick,
   which for Grbl must be less than 33.3usec (@30kHz ISR rate). Oscilloscope measured time in
   ISR is 5usec typical and 25usec maximum, well below requirement.
   NOTE: This ISR expects at least one step to be executed per segment.

In the case of RP2040 driver, the interrupt is raised by the PIO irq. I didn't look at PIO code but I guess it will be raised each 33.3us to enforce the 30kHz rate (eventually diminished according to user speed settings). The note above says the isr takes 5us to complete (with 25us peaks); what happen in the remaining time?

I mean, the irq fires the handler, the handler takes 5us to calc and turn on the gpios according to the calc result. Then the reset irq turns gpios off (let's say) after 2us (ie: minimum pulse width for common motor drivers). Then there are about 24us until next trigger... what are the motors doing during that time? Are they in hold or let the rotor spin or ...
I don't understand the mechanical implications. From my perspective the motion would be interrupted (fighting with inertial forces) if the motor holds, and it would let the head going off position if the motor doesn't hold for so long time.
Is everything pre-considered in the previous calculation?

When I compared the firmware a couple of years ago, I saw grbl and klipper being the fastest at about the same rate. Klipper speed is the fastest for single stepper (200kHz on avr 16mhz) and speed goes to grbl when increasing the number of steppers (4-5 steppers, a real scenario, equals grbl at 25-30kHz).
I was hoping that offloading math could lead to increasing the speed at 200kHz, allowing for higher microstepping ratios (1/256) without killing the overall cm/s speed. Is that note above talking about a reference speed or an hard limit?
I mean, if the board allows for speeds higher than 30kHz, will it make it or the 30kHz are enforced?

[mfp20]

Is there any existing code to make multiple boards working as one?
Any thoughts?

Ok, I think the overall plan is good enough to proceed with implementation. Any criticism at this stage would be very helpful (ie: before loose myself in details).

[terjeio]

In the case of RP2040 driver, the interrupt is raised by the PIO irq. I didn't look at PIO code but I guess it will be raised each 33.3us to enforce the 30kHz rate (eventually diminished according to user speed settings).

The 30KHz limit is an Arduino thing, no such limit is imposed by the code itself. The PR2040 seems to be able to output at least 200KHz, the iMXRT1062 driver has been pushed to 600KHz for simple moves (with tweaked code).

The note above says the isr takes 5us to complete (with 25us peaks); what happen in the remaining time?

All the other stuff, parsing and keeping the buffers stuffed. The planner buffer size is a bottleneck for the max step rate that can be achieved, at least when the g-code contains a lot of small movements (think laser engraving). This since the planner always plans to a full stop.

what are the motors doing during that time? Are they in hold or let the rotor spin or ...

Motors are spinning due to inertia. They act as a low pass filter for the step pulses.

I was hoping that offloading math could lead to increasing the speed at 200kHz, allowing for higher microstepping ratios (1/256) without killing the overall cm/s speed. Is that note above talking about a reference speed or an hard limit?

The max step rate is already significantly higher that the 8-bit Grbl code is able to output. More testing is needed for the RP2040 driver, high speed laser engraving with lots of short moves is the acid test and requires planner buffer tuning to establish what the real life max rate is.

I mean, if the board allows for speeds higher than 30kHz, will it make it or the 30kHz are enforced?

As shown above it is not enforced by code but by the MCU, a hard limit is the MCU IRQ latency and a softer is the planners ability to deliver blocks with high step rates (buffer size and processor speed plays into this).

[mfp20]

what are the motors doing during that time? Are they in hold or let the rotor spin or ...

Motors are spinning due to inertia. They act as a low pass filter for the step pulses.

Cool. I guess the "position error" can be evaluated in time...

I was hoping that offloading math could lead to increasing the speed at 200kHz, allowing for higher microstepping ratios (1/256) without killing the overall cm/s speed. Is that note above talking about a reference speed or an hard limit?

The max step rate is already significantly higher that the 8-bit Grbl code is able to output. More testing is needed for the RP2040 driver, high speed laser engraving with lots of short moves is the acid test and requires planner buffer tuning to establish what the real life max rate is.

I have a laser head, I might be able to setup that kind of testbed. I made the laser current driver board myself but never tested the output on a cartesian bot, so it will be the chance to test everything together ...

I mean, if the board allows for speeds higher than 30kHz, will it make it or the 30kHz are enforced?

As shown above it is not enforced by code but by the MCU, a hard limit is the MCU IRQ latency and a softer is the planners ability to deliver blocks with high step rates (buffer size and processor speed plays into this).

Well, once the I/O and the grbl core use 2 different mcu cores, most of the IRQs are on the I/O core; so the stepper IRQ is the only one triggering on the application core (no jitter). I'm also moving the last bits of Bresenham and AMASS from the interrupt handler to the end of the motion computing pipeline, so the irq handler have way fewer instructions to exec (ie: it just shift the stepper_t pointer to the right buffer head, instead of prepping the stepper_t with Bresenham and AMASS).

IRQ latency should be a little fraction of the upstream code; will it break the timings or it is just faster?

In order to do that I needed ram to make an array out of the existing stepper_t st, for a new "steps buffer". To get some ram I greatly reduced the size of the planner block buffer (36->9); half of the saved ram goes for augmented segment buffer (as I read that someway it limits the cpu time available to other core tasks) and half for the new steps buffer. So, instead of buffering (a few) complex objects (ie: blocks) at the beginning of the pipeline, it buffers (many) simple objects (steps) at the end of the pipeline; that's important once blocks and segments are on the host and steps are on the mcu: it absorbs eventual glitches on the input stream. In other words: cpu-time went from scarce to abundant, but we have another variable: input (steps) stream throughput, that's why the new buffer.

What are the side effects of this? In other words: are there particular reasons for the planner buffer to have 36 blocks instead of 2-3 just to (re)compute the entry/exit velocities? As far as I understand, in upstream code grbl is trying to compute as much as possible every time there's some cpu time available before the next stepper IRQ, so that on IRQ there's always fresh data to work with; and that's the only reason to have a large buffer right after the most cpu-intensive task. So, in theory, once the computation is offloaded, there's no need to have such big buffer there and there aren't side effects from reducing that buffer. Is it correct?

I'm currently studying the sys and hal calls in stepper.c, as well as the other code blocks in stepper.c (for probe, spindle and so on). Once computation and exec occur at different times, those calls can't be for real time flags/data/state.

[terjeio]

What are the side effects of this? In other words: are there particular reasons for the planner buffer to have 36 blocks instead of 2-3 just to (re)compute the entry/exit velocities?

The planner always plan to a full stop - the last entry for that. More blocks -> higher step rates for short moves (such as arcs or laser engraving bit maps). 36 was selected due to limited memory in the original 8-bit processors and is, as I understand it, a good compromise. For short moves you need a significantly larger buffer if the aim is to get high step rates. The optimal size will vary between drivers as it is dependant on processor speed and availability of a FPU.

I believe some are running grblHAL with buffer sizes of 500 entries or more for fast laser engraving. The 600MHz iMXRT1062 driver is capable of engraving speeds > 50000 mm/min when tuned. Changes to the grblHAL core should not degrade that capability.

I'm currently studying the sys and hal calls in stepper.c, as well as the other code blocks in stepper.c (for probe, spindle and so on). Once computation and exec occur at different times, those calls can't be for real time flags/data/state.

Well, it might be seemingly small details that will cause the most problems when changing the the way everything is glued together...

[mfp20]

What are the side effects of this? In other words: are there particular reasons for the planner buffer to have 36 blocks instead of 2-3 just to (re)compute the entry/exit velocities?

The planner always plan to a full stop - the last entry for that. More blocks -> higher step rates for short moves (such as arcs or laser engraving bit maps). 36 was selected due to limited memory in the original 8-bit processors and is, as I understand it, a good compromise. For short moves you need a significantly larger buffer if the aim is to get high step rates. The optimal size will vary between drivers as it is dependant on processor speed and availability of a FPU.

I believe some are running grblHAL with buffer sizes of 500 entries or more for fast laser engraving. The 600MHz iMXRT1062 driver is capable of engraving speeds > 50000 mm/min when tuned. Changes to the grblHAL core should not degrade that capability.

I believe there's no capability degradation (no tests yet, can't build yet): in the case of single cores cpu like the iMXRT on Teensy4, there are no changes as everything falls back to current upstream code workflow; and the ram distribution isn't changed. The driver must set BOARD_OFFLOAD_TO_CORE || BOARD_OFFLOAD_TO_HOST to enable the new code paths and change ram distribution.

The st_prep_buffer(void) is turned into st_prep_segment_buffer(bool refill, bool steps). First bool is to run exactly the same old code refilling the segment buffer; the second bool is to run the code to refill the new stepper_t buffer. Where used to be st_prep_buffer() (protocol.c, limits.c, state_machine.c, ...), now there is

st_prep_segment_buffer(!(BOARD_OFFLOAD_TO_HOST || BOARD_OFFLOAD_TO_CORE), false);

If both defines are 0 (ie: single core, no offloading), it become st_prep_segment_buffer(true, false), running the same exact old st_prep_buffer(void) code only. And something similar is happening in the interrupt handler: if both defines are 0, it runs the exact same Bresenham and AMASS.

About memory something similar is happening. In the case of multi-core and/or offloading to host instead, the RAM is always used as buffer, but it is used in a different way.

// in the case of offloaded mode we reduce this buffer to have spare RAM
#if (BOARD_OFFLOAD_TO_CORE || BOARD_OFFLOAD_TO_HOST)

#ifndef BLOCK_BUFFER_SIZE_OFFLOADED
#define BLOCK_BUFFER_SIZE_OFFLOADED (BLOCK_BUFFER_SIZE/4)
#endif

#undef BLOCK_BUFFER_SIZE
#define BLOCK_BUFFER_SIZE BLOCK_BUFFER_SIZE_OFFLOADED

#endif

// in the case of offloaded mode we make both segment and steps buffers bigger
#if (BOARD_OFFLOAD_TO_CORE || BOARD_OFFLOAD_TO_HOST)

// we use here half of the memory spared by the planner block buffer
#undef SEGMENT_BUFFER_SIZE
#define SEGMENT_BUFFER_SIZE (((sizeof(plan_block_t)*BLOCK_BUFFER_SIZE*2)/(sizeof(st_block_t)+sizeof(segment_t)))+10)

// we use here the second half of the memory spared by the planner block buffer
#undef STEPS_BUFFER_SIZE
#define STEPS_BUFFER_SIZE (((sizeof(plan_block_t)*BLOCK_BUFFER_SIZE*2)/sizeof(stepper_t))+1)

#endif

I'm buffering stepper_t(s) ready for execution instead of planner blocks to be further elaborated into segments first, and stepper_t last.

The workflow is different because the exact same planner-motioncontrol-stepper code is running in a loop on the second core instead of being triggered by the main system. This is the function running on core1

void motion_computing_loop(void) {
    bool running = true;
    char *block;
    char *message;

    while (running) {
        // get new block if any (non-blocking queue read)
        if (hal.pop_motion_data(block, message))
            execute_gcode(char *block, char *message);
        // prep segment and steps buffers
        st_prep_segment_buffer(true, true);
    }
}

The main system on core0 calls st_prep_segment_buffer(false, false) (because !(BOARD_OFFLOAD_TO_HOST || BOARD_OFFLOAD_TO_CORE) is false); so basically calls a function doing nothing because the motion computing runs 'elsewhere'. And the motion computing loop 'elsewhere' continuously refills both segment and steps buffers without the need to wait for grbl core's other tasks and I/O irqs.
The end result is that both grbl core and the motion computing have 1 mcu core all for themselves and are interrupted much less. As well as the interrupt handler don't have to compute Bresenham and AMASS because it is already computed by st_prep_segment_buffer(true, true) (the second true is Bresenham and AMASS now in the upstream interrupt handler instead).

I'm currently studying the sys and hal calls in stepper.c, as well as the other code blocks in stepper.c (for probe, spindle and so on). Once computation and exec occur at different times, those calls can't be for real time flags/data/state.

Well, it might be seemingly small details that will cause the most problems when changing the the way everything is glued together...

I know. That's why I was asking for info about those.