What is the best way to handle continuation packets?

[joeymukherjee]

I have packets making use of the CCSDS_SEQUENCE_FLAG and I need to put the packets together before using the rest of my parsing. What is the best way to do this?

I have to use split_packet_bytes or iter_packet_bytes so deal with the packets at a byte level, right?

Best practices welcome!

[ddasilva]

Hello @joeymukherjee

How do continuation packets combine in your case? Does a certain data field accumulate together?

[ddasilva]

This is what I would do too. Minor difference in what I would do would be that instead of keeping track of start_packet and last_packet, I would recommend you keep a buffer. With this buffer, you would:

• reset to empty when you see a start packet
• append to in a continuation segment
• finalize/process in a last segment

I think this pattern is a bit more standard, though both approaches are acceptable.

[ddasilva]

Did this help @joeymukherjee ? Let me know if there is anything else I can do

[joeymukherjee]

I'm not a fan of the buffer since appending to those are slower than keeping the segment counters, and for a large file, it might be significant. That was one thing I didn't like about my implementation either: my full_bytes is a buffer and I am reading the entire file into memory and keeping the indexes of the start of the packets. I also didn't like the fact that I am deleting the packets after it took time to parse them out. It seems like I should be able to do that before the parsing process.

Could there be a way to an array of dictionaries instead of a dictionary of parallel arrays? In some cases, this seems like it could be more useful?

However my code was working, so I didn't change it. :)

Thanks for the help though!
Joey

[ddasilva]

if you were thinking about the concatenation being slow from increasing copy overhead, it's not slower if you append each chunk to a list and do the concatenation before you finalize, or use list.extend or BytesIO. it's still O(n). what you're saying is true if you use a string or bytes object and do += each iteration (that is O(n^2))

[ddasilva]

Could there be a way to an array of dictionaries instead of a dictionary of parallel arrays? In some cases, this seems like it could be more useful?

That's a good idea to reflect on; we've been doing it this way because it was originally easier to plot time series of housekeeping values. I'll give this suggestion some thought