NumPy ndarray Output

[WilliamParsonsV]

I made a bit more progress and now have a full run-through of the extract/decode process and am now testing on actual .tlm files (.VC4 in my specific case). I’m new to using numpy arrays (my coding background is primarily in MATLAB and python-automation) so I could be interpreting this incorrectly but my output seems to be garbage at this point.

I tried both with a full script and with the command line split into separate .tlm files and got the same result for a specific packet test:
pkt1612Test_Output.docx

It’s possible the issue is being caused by the fact this packet has ACS_1612_TIME with the RTIME42 data_type, but I'm unsure.

My expected output array is [6691 x 7], 6691 samples for time + 6 parameters. In CCSDSPy format though this would be 6 key value pairs each 6691 x 1

[ddasilva]

From the packet definition you sent me earlier, I think the ACS_1612_TIME field was an alternate name for the ACS_1615_COARS_T and ACS_1612_FINE_T fields concatenated. When you create a packet definition, unless you specify the offset manually it's going to calculate the offset automatically from the sum of the preceding bit lengths. It might be getting confused this way.

Can you try specifying just one of ACS_1612_TIME or the ACS_1615_COARS_T/ACS_1612_FINE_T pair?

[WilliamParsonsV]

I thought that might be my issue as well, but commenting out ACS_1612_TIME produces similar outputs. I think what my actual issue is that my input file is a Frame-based file and after talking with a colleague it looks like CCSDSPy expects a Packet-based input?

I'll look into how to convert from frame-based to packet-based using framesync before feeding to CCSDSPy.

[ddasilva]

I'm not sure what level of frame you are referring to, but CCSDSPy expects only the "CCSDSPy APID Frame", which is the standardized header defined here:
https://ccsdspy.readthedocs.io/en/latest/user-guide/ccsds.html

If there is another frame on top of this, then your colleague is right that CCSDSPy will get confused. Can you tell me more about what the framesync tool is? It might be interesting to add some similar functionality to ccsdspy.utils

[WilliamParsonsV]

Sorry I should have gave more background in my question: Our data comes down in virtual channel files (back orbit, realtime, etc) that are blocks of transfer frames (TF), these then get parsed/de-muxed into a packet-based binary file. This is the step that CCSDSPy uses as the input for decoding but I was trying to input my virtual channel files as input. So I need to first write an api that does the de-muxing step and then I can utilize CCSDSPy's functions.

My colleague wrote a script many years ago to do this in Perl (what is currently used in Ops) but I need to rewrite this in Python and make it mission-generic.

Sorry for the links, Github wasn't allowing me to upload files to this reply:
(CCSDS Map) https://ibb.co/vqQQvZv
(CCSDS Packet Layer) https://ibb.co/Cb381Np (What I believe CCSDSPy uses as input)
(CCSDS TF Layer) https://ibb.co/0nMTcdB (The Transfer Frame Data Field is what makes up the Packet Layer)

[ddasilva]

Yes, you're correct in your understanding! We work at the packet layer and assume the header given in that diagram.

When you have multiplexed stream at the packet layer, we have a built in function to split it into one stream per packet type (APID):
https://ccsdspy.readthedocs.io/en/latest/user-guide/utils.html#splitting-mixed-streams-by-apid

Out of curiosity, is your code for going from the transfer layer to the packet layer more complex than stripping the transfer header?

[WilliamParsonsV]

Okay perfect, I am using the split_by_apid function. I have a pktBased file now from our current extract tool and am using that to test, I did some bit bashing and found that my test packet is ALMOST working, but is currently being read one byte shifted.

Going from tfBased to pktBased it uses frame sync and checks if any frames are missing, I can give a more educated answer once I start coding that step and understand it better, which is next after I fix the byte shift issue.

Thanks again for all your help, I'm going to test a custom converter soon as well for a pkt with a calibration curve.

Edit: The byte shift issue doesn't occur when using Include_Primary_Header = True, my data is correct, although not all there, its finding only certain instances of my pkt

[ddasilva]

Sounds like we’re close! Are you using FixedLength or VariableLength readers? It sounds like it could be an issue with the secondary header sometimes being there. Do you know if that’s the case?

[WilliamParsonsV]

We have liftoff! The existing Perl tfBased to pktBased conversion was adding newline characters which was messing it up. Next I'm working on a few scripts connecting our overall system with the decoder but I'll follow up when I make progress with a Python function for dealing with Transfer Frame files