v1.1.1 versus v1.2.0

[WilliamParsonsV]

Hello! I've noticed different behavior between versions 1.1.1 (your version history shows only 1.1.0 exists but I have 1.1.1 pip installed) and 1.2.0.

I have a baseline file created from our existing Perl extract tool and I am comparing it against my Python extract tool that uses ccsdspy. For v1.1.1 I get the assert result that my two dataframes are equivalent for apid=400, yet with no other changes other than upgrading to ccsdspy==1.2.0 the assertion errors that a few of my columns are now all 0's compared to the baseline which has those columns as all 32 or all 26.

I've narrowed down the reasoning:

• v1.1.1 gives me the error "Big-endian buffer not supported on little-endian compiler" when trying to do a df.drop(columns=) operation for apid=[40, 205]. Yet v1.2.0 gives me this error for apid=[40, 205, and 400] which is the apid I'm using in my tests.

• I get around this error using the below code, but that doesn't seem to actually change the values and therefore doesn't change the assertion result (v1.1.1 = SUCCESS, v1.2.0 = FAIL). But this does raise the question why I get this error for apid=400 only with v1.2.0?

• Worth noting that apid=400 does have polynomial conversions applied to its data

                    try:
                        df[apid] = df[apid].drop(CCSDSinfo, axis=1) # Drop CCSDS Info
                    except:
                        for key in df[apid].keys():
                            if df[apid][str(key)].dtype.byteorder == '>':
                                numpy_array = df[apid][str(key)].to_numpy()
                                swapped_array = numpy_array.byteswap().newbyteorder() # Use the byteswap method on the NumPy array
                                df[apid][str(key)] = pd.Series(swapped_array) # Assign the swapped NumPy array back to the DataFrame
    
                        df[apid] = df[apid].drop(CCSDSinfo, axis=1) # Drop CCSDS Info```


 

[ddasilva]

Can you please provide the packet definition for this packet, and some test data? You can email me privately if you don't want to share here. We fixed some issues with signed int handling in 1.2, are the affected fields int or uint?

[ddasilva]

Great! I'll wait to hear whether this for loop del code works for you.

I am tempted to say that the error you are getting is more of a pandas issue than a CCSDSPy one (maybe a bug in pandas, or just a
pandas quirk), and it is surfacing because of something that changed in CCSDSPy that isn't necessarily a CCSDSPy bug. Either way, I'm happy to help out.

[WilliamParsonsV]

So it seems the for loop del code works in the sense that it stops the "Big-endian buffer not supported on little-endian compiler" error from occurring, however, running my test_extract.py with v1.1.1 gives No.1 and changing to v1.2.0 and doing nothing else then re-running test_extract.py gives No.2, this is where I was at with my try/except code from the original post above.

1. Difference b/w BASELINE and ACTUAL: None
2. Traceback (most recent call last):
   File "c:\Users\wparsons\GMOC\GMOC_SCHM\Testing\test_extract.py", line 55, in
   test_overall_extract(station, orbit, method, tlmFileList=TESTFILE)
   File "c:\Users\wparsons\GMOC\GMOC_SCHM\Testing\test_extract.py", line 53, in test_overall_extract
   print("Difference b/w BASELINE and ACTUAL: " + str(assert_frame_equal(baselineDF, outputDF, check_dtype=False)))
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "C:\Users\wparsons.conda\envs\GMOC_SCHM\Lib\site-packages\pandas_testing\asserters.py", line 1224, in assert_frame_equal
   assert_series_equal(
   File "C:\Users\wparsons.conda\envs\GMOC_SCHM\Lib\site-packages\pandas_testing\asserters.py", line 1014, in assert_series_equal
   _testing.assert_almost_equal(
   File "pandas_libs\testing.pyx", line 52, in pandas._libs.testing.assert_almost_equal
   File "pandas_libs\testing.pyx", line 172, in pandas._libs.testing.assert_almost_equal
   File "C:\Users\wparsons.conda\envs\GMOC_SCHM\Lib\site-packages\pandas_testing\asserters.py", line 599, in raise_assert_detail
   raise AssertionError(msg)
   AssertionError: DataFrame.iloc[:, 2] (column name="REMOVING MNEMONIC NAME SORRY") are different

DataFrame.iloc[:, 2] (column name="REMOVING MNEMONIC NAME SORRY") values are different (100.0 %)
[index]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62]
[left]: [32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32]
[right]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
At positional index 0, first diff: 32 != 0

[WilliamParsonsV]

I agree its probably a Pandas bug but the fact I can get it to specifically occur between 1.1.1 and 1.2.0 means the changes made between those versions might help point to what the Pandas bug actually is

[ddasilva]

This is now a different issue, which is that the values changed. This looks like the bugfix we made. I suspect that the new values are correct, and the old ones are not. Can you double check this?

[WilliamParsonsV]

Sorry for the slow response, deadlines for other projects, I will double check this today/Monday and get back to you asap

[Hydravine]

Okay so after some manual bit-bashing comparing our existing extract tool vs new ccsdspy-based tool I've come back with 2 major findings:

1. We've been handling 12 bit unsigned, nibble aligned data wrong for 20+ years (only occurred on useless mnemonics) but thank you for bringing this to our attention!
2. For the specific issue I mention above it seems our existing tool is giving the correct result of 32 vs ccsdspy is giving 0. The specific mnemonic spans the byte boundary which may be a case that ccsdpy doesn't handle?
3. This only happens for v1.2.0, not v1.1.1

v1.1.1:

v1.2.0:

I've attached here a trimmed definitions .JSON file with the 2 problem parameters as well as the apid400.tlm source pkt file. There are no conversion applied for these 2 mnemonics.
pkt400trimmed.zip

[Hydravine]

(I recently switched to a Calian GitHub account as my GitHub CoPilot license is through there, but this is still WilliamParsonsV lol)

Let me know if there's anything else I can provide to help debug this!

[Hydravine]

Further development! I searched through the diffs of v1.1.1 and 1.2.0 on github (6bc8232) and found this removed code that would appear to solve my issue, so I inserted it back into my v1.2.0 decode.py function and tested, problem solved!!

1. Why was this code removed between versions?
2. I'd also suggest slightly altering the arithmetic for the rare scenarios where someone spans more than one byte boundary instead of simply adding =+1 (Ex: could actually span 2 and need =+2)

[ddasilva]

Thanks so much! I was able to reproduce your issue, and I pushed a fix in this branch. I had to do something other uncommenting that block (doing so made other tests fail), but you're right in that it was related to that. I also added some regression tests for this issue. I notice that this bug only affects the FixedLength decoder!
https://github.com/CCSDSPy/ccsdspy/tree/parsons-fix

Can you try this branch and confirm it fixes your issue? If so, I will merge it and make a bug fix release.

[Hydravine]

Tested and it works! Thank you! Ill stay on this branch until its implemented into the next release.

[ddasilva]

Hi @Hydravine @WilliamParsonsV: the new update (v1.2.1) has been pushed to PyPI. Let me know if there is anything else I can do!