File appears truncated-- missing X byte (or maybe garbage at the end)

[Hydravine]

Hello!
I'm wondering why the line "File appears truncated-- missing X byte (or maybe garbage at the end)" appears for everyone one of my telemetry extracts? Sometimes the number of bytes is quite large, does that mean CCSDSPy cuts it at that point and ignores the remaining X bytes? See log below for example of my code flow, where the CCSDSPy line is printed following the split_by_apid function call. For context I am on v1.2.1.

"Message":"Waiting for new files in the Input_files directory"}
"Message":"New file found: XXXXX.VC5"}
"Message":"Mission ODB Injested"}
"Message":"Initiating Extract on files: [XXXXX.VC5] with method: postpass from station contact SCHM 999999"
"Message":"Converting XXXXX.VC5 to pktBased TLM file"}
"Message":"Found 758 frames"}
"Message":"Converted to pktBased file: XXXXX.VC5.PKT"}
"Message":"Starting extract on file: XXXXX.VC5.PKT, splitting file into apids..."}

c:\User\gmoc-schm-repo.venv\Lib\site-packages\ccsdspy\utils.py:81: UserWarning: File appears truncated-- missing 60624 byte (or maybe garbage at end)
warnings.warn(message)

"Message":"XXXXX.VC5.PKT split into apids, parsing apids..."}
"Message":"Creating and manipulating Dataframes..."}
"Message":"Transferring data to InfluxDB..."}
"Message":"Transferred 1228 mnemonics to InfluxDB."}
"Message":"Transfer to Database Successful"}
"Message":"Total Extract Runtime: 23.9 sec"}
Extract Complete, see .LOG file for details!

[ddasilva]

When split_by_apid() sees an incomplete packet at the end it will ignore it and issue a warning. This happens when the last packet length in the last packet's header indicates a length of the final packet, but the stream ends before that many bytes can be collected. This may be an issue with the last packet, or corrupt packets before it that confuse the decoding. I recommend debugging your data with some of the utilities, e.g. utils.split_packet_bytes()​ and utils.get_packet_total_bytes()​.

…

On Wednesday, February 26th, 2025 at 1:29 PM, William Parsons ***@***.***> wrote: Hello! I'm wondering why the line "File appears truncated-- missing X byte (or maybe garbage at the end)" appears for everyone one of my telemetry extracts? Sometimes the number of bytes is quite large, does that mean CCSDSPy cuts it at that point and ignores the remaining X bytes? See log below for example of my code flow, where the CCSDSPy line is printed following the split_by_apid function call. For context I am on v1.2.1. "Message":"Waiting for new files in the Input_files directory"} "Message":"New file found: XXXXX.VC5"} "Message":"Mission ODB Injested"} "Message":"Initiating Extract on files: [XXXXX.VC5] with method: postpass from station contact SCHM 999999" "Message":"Converting XXXXX.VC5 to pktBased TLM file"} "Message":"Found 758 frames"} "Message":"Converted to pktBased file: XXXXX.VC5.PKT"} "Message":"Starting extract on file: XXXXX.VC5.PKT, splitting file into apids..."} c:\User\gmoc-schm-repo.venv\Lib\site-packages\ccsdspy\utils.py:81: UserWarning: File appears truncated-- missing 60624 byte (or maybe garbage at end) warnings.warn(message) "Message":"XXXXX.VC5.PKT split into apids, parsing apids..."} "Message":"Creating and manipulating Dataframes..."} "Message":"Transferring data to InfluxDB..."} "Message":"Transferred 1228 mnemonics to InfluxDB."} "Message":"Transfer to Database Successful"} "Message":"Total Extract Runtime: 23.9 sec"} Extract Complete, see .LOG file for details! — Reply to this email directly, [view it on GitHub](#133), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAEC6OWVHSADQ3JNOGNDM4T2RYB2LAVCNFSM6AAAAABX55RT5SVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZYGAYTOMRSGU). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ddasilva]

Did this help resolve your issue?

[Hydravine]

Sorry I haven't been able to fully investigate yet. This doesn't actually cause an issue with the code execution so I have to figure out a proper way to determine if this truncation is causing my code to lose any usable datapoints in the TLM extraction. I'll be sure to Mark as answer or follow-up if need be when I can look into it. Thank you!