Trying to use '_debug' metadata in view

[hello-adam]

I'm using the Kaitai Python runtime in https://github.com/Mahlet-Inc/hobbits to make a Kaitai runner and viewer plugin. It works in a few cases, but a user pointed out a pretty simple case in which it fails to calculate offsets correctly: Mahlet-Inc/hobbits#93

Here is the ksy:

meta:
  file-extension: bla
  id: servermod
  endian: le
  imports:
    - /serialization/google_protobuf
seq:
  - id: records
    type: record
    repeat: eos
types:
  main_header:
    seq:
      - id: ip1
        type: u1
      - id: ip2
        type: u1
      - id: ip3
        type: u1
      - id: ip4
        type: u1
      - id: body_len
        type: u4le
  record:
    seq:
      - id: len_record
        type: main_header
        doc: Size of whole record, including all headers, footers and data
      - id: ctype
        type: u1
        doc: ctype
      - id: flow
        type: u1
        doc: flow
      - id: cid
        type: u4le
        doc: cid
      - id: did
        type: u4le
        doc: did
      - id: seq1
        type: u8le
        doc: seq1
      - id: command
        type: u2le
        doc: command
      - id: protobuf
        size: len_record.body_len - 24
        type: google_protobuf

Here is where I hit a roadblock:

Sometimes the "_debug" metadata from the parse has an absolute offset, for example with the records[1].len_record section:

{ '_debug': defaultdict(<class 'dict'>,
                        { 'body_len': {'end': 46, 'start': 42},
                          'ip1': {'end': 39, 'start': 38},
                          'ip2': {'end': 40, 'start': 39},
                          'ip3': {'end': 41, 'start': 40},
                          'ip4': {'end': 42, 'start': 41}}),
  '_io': <kaitaistruct.KaitaiStream object at 0x7fff98348070>,
  '_parent': <servermod.Servermod.Record object at 0x7fff60bae8b0>,
  '_root': <servermod.Servermod object at 0x7fff98368df0>,
  'body_len': 34,
  'ip1': 47,
  'ip2': 254,
  'ip3': 184,
  'ip4': 155}

But for the records[1].protobuff section, everything resets to relative offsetting:

{ '_debug': defaultdict(<class 'dict'>,
                        { 'pairs': { 'arr': [ {'end': 2, 'start': 0},
                                              {'end': 10, 'start': 2}],
                                     'end': 10,
                                     'start': 0}}),
  '_io': <kaitaistruct.KaitaiStream object at 0x7fff60bae730>,
  '_parent': None,
  '_root': <google_protobuf.GoogleProtobuf object at 0x7fff60bae6d0>,
  'pairs': [ <google_protobuf.GoogleProtobuf.Pair object at 0x7fff60bae670>,
             <google_protobuf.GoogleProtobuf.Pair object at 0x7fff60bae580>]}

There might be a pattern to when it is relative (in which case I could get the absolute offset from the parent chunk), but I don't notice any such pattern.

I don't know much about kaitai, so it's likely that I'm missing something simple here, but I thought I'd ask for help since I can't find any documentation for parsing the python "_debug" section.

[KOLANICH]

I guess the key is '_root': <google_protobuf.GoogleProtobuf object at 0x7fff60bae6d0>, which has an own _io, because it is an imported type (and an importable type and the generated code should require no modifications for standalone/importable usage, so its instances are relative to its _root, so its _root must be another that the one of the type from which it is imported).

Thank you for bringing that in, it is likely that kaitai2WxHexEditor has the same bug.

[hello-adam]

ah, ok, I will try to rework it tomorrow with that in mind. Thank you!

[hello-adam]

Awesome, that was it

[generalmimon]

This question actually isn't directly related to the Python runtime implementation, so I'm moving it into the the main kaitai_struct issue tracker.

[Mingun]

In fact, this is a common problem, because of which the positions are counted from the beginning of the current stream, while it would be much more convenient to count them from the beginning of the enclosing type.

The record has no own _io, because it isn't limited in size in the root seq, so it reused the parent _io, which is equal to the main stream, represented the whole file. Therefore all offsets of the record fields (len_record include) relative to the main _io, and because that _io has no parent _io, actually that offsets is absolute.

As @KOLANICH said, imported types has its own _io, and all offsets calculated relatively to it. Because that _io matches the boundaries of the google_protobuf type, there may be a feeling that offsets are calculated relative to the type.

See also #709

[generalmimon]

@KOLANICH:

I guess the key is '_root': <google_protobuf.GoogleProtobuf object at 0x7fff60bae6d0>, which has an own _io, because it is an imported type (and an importable type and the generated code should require no modifications for standalone/importable usage, so its instances are relative to its _root, so its _root must be another that the one of the type from which it is imported).

@Mingun:

As @KOLANICH said, imported types has its own _io, and all offsets calculated relatively to it. Because that _io matches the boundaries of the google_protobuf type, there may be a feeling that offsets are calculated relative to the type.

Not true. The google_protobuf does not have its own _io substream because it's an imported type. Please look closely at how the google_protobuf type is instantiated:

      - id: protobuf
        size: len_record.body_len - 24 # !!!
        type: google_protobuf

It has its own _io only because it's invoked with size delimitation. Whether it is an imported type or isn't doesn't matter.

You can easily test this in the Web IDE - create these .ksy files there:

hello_world.ksy

meta:
  id: hello_world
seq:
  - id: one
    type: u1

imports0.ksy

meta:
  id: imports0
  imports:
    - hello_world
seq:
  - id: two
    type: u1
  - id: hw
    type: hello_world
instances:
  hw_one:
    value: hw.one
  is_hw_io_same:
    value: false
    if: false

and open imports0.ksy (on the first load it fails with TypeError: HelloWorld is not a constructor, but just ignore it for now).

The is_hw_io_same is only a placeholder, because KSC doesn't allow testing KaitaiStreams for equality (well, why would it actually), so you also need to edit these lines in the JS code (debug) tab:

      if (false) {
        this._debug._m_isHwIoSame = {  };
        this._m_isHwIoSame = false;
      }

so they look like this:

      if (true) {
        this._debug._m_isHwIoSame = {  };
        this._m_isHwIoSame = (this.hw._io === this._io);
      }

and then press Ctrl+Enter to reparse the file using the modified parsing code. The === operator for two objects returns true only if the variables point to the same object. You can see that it evaluates to true once you run the edited code, which means that the hello_world type actually reuses the same substream as imports0.

You can also check that if you put hello_world into a substream using size: 1, size-eos: true or terminator and edit the JS code (debug) again, it will return false because then the hello_world now really has its own _io.

---

Sorry for coming late into this discussion, but I still felt the need to eradicate this misinformation so that it doesn't confuse people (like me 🤦). I didn't realize it before since I didn't bother to use my brain and had a faith in you that you've done some research. It hadn't hit me until I was answering @CandyAngel's question on Gitter about how do they shift the instance pos offsets to be relative to the start offset of the imported type while the size of the imported type is not known in advance. If every imported type was automatically put into a substream with unknown size, it would work just like they wanted. But it's not that simple, because it's not possible to create an auto-growing substream (yet).

[KOLANICH]

@generalmimon, thanks for correcting me.

I have done some tests and it turns out that imported types really don't have an own _io. It is a bug, because it would break every embedded type using pos-instances without doing measures to ensure creation of a substream for them.

I apologize to everyone for the misinformation.

[hello-adam]

ok, so was my adjustment for this issue that added the root offset if the _io was different the wrong thing to do?

Mahlet-Inc/hobbits@03067d4

It fixed this specific issue, but it sounds like it's not a generic fix.

[generalmimon]

ok, so was my adjustment for this issue that added the root offset if the _io was different the wrong thing to do?

Nope, your adjustment looks perfectly fine. The dump_struct function is generic, it simply checks if the section._io is same as the current root_io and if isn't, then sets it as a new "root_io" for all descendants. There is no special adjustment that you need to do specifically for imported types, that's how the imported types are designed. The adjustment you did was for any types that have its own substream.

The only tricky thing in the light of the new finding is that the naming convention of root_io and root_offset, it's basically OK to call it like that but you need to keep in mind that it has little in common with _root property and what we usually call "root type". As long as you and all developers that may come across the code understand that it's more like "root stream" in the meaning as used in this PR (kaitai-io/kaitai_struct_java_runtime#28) for example, there's nothing wrong with that and I'd say it's an appropriate name. The point is that this "root stream" can be created on any type at any level in the KSY spec just by instantiating it with size: ..., size-eos: true or terminator: ..., regardless

I like to say that importing a type is no different (except for the _root pointer, which does not directly affect the parsing though - it would only take effect in expressions) than inlining the entire imported spec into the root spec's types, using meta/id as the key in types map. For example:

imports0.ksy

meta:
  id: imports0
  imports:
    - hello_world
seq:
  - id: two
    type: u1
  - id: hw
    type: hello_world
instances:
  hw_one:
    value: hw.one

hello_world.ksy

meta:
  id: hello_world
seq:
  - id: one
    type: u1

... is equivalent to this:

meta:
  id: imports0
#  imports:
#    - hello_world
seq:
  - id: two
    type: u1
  - id: hw
    type: hello_world
instances:
  hw_one:
    value: hw.one
types:
  hello_world: #  <-----------+
#    meta:                    |
#      id: hello_world  ------+
    seq:
      - id: one
        type: u1

Always think of meta/imports like this and you'll be fine.

---

It fixed this specific issue, but it sounds like it's not a generic fix.

It is a generic fix, ... as long as we don't consider starting a new root stream from a totally unconnected array of bytes which is the case of process. If you have a byte array with zlib-compressed data which has some internal structure that you'd want to parse with KS, you can do this:

seq:
  - id: body
    size: len_body
    process: zlib
    type: inner_structure
types:
  inner_structure:
    seq:
      # ...

Now, inner_structure's _io is a brand new block of data and the bytes and stream positions don't map to anything in the original root file stream.

Let's be honest: none of hex editors supporting Kaitai Struct that we know of can properly handle that, but it would be certainly cool and useful. It could be visualized by splitting the hex view and showing the uncompressed bytes there, as suggested in kaitai-io/kaitai_struct_webide#87.

Sorry for putting ideas into your head, but I couldn't help it 🙂

[hello-adam]

ok, thank you for the clarification. this project has a super helpful community!

Let's be honest: none of hex editors supporting Kaitai Struct that we know of can properly handle that, but it would be certainly cool and useful. It could be visualized by splitting the hex view and showing the uncompressed bytes there, as suggested in kaitai-io/kaitai_struct_webide#87.

haha, I'd love for hobbits-kaitai integration to one day be extra classy like that. I'm hoping to spend more time on efforts like that once I finalize the core application and interfaces this Summer.