How to use the AnyPointer type?

[jonathan-roy]

Hi,

I have a Request struct with the following schema:

struct Request {
	timestamp @0: UInt64;
	payload @1: AnyPointer;
}

The payload can either be a capnp.Data struct filled from a Node.js Buffer, or another user-defined struct (let's say a Todo struct for example). How am I supposed to create a message with such payload?

I tried the following:

const msg = new capnp.Message();
const req = msg.initRoot(Request);
req.setTimestamp(capnp.Uint64.fromNumber(0));

// With a capnp.Data struct:
const payload = capnp.Struct.initStructAt(0, capnp.Data, req);
payload.copyBuffer(Buffer.from("coucou"));
req.setPayload(payload);
// or with a Todo struct:
const payload = capnp.Struct.initStrutAt(0, Todo, req);
payload.setText("Masterize capnp-ts");
payload.setDone(false);
res.setPayload(payload);

The payload is transmitted but the timestamp field seems corrupted when I set the payload. So, what is the correct way to fill this AnyPointer payload field?

[popham]

Your payload should probably use an union instead of an AnyPointer, e.g.

struct Request {
  timestamp @0 :UInt64;
  payload :union {
    buffer @1 :Data;
    todo @2 :Todo;
  }
}

And the name payload seems a little unnecessary. You might use an anonymous union instead of naming it "payload".

[jonathan-roy]

Thank you for your answer but I can't change the schemas since I don't have control on them, they are not provided by my application. Is there really no way to use an AnyPointer field?

[SeanTasker]

I've been doing the following:

const msg = new capnp.Message();
const req = msg.initRoot(Request);
req.setTimestamp(capnp.Uint64.fromNumber(0));

// With a capnp.Data struct:
let dataMsg = new capnp.Message();
let dataRoot = dataMsg.initRoot(capnp.Data);
dataRoot.copyBuffer(Buffer.from("coucou"));
req.setPayload(dataRoot);

// or with a Todo struct:
let todoMsg = new capnp.Message();
let todoRoot = dataMsg.initRoot(Todo);
todoRoot.setText("Masterize capnp-ts");
todoRoot.setDone(false);
req.setPayload(todoRoot);

I have not tested this but this is how I'm getting things to work.

I haven't used Data types directly with this implementation yet, but looking at the code copyBuffer looks as though it is allocating as necessary.

When you set an AnyPointer the implementation looks like it copies the data from the source. Since there are copies being made you don't need to keep the internal message references.

[jonathan-roy]

Thank you @SeanTasker, unluckily it does not seem to work with capnp.Data because the Data struct has a size of 0, thus copyBuffer() ignores my buffer:

From capnp.Data.copyBuffer():

if (dstLength < srcLength) {
    trace(
        "Truncated %d bytes from source buffer while copying to %s.",
        srcLength - dstLength,
        this
    );
}

[SeanTasker]

Hmm, unfortunate. Since these structures are designed to be initialised around buffers, can you create an ArrayBuffer out of your data first, initialise a message with it then getRoot(capnp.Data) then set that as a payload?

const msg = new capnp.Message();
const req = msg.initRoot(Request);
req.setTimestamp(capnp.Uint64.fromNumber(0));

// With a capnp.Data struct:
let yourReadyData = new ArrayBuffer(1234);


//Ready buffer here


let dataMsg = new capnp.Message(yourReadyData);
let dataRoot = dataMsg.getRoot(capnp.Data);
req.setPayload(dataRoot);

[jonathan-roy]

let dataMsg = new capnp.Message(yourReadyData);

throws the following error:

Uncaught (in promise) Error: CAPNP-TS001 Attempted to parse an invalid message frame header; are you sure this is a Cap'n Proto message?
    at getFramedSegments (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/message.js:194:15)
    at initMessage (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/message.js:170:46)
    at new Message (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/message.js:51:23)

I also tried to pass an empty ArrayBuffer then fill it with the real buffer:

const msg = new capnp.Message(new ArrayBuffer(yourReadyData.length));
const payloadStruct = msg.getRoot(capnp.Data);

But then I have the following error:

Uncaught (in promise) RangeError: Offset is outside the bounds of the DataView
    at DataView.getFloat64 (<anonymous>)
    at Segment.isWordZero (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/segment.js:199:25)
    at isNull (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/pointers/pointer.js:571:22)
    at Object.validate (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/pointers/pointer.js:683:9)
    at getRoot (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/message.js:274:15)
    at Message.getRoot (/Users/jroy/Workspace/test/test/node_modules/capnp-ts/lib/serialization/message.js:78:16)

[SeanTasker]

Damn. But that does make sense since it is looking for a correctly formatted buffer.

Lack of build examples is actually my main issue with this implementation, capnp too for more unusual use cases - of building structures.

I just opened an issue about creating structures and the problem I'm having along my no-guide path.

Reading is arguably very trivial, but there aren't many examples of creating structures beyond the address book one, which is very direct and single use case.

@jdiaz5513 are you able to enlighten us?

[jonathan-roy]

I finally found a very hacky way to fill the capnp.Data struct. First, I created a DataBox struct with the following schema:

@0x915f3367709de788;
  
struct DataBox {
        value @0 :Data;
}

Then I use it like this:

const msg = new capnp.Message();
const dataBox = msg.initRoot(DataBox);
dataBox.initValue(yourReadyData.length);
const payload = dataBox.getValue();
payload.copyBuffer(yourReadyData);
req.setPayload(payload);

It is obviously not ideal so I won't close this issue, but it seems to work.

[ndisidore]

Sorry for the necro thread, but this still seems to be an issue: this could definitely be user error, but as far as I can tell a few data types (Data & AnyPointer are the ones tested) are near impossible to use because copyBuffer has some fundamental issues in that it assumes your destination buffer is (at least) the right size.

In my setup, I'm working with a protobuf struct like

struct MyStruct {
  value @0 :Data;
}

Accompanying JS code looks like

const prepopulatedBuffer = new TextEncoder().encode('test')

const msg = new capnp.Message()
const myStruct = msg.initRoot(MyStruct)
const data = new capnp.Message().initRoot(capnp.Data)
...

I'd like to copy an externally populated Uint8Array prepopulatedBuffer to data. However, passing both the raw Uint8Array and the Uint8Array as an ArrayBuffer (via prepopulatedBuffer.buffer) don't accomplish this because copyBuffer truncates based on the destination size (which I haven't found a way to not be 0 yet) and results in the same behavior @jonathan-roy was seeing.