Broadcasting camera data through network

[kenavolic]

Hi,

As discussed in #849 , I'm testing the broadcasting of camera data from a PC where the camera is connected to a PC used for processing.

Here is setup:

• PC1 with Azure Kinect connected and PlusServer running with the default config file
• PC2 with:
  • server running using the configuration file detailed below
  • a client connected to the local server

If I remove ImageMessageEmbeddedTransformName="VideoStreamRGBToVideoStreamRGB" from the PC2 server config file, I never receive any data:

|WARNING|001.203191| No data is broadcasted, as no data is available yet.| in source_subfolder/src/PlusServer/vtkPlusOpenIGTLinkServer.cxx(440)

If I set it, I correctly receive the image. But I've some error logs that make me worry:

ERROR|001.234576| Setting a transform to itself is not allowed: VideoStreamRGBToVideoStreamRGB| in /source_subfolder/Source/IGSIOCommon/vtkIGSIOTransformRepository.cxx(153)

My goal is to receive the image without any error log bound to the receiving process :)

NOTE: I have no problem retrieving the camera data through a direct connection to the PC1 server with the PC2 client.

Here are some details about the configuration and code I use.

PC2 server config:

<PlusConfiguration version="2.4">
  <DataCollection StartupDelaySec="1.0">
    <Device Id="KinectRemoteRGBDevice" Type="OpenIGTLinkVideo" ImageMessageEmbeddedTransformName="VideoStreamRGBToVideoStreamRGB" MessageType="IMAGE" ServerAddress="195.83.246.66" ServerPort="18945" IgtlMessageCrcCheckEnabled="false">
      <DataSources>
        <DataSource Type="Video" Id="Video" PortUsImageOrientation="UN" />
      </DataSources>
      <OutputChannels>
        <OutputChannel Id="VideoStreamRGB" VideoDataSourceId="Video" />
      </OutputChannels>
    </Device>
  </DataCollection>
  <PlusOpenIGTLinkServer MaxNumberOfIgtlMessagesToSend="1" MaxTimeSpentWithProcessingMs="50" ListeningPort="18944" SendValidTransformsOnly="true" OutputChannelId="VideoStreamRGB">
    <DefaultClientInfo>
      <MessageTypes>
        <Message Type="IMAGE" />
      </MessageTypes>
      <ImageNames>
        <Image Name="VideoStreamRGB" EmbeddedTransformToFrame="VideoStreamRGB" />
      </ImageNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>
</PlusConfiguration>

PC2 client code snippet (without detailed error handling):

/*
main.cpp
*/
// Launch server
dataCollector->ReadConfiguration(inputConfigFileName)
vtkXMLDataElement *configRootElement = vtkPlusConfig::GetInstance()->GetDeviceSetConfigurationData();
vtkNew<vtkIGSIOTransformRepository> transformRepository;
transformRepository->ReadConfiguration(configRootElement);
dataCollector->Connect();
dataCollector->Start();
// ...
server->Start(dataCollector.GetPointer(), transformRepository.GetPointer(), serverElement, vtkPlusConfig::GetInstance()->GetDeviceSetConfigurationFileName());
// In a loop:
server->ProcessPendingCommands();

// ...

// Run client
auto client = vtkSmartPointer<vtkPlusOpenIGTLinkClientWithTransformLogging>::New();
client->SetServerHost("127.0.0.1");
client->SetServerPort(server->GetListeningPort());
client->Connect(2.0) ;

/*
vtkPlusOpenIGTLinkClientWithTransformLogging.cpp
*/
bool OnMessageReceived(igtl::MessageHeader::Pointer messageHeader)
{
    bool messageBodyReceived = false;
    igtl::MessageBase::Pointer bodyMsg = this->IgtlMessageFactory->CreateReceiveMessage(messageHeader);
    if (bodyMsg.IsNull())
    {
      LOG_ERROR("Unable to create message of type: " << messageHeader->GetMessageType());
      return false;
    }

    if (typeid(*bodyMsg) == typeid(igtl::ImageMessage))
    {
      igtl::ImageMessage::Pointer imageMsg = dynamic_cast<igtl::ImageMessage *>(bodyMsg.GetPointer());
      imageMsg->SetMessageHeader(messageHeader);
      imageMsg->AllocatePack();
      SocketReceive(imageMsg->GetBufferBodyPointer(), imageMsg->GetBufferBodySize());
      messageBodyReceived = true;

      int c = imageMsg->Unpack(1);
      if (!(c & igtl::MessageHeader::UNPACK_BODY))
      {
        LOG_ERROR("Failed to receive IMAGE reply (invalid body)");
        return messageBodyReceived;
      }
      LOG_INFO("Matrix for " << imageMsg->GetDeviceName() << " IMAGE received: ");
      }
    }
    else if (typeid(*bodyMsg) == typeid(igtl::StatusMessage))
    {
      //std::cout << "Device type " << std::string(messageHeader->GetDeviceType()) << std::endl;
      igtl::StatusMessage::Pointer statusMessage = dynamic_cast<igtl::StatusMessage *>(bodyMsg.GetPointer());
      statusMessage->SetMessageHeader(messageHeader);
      statusMessage->AllocatePack();
      SocketReceive(statusMessage->GetBufferBodyPointer(), statusMessage->GetBufferBodySize());
      messageBodyReceived = true;

      int c = statusMessage->Unpack(1);
      if (!(c & igtl::MessageHeader::UNPACK_BODY))
      {
        LOG_ERROR("Failed to receive STATUS reply (invalid body)");
        return messageBodyReceived;
      }

      LOG_INFO("Status message code " << statusMessage->GetCode());
      LOG_INFO("Status message subcode " << statusMessage->GetSubCode());
      LOG_INFO("Status error name " << statusMessage->GetErrorName());
      LOG_INFO("Status status string " << statusMessage->GetStatusString());
    }
    else
    {
      LOG_INFO("Received " << messageHeader->GetMessageType() << " message.");
    }

    return messageBodyReceived;
 }

[Sunderlandkyl]

The embedded transform name should be just VideoStreamRGB not VideoStreamRGBToVideoStreamRGB.
ImageMessageEmbeddedTransformName should use AToB form.

From DeviceOpenIGTLinkVideo:

ImageMessageEmbeddedTransformName If IMAGE message is received and this attribute is defined then the transform embedded in the message will be recorded as a transform, with the specified name (e.g., "ImageToReference"). If the attribute is not defined then the embedded transform is ignored. If the message type is not IMAGE then the attribute is ignored. optional.png Default = " "

[Sunderlandkyl]

Can you try removing ImageMessageEmbeddedTransformName="VideoStreamRGBToVideoStreamRGB", and setting MessageType to "TRACKEDFRAME" in "KinectRemoteRGBDevice"?

[kenavolic]

Unfortunatly, it doest not work. I got:

|WARNING|001.204058| No data is broadcasted, as no data is available yet.| in /home/user/.conan/data/PlusLib/2.9.0/latim/stable/build/88131246a5a853bdb23bd09d9d3d766a01f9b668/source_subfolder/src/PlusServer/vtkPlusOpenIGTLinkServer.cxx(440)

[kenavolic]

As broadcasting data is our main use case, I tried also with our Telemed US device.

First with the default configuration file modified (reference coordinates removed) on the server side, and I get the exact same result (only way to retrieve the image is to set ImageMessageEmbeddedTransformName="ImageToImage" but then I get the same error log.

Server side:

<PlusConfiguration version="2.1">
  <DataCollection StartupDelaySec="1.0" >
    <DeviceSet
      Name="PlusServer: Telemed ultrasound device"
      Description="Broadcasting acquired video through OpenIGTLink"
    />
    <Device
      Id="VideoDevice"
      Type="TelemedVideo" >
      <DataSources>
        <DataSource Type="Video" Id="Video" PortUsImageOrientation="UN"  />
      </DataSources>
      <OutputChannels>
        <OutputChannel Id="VideoStream" VideoDataSourceId="Video" />
      </OutputChannels>
    </Device>
  </DataCollection>
  <PlusOpenIGTLinkServer
    MaxNumberOfIgtlMessagesToSend="1"
    MaxTimeSpentWithProcessingMs="50"
    ListeningPort="18944"
    SendValidTransformsOnly="true"
    OutputChannelId="VideoStream" >
    <DefaultClientInfo>
      <MessageTypes>
        <Message Type="IMAGE" />
      </MessageTypes>
      <ImageNames>
        <Image Name="Image" EmbeddedTransformToFrame="Image" />
      </ImageNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>

</PlusConfiguration>

Client side:

<PlusConfiguration version="2.4">
  <DataCollection StartupDelaySec="1.0">
    <Device Id="TelemedDevice" Type="OpenIGTLinkVideo" MessageType="IMAGE" ServerAddress="195.83.246.66" ImageMessageEmbeddedTransformName="ImageToImage" ServerPort="18944" IgtlMessageCrcCheckEnabled="false">
      <DataSources>
        <DataSource Type="Video" Id="UsImages" PortUsImageOrientation="UN" />
      </DataSources>
      <OutputChannels>
        <OutputChannel Id="TelemedVideo" VideoDataSourceId="UsImages" />
      </OutputChannels>
    </Device>
  </DataCollection>
  <PlusOpenIGTLinkServer MaxNumberOfIgtlMessagesToSend="100" MaxTimeSpentWithProcessingMs="50" ListeningPort="18944" SendValidTransformsOnly="true" OutputChannelId="TelemedVideo">
    <DefaultClientInfo>
      <MessageTypes>
        <Message Type="IMAGE" />
      </MessageTypes>
      <ImageNames>
        <Image Name="Image" EmbeddedTransformToFrame="Image" />
      </ImageNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>
</PlusConfiguration>

Then I decided to test with the default Telemed config file as we will have some coordinates transform in the future and I get:

|ERROR|001.220105| Unable to get custom transform status from name: ImageToReferenceTransformStatus| in /home/user/.conan/data/IGSIO/latest/latim/stable/build/3fc334fd88a822bca2e5dd3461277d06c55c4281/source_subfolder/Source/IGSIOCommon/igsioTrackedFrame.cxx(556)
|ERROR|001.220340| Failed to get frame transform from tracked frame: ImageToReference| in /home/user/.conan/data/IGSIO/latest/latim/stable/build/3fc334fd88a822bca2e5dd3461277d06c55c4281/source_subfolder/Source/IGSIOCommon/vtkIGSIOTransformRepository.cxx(168)
ImageMessageEmbeddedTransformName/3fc334fd88a822bca2e5dd3461277d06c55c4281/source_subfolder/Source/IGSIOCommon/vtkIGSIOTransformRepository.cxx(168)
|ERROR|001.220724| Transform path not found from Image to Reference coordinate system. Available transforms in the repository (including the inverse of these transforms): | in /home/user/.conan/data/IGSIO/latest/latim/stable/build/3fc334fd88a822bca2e5dd3461277d06c55c4281/source_subfolder/Source/IGSIOCommon/vtkIGSIOTransformRepository.cxx(543)
|WARNING|001.220912| Failed to create IMAGE message: cannot get image transform. ToolStatus: 10| in /home/user/.conan/data/PlusLib/2.9.0/latim/stable/build/88131246a5a853bdb23bd09d9d3d766a01f9b668/source_subfolder/src/PlusOpenIGTLink/vtkPlusIgtlMessageFactory.cxx(419)
|WARNING|001.221021| Failed to pack all IGT messages| in /home/user/.conan/data/PlusLib/2.9.0/latim/stable/build/88131246a5a853bdb23bd09d9d3d766a01f9b668/source_subfolder/src/PlusServer/vtkPlusOpenIGTLinkServer.cxx(1061)

with a client configuration file:

<PlusConfiguration version="2.4">
  <DataCollection StartupDelaySec="1.0">
    <Device Id="TelemedDevice" Type="OpenIGTLinkVideo" MessageType="IMAGE" ServerAddress="195.83.246.66" ImageMessageEmbeddedTransformName="ImageToReference" ServerPort="18944" IgtlMessageCrcCheckEnabled="false">
      <DataSources>
        <DataSource Type="Video" Id="UsImages" PortUsImageOrientation="UN" />
      </DataSources>
      <OutputChannels>
        <OutputChannel Id="TelemedVideo" VideoDataSourceId="UsImages" />
      </OutputChannels>
    </Device>
  </DataCollection>
  <PlusOpenIGTLinkServer MaxNumberOfIgtlMessagesToSend="100" MaxTimeSpentWithProcessingMs="50" ListeningPort="18944" SendValidTransformsOnly="true" OutputChannelId="TelemedVideo">
    <DefaultClientInfo>
      <MessageTypes>
        <Message Type="IMAGE" />
      </MessageTypes>
      <ImageNames>
        <Image Name="Image" EmbeddedTransformToFrame="Reference" />
      </ImageNames>
    </DefaultClientInfo>
  </PlusOpenIGTLinkServer>
</PlusConfiguration>

[Sunderlandkyl]

Thanks for your patience. I've also created an issue (#873) tracking the problems that I've found. I've created a PR here that should fix most of the bugs: #874.

[kenavolic]

Thanks for the fixes. I gave it a run and it works (though "ImageMessageEmbeddedTransformName="VideoStreamRGBToVideoStreamRGB" or "ImageMessageEmbeddedTransformName="ImageToReference" is MANDATORY). I receive the camera image or us image without error log.