io.grpc.StatusRuntimeException: UNIMPLEMENTED error

[srikanthjoshi]

Hi,
I am planning to implement this on IBM PPC64 based systems. I got trouble in starting tensorflow-serving server, I got help from https://github.com/thammegowda/tensorflow-grpc-java and have tensorflow-serving running with inception inference model in place:

(srikanth) [root@brazossrik01 bin]# nohup tensorflow_model_server --model_name=inception --model_base_path=/root/srikanth/SERVING_INCEPTION/SERVING_INCEPTION --port=9000 2>&1 &
[1] 357
(srikanth) [root@brazossrik01 bin]# nohup: ignoring input and appending output to 'nohup.out'

(srikanth) [root@brazossrik01 bin]# ps -eaf |grep 9000
root 357 32464 27 23:03 pts/1 00:00:01 tensorflow_model_server --model_name=inception --model_base_path=/root/srikanth/SERVING_INCEPTION/SERVING_INCEPTION --port=9000
root 574 32464 0 23:03 pts/1 00:00:00 grep --color=auto 9000
(srikanth) [root@brazossrik01 bin]#

However, when I am trying to use the client code, I am getting UNIMPLEMENTED Error when I bind to kafakcat:

(srikanth) [root@brazossrik01 bin]# echo -e "src/main/resources/example.jpg" | /root/srikanth/kafkacat-master/kafkacat -b localhost:9092 -P -t ImageInputTopic
(srikanth) [root@brazossrik01 bin]#

(srikanth) [root@brazossrik01 bin]# java -cp target/tensorflow-serving-java-grpc-kafka-streams-1.0-jar-with-dependencies.jar com.github.megachucky.kafka.streams.machinelearning.Kafka_Streams_TensorFlow_Serving_gRPC_Example
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Image Recognition Microservice is running...
Input images arrive at Kafka topic ImageInputTopic; Output predictions going to Kafka topic ImageOutputTopic
Image path: src/main/resources/example.jpg
Image = src/main/resources/example.jpg
io.grpc.StatusRuntimeException: UNIMPLEMENTED
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at com.github.megachucky.kafka.streams.machinelearning.InceptionBlockingStub.classify(InceptionBlockingStub.java:63)
at com.github.megachucky.kafka.streams.machinelearning.TensorflowObjectRecogniser.recognise(TensorflowObjectRecogniser.java:66)
at com.github.megachucky.kafka.streams.machinelearning.Kafka_Streams_TensorFlow_Serving_gRPC_Example.lambda$main$0(Kafka_Streams_TensorFlow_Serving_gRPC_Example.java:91)
at org.apache.kafka.streams.kstream.internals.AbstractStream$2.apply(AbstractStream.java:111)
at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:40)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:115)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:146)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:129)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:93)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:351)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:104)
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:413)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:862)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)

I got suggestion from Thammegowda that it could be a version mismatch. So, I wanted to check what is the version of tensorflow-serving you have used so that I try to use same version.

If you feel that it isn't version problem, please let me know if I can get any insights from you to solve this issue.

Thanks in advance for your kind support.

BTW - I am able to run this on my Mac OS by creating docker using Thammegowda's github. That isn't getting supported on PPC64 system as libraries are x86 compiled.

[kaiwaehner]

Honestly, I don't remember what version of TensorFlow Serving was used (and sorry for not documenting it). As different TensorFlow versions sometimes creates issues (like other Python libraries etc.), this might be the reason.

I would expect that you can find out the version by getting into the Docker container and using a specific command or checking the config files. I recommend to doublecheck with TensorFlow community to find out how you can find out which version is used.

Having said this, running on hardware such as IBM PPC64 could also be the issue, of course. Especially if you get it working on Mac succesfully.

[srikanthjoshi]

@kaiwaehner - Thanks for your suggestions. I believe tensorflow serving server is started successfully on 9000 port on my ppc64 system. However, I feel I am getting problem in client code which I got from this git. Docker is having only serving and we need to compile client code once server is up in 9000 port.. So, when I am running client code, I am getting the error.

[kaiwaehner]

For the client side, I would first try out the client I built with Maven:
https://github.com/kaiwaehner/tensorflow-serving-java-grpc-kafka-streams/blob/master/pom.xml

The pom.xml shows all dependencies and versions.

[srikanthjoshi]

Thanks @kaiwaehner . I see grpc version used 1.13.1. Will check if I can use same version.

[srikanthjoshi]

Hi @kaiwaehner - I tried 1.13.1 grpc version. Still seeing same issue.

(srikanth) [root@brazossrik01 srikanth]# git clone https://github.com/grpc/grpc-java.git
Cloning into 'grpc-java'...
remote: Enumerating objects: 106, done.
remote: Counting objects: 100% (106/106), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 93247 (delta 22), reused 71 (delta 13), pack-reused 93141
Receiving objects: 100% (93247/93247), 35.14 MiB | 1.07 MiB/s, done.
Resolving deltas: 100% (44333/44333), done.
(srikanth) [root@brazossrik01 srikanth]# cd grpc-java
(srikanth) [root@brazossrik01 grpc-java]# ls
AUTHORS NOTICE.txt all auth codecov.yml documentation gradlew.bat okhttp run-test-server.sh testing-proto
BUILD.bazel README.md alts benchmarks compiler examples grpclb protobuf services xds
COMPILING.md RELEASING.md android bom context gae-interop-testing interop-testing protobuf-lite settings.gradle
CONTRIBUTING.md SECURITY.md android-interop-testing build.gradle core gradle java_grpc_library.bzl repositories.bzl stub
LICENSE WORKSPACE api buildscripts cronet gradlew netty run-test-client.sh testing
(srikanth) [root@brazossrik01 grpc-java]# git checkout v1.13.1
Note: checking out 'v1.13.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at 9a3e0705b Bump version to 1.13.1
(srikanth) [root@brazossrik01 grpc-java]# export CXXFLAGS="-I/root/srikanth/protobuf/src" LDFLAGS="-L/root/srikanth/protobuf/src/.libs"
(srikanth) [root@brazossrik01 grpc-java]# cd compiler
(srikanth) [root@brazossrik01 compiler]# GRPC_BUILD_CMD="../gradlew java_pluginExecutable"
(srikanth) [root@brazossrik01 compiler]# eval $GRPC_BUILD_CMD
Downloading https://services.gradle.org/distributions/gradle-4.7-bin.zip
.......................................................................
Unzipping /root/.gradle/wrapper/dists/gradle-4.7-bin/73xux3q4721jgr4pjvqf9o30/gradle-4.7-bin.zip to /root/.gradle/wrapper/dists/gradle-4.7-bin/73xux3q4721jgr4pjvqf9o30
Set executable permissions for: /root/.gradle/wrapper/dists/gradle-4.7-bin/73xux3q4721jgr4pjvqf9o30/gradle-4.7/bin/gradle

Welcome to Gradle 4.7!

Here are the highlights of this release:

• Incremental annotation processing
• JDK 10 support
• Grouped non-interactive console logs
• Failed tests are re-run first for quicker feedback

For more details see https://docs.gradle.org/4.7/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)
Download https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-apt-plugin/0.13/gradle-apt-plugin-0.13.pom
Download https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-errorprone-plugin/0.0.13/gradle-errorprone-plugin-0.0.13.pom
Download https://plugins.gradle.org/m2/ru/vyarus/gradle-animalsniffer-plugin/1.4.0/gradle-animalsniffer-plugin-1.4.0.pom
Download https://plugins.gradle.org/m2/org/codehaus/mojo/animal-sniffer/1.15/animal-sniffer-1.15.pom
Download https://plugins.gradle.org/m2/org/codehaus/mojo/animal-sniffer-parent/1.15/animal-sniffer-parent-1.15.pom
Download https://plugins.gradle.org/m2/org/ow2/asm/asm-all/5.0.3/asm-all-5.0.3.pom
Download https://plugins.gradle.org/m2/org/ow2/asm/asm-parent/5.0.3/asm-parent-5.0.3.pom
Download https://plugins.gradle.org/m2/io/airlift/airline/0.7/airline-0.7.pom
Download https://plugins.gradle.org/m2/io/airlift/airbase/28/airbase-28.pom
Download https://plugins.gradle.org/m2/com/google/code/findbugs/annotations/2.0.3/annotations-2.0.3.pom
Download https://plugins.gradle.org/m2/com/google/code/findbugs/annotations/2.0.3/annotations-2.0.3.jar
Download https://plugins.gradle.org/m2/io/airlift/airline/0.7/airline-0.7.jar
Download https://plugins.gradle.org/m2/ru/vyarus/gradle-animalsniffer-plugin/1.4.0/gradle-animalsniffer-plugin-1.4.0.jar
Download https://plugins.gradle.org/m2/org/codehaus/mojo/animal-sniffer/1.15/animal-sniffer-1.15.jar
Download https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-apt-plugin/0.13/gradle-apt-plugin-0.13.jar
Download https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-errorprone-plugin/0.0.13/gradle-errorprone-plugin-0.0.13.jar
Download https://plugins.gradle.org/m2/org/ow2/asm/asm-all/5.0.3/asm-all-5.0.3.jar
Download https://maven-central.storage-download.googleapis.com/repos/central/data/com/google/protobuf/protobuf-gradle-plugin/0.8.5/protobuf-gradle-plugin-0.8.5.pom
Download https://maven-central.storage-download.googleapis.com/repos/central/data/com/google/protobuf/protobuf-gradle-plugin/0.8.5/protobuf-gradle-plugin-0.8.5.jar
Download https://maven-central.storage-download.googleapis.com/repos/central/data/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.7/jetty-alpn-agent-2.0.7.pom
Download https://maven-central.storage-download.googleapis.com/repos/central/data/org/mortbay/jetty/alpn/jetty-alpn-agent/2.0.7/jetty-alpn-agent-2.0.7.jar

Configure project :grpc-compiler
*** Building codegen requires Protobuf version 3.5.1-1
*** Please refer to https://github.com/grpc/grpc-java/blob/master/COMPILING.md#how-to-build-code-generation-plugin
Download https://jcenter.bintray.com/com/github/jengelman/gradle/plugins/shadow/2.0.2/shadow-2.0.2.pom
Download https://jcenter.bintray.com/com/github/jengelman/gradle/plugins/shadow/2.0.2/shadow-2.0.2.jar

Task :grpc-compiler:compileJava_pluginExecutableJava_pluginCpp
Task :grpc-compiler:linkJava_pluginExecutable
Task :grpc-compiler:java_pluginExecutable

BUILD SUCCESSFUL in 1m 17s
2 actionable tasks: 2 executed
(srikanth) [root@brazossrik01 compiler]#

(srikanth) [root@brazossrik01 bin]# echo -e "src/main/resources/example.jpg" | /root/srikanth/kafkacat-master/kafkacat -b localhost:9092 -P -t ImageInputTopic
(srikanth) [root@brazossrik01 bin]#

(srikanth) [root@brazossrik01 bin]# java -cp target/tensorflow-serving-java-grpc-kafka-streams-1.0-jar-with-dependencies.jar com.github.megachucky.kafka.streams.machinelearning.Kafka_Streams_TensorFlow_Serving_gRPC_Example
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Image Recognition Microservice is running...
Input images arrive at Kafka topic ImageInputTopic; Output predictions going to Kafka topic ImageOutputTopic
Image path: src/main/resources/example.jpg
Image = src/main/resources/example.jpg
io.grpc.StatusRuntimeException: UNIMPLEMENTED
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at com.github.megachucky.kafka.streams.machinelearning.InceptionBlockingStub.classify(InceptionBlockingStub.java:63)
at com.github.megachucky.kafka.streams.machinelearning.TensorflowObjectRecogniser.recognise(TensorflowObjectRecogniser.java:66)
at com.github.megachucky.kafka.streams.machinelearning.Kafka_Streams_TensorFlow_Serving_gRPC_Example.lambda$main$0(Kafka_Streams_TensorFlow_Serving_gRPC_Example.java:91)
at org.apache.kafka.streams.kstream.internals.AbstractStream$2.apply(AbstractStream.java:111)
at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:40)
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:115)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:146)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:129)
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:93)
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:351)
at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:104)
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:413)
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:862)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:777)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:747)

Please let me know if there are any insights you can get for helping me.

[kaiwaehner]

Sorry to hear this. Maybe it is really a OS issue then.

I am sorry, but I don't know how to fix this.