NoSuchMethodError in HDF 3.4 cluster when using Storm HDFS sink #1348
Description
java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;) is thrown in Storm worker in HDF 3.4 cluster after starting topology when HDFS sink is used.
2019-04-03 10:07:40.509 c.h.r.a.KerberosLogin kerberos-refresh-thread [INFO] TGT refresh sleeping until: 2019-04-04T06:16:40.730+0000
2019-04-03 10:07:40.861 c.h.r.s.c.SchemaVersionInfoCache Thread-7-7-KAFKA-SOURCE-executor[2 2] [INFO] Loading entry for cache with key [Key {schemaVersionKey=null, schemaIdVersion=SchemaIdVersion{schemaMetadataId=null, version=null, schemaVersionId=3}}] from target service
2019-04-03 10:07:40.861 c.h.r.s.c.SchemaRegistryClient Thread-7-7-KAFKA-SOURCE-executor[2 2] [INFO] Getting schema version from target registry for [3]
2019-04-03 10:07:41.024 o.a.s.util Thread-7-7-KAFKA-SOURCE-executor[2 2] [ERROR] Async loop died!
com.google.common.util.concurrent.ExecutionError: java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-16.0.1.jar:?]
at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.1.jar:?]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.1.jar:?]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.1.jar:?]
at com.hortonworks.registries.schemaregistry.cache.SchemaVersionInfoCache.getSchema(SchemaVersionInfoCache.java:103) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getSchemaVersionInfo(SchemaRegistryClient.java:654) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:146) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.streamline.streams.runtime.storm.spout.AvroKafkaSpoutTranslator.apply(AvroKafkaSpoutTranslator.java:61) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at org.apache.storm.kafka.spout.KafkaSpout.emitOrRetryTuple(KafkaSpout.java:475) ~[dep-org.apache.storm-storm-kafka-client-jar-1.2.1.3.4.0.0-152.jar.1554130513000:1.2.1.3.4.0.0-152]
at org.apache.storm.kafka.spout.KafkaSpout.emitIfWaitingNotEmitted(KafkaSpout.java:440) ~[dep-org.apache.storm-storm-kafka-client-jar-1.2.1.3.4.0.0-152.jar.1554130513000:1.2.1.3.4.0.0-152]
at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:308) ~[dep-org.apache.storm-storm-kafka-client-jar-1.2.1.3.4.0.0-152.jar.1554130513000:1.2.1.3.4.0.0-152]
at org.apache.storm.daemon.executor$fn__9593$fn__9608$fn__9641.invoke(executor.clj:660) ~[storm-core-1.2.1.3.4.0.0-152.jar:1.2.1.3.4.0.0-152]
at org.apache.storm.util$async_loop$fn__555.invoke(util.clj:484) [storm-core-1.2.1.3.4.0.0-152.jar:1.2.1.3.4.0.0-152]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
at org.glassfish.jersey.client.ClientRequest.accept(ClientRequest.java:327) ~[jersey-client-2.25.1.jar:?]
at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:229) ~[jersey-client-2.25.1.jar:?]
at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:59) ~[jersey-client-2.25.1.jar:?]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$17.run(SchemaRegistryClient.java:1135) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$17.run(SchemaRegistryClient.java:1132) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_171]
at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_171]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getEntity(SchemaRegistryClient.java:1132) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.doGetSchemaVersionInfo(SchemaRegistryClient.java:681) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.access$100(SchemaRegistryClient.java:145) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$1.retrieveSchemaVersion(SchemaRegistryClient.java:248) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.cache.SchemaVersionInfoCache$1.load(SchemaVersionInfoCache.java:71) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.hortonworks.registries.schemaregistry.cache.SchemaVersionInfoCache$1.load(SchemaVersionInfoCache.java:63) ~[stormjar.jar:0.6.0.3.4.0.0-152]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.1.jar:?]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.1.jar:?]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.1.jar:?]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.1.jar:?]
... 14 more
this is the first part of the storm jar command generated by Streamline during topology deployment:
/usr/hdf/current/storm-client/bin/storm jar
/tmp/storm-artifacts/streamline-2-ProjectionProcHdfsSinkTest2019-04-03-09-54/artifacts/streamline-runtime-storm-0.6.0.3.4.0.0-152.jar
--artifacts
org.apache.kafka:kafka-clients:0.10.2.1,
org.apache.storm:storm-kafka-client:1.2.1.3.4.0.0-152^org.slf4j:slf4j-log4j12^log4j:log4j^org.apache.zookeeper:zookeeper^org.apache.kafka:kafka-clients,
org.apache.storm:storm-hdfs:1.2.1.3.4.0.0-152^org.slf4j:slf4j-log4j12^org.apache.curator:curator-client^org.apache.curator:curator-framework^org.apache.avro:avro
when adding these Maven artifacts to a sample project and checking dependency tree, I can see that javax.ws.rs:jsr311-api:jar:1.1.1 is a dependency of org.apache.storm:storm-hdfs:jar:1.2.1.3.4.0.0-152, so I assume, this should be excluded from the dependencies, since this colludes with new rs-api (javax.ws.rs:javax.ws.rs-api:jar:2.0.1) which is also a dependency.