why i can't connect iotdb 0.12.1 service by pyspark?

[tvboy20082018]

i have tried to connect iotdb 0.12.1 service by pyspark with all kinds of methods, also i'm sure i have written python code correctly and executed spark-submit commands with params correctly. however, i have been receiving the spark-submit execution result with "cannot establish connection with jdbc:iotdb://172.24.4.42:6668/" and the server error with "org.apache.thrift.transport.TTransportException: Read a negative frame size" all the time. I've been tortured by this problem for a whole day. Could somebody give a hand please?

[SilverNarcissus]

From you error log. The cause of this error is the port of JDBC connection. If you use default config, you should connect to port NO.6667 rather than 6668.
For trouble shooting, you can firstly use command line or java example to connect iotdb without spark. Then you can try use spark locally in our spark example. Finally, submit this job by spark-submit. During these step you can figure out where is the problem.

[tvboy20082018]

Sorry, i attached wrong shot with python code, in fact the right jdbcurl = "jdbc:iotdb://172.24.4.42:6668/".
i've configured rpc_address and rpc_port for iotdb 0.12.1 service. it has been okay that i connected to iotdb 0.12.1 service by iotdb 0.12.1 client.
i've also followed your guide to use jdbc to connect iotdb 0.12.1 service without spark successfully, but still couldn't connect it with pyspark.
do you think maybe there is something wrong with the protocol between thrift and spark-iotdb-connector?

[SilverNarcissus]

I will try it later. You can try with spark-shell as follow:

spark-shell --jars [your jars path]

import org.apache.iotdb.spark.db._

val df = spark.read.format("org.apache.iotdb.spark.db").option("url","jdbc:iotdb://127.0.0.1:6667/").option("sql","select * from root").load

df.printSchema()

df.show()

[tvboy20082018]

i followed official case for spark iotdb with scala

however, i got the same result with pyspark-connecting-iotdb: java.sql.SQLException: Can not establish connection with jdbc:iotdb://XXX

i'm confused where i was stuck.

[SilverNarcissus]

I have tried it and it can work on my computer. You may use 0.11 iotdb to try again?

[tvboy20082018]

my comrade replaced jars of the right version on Spark, then it worked and the error "java.sql.SQLException: Can not establish connection with jdbc:iotdb://172.24.4.42:6668/" didn't occur. however, another problem happened that "WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, node03, executor 2): org.apache.spark.util.TaskCompletionListenerException: null" occurs in the output of spark-submit execution. i'm confused with the problem and have no clue about it.

[tvboy20082018]

[SilverNarcissus]

You can also run it in local mode without submit. Is there any problem on the iotdb server?

[tvboy20082018]

I've solved the problem by replacing spark cluster server and using spark-iotdb-connector-0.12.0 to connect iotdb 0.12.1. thanks for SilverNarcissus and qiaojialin's help and patience!