Failed to construct kafka consumer due to the Kafka client code does not currently support obtaining a password from the user

[sahabcse]

We follows the steps, our environment we use SASL_SSL protocol and kerberos authentication.

When ever we execute from user we are getting below exception.

Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
ERROR streaming.StreamingContext: Error starting the context, marking it as stopped
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

[osboo]

hi @sahabcse
I faced the same issue after applying steps from the instruction. This error means that Kafka client cannot find the .keytab in HDFS which is unforunately happens in Cloudera CDH 5.14.2. Somehow the --keytab works for many services like HDFS, Spark etc (see this SO) but not for Kafka.
The solution is pretty simple: I made a copy of my keytab and distributed it via --files. Then I referenced the copy in my .jaas file for Kafka:

# Call
spark2-submit --keytab osboo.keytab \
 --principal osboo \
--files osboo-copy-for-kafka.keytab#osboo-copy-for-kafka.keytab,kafka.jaas#kafka.jaas

# kafka.jaas
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="osboo-copy-for-kafka.keytab"
principal="osboo@REALM.COM"
serviceName="kafka";
};

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="osboo-copy-for-kafka.keytab"
serviceName="zookeeper"
principal="osboo@REALM.COM";
};

[microeastcowboy]

hi, @osboo
I faced the same issue with you.
when you assign keytab file path --keytab through spark-submit ,the keytab file in yarn container is renamed by launch_container.sh when you deploy spark streaming with kafka Application.

it's amazing that, but when you deploy appliction via --files option, the keytab file name will not change.