librdkafka with SASL support

[prashantochatterjee]

Followed the steps below to build librdkafka and subsequently confluent-kafka Python library. Note that I have not configured Kerberos.

• Download librdkafka source
  $ git clone https://github.com/edenhill/librdkafka.git

• Install prerequisite libraries
  $ sudo apt-get install libssl-dev libsasl2-dev

• Build and install librdkafka from source
  Refer: https://github.com/edenhill/librdkafka/blob/master/README.md

• Install confluent-kafka from source
  $ pip install --no-binary :all: confluent-kafka
  Refer: https://github.com/confluentinc/confluent-kafka-python/blob/master/README.md

The confluent-kafka library still complains of unsupported JAAS configuration:
`$ python
Python 3.7.4 (default, Dec 17 2021, 20:49:59)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

from confluent_kafka import Consumer
consumer = Consumer({
... 'bootstrap.servers': 'localhost:9093',
... 'group.id': 'test-consumer-group',
... 'security.protocol': 'SASL_PLAINTEXT',
... 'sasl.mechanism': 'SCRAM-SHA-256',
... 'sasl.jaas.config': 'org.apache.kafka.common.security.scram.ScramLoginModule required
... username="9846c8dc-5b23-4e1a-a122-32d2b54ddb2f.user"
... password="9846c8dc-5b23-4e1a-a122-32d2b54ddb2f";'
...
... })
Traceback (most recent call last):
File "", line 8, in
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Java JAAS configuration is not supported, see https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka for more information."}
The logs from configure indicating SASL library is available:$ ./configure
using cache file config.cache
checking for OS or distribution... ok (linuxmint)
checking for C compiler from CC env... failed
checking for gcc (by command)... ok
checking for C++ compiler from CXX env... failed
checking for C++ compiler (g++)... ok
checking executable ld... ok
checking executable nm... ok
checking executable objdump... ok
checking executable strip... ok
checking executable libtool... failed (disable)
checking executable ranlib... ok
checking for pkgconfig (by command)... ok
checking for install (by command)... ok
checking for GNU ar... ok
checking for PIC (by compile)... ok
checking for GNU-compatible linker options... ok
checking for GNU linker-script ld flag... ok
checking for __atomic_32 (by compile)... ok
checking for __atomic_64 (by compile)... ok
checking for socket (by compile)... ok
parsing version '0x010900ff'... ok (1.9.0)
checking for librt (by pkg-config)... failed
checking for librt (by compile)... ok
checking for libpthread (by pkg-config)... failed
checking for libpthread (by compile)... ok
checking for c11threads (by pkg-config)... failed
checking for c11threads (by compile)... ok
checking for libdl (by pkg-config)... failed
checking for libdl (by compile)... ok
checking for zlib (by pkg-config)... ok
checking for libcrypto (by pkg-config)... ok
checking for libssl (by pkg-config)... ok
checking for libsasl2 (by pkg-config)... ok
checking for libzstd (by pkg-config)... failed
checking for libzstd (by compile)... failed (disable)
checking for libcurl (by pkg-config)... failed
checking for libcurl (by compile)... failed (disable)
checking for libm (by pkg-config)... failed
checking for libm (by compile)... ok
checking for liblz4 (by pkg-config)... failed
checking for liblz4 (by compile)... failed (disable)
checking for syslog (by compile)... ok
checking for rapidjson (by compile)... failed (disable)
checking for crc32chw (by compile)... ok
checking for regex (by compile)... ok
checking for rand_r (by compile)... ok
checking for strndup (by compile)... ok
checking for strlcpy (by compile)... failed (disable)
checking for strerror_r (by compile)... ok
checking for strcasestr (by compile)... ok
checking for pthread_setname_gnu (by compile)... ok
checking for nm (by env NM)... ok (cached)
checking for python3 (by command)... ok
checking for getrusage (by compile)... ok
Generated Makefile.config
Generated config.h

Configuration summary:
prefix /usr/local
MKL_DISTRO linuxmint
SOLIB_EXT .so
ARCH x86_64
CPU generic
GEN_PKG_CONFIG y
MKL_APP_NAME librdkafka
MKL_APP_DESC_ONELINE The Apache Kafka C/C++ library
CC gcc
CXX g++
LD ld
NM nm
OBJDUMP objdump
STRIP strip
RANLIB ranlib
CPPFLAGS -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align
PKG_CONFIG pkg-config
INSTALL /usr/bin/install
HAS_GNU_AR y
LIB_LDFLAGS -shared -Wl,-soname,$(LIBFILENAME)
LDFLAG_LINKERSCRIPT -Wl,--version-script=
RDKAFKA_VERSION_STR 1.9.0
MKL_APP_VERSION 1.9.0
LIBS -lm -lsasl2 -lssl -lcrypto -lz -ldl -lpthread -lrt -lpthread -lrt
MKL_PKGCONFIG_LIBS_PRIVATE -lm -ldl -lpthread -lrt -lpthread -lrt
MKL_PKGCONFIG_REQUIRES_PRIVATE zlib libcrypto libssl libsasl2
CFLAGS
MKL_PKGCONFIG_REQUIRES zlib libcrypto libssl libsasl2
CXXFLAGS -Wno-non-virtual-dtor
SYMDUMPER $(NM) -D
MKL_DYNAMIC_LIBS -lm -lsasl2 -lssl -lcrypto -lz -ldl -lpthread -lrt -lpthread -lrt
exec_prefix /usr/local
bindir /usr/local/bin
sbindir /usr/local/sbin
libexecdir /usr/local/libexec
datadir /usr/local/share
sysconfdir /usr/local/etc
sharedstatedir /usr/local/com
localstatedir /usr/local/var
runstatedir /usr/local/var/run
libdir /usr/local/lib
includedir /usr/local/include
infodir /usr/local/info
mandir /usr/local/man
BUILT_WITH GCC GXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL SASL_CYRUS HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW
Generated config.cache

Now type 'make' to build
`

[edenhill]

Traceback (most recent call last):
File "", line 8, in
cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Java JAAS configuration is not supported, see https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka for more information."}

You can't use JAAS (Java-specific) configuration in the Python client.
Try setting sasl.username and sasl.password in your conf dict instead.

[prashantochatterjee]

That was STUPID on my part :(. Thanks for correcting me.

Works fine and reports an authentication error when correct user is not used

Python 3.7.4 (default, Dec 17 2021, 20:49:59) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from confluent_kafka import Consumer
>>> consumer = Consumer({
...         'bootstrap.servers': 'localhost:9093',
...         'group.id': 'test-consumer-group',
...         'security.protocol': 'SASL_PLAINTEXT',
...         'sasl.mechanism': 'SCRAM-SHA-256',
...         'sasl.username': 'foo.user',
...         'sasl.password': '9846c8dc-5b23-4e1a-a122-32d2b54ddb2f'
...     })
>>> %3|1643885693.317|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://localhost:9093/bootstrap]: sasl_plaintext://localhost:9093/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (after 338ms in state AUTH_REQ)
%3|1643885693.906|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://localhost:9093/bootstrap]: sasl_plaintext://localhost:9093/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256 (after 301ms in state AUTH_REQ, 1 identical error(s) suppressed)```