KafkaConnect and Kafkaconnector for google cloud pub sub unable to work for k8s.

[qafro1]

Am unable to get the Kafka connector google pubsub to work in k8s but works locally.
kubectl create secret generic pubsub-key --from-file=key.json=key.json -n kafka

• Infrastructure: [ Minikube]

YAML files and logs
Strimzi version: 0.24
Helm charts install

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi
  jvmOptions:
    gcLoggingEnabled: false
  config:
    group.id: my-connect-cluster
    config.providers: file 
    config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
    offset.storage.topic: my-connect-cluster-offsets
    config.storage.topic: my-connect-cluster-configs
    status.storage.topic: my-connect-cluster-status
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    config.storage.replication.factor: 3 #n for production per n brokers
    offset.storage.replication.factor: 3 #per broker
    status.storage.replication.factor: 3 #per broker
  build:
    output:
      type: docker
      image: docker.io/simplifaisystems/kafkaconnect:v2
      pushSecret: my-registry-credentials
    plugins:
      - name: pubsub-sink-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-pubsub-kafka-connector/0.10.1/camel-google-pubsub-kafka-connector-0.10.1-package.tar.gz
      - name: storage-sink-connector
        artifacts:
          - type: tgz
            url: https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-google-storage-kafka-connector/0.10.1/camel-google-storage-kafka-connector-0.10.1-package.tar.gz
  template:
    pod:
      imagePullSecrets:
        - name: my-registry-credentials
  externalConfiguration:
    volumes:
      - name: google
        secret:
          secretName: pubsub-key 

Kafkaconnector yaml

kind: KafkaConnector
apiVersion: kafka.strimzi.io/v1beta2
metadata:
  name: camelgoogle-pubsubsinkconnector
  namespace: kafka
  labels:
    # must match connect cluster name
    strimzi.io/cluster: my-connect-cluster
spec:
  tasksMax: 5
  class: org.apache.camel.kafkaconnector.googlepubsub.CamelGooglepubsubSinkConnector
  
  # use the kafka converters that better suit your needs, these are just defaults:
  #key.converter: org.apache.kafka.connect.storage.StringConverter
  #value.converter: org.apache.kafka.connect.storage.StringConverter
  config:
    # comma separated topics to get messages from
    topics: test-topic
    # mandatory properties (for a complete properties list see the connector documentation):
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    # The Service account key that can be used as credentials for the PubSub publisher/subscriber. It can be loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems.
    camel.component.google-pubsub.serviceAccountKey: ${file:/opt/kafka/external-configuration/google/pubsub-key:key.json}
    # The Service account key that can be used as credentials for the PubSub publisher/subscriber. It can be loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems.
    camel.sink.endpoint.serviceAccountKey: ${file:/opt/kafka/external-configuration/google/pubsub-key:key.json}
    # The Destination Name. For the consumer this will be the subscription name, while for the producer this will be the topic name.
    camel.sink.path.destinationName: test-topic
    # The Google Cloud PubSub Project Id
    camel.sink.path.projectId: <PROJECTID>

error msg:

2021-07-15 11:02:14,157 ERROR Uncaught exception in REST call to /connectors/camelgoogle-pubsubsinkconnector/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper) [qtp1887]
org.apache.kafka.common.config.ConfigException: Could not read properties from file /opt/kafka/external-configuration/google/pubsub-key
	at org.apache.kafka.common.config.provider.FileConfigProvider.get(FileConfigProvider.java:92)
	at org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)
	at org.apache.kafka.connect.runtime.WorkerConfigTransformer.transform(WorkerConfigTransformer.java:58)
	at org.apache.kafka.connect.runtime.WorkerConfigTransformer.transform(WorkerConfigTransformer.java:53)
	at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:361)

Any ideas cos I've followed the doc

[scholzj]

Is /opt/kafka/external-configuration/google/pubsub-key a properties file (https://en.wikipedia.org/wiki/.properties)? If not the error means exactly what it says it means.

[qafro1]

I used kubectl create secret generic pubsub-key --from-file=key.json=key.json -n kafka to create the secret in k8s and it works locally for gcp pubsub.
Please how do I create the properties file for the key.json?

[qafro1]

I've created the json properties and now there are many key values. Please what's the best way to use the property for the file?

[perebaj]

did you find the solution?