OPA, Kubernetes role/rolebinding and Strimzi

[gwenael-lebarzic]

Hello.

I start this discussion because I am trying to use OPA as an authorization system for apache Strimzi (kafka cluster on Kubernetes).

I'm working on a Kubernetes cluster where I am not admin.
I work in a namespace my-namespace, where I am admin of the namespace.

I try to deploy OPA like this :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: opa
  labels:
    app: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      securityContext:
        runAsUser: 1001
        fsGroup: 1001
      containers:
      - name: kube-mgmt
        image: openpolicyagent/kube-mgmt:8.3.0
        args:
          - "--replicate=kafka.strimzi.io/v1beta2/kafkatopics"
          - "--replicate=kafka.strimzi.io/v1beta2/kafkausers"
          - "--namespaces=my-namespace"
        resources:
          requests:
            memory: 500Mi
            cpu: "1"
          limits:
            memory: 500Mi
            cpu: "1"
      - name: opa
        image: openpolicyagent/opa:0.57.0
        resources:
          requests:
            memory: 1Gi
            cpu: "4"
          limits:
            memory: 1Gi
            cpu: "4"
        ports:
        - name: http
          containerPort: 8181
        args:
        - "run"
        - "--ignore=.*"  # exclude hidden dirs created by Kubernetes
        - "--server"
        - "/policies"
        volumeMounts:
        - readOnly: true
          mountPath: /policies
          name: kafka-policies
      volumes:
      - name: kafka-policies
        configMap:
          name: kafka-policies

When I deploy the opa container like this, with kube-mgmt container, I can see the following errors in the logs of opa container :

E1018 08:27:20.433570       1 reflector.go:138] k8s.io/client-go@v0.23.8/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: kafkatopics.kafka.strimzi.io is forbidden: User "system:serviceaccount:my-namespace:default" cannot list resource "kafkatopics" in API group "kafka.strimzi.io" at the cluster scope
W1018 08:27:26.511207       1 reflector.go:324] k8s.io/client-go@v0.23.8/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: kafkausers.kafka.strimzi.io is forbidden: User "system:serviceaccount:my-namespace:default" cannot list resource "kafkausers" in API group "kafka.strimzi.io" at the cluster scope

I created my custom resources kafkaTopics and kafkaUsers in my namespace my-namespace.
Is it possible to make OPA work with a service account without clusterRole ?

I tried to create a role and a rolebinding for the default service account of my namespace :

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: strimzi-view
  labels:
    app: strimzi
    # Add these permissions to the "view" default role.
    rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
  - apiGroups:
      - "kafka.strimzi.io"
    resources:
      - kafkas
      - kafkaconnects
      - kafkamirrormakers
      - kafkausers
      - kafkatopics
      - kafkabridges
      - kafkaconnectors
      - kafkamirrormaker2s
      - kafkarebalances
    verbs:
      - get
      - list
      - watch

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: strimzi-opa
  labels:
    app: strimzi
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: strimzi-view
  apiGroup: rbac.authorization.k8s.io

But it does not help sadly.

Is it possible to use OPA like I try, only to handle authorizations for a kafka cluster with strimzi operator and not being admin of the kubernetes cluster ?

[anderseknert]

Hi there!

The error you're seeing is not really related to OPA, as it's kube-mgmt trying to replicate data from the cluster, and failing with a permissions error. Can you list those resources using the same service account and kubectl? That'd be the first thing to make sure it works, as if that works, there's probably some issue in kube-mgmt. If that doesn't work, you'll need to have that fixed first.

[gwenael-lebarzic]

Hello.

Thank you for your first answer !

I specified a specific service account for my opa deployment : opa-deployer.
I checked that I was encountering the same error.

Then I retrieve the secret of the service account and use it in a kubectl command :

kubectl -n my-namespace --token=<thetoken> get kafkatopics
NAME                                                                                               AGE
consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a                                        12d
strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55                                     12d
strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b   12d

Same with kafkausers, it works.
So the serviceaccount can list custom resources kafkatopics and kafkausers in my namespace, but it cannot do it on other namespaces.

Is there a parameter to specify in the deployment.yaml to tell opa to not watch all the namespaces ?

[gwenael-lebarzic]

If I check the documentation for kube-mgmt for custom resources :
https://github.com/open-policy-agent/kube-mgmt/blob/master/docs/admission-control-crd.md#admission-control-for-custom-resources

There are three items :

1. Define a role for reading Kubernetes custom resources.
2. Grant OPA/kube-mgmt permissions to read Kubernetes custom resources.
3. Configure kube-mgmt to load Kubernetes custom resources into OPA.

1 is done thanks to the role indicated in my original post (strimzi-view)
2 is done thanks to the rolebinding indicated in my original post (strimzi-opa)
3 is done by adding the --replicate options in the args key in the deployment.yaml (I think)

I wonder what could be the cause of this problem.

[anderseknert]

Yeah, I'm afraid I'm not too familiar with how that all works either. Perhaps open an issue in the kube-mgmt project, to see if anyone else is able to help you out.

[charlieegan3]

Is there a parameter to specify in the deployment.yaml to tell opa to not watch all the namespaces ?

There is the --namespaces flag on kube-mgmt that might do the trick if you set it to just your namespace?

[gwenael-lebarzic]

Is there a parameter to specify in the deployment.yaml to tell opa to not watch all the namespaces ?

There is the --namespaces flag on kube-mgmt that might do the trick if you set it to just your namespace?

Thank you for your answer. I already set --namespaces arguments but I still have the same error.

[gwenael-lebarzic]

Yeah, I'm afraid I'm not too familiar with how that all works either. Perhaps open an issue in the kube-mgmt project, to see if anyone else is able to help you out.

Thank you, I'm going to try that.