Broker didn't receive the message from inference service

[uraychang]

Hello team, I was following the instruction here to setup the integration of inference service and drift detector (on my local minikube). If I understand it right, the broker captures the logs from inference service. Then detector as the subscriber consumes the message from broker. So as the first step, I wanna make sure the event is published successfully by the inference service. But, there's no any log in broker ingress pod after I triggered a request to the inference service. Same for the event-display service which is deployed for testing the message consumption. I've check all the status of my components. They looked up and running well. Hope anyone with any idea can provide some thoughts, thanks! Willing to provide more info if needed.

K8s resources:

% k get pods
NAME                                                   READY   STATUS    RESTARTS   AGE
debug-pod                                              1/1     Running   0          30m
hello-display-85f6789687-vttj7                         1/1     Running   0          97m
iris-demo-predictor-00001-deployment-d875664d8-n6dv6   3/3     Running   0          103m

% k get pods -n knative-eventing
NAME                                    READY   STATUS    RESTARTS       AGE
eventing-controller-cc45869cd-lfht8     1/1     Running   3 (112m ago)   131m
eventing-webhook-67fcc6959b-gqnmq       1/1     Running   4 (112m ago)   131m
imc-controller-78848895c-sw88q          1/1     Running   5 (112m ago)   128m
imc-dispatcher-79f95cc86c-p2b6g         1/1     Running   6 (111m ago)   128m
job-sink-7f5d754db-qjm4g                1/1     Running   6 (111m ago)   131m
mt-broker-controller-5f64b86f8c-7kxgk   1/1     Running   0              128m
mt-broker-filter-5b46f6945d-ffm2j       1/1     Running   6 (112m ago)   128m
mt-broker-ingress-58ffc5448-vxmcs       1/1     Running   0              128m

% k get triggers.eventing.knative.dev event-display-trigger
NAME                    BROKER    SUBSCRIBER_URI                                   AGE   READY   REASON
event-display-trigger   default   http://hello-display.default.svc.cluster.local/   58m   True

% k get brokers.eventing.knative.dev default
NAME      URL                                                                        AGE    READY   REASON
default   http://broker-ingress.knative-eventing.svc.cluster.local/default/default   107m   True

Inference yaml:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: iris-demo
spec:
  predictor:
    imagePullSecrets:
        - name: my-dockerhub-secret
    model:
      modelFormat:
        name: sklearn
      storageUri: kit://docker.io/<my-docker-repo>/iris-demo:sk-learn
    logger:
      mode: all
      url: http://broker-ingress.knative-eventing.svc.cluster.local/default/default
     

Broker yaml

apiVersion: eventing.knative.dev/v1
kind: broker
metadata:
 name: default

Event-display yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-display
spec:
  replicas: 1
  selector:
    matchLabels: &labels
      app: hello-display
  template:
    metadata:
      labels: *labels
    spec:
      containers:
        - name: event-display
          image: gcr.io/knative-releases/knative.dev/eventing-contrib/cmd/event_display

---

kind: Service
apiVersion: v1
metadata:
  name: hello-display
spec:
  selector:
    app: hello-display
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Event-display trigger yaml

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: event-display-trigger
spec:
  broker: default
  # filter:
  #   attributes:
  #     type: org.kubeflow.serving.inference.request
  subscriber:
    ref:
      apiVersion: v1
      kind: Service
      name: hello-display