Unable to Access Kibana from Seqr Web application

[DeepakRamchandani1]

I am trying to access Kibana from the Seqr web application which is deployed on a Google Kubernetes Engine Cluster. When I am trying to click on the Kibana button which is present on the Seqr UI, it's redirecting me back to the Seqr login page. When I am checking the logs it's showing me the below error:

login/?msg=UNAUTHENTICATED&next=%2Fapp%2Fkibana" , status: 401, "severity": "WARNING",

{"type":"response","@timestamp":"2022-06-03T07:56:37+00:00","tags":[],"pid":8,"method":"get","statusCode":401,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36","accept-encoding":"gzip, deflate, br","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","connection":"keep-alive","host":"kibana-kb-http:5601",}

Also, I am able to access the Kibana directly from the Kibana Pod's IP Address provided I am in the same network but, when I am trying to access it through the Seqr web application, it's throwing me back at the Seqr Login page. and stating the above-mentioned errors.

[hanars]

This looks like an error with the kibana configuration we provide in the open source version of seqr, so I'm sorry for that. Try adding the following to the environment section of the kibana configuration in your docker-compose:

- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- XPACK_SECURITY_SAMESITECOOKIES=None
- XPACK_SECURITY_AUTHC_PROVIDERS_ANONYMOUS_ANONYMOUS1_ORDER=0
- XPACK_SECURITY_AUTHC_PROVIDERS_ANONYMOUS_ANONYMOUS1_CREDENTIALS=elasticsearch_anonymous_user

[DeepakRamchandani1]

Any update on this?

[hanars]

Hi Deepak, seqr is a freely available piece of open source software with a one person support team. Please have some patience with us, we try to respond within a week and faster if possible, but we do not have the resources to respond as quickly as you would like.

You are also using seqr in an unsupported manner, which makes helping you very difficult. The servctl code is not suitable for local installs which is why it is not included in our local install documentation, and as you have not followed our deployment instructions it is very hard for us to get this to work for you. I am trying my best to figure out your issue without asking you to completely redeploy the system using the recommended docker compose approach, but it means this is a lot of trial and error and it will take a while.

I will try to get back to you by EOD tomorrow with some updated suggestions

[hanars]

also, could you please provide the elasticsearch.yml file you are using?

[DeepakRamchandani1]

Hi, I am running the servctl code for installing the Seqr on the Google Kubernetes Engine. I am not using this for local installs. Also, while running the servctl code it automatically installed the CRDs(Custom Resource Definition) and Operators for the Elastic-Search and Kibana.

CRDs and Operator links are present in deploy_commands_utils.py file present in seqr/deploy/servctl_utils directory on line number 152:

run('kubectl create -f https://download.elastic.co/downloads/eck/1.9.1/crds.yaml')

run('kubectl apply -f https://download.elastic.co/downloads/eck/1.9.1/operator.yaml') 

After installing that, it runs Elasticsearch.yml file and the same approach goes for kibana as well.

I am adding the contents of the Elasticsearch.yml file below

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
  namespace: {{ NAMESPACE }}
  labels:
    name: elasticsearch
    deployment: {{ DEPLOY_TO }}
spec:
  version: {{ ELASTICSEARCH_VERSION }}
  auth:
    roles:
      - secretName: elasticsearch-secrets
    fileRealm:
      - secretName: elasticsearch-secrets
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  {% if ES_CONFIGURE_SNAPSHOTS %}
  secureSettings:
    - secretName: es-snapshot-gcs-secrets
  {% endif %}
  nodeSets:
  - name: master-node
    count: {{ ES_MASTER_NUM_PODS }}
    config:
      node.master: true
      node.data: false
      node.ingest: false
      node.ml: false
      node.remote_cluster_client: false
      cluster.routing.allocation.disk.watermark.low: "91%"
      cluster.routing.allocation.disk.watermark.high: "95%"
      cluster.routing.allocation.disk.watermark.flood_stage: "97%"
    podTemplate:
      metadata:
        labels:
          name: elasticsearch
          deployment: {{ DEPLOY_TO }}
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: nodeType
                  operator: NotIn
                  values:
                  - data
        initContainers:
          - name: sysctl
            securityContext:
              privileged: true
            command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          - name: install-plugins
            command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-gcs']
        containers:
        - name: elasticsearch
          imagePullPolicy: {{ IMAGE_PULL_POLICY }}
    volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: standard
            resources:
              requests:
                storage: {{ ES_POD_VOLUME_SIZE }}
  - name: client-node
    count: {{ ES_CLIENT_NUM_PODS }}
    config:
      node.master: false
      node.data: false
      node.ingest: false
      node.ml: false
      node.remote_cluster_client: false
      cluster.routing.allocation.disk.watermark.low: "91%"
      cluster.routing.allocation.disk.watermark.high: "95%"
      cluster.routing.allocation.disk.watermark.flood_stage: "97%"
    podTemplate:
      metadata:
        labels:
          name: elasticsearch
          deployment: {{ DEPLOY_TO }}
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: nodeType
                  operator: NotIn
                  values:
                  - data
        initContainers:
          - name: sysctl
            securityContext:
              privileged: true
            command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          - name: install-plugins
            command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-gcs']
        containers:
        - name: elasticsearch
          imagePullPolicy: {{ IMAGE_PULL_POLICY }}
    volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: standard
            resources:
              requests:
                storage: {{ ES_POD_VOLUME_SIZE }}
  - name: data
    count:  {{ ES_DATA_NUM_PODS }}
    config:
      node.master: false
      node.data: true
      node.ingest: false
      node.ml: false
      node.remote_cluster_client: false
      cluster.routing.allocation.disk.watermark.low: "91%"
      cluster.routing.allocation.disk.watermark.high: "95%"
      cluster.routing.allocation.disk.watermark.flood_stage: "97%"
      cluster.routing.allocation.total_shards_per_node: {{ ELASTICSEARCH_MAX_SHARDS_PER_NODE }}
    podTemplate:
      metadata:
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
        labels:
          name: elasticsearch
          deployment: {{ DEPLOY_TO }}
      spec:
      {% if ES_DATA_NUM_PODS > 1 %}
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: nodeType
                  operator: In
                  values:
                  - data
      {% endif %}
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        - name: install-plugins
          command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-gcs']
        containers:
        - name: elasticsearch
          imagePullPolicy: {{ IMAGE_PULL_POLICY }}
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms{{ ELASTICSEARCH_JVM_MEMORY }} -Xmx{{ ELASTICSEARCH_JVM_MEMORY }}"
          resources:
            requests:
              memory: {{ ELASTICSEARCH_CONTAINER_MEMORY }}
            limits:
              memory: {{ ELASTICSEARCH_CONTAINER_MEMORY }}
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: {{ ES_DATA_DISK_TYPE }}-storage-class
        resources:
          requests:
            storage: {{ ES_DATA_DISK_SIZE }}
{% if ES_LOADING_NUM_PODS > 0 %}
  - name: data-loading-node
    count: {{ ES_LOADING_NUM_PODS }}
    config:
      node.master: false
      node.data: true
      node.ingest: true
      node.ml: false
      node.remote_cluster_client: false
      cluster.routing.allocation.disk.watermark.low: "91%"
      cluster.routing.allocation.disk.watermark.high: "95%"
      cluster.routing.allocation.disk.watermark.flood_stage: "97%"
    podTemplate:
      metadata:
        labels:
          name: elasticsearch
          deployment: {{ DEPLOY_TO }}
      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: nodeType
                  operator: NotIn
                  values:
                  - data
        initContainers:
          - name: sysctl
            securityContext:
              privileged: true
            command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
          - name: install-plugins
            command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-gcs']
        containers:
        - name: elasticsearch
          imagePullPolicy: {{ IMAGE_PULL_POLICY }}
          #environment:
          #- LS_JAVA_OPTS=-Xms2048m -Xmx2048m
    volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: standard
            resources:
              requests:
                storage: {{ ES_LOADING_DISK_SIZE }}
{% endif %}
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-es-data-nodes
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
    #cloud.google.com/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  #{% if ES_DATA_SERVICE_IP %}
  #loadBalancerIP: {{ ES_DATA_SERVICE_IP }}
  #{% endif %}
  ports:
    - name: http
      port: 9200
      targetPort: 9200
  selector:
    elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
    elasticsearch.k8s.elastic.co/node-data: "true"

[hanars]

Hi Deepak,

Just to be clear, you are running a local installation of seqr, regardless of how or where you deploy it. Every installation that is not the Broad's installation is considered a local installation. Again, the fact that servctl is not publicly documented is meant to be the first indicator that it is not meant to be used outside if the Broad. We are in the process of trying to create more flexible deployment options for users who wish to use kubernetes, but as I said before we have a small development team and have not finished yet. As an FYI, servctl will be deprecated fully when this happens, hopefully by the end of the year.

Since you are committed to using the servctl code in the meantime I will try to help you get this working, but I can't promise it will work, as I will once again stress that this is not meant to be used as a public tool. I also once again ask you for your patience and grace as I help you with this.

Try adding the following to the config sections for all the nodeSets in elasticsearch.yml

xpack.security.authc.anonymous.roles: superuser, kibana_admin