Containerized apb doesn't work with OKD 3.11

[aliok]

Followed the docs from:

• https://github.com/openshift/ansible-service-broker#getting-started-on-openshift
• https://github.com/ansibleplaybookbundle/ansible-playbook-bundle/blob/master/docs/apb_cli.md#running-from-a-container

1. Start OKD with ASB:

> oc cluster up --enable=service-catalog,automation-service-broker
Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Pulling image openshift/origin-cli:v3.11
Image pull complete
Checking type of volume mount ...
Determining server IP ...
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11 ...
I1120 11:41:13.529522    7773 config.go:40] Running "create-master-config"
I1120 11:41:15.473939    7773 config.go:46] Running "create-node-config"
I1120 11:41:16.437703    7773 flags.go:30] Running "create-kubelet-flags"
I1120 11:41:17.203978    7773 run_kubelet.go:49] Running "start-kubelet"
I1120 11:41:17.465986    7773 run_self_hosted.go:181] Waiting for the kube-apiserver to be ready ...
I1120 11:41:35.508850    7773 interface.go:26] Installing "kube-proxy" ...
I1120 11:41:35.508897    7773 interface.go:26] Installing "kube-dns" ...
I1120 11:41:35.508915    7773 interface.go:26] Installing "openshift-service-cert-signer-operator" ...
I1120 11:41:35.508928    7773 interface.go:26] Installing "openshift-apiserver" ...
I1120 11:41:35.508982    7773 apply_template.go:81] Installing "kube-proxy"
I1120 11:41:35.509030    7773 apply_template.go:81] Installing "openshift-service-cert-signer-operator"
I1120 11:41:35.509271    7773 apply_template.go:81] Installing "kube-dns"
I1120 11:41:35.510481    7773 apply_template.go:81] Installing "openshift-apiserver"
I1120 11:41:37.505082    7773 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-service-cert-signer-operator" "openshift-apiserver"
I1120 11:43:03.523414    7773 run_self_hosted.go:242] openshift-apiserver available
I1120 11:43:03.523445    7773 interface.go:26] Installing "openshift-controller-manager" ...
I1120 11:43:03.523464    7773 apply_template.go:81] Installing "openshift-controller-manager"
I1120 11:43:05.355669    7773 interface.go:41] Finished installing "openshift-controller-manager"
Adding default OAuthClient redirect URIs ...
Adding service-catalog ...
Adding automation-service-broker ...
I1120 11:43:05.370220    7773 interface.go:26] Installing "openshift-service-catalog" ...
I1120 11:43:05.370238    7773 interface.go:26] Installing "automation-service-broker" ...
I1120 11:43:05.372586    7773 apply_template.go:81] Installing "automation-service-broker"
I1120 11:43:05.400350    7773 apply_template.go:81] Installing "service-catalog"
I1120 11:43:48.266523    7773 interface.go:41] Finished installing "openshift-service-catalog" "automation-service-broker"
Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.

The server is accessible via web console at:
    https://127.0.0.1:8443

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

2. Give cluster admin permissions to developer

>  oc login -u system:admin
> oc adm policy add-cluster-role-to-user cluster-admin developer
cluster role "cluster-admin" added: "developer"

3. Login with developer

oc login -u developer

4. Have apb in path with the content of https://raw.githubusercontent.com/ansibleplaybookbundle/ansible-playbook-bundle/master/scripts/apb-docker-run.sh

#!/bin/bash
# Script for running apb with a container.
# Recommended to copy this to somewhere in your PATH as "apb"
APB_IMAGE=${APB_IMAGE:-docker.io/ansibleplaybookbundle/apb-tools:canary}

if [[ $(id -u) = 0 ]]; then
  echo "apb should not be run as root!"
  exit 1
fi

echo "Running APB image: ${APB_IMAGE}"

if ! [[ -z "${DOCKER_CERT_PATH}" ]] && [[ ${DOCKER_CERT_PATH} = *"minishift"* ]]; then
  IS_MINISHIFT=true
  echo "Targetting minishift host: ${DOCKER_HOST}"
fi

KUBECONFIG_ENV="${KUBECONFIG:+-v ${KUBECONFIG}:${KUBECONFIG} -e KUBECONFIG=${KUBECONFIG}}"

if [[ $IS_MINISHIFT = true ]]; then
  # If targetting minishift, there are some unique issues with using the apb
  # container. Need to capture the minishift docker-env vars, unset them for the
  # purposes of this command, and pass them through to the docker container along
  # with mounting the minishift docker certs.
  # The minishift docker-env must be unset so the apb container is run by the *host*
  # daemon instead of the minishift daemon. However, It will still be configured
  # to operate on the minishift registry. This is required, as the volume mounts
  # must be mounted into the apb container from the host system.
  # If the minishift daemon is used, they will be empty mounts.
  MINISHIFT_DOCKER_CERT_SRC="${DOCKER_CERT_PATH}"
  MINISHIFT_DOCKER_CERT_DEST="/var/run/minishift-certs"
  MINISHIFT_DOCKER_HOST="${DOCKER_HOST}"

  unset DOCKER_TLS_VERIFY
  unset DOCKER_HOST
  unset DOCKER_CERT_PATH

  docker run --rm --privileged \
    -v $PWD:/mnt -v $HOME/.kube:/.kube \
    -v $MINISHIFT_DOCKER_CERT_SRC:$MINISHIFT_DOCKER_CERT_DEST \
    -e DOCKER_TLS_VERIFY="1" \
    -e DOCKER_HOST="${MINISHIFT_DOCKER_HOST}" \
    -e DOCKER_CERT_PATH="${MINISHIFT_DOCKER_CERT_DEST}" \
    -e MINISHIFT_REGISTRY=$(minishift openshift registry) \
    ${KUBECONFIG_ENV} \
    -u $UID $APB_IMAGE "$@"
else
  docker run --rm --privileged \
    -v $PWD:/mnt -v $HOME/.kube:/.kube \
    -v /var/run/docker.sock:/var/run/docker.sock \
    ${KUBECONFIG_ENV} \
    -u $UID $APB_IMAGE "$@"
fi

5. Run apb list

> apb list
Running APB image: docker.io/ansibleplaybookbundle/apb-tools:canary
2018-11-20 08:45:51,913 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f70093de950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /oapi/v1/namespaces/ansible-service-broker/routes
2018-11-20 08:45:51,914 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f70093de490>: Failed to establish a new connection: [Errno 111] Connection refused',)': /oapi/v1/namespaces/ansible-service-broker/routes
2018-11-20 08:45:51,914 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f70093de150>: Failed to establish a new connection: [Errno 111] Connection refused',)': /oapi/v1/namespaces/ansible-service-broker/routes
Exception occurred! HTTPSConnectionPool(host='127.0.0.1', port=8443): Max retries exceeded with url: /oapi/v1/namespaces/ansible-service-broker/routes (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f70093de590>: Failed to establish a new connection: [Errno 111] Connection refused',))

Tried with canary and latest. Same error in both.
Tried passing --net=host to Docker command, other errors occurred in that case.

[dymurray]

Hi,

This project is deprecated and is not supported for OKD 3.11. We have moved repositories and the relevant documentation for running apb 2.0+ on 3.11 in a container is here: https://github.com/automationbroker/apb/blob/master/docs/apb_cli.md#running-from-a-container.

It does appear that the canary image is out of date on Dockerhub which I will work to resolve. For now you can use the nightly tag which is what canary should be.

Please confirm for me that you are aware of the new repository as there was a significant refactoring of this project and apb list is now apb bundle list so it appears that the output you are seeing is out of date.

[aliok]

Thanks for the help @dymurray

This project is deprecated and is not supported for OKD 3.11. We have moved repositories and the relevant documentation for running apb 2.0+ on 3.11 in a container is here: https://github.com/automationbroker/apb/blob/master/docs/apb_cli.md#running-from-a-container.

That documentation is the 'old' documentation which instructs setting up apb as this script: https://raw.githubusercontent.com/ansibleplaybookbundle/ansible-playbook-bundle/master/scripts/apb-docker-run.sh
And, that script uses APB_IMAGE=${APB_IMAGE:-docker.io/ansibleplaybookbundle/apb-tools:canary}

Anyway, I tried with docker.io/ansibleplaybookbundle/apb-tools:nightly and I can now see the new tool with bundle subcommand.

Both canary and stable are pointing to the old tool.

Thanks for the help. I am OK to resolve this issue. I am having some other problems with the new tool like following but I will create new tickets (or check existing tickets):

$ apb registry add lo --type local_openshift --namespaces openshift
Running APB image: docker.io/ansibleplaybookbundle/apb-tools:nightly
level=warning msg="Didn't find config file /.apb/registries.json, creating."
level=error msg="open /.apb/registries.json: no such file or directory"
$ apb bundle list
Running APB image: docker.io/ansibleplaybookbundle/apb-tools:nightly
level=warning msg="Didn't find config file /.apb/registries.json, creating."
level=error msg="open /.apb/registries.json: no such file or directory"

[aliok]

Created automationbroker/apb#142 for the problems I am having with the new apb