Docker Swarm v1.13 with Docker Compose Format v3

[djeeg]

Hi,

I am trying to get letsencrypt-nginx-proxy-companion to work with the latest docker swarm/compose

Unfortunately volumes_from can not be used with stacks

Compose file contains unsupported options:

volumes_from: To share a volume between services, define it using the top-level `volumes` option and reference it from each service that shares it using the service-level `volumes` option.

And it doesn't look like NGINX_PROXY_CONTAINER=**** can be used, as container_name cant be set either

Ignoring deprecated options:

container_name: Setting the container name is not supported.

I am working on a way to dynamically look up the containerId from a label

Set up the containers like this

  nginx:
    image: jwilder/nginx-proxy
    labels:
      - "NGINX_PROXY_SWARM_CONTAINER=true"

  letsencrypt-nginx-proxy-companion:
    image: ${DOCKER_REGISTRY}/hdtl-proxy-letsencrypt
    environment:
      - NGINX_PROXY_SWARM_CONTAINER=true

And then lookup the containerId in entrypoint.sh like this

function get_nginx_proxy_cid {
    # Look for a NGINX_VERSION environment variable in containers that we have mount volumes from.
    local volumes_from=$(docker_api "/containers/$CONTAINER_ID/json" | jq -r '.HostConfig.VolumesFrom[]' 2>/dev/null)
    for cid in $volumes_from; do
        cid=${cid%:*} # Remove leading :ro or :rw set by remote docker-compose (thx anoopr)
        if [[ $(docker_api "/containers/$cid/json" | jq -r '.Config.Env[]' | egrep -c '^NGINX_VERSION=') = "1" ]];then
            export NGINX_PROXY_CONTAINER=$cid
            break
        fi
    done

    # TEST: curl --unix-socket /var/run/docker.sock -G -XGET "http://localhost/containers/json" -d 'all=0&filters=%7B%22label%22%3A%5B%22NGINX_PROXY_SWARM_CONTAINER%22%5D%7D' | jq -r ".[] | .Id"
    echo "checking for swarm mode ... "
    if [[ -n "${NGINX_PROXY_SWARM_CONTAINER:-}" ]]; then
        echo "... swarm mode detected"
        echo ""
        echo "containers found:"
        docker_api "/containers/json?all=0&filters=%7B%22label%22%3A%5B%22NGINX_PROXY_SWARM_CONTAINER%22%5D%7D" | jq -r ".[] | .Names[0]"
        echo "---------------------------"
        echo ""
        nginxswarmcontainerid=$(docker_api "/containers/json?all=0&filters=%7B%22label%22%3A%5B%22NGINX_PROXY_SWARM_CONTAINER%22%5D%7D" | jq -r ".[] | .Id")
        echo "nginxswarmcontainerid: [$nginxswarmcontainerid]"
        echo "---------------------------"
        echo ""
        export NGINX_PROXY_CONTAINER=$nginxswarmcontainerid
    fi

    if [[ -z "${NGINX_PROXY_CONTAINER:-}" ]]; then
        echo "Error: can't get nginx-proxy container id !" >&2
        echo "Check that you use the --volumes-from option to mount volumes from the nginx-proxy." >&2
        exit 1
    fi
}

This seems to work:

checking for swarm mode ...
... swarm mode detected

containers found:
/proxy_nginx.1.bh6cty5dz9nlxvmw0znnmrep1
---------------------------

nginxswarmcontainerid: [06883fe069376656c3943b8ce2ba3710d0b174a7df0ab42bbafb55d641259317]
---------------------------

Sleep for 3600s
2017/01/01 20:52:55 Generated '/app/letsencrypt_service_data' from 9 containers
2017/01/01 20:52:55 Running '/app/update_certs'
2017/01/01 20:52:55 Watching docker events
2017/01/01 20:52:55 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/update_certs'

However then i get hit by #144, so can't confirm if this fully works

Is there a better way to do this?

[Hermsi1337]

+1

Experiencing the same problem.. Your trial works fine, maybe this one is worth a pull-request since no one seems to have an alternative solution?

[pi0]

@Hermsi1337 I can also confirm this works well with latest docker version 1.13.0 :)
Maybe we can make a PR from #741c683 with some docs and of course permissions of @djeeg :)
Meanwhile you can try using this patched image : banian/letsencrypt

[Hermsi1337]

Sorry for bothering @JrCs ... what do you think about this?

[mantensteiner]

Hi, I'm still not able to get the simple setup (2 containers) running in Docker Swarm v1.13. I'd be happy if anyone could help me out. I'm not sure if I need to specify the full volumes with same path in the letsencrypt-nginx-proxy-companion service, or just the name (I'm a little confused by the new volume-syntax to be honest).

Here my stack config:

version: "3"

volumes:
  nginx-conf-vol:
  nginx-vhost-vol:
  nginx-share-vol:
  nginx-certs-vol:

networks:
  test-net:
    driver: overlay

services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    networks:
      - test-net
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - nginx-conf-vol:"/etc/nginx/conf.d"
      - nginx-vhost-vol:"/etc/nginx/vhost.d"
      - nginx-share-vol:"/usr/share/nginx/html"
      - nginx-certs-vol:"./volumes/proxy/certs:/etc/nginx/certs:ro"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.labels.servertype == web]

  letsencrypt-nginx-proxy-companion:
    image: banian/letsencrypt
    networks:
      - test-net
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./volumes/proxy/certs:/etc/nginx/certs:rw"
      - nginx-conf-vol
      - nginx-vhost-vol
      - nginx-share-vol
      - nginx-certs-vol
    environment:
      - NGINX_DOCKER_GEN_CONTAINER=nginx-gen
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.labels.servertype == web]

  my-testapp:
    image: testapp
    networks:
      - test-net
    environment:
      - VIRTUAL_HOST=mydomain.com
      - VIRTUAL_NETWORK=test-net
      - VIRTUAL_PORT=80
      - LETSENCRYPT_HOST=mydomain.com
      - LETSENCRYPT_EMAIL=mymail@mydomain.com      
    ports:
      - "3000:3000"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.labels.servertype == app]        

[mbaig25]

@mantenpanther I'm having the same problem as you. Very confused about the new volume syntax as well. Were you able to find a solution at all?

[pi0]

@mantenpanther @mbaig25 What image are you currently using?

[mantensteiner]

I've tried the docker-letsencrypt-nginx-proxy-companion and as you can see from my stack config the banian/letsencrypt images.

[pi0]

@mantenpanther The problem is you are missing NGINX_PROXY_SWARM_CONTAINER label. It is important because this way only we can discover main proxy service in swarm from lets-encrypt container. Here is our working compose file :

version: '3'

services:

    proxy:
        image: banian/nginx-proxy
        deploy:
           placement:
                constraints:
                    - node.hostname==master_node_hostname_important
        labels:
            - NGINX_PROXY_SWARM_CONTAINER=true
        volumes:
            - /var/run/docker.sock:/tmp/docker.sock:ro
            - /data/local/nginx/vhost.d:/etc/nginx/vhost.d
            - /data/local/nginx/html:/usr/share/nginx/html
            - /data/local/certs:/etc/nginx/certs
            - /data/local/nginx/nginx_log:/var/log/nginx
        ports: ["80:80","443:443"]
        networks: [edge]
        restart: always

    letsencrypt:
        image: banian/letsencrypt
        deploy:
           placement:
                constraints:
                    - node.hostname==node2
        environment:
            - NGINX_PROXY_SWARM_CONTAINER=true
            #- DEBUG=true
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock:ro
            - /data/local/nginx/vhost.d:/etc/nginx/vhost.d
            - /data/local/nginx/html:/usr/share/nginx/html
            - /data/local/certs:/etc/nginx/certs
            - /data/local/nginx/nginx_log:/var/log/nginx
        restart: always

networks:
    edge:
        external: true

[mbaig25]

@pi0 Thanks for responding. I tried the config you provided but the letsencrypt image fails with the following error message:

"task: non-zero exit (1)" 

Here is my compose file

version: '3'
services:
    nginx:
        image: jwilder/nginx-proxy
        networks:
            - proxy
        ports:
            - "80:80"
            - "443:443"
        labels:
            - NGINX_PROXY_SWARM_CONTAINER=true
        volumes:
            - /etc/nginx/certs:/etc/nginx/certs:ro
            - /etc/nginx/vhost.d:/etc/nginx/vhost.d
            - /usr/share/nginx/html:/usr/share/nginx/html
            - /var/run/docker.sock:/tmp/docker.sock:ro
        deploy:
            placement:
                constraints:
                    - node.role==manager
    letsencrypt:
        image: jrcs/letsencrypt-nginx-proxy-companion
        networks:
            - proxy
        labels:
            - NGINX_PROXY_SWARM_CONTAINER=true
        depends_on:
            - nginx
        volumes:
            - /etc/nginx/certs:/etc/nginx/certs:rw
            - /etc/nginx/vhost.d:/etc/nginx/vhost.d
            - /usr/share/nginx/html:/usr/share/nginx/html
            - /var/run/docker.sock:/var/run/docker.sock:ro
        deploy:
            placement:
                constraints:
                    - node.role==manager

Any ideas what could be causing this?

[pi0]

@mbaig25 Please note that this feature is not merged to this repo! So you have to (temporary) use patched image banian/letsencrypt :)

[mantensteiner]

@pi0 Thanks for your answers, but still no luck for me. Containers do not start (tried on different hosts/docker engines; tried your file and the fixed one from mbaig25). I think I'll check again when this Issue is solved, can't spend more time ATM.

[mbaig25]

@pi0 I tried your suggested image and I'm still getting the same error.

I used the following command to try it:

docker service create --name letsencrypt-companion \
    -l NGINX_PROXY_SWARM_CONTAINER=true \
    --network proxy \
    --mount "type=bind,source=/etc/nginx/certs,target=/etc/nginx/certs" \
    --mount "type=bind,source=/etc/nginx/vhost.d,target=/etc/nginx/vhost.d" \
    --mount "type=bind,source=/usr/share/nginx/html,target=/usr/share/nginx/html" \
    --mount "type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock" \
    --constraint 'node.role == manager' \
    banian/letsencrypt

I started nginx-proxy prior to this and already have the volumes created on the manager machine.

Still no luck. 😕

[JrCs]

As i don't support docker-swarm, docker-compose and other tools to start the container, i think the best is to use a dedicated variable like in PR #126 to force the id of the nginx proxy container.
PR are welcome.