Page tree
Skip to end of metadata
Go to start of metadata

The compose templates provided by the SSDT do not publish ports for the applications outside of the docker host.  Each ITC must decide how they are going to be mapping the Tomcat port from each application (port 8080) to make it available to external web browsers.

This article explains how to use nginx-proxy to create a reverse proxy which automatically updates as containers are started and stopped. Note this is just one option for the reverse proxy.

The goal of this article is to

  • start with a basic reverse proxy
  • add SSL encryption with a signed (or self-signed) certificate
  • Optionally, use LetsEncrypt.org to automatically generate signed certificates

Tutorial

Below are steps to configure nginx-proxy on your docker host.  The tutorial assumes the following:



Docker hostdocker-01.example.com
sub-domain for virtual hostsdemo.example.com

Create a Reverse Proxy on port 80

  1. Wildcard DNS: Create DNS records in the domain to point to the docker host. 

    *.demo.example.com   IN    CNAME docker-01.example.com

    This wildcard will be used for all virtual hosts on this machine.  If you intend to also use other subdomains for this host, you can add hostnames in other domains.  For example, if "sampletown" has their own domain, then you could also add these RR's to sampletown's domain: 

    usas                 IN CNAME docker-01.example.com
    usps                 IN CNAME docker-01.example.com

    Because USAS and USPS run in separate containers they each must have their own virtual domain.

  2.  Create a directory on the docker host, e.g. /data/proxy/ and create the following docker-compose.yml:

  3. Some newer versions of nginx may contribute to some network issues.  This has not been verified, but to pull an older version use:

    image: jwilder/nginx-proxy:0.9.1


    image: jwilder/nginx-proxy:0.9.1-alpine

    Different information about the images can be seen here:  https://hub.docker.com/r/jwilder/nginx-proxy    go to the tags column to see the versions



    version: "3"
    services:
      proxy:
         image: jwilder/nginx-proxy:alpine
         restart: always
         volumes:
           - /var/run/docker.sock:/tmp/docker.sock:ro
           - ./certs:/etc/nginx/certs:ro
           - ./vhost.d:/etc/nginx/vhost.d
           - ./html:/usr/share/nginx/html
           - ./conf.d:/etc/nginx/conf.d
         environment:
           - DEFAULT_HOST=demo.example.com
         ports:
           - "443:443"
           - "80:80"

    The volume definitions are not strictly necessary at this point. However, adding them here is harmless and will make the subsequent instructions easier.

  4. Run the proxy:

    > docker-compose up -d

    You should now be able to visit:  http://sampletown.demo.example.com/. However, you'll receive a "503 Service Temporarily Unavailable" because nginx-proxy does not know how to discover our service containers.

  5. In your district's compose project, add (or modify) docker-compose.override.yml file to define VIRTUAL_HOST and VIRTUAL_PORT environments for each service: 

    usasapp:
      environment:
        - VIRTUAL_HOST=sampletown-usas.demo.example.com
        - VIRTUAL_PORT=8080
    uspsapp:
      environment:
        - VIRTUAL_HOST=sampletown-usps.demo.example.com
        - VIRTUAL_PORT=8080
    
    

    The nginx-proxy container will monitor docker events. Each time a container starts or stops, which has a VIRTUAL_HOST variable, it will create a new nginx configuration which reverse proxies port 80 for the virtual domain to 8080 of the container. 

  6. Rebuild the district's containers with: 

    docker-compose up -d

    After the apps start, you should be able to reach the applications at: http://sampletown-usas.demo.example.com and http://sampletown-usps.demo.example.com.

  7. Repeat the previous two steps for each school district. 
  8. If the host is not working, you can check the nginx-proxy log files with: 

    cd /data/proxy
    docker-compose logs


HTTPS proxy on port 443

If you wish to use LetsEncrypt.org for automated certificate signing, then skip this section.

Now that we have a reverse proxy, we can secure the port using HTTPS.  In this example, we are creating a wildcard certificate to match the wildcard DNS entry.  In this example, the "Common Name" is "*.demo.example.com".

A wildcard certificate only covers one level of subdomains.  For example, you cannot use *.example.com as a wildcard certificate for sampletown-usas.demo.example.com because, in this case there are two subdomain levels. The wildcard certificate needs to be *.demo.example.com.


  1. Create a certificate and CSR in the proxy's ./certs directory (this volume was mounted in the proxy's docker-compose.yml file above). 

    data/proxy# mkdir -p certs
    data/proxy# cd certs
    data/proxy/certs# # Create a private key:
    data/proxy/certs# openssl genrsa -out demo.example.com.key 2048
    data/proxy/certs# # Create a CSR from the new key:
    data/proxy/certs# openssl req -new -sha256 -key demo.example.com.key -out demo.example.com.csr
    -----
    Country Name (2 letter code) [AU]:US
    State or Province Name (full name) [Some-State]:Ohio
    Locality Name (eg, city) []:Archbold
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:Your Organization name
    Organizational Unit Name (eg, section) []:Your OU
    Common Name (e.g. server FQDN or YOUR name) []:*.demo.example.com.
    Email Address []:hostmaster@example.com
    Please enter the following 'extra' attributes
    to be sent with your certificate request
    A challenge password []:
    An optional company name []:
    
    
  2. Send the CSR to your favorite signing authority, or self sign it:

     data/proxy/certs# openssl x509 -req -sha256 -days 3650 -in  demo.example.com.csr -signkey  demo.example.com.key -out  demo.example.com.crt
  3. Configure nginx to listen on port 443.  Add port mapping to the proxy's docker-compose.yml file: 

    proxy:
       image: jwilder/nginx-proxy
       restart: always
       volumes:
         - /var/run/docker.sock:/tmp/docker.sock:ro
         - ./certs:/etc/nginx/certs:ro
         - ./vhost.d:/etc/nginx/vhost.d
         - ./html:/usr/share/nginx/html
       environment:
         - DEFAULT_HOST=demo.example.com
       ports:
         - "80:80"
         - "443:443"
  4. Recreate the proxy container with: 

    docker-compose up -d 

This exposes port 443 for SSL.  We are leaving port 80 exposed because the nginx-proxy will automatically redirect port 80 to 443.  Now we can access our application at: https://sampletown.demo.example.com/.  If the cert is self-signed, you'll get a browser warning.  It will go away when/if you have the certificate signed.  

After receiving the signed certificate from the signing authority, replace the .crt file created above with the signed certificate.  In the example above, the self-signed certificate is named demo.example.com.crt.  That file should be replaced with the file from the signing authority.  Note: The names of the certificate files are important.  The certificate file name must be must match the domain name it applies to.   Again, from the above example, the wildcard domain name is *.demo.example.com so the certificate and keys must be named demo.example.com.crt and demo.example.com.key

By convention, nginx-proxy will use the domain name to find the most specific certificate first and then drop prefixes until it finds a match.   In this case, it will look for sampletown.demo.example.com.crt and then demo.example.com.crt . This allows you to have different signed certificates for different domain names on the same proxy. 

Automatic Signed Certificates with LetsEncrypt.org

LetsEncrypt is a service which issues free automated certificates. Before using the service, you should review the https://letsencrypt.org/about/ and current rate limits.


LetsEncrypt.org is a service which automates the process of creating, signing, installing and renewing Domain Validation (DV) Certificates.  These are certificates provide the lowest level of host verification but do ensure encrypted traffic to the users browser.

The steps below show how to configure an extra container to automatically create and install certificates using jrcs/letsencrypt-nginx-proxy-companion.  This is a non-intrusive way to add letsencrypt to an existing proxy configuration.

Pulling new images

 If the certificate renewal appears to stop working, make sure you have the most current version of the image(s) and restart the application. 
docker pull jwilder/nginx-proxy:alpine
docker pull jrcs/letsencrypt-nginx-proxy-companion

docker-compose up -d

Requirements:

You must expose port 80 and 443 of your docker host to the outside via your firewall.  That is, the docker host must have a public IP address and be accessible on both port 80 and 443 to the outside. DNS entries must exist in the global DNS for the virtual host(s) which point to the docker host's IP address.  When your host makes a certificate request,  LetsEncrypts service will callback to your host for verification. If the remote service can not reach your host, then they can not verify your control of the domain name and the signing request will fail. 

Steps to enable LetsEncrypt

  1. First, if you haven't already exposed port 80 and 443 on the nginx-proxy. The proxy's docker-compose.yml should look like this: 

    version: "3"
    services:
      proxy:
         image: jwilder/nginx-proxy:alpine
         restart: always
         volumes:
           - /var/run/docker.sock:/tmp/docker.sock:ro
           - ./certs:/etc/nginx/certs:ro
           - ./vhost.d:/etc/nginx/vhost.d
           - ./html:/usr/share/nginx/html
           - ./conf.d:/etc/nginx/conf.d
         environment:
           - DEFAULT_HOST=demo2.ssdt.io
         ports:
           - "443:443"
           - "80:80"
         labels:
           - "com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true"
      le:
        image: jrcs/letsencrypt-nginx-proxy-companion
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock:ro
          - ./certs:/etc/nginx/certs:rw
          - ./vhost.d:/etc/nginx/vhost.d:rw
          - ./html:/usr/share/nginx/html:rw
        environment:
          - NGINX_PROXY_CONTAINER=proxy_proxy_1
    
    

    Notice that the companion shares the volumes with the nginx-proxy container.  The companion container will monitor docker events and automatically create certificates and place them in the correct directories for the primary nginx server. 

  2. Recreate the proxy with 

    docker-compose up -d
  3. For each district you want to have a certificate created for, edit the docker-compose.override.yml file to create the LETSENCRYPT_HOST and LETSENCRYPT_EMAIL variables, like: 

    usasapp:
      environments:
        - VIRTUAL_HOST=usas.sampletown.org
        - VIRTUAL_PORT=8080
        - LETSENCRYPT_HOST=usas.sampletown.org
        - LETSENCRYPT_EMAIL=hostmaster@example.com

    The companion image will create, sign and install a certificate for any service with the LETSENCRYPT_HOST variable.  In most cases, the VIRTUAL_HOST and LETSENCRYPT_HOST will be set to the same value.

  4. Restart the district's services with: 

    docker-compose up -d

Because nginx-proxy and the companion container use separate environment variables, you can use a traditionally signed certificate for some hosts (see previous section) and letsencrypt certificates for others.  For example, you might use a wildcard certificate for "*.demo.example.com" hosts and a letsencrypt certificate for a district's "vanity domain" (usas.sampletown.org and usps.sampletown.org).

Updating NGINX Proxy

If you want to update the version of NGINX, possibly due to a security advisory, the following steps are suggested

Determine the current image being used and check the NGINX version

Get image from docker-compose file

Go to the directory containing the docker-compose files for the proxy. In this example, it is /data/proxy

cd /data/proxy
cat docker-compose.yml | grep 'image'

Example Result
 image: jwilder/nginx-proxy
 image: jrcs/letsencrypt-nginx-proxy-companion

Check the nginx version of that image

 Get the image ID:
 docker image ls | grep 'nginx'
 Result
 jwilder/nginx-proxy latest fcb5a96e19c1 20 months ago 161MB

Check that image to see what the ngixn version is using the ID from the previous command
 docker image inspect fcb5a96e19c1 | grep 'NGINX' <-- this uses the image id from the docker image ls command
 Result:
 docker image inspect fcb5a96e19c1 | grep 'NGINX'
 "NGINX_VERSION=1.17.6", 
 "NGINX_VERSION=1.17.6",

Note this shows a compromised version

Pull new image and check version

Note the version to pull depends on the version of nginx proxy being used.  The image to pull will match the image name found in the docker-compose file

docker pull jwilder/nginx-proxy:latest
 latest: Pulling from jwilder/nginx-proxy
 07aded7c29c6: Pull complete
 bbe0b7acc89c: Pull complete
 44ac32b0bba8: Pull complete
 91d6e3e593db: Pull complete
 8700267f2376: Pull complete
 4ce73aa6e9b0: Pull complete
 6e90715ff75c: Pull complete
 8291a33eed7e: Pull complete
 6ddfe39af3a9: Pull complete
 fa40559ef5bc: Pull complete
 0db4d49e6f76: Pull complete
 062b69f5d6b6: Pull complete
 Digest: sha256:0eba2c7c08c6dd26cdd8209223ce11ea39906a6b677adc1e076401a637dd1043
 Status: Downloaded newer image for jwilder/nginx-proxy:latest

Repeat these steps to verify the version - note the old image is still there with no tag; that is normal

docker image ls | grep 'nginx' <-- to get the image id and make sure a newer one is downloaded
jwilder/nginx-proxy latest 2ffbdd7281bf 4 days ago 149MB
jwilder/nginx-proxy <none> fcb5a96e19c1 20 months ago 161MB

Inspect the new image to get the NGINX version:
docker image inspect 2ffbdd7281bf | grep 'NGINX'"NGINX_VERSION=1.21.3",

Activate it in your container

docker-compose down
docker-compose up -d
docker-compose ps <-- to make sure it came up

Make sure the container shows the new NGINX version

Note that the container name includes the name of the directory where the proxy files are placed.  In this example, the directory has /proxy, so we can grep on this.

Find the container ID
docker ps  | grep 'proxy' <- to get the container ID

e27b852c849a jwilder/nginx-proxy "/app/docker-entrypo…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp proxy_proxy_1

docker inspect e27b852c849a | grep 'NGINX' <-- using the container ID
"NGINX_VERSION=1.21.3",

Miscellaneous tips and tidbits:

Issues with LetsEncrypt

ACMEv1/ACMEv2 error

LetsEncrypt no longer supports ACMEv1 for certificate management. If your site stops automatically renewing/generating certificates, this may appear in the logs:

docker-compose logs --tail=50 le
...
le_1 | 2020-01-09 11:48:43,395:INFO:simp_le:1382: Generating new account key
le_1 | ACME server returned an error: urn:acme:error:unauthorized :: The client lacks sufficient authorization ::
Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555.
See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.

To resolve this, make sure you have the latest images.  See https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion

To check if it successful, go to the certs/accounts directory.  You will see something like this if it is using ACMEv2 - in this case our proxy docker-compose.yml is in /data/proxy:

/data/proxy/certs/accounts# ls
acme-v02.api.letsencrypt.org

Checking site configuration

Useful Commands:

Force certificate renewal for all:

docker exec name_of_lets_encrypt_container/app/force_renew
Example:
 docker exec proxy_le_1 /app/force_renew

Get certificate status - note in this case, I did a force renew on 3/30, so 90 days is 6/28.

docker exec name_of_lets_encrypt_container/app/cert_status
Example:
 docker exec proxy_le_1 /app/cert_status
##### Certificate status #####
/etc/nginx/certs/files.ssdt.io/fullchain.pem: no corresponding chain.pem file, unable to verify certificate
Certificate was issued by Let's Encrypt Authority X3
Certificate is valid until Jun 28 13:11:46 2020 GMT
...

Timeout

We recommend making adjustments to the timeout configuration.