How to certbot renew a subdomain that’s using proxy_pass, like for jira or confluence

  linux, nginx

he problem: “certbot renew” doesn’t work as nginx forwards to another service using proxy_pass

The error looks something like this:

Processing /etc/letsencrypt/renewal/wiki.lupsha.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wiki.lupsha.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (wiki.lupsha.com) from /etc/letsencrypt/renewal/wiki.lupsha.com.conf produced an unexpected error: Failed authorization procedure. wiki.lupsha.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://wiki.lupsha.com/.well-known/acme-challenge/A-AhEux1R51j3yqPfq7Z4rNb18jONY_RtdYrIVV_3tM: Error getting validation data. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wiki.lupsha.com/fullchain.pem (failure)

The solution:

Comment out the location entry in /etc/nginx/sites-enabled/wiki.lupsha.com and restart nginx

#        location / {
#                proxy_bind 127.0.0.1;
#                proxy_pass http://127.0.0.1:8090;
#                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#                proxy_set_header X-Forwarded-Proto $scheme;
#        }

The full config at /etc/nginx/sites-enabled/wiki.lupsha.com

server {

#       listen 80;
#       listen   [::]:80 ipv6only=on;

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/wiki.lupsha.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/wiki.lupsha.com/privkey.pem;
        include /etc/nginx/snippets/ssl.conf;

        root /opt/web/wiki.lupsha.com;
        index index.html index.htm;

        server_name wiki.lupsha.com;

#        location / {
#                proxy_bind 127.0.0.1;
#                proxy_pass http://127.0.0.1:8090;
#                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#                proxy_set_header X-Forwarded-Proto $scheme;
#        }

}

# force the user to https
server {
    listen 80;
    server_name wiki.lupsha.com;
    return 301 https://$host$request_uri;
}

Restart nginx:

sudo /etc/init.d/nginx restart

Try the renew again:

certbot certonly -d wiki.lupsha.com

Output:

root@web:/opt/confluence/current/conf# certbot certonly -d wiki.lupsha.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
-------------------------------------------------------------------------------
1: Nginx Web Server plugin - Alpha (nginx)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
-------------------------------------------------------------------------------
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator nginx, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for wiki.lupsha.com
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/wiki.lupsha.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/wiki.lupsha.com/privkey.pem
   Your cert will expire on 2019-08-14. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Add back the /location entry, final file /etc/nginx/sites-enabled/wiki.lupsha.com is back to normal:

server {

#       listen 80;
#       listen   [::]:80 ipv6only=on;

        listen 443 ssl;
        ssl_certificate /etc/letsencrypt/live/wiki.lupsha.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/wiki.lupsha.com/privkey.pem;
        include /etc/nginx/snippets/ssl.conf;

        root /opt/web/wiki.lupsha.com;
        index index.html index.htm;

        server_name wiki.lupsha.com;

        location / {
                proxy_bind 127.0.0.1;
                proxy_pass http://127.0.0.1:8090;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
}

# force the user to https
server {
    listen 80;
    server_name wiki.lupsha.com;
    return 301 https://$host$request_uri;
}

Restart nginx:

/etc/init.d/nginx restart

Check the site:

https://wiki.lupsha.com

Looks good.