I (asd) was doing a routine docker compose {pull, down, up -d} upgrade on loomio.
Ever since the server upgrade in October 2025, we’ve been having docker warning us the following:
WARN[0000] volume "loomio-deploy_certs" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
WARN[0000] volume "loomio-deploy_vhost" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
WARN[0000] volume "loomio-deploy_html" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
[+] Running 11/11 "loomio-deploy_acme" already exists but was not created by Docker Compose. Use `external: true` to use an existing volume
My hypothesis is that this was because we upgraded docker-compose plugin and the newer version isn’t detecting the older version’s volumes as its own.
Since these volumes seemed like recreatable data, I decided to remove those volumes and let docker compose recreate them (and start managing them as non-external).
docker volume rm loomio-deploy_certs loomio-deploy_vhost loomio-deploy_html loomio-deploy_acme
After doing this and restarting the service, codema.in was giving a 502 error in the browser.
It was tricky to debug this as docker compose ps showed the app was running healthily. docker compose logs also didn’t show any errors.
Here’s a brief aside on how the server is set up. It is like caddy -> exposed ports on docker -> nginx-proxy (inside docker) -> loomio app (inside docker). The Caddyfile config looks like:
http://codema.in, http://hocuspocus.codema.in {
reverse_proxy http://127.0.0.1:8088
}
codema.in, hocuspocus.codema.in {
reverse_proxy https://127.0.0.1:4443 {
transport http {
tls
tls_insecure_skip_verify
}
}
}
Ports 8088 and 4443 are exposed from docker with nginx-proxy handling those.
So, I looked at caddy logs with sudo journalctl -u caddy -f and it showed something like remote error: tls: unrecognized name when trying to connect to nginx-proxy.
I assumed that nginx-proxy is somehow not matching it to the right domain and tried fiddling with environment variables being passed to nginx-proxy. It didn’t help.
In nginx-proxy’s README there’s a section talking about missing certificates causing unrecognized name alert. So I checked inside the loomio-deploy_certs whether the certificates were actually present. And they were. The nginx-proxy logs also showed lines like the following which suggested that certs were available to nginx-proxy.
"ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/nginx/certs/codema.in.crt"
"ssl_stapling" ignored, no OCSP responder URL in the certificate "/etc/nginx/certs/hocuspocus.codema.in.crt"
Then I tried doing curl -H 'Host: codema.in' https://127.0.0.1:4443 and it was giving a similar error: error:0A000458:SSL routines::tlsv1 unrecognized name.
An issue on a different nginx-proxy-manager gave me the idea that this could be because nginx-proxy is using SNI for identifying server.
So, with SNI, it is not enough to pass a header for the original host (which caddy does by default), but also pass tls_server_name option as follows:
codema.in, hocuspocus.codema.in {
reverse_proxy https://127.0.0.1:4443 {
transport http {
tls
tls_insecure_skip_verify
tls_server_name {host}
}
}
}
With that set, codema.in started working again.
But as always, I check browser console for any errors and I saw Firefox can’t establish a connection to the server at wss://codema.in/cable.
The request to /cable was getting a 502.
First I started wondering whether I should be configuring caddy to handle “wss” scheme. But on reading about websockets I realized it’s probably going over HTTP itself and caddy should be already working without any configuration.
Aside: /cable was introduced recently in 3.0.20 version of loomio which I upgraded loomio to 2 days before on Friday. There’s a very good chance that /cable was broken even before I started today’s maintenance. But unfortunately I did not know whether it was the case or not.
Again I started looking for logs from inside out. Found no log in docker compose that was useful. Eventually in caddy logs, I found something like:
http2: invalid Upgrade request header: ["websocket"]
Fortunately there was an issue on caddy repository about the same and turns out it was something to do with the HTTP version used for backend requests. There was a fix available which was released in 2.11 of caddy (and we were on 2.10).
So I did apt update and apt upgrade hoping to see the problem go away
Well. What?
Now there was an nginx error page showing up saying the service was unavailable.
Again I started looking at all logs from inside out. Nothing had changed with the loomio app, it was running healthy. The nginx-proxy logged showed nothing except a 503. The caddy logs showed nothing either. What?
So I went inside the nginx-proxy container with docker compose exec nginx-proxy bash and ran curl https://codema.in/dashboard --resolve codema.in:443:127.0.0.1 (this is the way to do SNI with curl) and it was responding fine. Next, I looked at the nginx configuration and saw that there was a default server (in addition to codema.in server) which returned 503 when matched. I changed the return code to 504, did nginx -s reload and refreshed the browser and started seeing 504.
After breaking the glass I was drinking tea in I was like the only thing that changes is caddy version, what if there’s something new with caddy and SNI?.
So I looked at caddy releases and turns out the latest release 2.11.2 was from 2 days ago and had this highlight:
Reverse proxy got a lot of love with certain edge cases related to PROXY protocol, health check port, and closing body on retries. Dynamic upstreams are now tracked which enables passive health checking.
Convinced that this was a caddy issue, I immediately downgraded caddy apt install caddy=2.10.2 and the site was loading again.
In the issue about websocket error there was a workaround suggested, to set http version to 1.1 like:
transport http {
tls
tls_insecure_skip_verify
tls_server_name {host}
versions 1.1
}
Indeed that works.
But I didn’t like fixing the version like this. So I digged in the caddy release notes and discovered that in caddy 2.11.1 they changed the default behavior of reverse_proxy to pass host as upstream_hostport.
So eventually, the setting that works for now is:
codema.in, hocuspocus.codema.in {
reverse_proxy https://127.0.0.1:4443 {
header_up Host {host}
transport http {
tls
tls_insecure_skip_verify
tls_server_name {host}
}
}
}
Last updated: March 8, 2026 at 2:06 PM