• Ops

An obscure intermitent SSH connection issue with Coolify

While setting up a self-hosted single-server Coolify installation to test it as an alternative to Dokku, I was struggling with a strange intermitent SSH connection issue: when saving the configuration of the server, I would sometimes get a success message, but most of the time the following error:

ssh: connect to host host.docker.internal port 22: Connection refused

I tried to change the server IP/domain to something else, saved, got an expected error, changed back to the right value: it worked! Then I went to the proxy tab, and I got an alert with the error again. It was impossible to understand why it seemed to work and at the same time didn’t work.

To reproduce the issue, I entered the running coolify container, copied the SSH keys and tried to ssh on the host: it worked. No issue.

At this point, it was impossible to understand this behavior; even reinstalling Coolify didn’t fix it.

But I got lucky: I tried the SSH command again just after I got the error in the GUI: Connection refused! I tried again, and got the error. The exact same command that worked some time before now was failing: I was reproducing the behavior of the GUI.
After switching to something else and coming back, I tried again: it worked, no error. I tried again and it didn’t. To understand what happened, I ran dmesg -w on the host server to follow my connection attempts. After a failed attempt, a line like this caught my attention:

[1921181.417888] [UFW LIMIT BLOCK] IN=br-44b6a… SRC=10.0.1.4 DST=10.0.0.1 … DPT=22

What this line is saying is that my connection attempt was blocked by UFW. This LIMIT BLOCK rang a bell: our basic setup for servers is to open the port 22 with a general rate limit to reduce the attack potential. To circumvent this, I added a rule of higher priority to accept unlimited connections on this port from the Docker interface:

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       10.0.1.0/24
OpenSSH                    LIMIT       ...
...

After I did this, the intermitent issues went away!

In the end it seems that Coolify makes a heavy use of SSH connections: just opening the “Configuration” tab of the server in the GUI results in two new SSH connections, then switching to the “Proxy” tab opens three more. No wonder this was triggering the rate limit!