Let's Encrypt Certificate renewals failing

Thanks Lorne!

@lgaetz, This would be fantastic. Any way to fast track this to get it into the production tracks?

@jsmith very much this.

It is only recently that this was enforced. But it has been know for a long time.

This is a failure of the Sangoma team to stay current. This is not any different than the recent thread about fail2ban being so old. Or the entire fiasco as to why php7 is not standard yet.

And of course the core OS is now even getting out of date. RHEL 8 was released almost a year ago (May 7, 2019). Yet we have heard zero about plans to update.

Schmoozecom chose to go their own way and build SNG7 for their own reasons. Honestly, there is not much wrong with that, but it does absolutely require that you dedicate resources to maintaining and moving forward. In theory Sangoma knew this when they bought Schmoozecom.

just want to say thanks. this custom firewall ruleset worked for me and was easy enough to follow.
now i need to sit down and run this on several more systems.

1 Like

Hi @thx2000

Development is looking at your feature suggestion now and found a fatal flaw. I said a month ago:

well that’s changed (a bit anyway) and I now realize that adding static known rules to iptables is not a solution to this problem. The flaw is that I, and malicious users, are free to set the user agent to match the rule string, GET /.well-known/acme-challenge, and when I do, I end up with full access to all resources on port 80, which is definitely not what you want. Test for yourself by doing the following curls from an untrusted host:

curl http://<pbx_ip>
curl http://<pbx_ip>  -A "GET /.well-known/acme-challenge"

The first will time out, the second won’t. I would advise everyone against using these rules as currently written. At this point the only recommended method of using LE is dedicating port 80 to LE validation.

The rules are also incomplete. In addition to allowing access to /.well-known/acme-challenge it is also necessary to allow access to /.freepbx-known, at least for cert creation.

edit:

Cause is not hopeless. Static, non-unique iptables rules are out, but you could create a rule unique to your system that allows all inbound access from world using the fqdn. In which case you can change the ACCEPT rule to:

 -A lefilter -m string --string "pbx.example.com" --algo kmp -j ACCEPT
1 Like

What i did for this (since I have over 40 PBX’s with the same issue) was I changed the port management for http to 8888 and the LetsEncrypt to 80. Then I went into the firewall rules, then went to status and deleted all of the existing blocked connections (you’ll find many ips that start in 45, that is, i believe, the letsencrypt IPs) but remove them all and wait a few minutes then try, you should be all set after. I didn’t have to mess with Iptables.

Not sure I like the approach, but for the sake of discussion:

That still opens up to anyone that can derive the fqdn. Not sure if it’s much better.

Require the Get and Host headers to match and limit to the beginning of the packet:

iptables -A lefilter -m string --hex-string "GET /.well-known/acme-challenge HTTP/1.1|0D0A|Host: pbx.example.com|0D0A|" --algo kmp --to 40 -j ACCEPT

Should only allow access to the specific resource and require a proper fqdn. A header added later (like the above user agent hack) won’t match.

Above is untested - might need to be tweaked to match LE’s exact request syntax.

1 Like

Why do this the hard way?

FreePBX knows when it is issuing a request to LE.

Simply have it add the rule above prior to executing the the request and then remove it again.

Hi Jared:

To be clear, my suggestion above was for individual users who want to test with their own custom firewall rules. Whatever gets done with certman will probably be along the lines you suggest, but obviously there’s a lot more moving parts than having firewall define static rules.

2 Likes

Good catch Lorne. What about this…

#Lets Encrypt
#Create lefilter chain
-N lefilter

#Remove FreePBX rule that allows all established states through
-D fpbxfirewall -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT

#Insert rule into INPUT chain to pass all port 80 traffic to lefilter
-I INPUT -p tcp -m tcp --dport 80 -j lefilter

#Insert rule back into fpbxfirewall chain to allow all established traffic except for traffic on port 80
-I fpbxfirewall -p tcp ! --dport 80 -m state --state RELATED,ESTABLISHED -j ACCEPT

#Allow new port 80 states to be generated
-A lefilter -m state --state NEW -j ACCEPT

#Filter subsequent traffic to allow access to /.well-known/acme-challenge
-A lefilter -m string --from 52 --to 53 --string "GET /.well-known/acme-challenge/" --algo kmp -j ACCEPT

#Return back to the INPUT chain for further processing
-A lefilter -j RETURN
#End of Lets Encrypt

This modification looks for “GET /.well-known/acme-challenge/” at the start of the HTTP header, and only at the start of the http header. So any malformed headers, or packets with that string anywhere else, will be dropped.

My initial interpretation of the --from and --to offsets seems to be a little off. I thought the --to offset would be where the matched string would have to end, but it appears that it will attempt to match the entire string, regardless of where it ends, from that starting point.

From my tests, this appears to solve the user-agent problem you mentioned, and still allows legitimate LE requests through.

3 Likes

From memory, I implemented the LE-Only port in sysadmin about 3 years ago, when LE said they were going to send requests from anywhere. Because of that, Sysadmin already does this for you, there’s no need to mess around with string matching in packets.

I’m kinda surprised that no-one else mentioned this. That’s what it’s there for. It 404’s and fail2bans anything that tries to scan it, and responds happily to LE.

1 Like

It has been mentioned several times, but you have to be ok with only using port 80 for LE. I haven’t reviewed the fail2ban code relevant in your example, but this also sounds likely to ban legitimate users who aren’t specifically entering https or port 81 in the url.

That’s the only port LE uses. So if you want to use LE, you have to use port 80.

Secondly, if legitimate users are going there, read what the screenshot says.

I did mention this. Apparently nobody read it.

1 Like

Just to be complete, LE only needs port 80 when using HTTP-01 protcol No open ports thus no firewall fiddling is needed for LE when using DNS-01 .

Is there a way to issue CLI commands to change the Lets Encrypt port to 80, change the admin port to 81, and then set the Lets Encrypt Services to the Internet Zone? I have quite a few PBXs and it would be great if these changes could be made in mass.

Thanks so much thx2000!!!

Your solution is what I was looking for–and it works—and it’s secure enough for me.

1 Like

The other thing to note (if like me you enforced LE connections on the border firewalls) that LE have started verification of the .well-known/acme-challenge from multiple global addresses - to stop (minimise) local redirection of the challenge. The solution for me was [but I only have one service] was open 80 to all incoming addresses, refresh the LE cert and lock down again.
Not ideal, and I wish LE would allow you to specify a query port when making the first request (and FreePBX open up Fail2Ban for the duration of the queries).

In my recent testing of LE creation and validation, I noted that when LE validates from their servers, they have a very identifiable user agent string, as seen in the apache log:

52.28.236.88 - - [08/Apr/2020:19:40:11 -0300] "GET /.well-known/acme-challenge/<redacted> HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

So one could whitelist the Sangoma servers and the PBX by source IP and then use @thx2000’s earlier rule to allow inbound from LE’s user agent. I don’t know if LE has a stated position on whether or not their user agent string will remain fixed or not, but I’ve not really investigated since user agent is trivial to spoof.

The next version of the PBX Firewall module will have a feature that allows world access to the LE token folders if you don’t wish to dedicate port 80 to LE.

2 Likes

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.