Letsencrypt certificate renewal causes phone outage

FreePBX Security
1

Hi all,

We’re encountering the same issue as described on the topic “letsencrypt-certificate-renewal-caused-phone-outage” - on 3 servers, each every 2 months

Unfortunately, the suggested solutions of restarting Asterisk whenever we renew our certificates - aren’t applicable, because this renewal can happen in off hours and during high load on our Asterisk servers.
Any other suggestions on how to handle this? Currently we’re setting alerts, and getting ready for renewal events, which is weird.

2

Quick fix is to rewrite the cronjob for the acme client to run only on Sunday at 4 am, that gives you time to check it all before Monday or add a ;/usr/sbin/fwconsole restart to the just edited cronjob.

2 Likes
3

Thanks @dicko - we’ll probably try these suggestions, for lack of other options at the moment

  • There is a nuance - that the restart should happen only if a renewal actually occurred. With Let’s Encrypt renewals happen roughly once every 2 months, and not weekly as the cron is going to run.
  • We work with a lot of timezones - and for a bunch 4AM in one timezone - is actually deep into the morning already
  • What about a not-so-quick-fix - I think we should open a bug for this issue, no?

Maybe people have other ideas?

4

For less than $20 you can get a five-year commercial SSL certificate that you only need to renew once a year.

5

It’s doesn’t scale so great… Trying to decrease the toil where possible…

6

Not so quick fix, but the one I use

fwconsole cert --help
fwconsole cert --import
fwconsole cert --updateall
.
.

You do not need to use FreePBX as your acme client, I use

If you are in control of your name-service then I suggest you use DNS-01 as a preferred protocol

Asterisk does not need to be restarted, but the webserver’s configs need to be reloaded and I have set mine to use the ones in /etc/asterisk/keys

so my arguments to the issuance include

  --dns dns_dgon  ; for digitalocean  api
  --cert-file /etc/asterisk/keys/$(hostname -f).crt              ;  Path to copy the cert file to after issue/renew..                                     
  --key-file /etc//asterisk/keys/$(hostname -f).key            ; Path to copy the key file to after issue/renew.                                       
  --reloadcmd fwconsole cert import;fwconsole cert updateall;apache2 reload            ;  Command to execute after issue/renew to reload the server.

The cron job acme.sh creates takes care of everything from that point on, so remove the one FreePBX used

1 Like
7

This is from 2020 - so everything here is in effect now. We have close to 100 PBXes and do not want to be logging in to each every 397 to update certs.

Maximum SSL certificate validity reduced to 1 year

This change was first announced by Apple at the CA/Browser Forum Spring Face-to-Face event in Bratislava back in March of this year. Then, at the CA/Browser Forum’s Summer event (held virtually), Google announced its intention to match Apple’s changes with its own root program.

Starting on September 1, SSL/TLS certificates cannot be issued for longer than 13 months (397 days).

8

I was answering the OP concern. As a business owner, a phone system interruption for a day will cost me more than $20. Updating the certificate on 3 servers is not that difficult.

9

You can use in the Cron job:

asterisk -x "core restart when convenient"

or

asterisk -x "core restart gracefully"

to only restart asterisk when no calls are in progress.

This obviously does not restart other services (like the web interface) but for a basic usage should be enough. I’m trying to find a way to execute this only when the cert is actually updated and will update you when I get it.

I agree that even this is a temporary solution and something better should be really implemented.

Edit: Asterisk forum talks about this, though no conclusion here too

10

There is no need to restart Asterisk or FreePBX, reloading any updated certs is all that is needed in any ‘secure’ apps that rely on them webrtc, https, provisioning etc. . . .

I suggest that the renewal process is broken at source if it exhibits this behavior.

edit:

reading the linked thread I would ask @jcolp to further comment, I personally do not find a restart necessary either.

11

So, PJSIP didn’t originally have support for reloading/restarting the TLS transport. You had to basically tear it all down in an asynchronous manner which sometimes worked and sometimes left you without working TLS (or TCP, or UDP) support. This is the origin of the “allow_reload” option and why it defaults to disabled.

In recent times PJSIP has gained the ability to “restart” the transport so it should pick up new TLS certificate and key. This is not yet taken advantage of in Asterisk, but I’ve been working on a patch to do so.

3 Likes
12

Hi @dicko , I have the same exact problem. After a renewal a notification (a red alert, not a yellow warning) in the GUI saying that some certificate is renewed and a reload may be necessary. This should already indicate that certman doesn’t deal with services restart itself.

Anyway I left the old certificate expire one time and indeed it was not replaced with the new one and phones stopped connecting. Checking with openssl the certificate was still the old one.

So this really should not work for anyone and the reload seems to be only manual at this time

1 Like
13

Then I agree that your suggestion is proper

14

I read your interesting blog posts about this.
I should point out that beyond asterisk this is also a FreePBX bug.

Someone should check if this applies only to the SIP stack or also the web interface (if enabled) and other protocol. I personally use TLS only for SIP so I don’t know if anything else needs a reload but I will have a look at the source of certman to try understanding more.

Anyway when you have news about the patch I would be happy to know more!

15

The built-in HTTP server in Asterisk does not require a restart to reload its TLS information. I can’t comment on FreePBX.

16

With the FreePBX distro http(s) provided by Apache doesn’t require any intervention when the cert updates. I believe that nodejs does require a restart, but going from faulty memory there.

17

As of now certman only issue a reload of asterisk (see here) so this will probably work when the patch is implemented, but as of now it is not useful at all.

Should we think about a PR changing the behavior to a core restart "when convenient"?
What do you think?

18

The JIRA issue has been created[1] and I’ve put a code review up[2] so you can follow the JIRA issue to see when it gets closed, and then ultimately when it gets released.

[1] [ASTERISK-30186] res_pjsip: Add support for reloading TLS certificate and key information - Digium/Asterisk JIRA
[2] https://gerrit.asterisk.org/c/asterisk/+/18987

3 Likes
19

I saw the fix has been merged. For anyone wondering we must wait for a system update to appear or upgrade asterisk manually (when it will be released), then everything should work without any further update to the modules

20

The next Asterisk release won’t occur until the end of October, at which point updating the FreePBX RPMs would then follow.

2 Likes