Let's Encrypt (LE) certificate request and installation / auto-renewal configuration

regexaurus · March 19, 2024, 12:12am

When my LE certificate had expired in January, I posted a comment on a now-closed topic where the op was reporting a problem with requesting LE certificates. I replied to say that ‘fwconsole cert --updateall --force’ succeeded as a workaround for me. Today, I received an LE email notification indicating the certificate for a FreePBX system I manage would expire in about 20 days. If native FreePBX LE renewal were working as expected, I don’t think I should receive such a notification. So after reading various posts here, I decided to use the official acme.sh script along with DNS API integration for LE certs. Thank you to others who have suggested this. You know who you are.
In case it is helpful to others, here is how I went about it. You will need to adjust certain steps for your requirements. I include a prompt character (#) in my example commands.

Installed acme.sh:

# curl https://get.acme.sh | sh -s [email protected]

Signed on NameSilo and enabled/generated API key and saved key in my trusty password manager. This will obviously be different for many of you. See the acme.sh DNS API doc.
Edited ~/.acme.sh/acme.sh.env, adding the line:

export Namesilo_Key="MY_NameSilo_API_KEY"

Closed (exited) my SSH session and signed on again, to reload bash. This would work too:

# source ~/.acme.sh/acme.sh.env

Listed and removed all certificates from FreePBX:

# fwconsole certificate --list
# fwconsole certificate --delete 0
# fwconsole certificate --delete 0

I had a “default” self-signed certificate and an LE certificate, so repeated the command to delete both certificates. After deleting the first (“default”) certificate with ID 0, the remaining certificate was assigned ID 0.

Backed up (just in case) and deleted contents of /etc/asterisk/keys:

# cp -r /etc/asterisk/keys ~/asteriskKeys.bak
# rm -rf /etc/asterisk/keys/*

Be cautious with the rm command! You can seriously fubar your system. Dropping the ‘f’ option is safer and will prompt you to confirm deletion of each file.

Submitted LE certificate request:

# acme.sh --issue --dns dns_namesilo -d primary.name.tld -d secondary.name.tld

I have 2 DNS A records for this FreePBX system, and want to include both FQDNs in the certificate. secondary.name.tld is included in the SAN (subject alternative name) field. After 10 minutes or so of the script doing some DNS “magic” and checking DNS records every 10 seconds, I had a new certificate. The DNS API doc suggests including --dnssleep 900 in the command for NameSilo. In hindsight, it would have been wise to include that.

Installed issued certificate and imported to FreePBX:

# acme.sh -i -d primary.name.tld -d secondary.name.tld --cert-file /etc/asterisk/keys/$(hostname -f).crt --key-file /etc/asterisk/keys/$(hostname -f).key --fullchain-file /etc/asterisk/keys/$(hostname -f).pem --renew-hook 'systemctl reload httpd.service'
# systemctl reload httpd.service
# fwconsole certificate --import

Set imported certificate as default and disabled responsive LE rules:

# fwconsole certificate --default 0
# fwconsole firewall lerules disable

Confirmed acme.sh renewal cron job was created:

# crontab -l

Dumped (displayed) config info for the cert:

# acme.sh --info -d primary.name.tld

I noticed that Le_RenewHook was unexpectedly set to a null string, so I edited ~/.acme.sh/primary.name.tld_ecc/primary.name.tld.conf and manually set Le_RenewHook:

Le_RenewHook='__ACME_BASE64__START_c3lzdGVtY3RsIHJlbG9hZCBodHRwZC5zZXJ2aWNlCg==__ACME_BASE64__END_'

c3lzdGVtY3RsIHJlbG9hZCBodHRwZC5zZXJ2aWNlCg== is base64-encoded: systemctl reload httpd.service. The acme.sh --info command will show the human-readable/unencoded string. You can use the base64 command (base64 -d to decode) to base64 encode a command string. For example:

# echo 'systemctl reload httpd.service' | base64

So…I’m not sure if LE certificate auto-renewal as currently configured (LE_RenewHook, in particular) will work as expected. A recent (January 2024) comment on an acme.sh issue (GitHub) indicates that deploy hooks aren’t working for the commenter, specifically for ECC certificates. Switching to RSA certificates worked for the commenter. If moderators leave this post open that long, I will try to remember to post an update with results and adjustments if required, for the next LE certificate renewal.

dicko · March 19, 2024, 6:42am

acme.sh -cron will show it working, if you use the default zerossl CA , then you can ‘force’ renewal without jeopardy, otherwise as you see it works ‘just as advertised’

regexaurus · March 19, 2024, 9:46pm

That’s helpful, @dicko. Thank you!

# acme.sh --cron --force

Worked as suggested. It took about 7 minutes for the certificate to be renewed. I also confirmed that my renew hook systemctl reload httpd.service runs as expected. However, this is apparently insufficient for the renewed certificate to be used by FreePBX (Apache?).
At this point, acme.sh --info -d primary.name.tld and fwconsole certificates --details 0 displayed info for the renewed certificate. However, the certificate details that appear in a browser when accessing the FreePBX control panel, are for the certificate that was issued yesterday, on 3/18/2024. I tried multiple browsers and devices, including incognito/private modes, clearing all history, cookies, etc. At some point I noticed the directory /etc/asterisk/keys/integration, which contain the files certificate.pem, webserver.crt, and webserver.key. The filesystem modified timestamps on the files were for yesterday and I assume these were for the certificate issued yesterday. I tried:

# fwconsole certificates --import
# fwconsole certificates --updateall

After this, the filesystem timestamps on the files in /etc/asterisk/keys/integration matched the time the import command was run, and serial numbers matched when comparing certificates:

# acme.sh --info -d primary.name.tld
# openssl x509 -in /etc/asterisk/keys/primary.name.tld.pem -noout -text
# openssl x509 -in /etc/asterisk/keys/integration/certificate.pem -noout -text
# fwconsole certificates --details 0

But the certificate seen by the browser is still the certificate issued yesterday! After the import and updateall commands, I tried:

# systemctl httpd.service reload
# systemctl httpd.service restart
# fwconsole reload
# fwconsole restart

and finally!:

# reboot

None of this has helped. The certificate issued on 3/18/2024 is persistently presented to the browser. I’m probably missing something painfully obvious… Suggestions?

dobrosavljevic · March 19, 2024, 9:48pm

Did you actually assign the new certificate to HTTPS under Admin -> HTTPS Setup -> Settings?

regexaurus · March 19, 2024, 10:00pm

@dobrosavljevic that took care of it! Is that required each time after acme.sh renews a certificate? There doesn’t seem to be an fwconsole equivalent… I’m wondering what is the minimum required renew hook, for unattended acme.sh renewal.

fwconsole certificates --import && fwconsole certificates --updateall

Something else/additional?

dobrosavljevic · March 19, 2024, 10:06pm

I never setup the script like you did but presumably the only way it’s automatically installed is through the builtin Let’s Encrypt functionality from FreePBX. Not sure if there is a way to make this happen through the CLI but I am sure somebody else here has some suggestions.

dicko · March 19, 2024, 10:39pm

I use the base64 encoded version of

fwconsole certificate --import; fwconsole certificate --default=my.domain.name;systemctl reload apache2

after updating my.domain.name.crt, my.domain.name.key and my.domain.name.pem in /etc/asterisk/keys, no reboot needed nor fwconsole reload

Note that you don’t ‘need’ the pem, and the generated .cer mutates into a .crt , some folks prefer setting the Le_RenewHook in the my.domain.name.conf file which gives some folks more comfort. I don’t use LetsEncrypt’s CA for a number of reasons but haven’t had a glitch in several years with many domains with the ZeroSSL ones. It makes using different domain names for TLS, provisioning , UCP and admin much easier ( and of course , much more secure )

regexaurus · March 19, 2024, 11:39pm

Hmmm…
Adding systemctl reload httpd.service back to my renew hook (base64-encoded), in addition to the fwconsole certificates import and default commands isn’t helping here. I tried acme.sh --cron --force --dnssleep again, and although the certificate renewed and the renew hook commands ran successfully, I have the same problem as before (wrong cert appears in browser). I’m running FreePBX 16.0.40.7 (up-to-date). I wonder if that’s the difference…

regexaurus · March 20, 2024, 12:30am

OK, so I think all I needed yet was to do:

# /usr/bin/cp -f -s /etc/asterisk/keys/integration/webserver.* /etc/httpd/pki/
# ln -s -f /etc/asterisk/keys/integration/certificate.pem /etc/httpd/pki/webserver.pem

On my system, the certificates that Apache actually uses apparently live in /etc/httpd/pki, which contained these files on my FreePBX system:
ca-bundle.crt
webserver.crt
webserver.key
certificate.pem

After doing the above commands, and # systemctl reload httpd.service, the correct certificate appears in the browser.
I think replacing the files at /etc/httpd/pki with symbolic links to the appropriate files in /etc/asterisk/keys/integration should be a one-time step. I need to test, but I think if I renew now, my renew hook should be sufficient to make the renewed certificate active. I’m not sure of possible side effects…

dicko · March 20, 2024, 1:13am

That depends on the highest level that defines where to look for the cert and key

grep -r ssl /etc/{apache2,httpd,nginx}

would in many cases show in /etc/*/sites-enabled .

But now you have the certs generation nailed , well almost, you might want to think about answering all http/https/tls/tcp connections with a reverse proxy , I use haproxy (and the newest FreePBX comes with that pre-installed)

Conceptually all http/https/tcp/tls connection attempts are answered by the http and tcp ‘front-ends’ of haproxy, here any certificates are handled , and acme.sh has a handy .acme.sh/deploy/haproxy.sh to do that,

You apply ‘Strict SNI’ to connections so ip-address based connections are silently dropped, http is forced to https and eventually everything coming out is handled by lotsa ‘back-ends’ one for admin, one for UCP, one for provisioning one for SIP but none need to worry themselves about secure connections, end result

http(s)://my.domain.name goes to UCP now at 127.0.0.1:80/ucp (but only if the domain name is used)
https://qwerasdesd.cyou gets answered by 127.0.0.1:80/admin (ditto , but you knew that)
tcp://yytytytyteetdss.icu:5060 and tls://yytytytyteetdss.icu:5061 get answered by FreePBX tcp transport listening on 127.0.0.1:5060 with the cert for TLS being handled by haproxy.
and so on
.

you can buy domains for less than 4 bucks a year from
namecheap and who will guess that their extension at my.domain.name needs to get it’s provisioning from asfkhkahlaskfjljl.link and register to yytytytyteetdss.icu unless you tell them and to mess with their extension they would need to login to FreePBX at qwerasdesd.cyou

There you go, my skeletal recipe for allround TLS/SSL certification with no need to mess with TLS in FreePBX or SSL in your webserver and as a result all ‘drive-byers’ won’t even see you.

franckdanard · March 21, 2024, 5:45pm

Weird, for my part on one of my server, that works fine.
Maybe something specific is present getting this error.
The all is to found out it.

regexaurus · March 21, 2024, 11:02pm

@dicko, very interesting, thank you!
That grep command found results only in /etc/httpd/conf.d/ssl.conf and /etc/httpd/conf.modules.d/00-ssl.conf, for me. I’m curious…do you use the SNG7 distro, “roll your own” FreePBX, or something else? Are you saying FreePBX 17 includes HAProxy? Is it bundled with FreePBX 17? I don’t think it’s installed by default on base Debian 12, and I don’t see anything about installing HAProxy in the FreePBX 17 manual install instructions…
I have a little experience with HAProxy and SNI on pfSense, and what you’re suggesting is definitely interesting.

dicko · March 22, 2024, 2:25am

I use FreePBX in Debian with nginx (props to @billsimon
for that) , Sangoma are eventually moving to Debian , their recently released beta install script is based on Debian 12 and includes Haproxy.

You are in a fine position to kick the tires of my suggestions, I don’t think you will be disappointed with what I find to be the greatly improved security and lack of fussiness.

FreerPBXer · March 22, 2024, 9:38pm

So much effort and frustration over what should be entirely nothing at all. It’s an integral part of FreePBX, was broken somewhere along the way and never properly fixed.

system · April 21, 2024, 9:39pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.