What is the GPG issue?

Could someone explain the GPG issue with FreePBX and how to resolve it?

My “Apply” button in FreePBX 15 was working fine for a time. Now it waits and times out every time.

What do you mean by GPG issue?

https://issues.freepbx.org/browse/FREEPBX-20698

https://issues.freepbx.org/browse/FREEPBX-20559

When I reload from the GUI, I see several gpg-related processes on the server taking a long time, and the reload eventually times out. When I use fwconsole r at the shell instead, it works fine.

1 Like

Looks like the key servers are useless, and the FreePBX key has been poisoned (evidenced by the failures).

So as I understand it, module signature checking can only happen against signatures that are local on the PBX. And those could get corrupted if someone gets access to your PBX via exploit, making signature checking useless.

Seems like there’s a serious problem here. Is the only mitigation at this time to disable signature checking?

(bump)

I would say yes because at this point it doesn’t look like much is going to be done. Both the authors of this seem to be happy enough to let it die. The fact that they’ve made it clear they don’t even think this is salvageable and the fact that in six months not much as been done should give an indication of what the future holds.

@mattf Perhaps you could let us know what FreePBX has on the roadmap for this? Signature checking, I thought, was considered to be a major security feature in recent versions of FreePBX. If it’s dead, what next?

Hey Bill,

Since we ship the FreePBX module signing key with FreePBX itself, I don’t think that GPG should be required to go lookup any of the module signing keys via public keyservers and shouldn’t be having problems due to the key being poisoned on the public keyserver. I’d be curious as to what gpg is actually doing in your case that is taking a long time to be able to say anything further.

Matt

1 Like

If the local key gets corrupted then FreePBX would tell you so. It doesn’t make the key useless. GPG still works locally around the globe (it’s used in Ubuntu even for packages and yum!). The SKS key servers are hosed, but they aren’t the only network. Ubuntu runs their own for example.

What should happen here is that FreePBX should be using https://keys.openpgp.org/. Which is the newest version of GPG and is not poison-able. Upload the pure FreePBX key there and then verify by the key’s email address that you own said key and you can maintain the state of it. This means that you can always upload the not poisoned key to openpgp. (Also it uses 2.17.x of GPG so it can’t be poisoned) The FreePBX Master GPG key’s email address is [email protected]. So someone would have to use that email to manage the key. Then you have FreePBX system look to openpgp ONLY.

Additionally,

Matt is right. The master keys are looked up locally and skipped from checking on the network.

@jfinstrom pointed out that the FreePBX Master key is already on openpgp and is only 5KB (the poisoned key is 5mb)

Master Key
https://keys.openpgp.org/vks/v1/by-fingerprint/2016349F5BC6F49340FCCAF99F9169F4B33B4659

Sub Signing Key 1
https://keys.openpgp.org/vks/v1/by-fingerprint/1013D73FECAC918A0A25823986CE877469D2EAD9

Sub Signing Key 2
https://keys.openpgp.org/vks/v1/by-fingerprint/072410D159E9DA63A459AB203DDB2122FE6D84F7

So all someone needs to do is remove all servers here: https://github.com/FreePBX/framework/blob/release/14.0/amp_conf/htdocs/admin/libraries/BMO/GPG.class.php#L52

and only add hkp://keys.openpgp.org

Additionally

[[email protected] root]$ gpg --keyserver hkps://keys.openpgp.org --refresh-keys
gpg: refreshing 3 keys from hkps://keys.openpgp.org
gpg: requesting key 69D2EAD9 from hkps server keys.openpgp.org
gpg: requesting key B33B4659 from hkps server keys.openpgp.org
gpg: requesting key FE6D84F7 from hkps server keys.openpgp.org
gpg: key 69D2EAD9: no user ID
gpg: key B33B4659: no user ID
gpg: key FE6D84F7: no user ID
gpg: Total number processed: 3

Someone needs to go and validate they own they keys (Two go to [email protected] and one goes to [email protected]) then they can allow the user IDs of the keys to be published so you don’t get the no user id warning:

You are probably using the keys.openpgp.org keyserver, which has an owner approval system – it will strip all user IDs unless the owner of the corresponding email address has allowed them to be published.

However I believe just changing the servers to keys.openpgp.org would immediately improve performance.

However during refresh the keys that are already known locally are NOT rechecked (see code here: https://github.com/FreePBX/framework/blob/release/14.0/amp_conf/htdocs/admin/libraries/BMO/GPG.class.php#L592).

So if the key is already local it’s not rechecked.

2 Likes

When I do an apply from the GUI, these processes appear and eventually time out.

19565 ?        S      0:00 sh -c /usr/bin/gpg --homedir /var/lib/asterisk/.gnupg --no-permission-warning --keyserver-options auto-key-retrieve=true,timeout=10 --no-tty --status-fd 3 --keyserver keyserver.pgp.com --refresh-keys C5C26
19566 ?        SL     0:00 /usr/bin/gpg --homedir /var/lib/asterisk/.gnupg --no-permission-warning --keyserver-options auto-key-retrieve=true timeout=10 --no-tty --status-fd 3 --keyserver keyserver.pgp.com --refresh-keys C5C26167A09

@billsimon it’s only refreshing “C5C26167A09” (that key is not known to FreePBX), However. Go into the GPG class and change the servers out to what I proposed. GPG is different on all systems so yours might actually ignore the fact that FreePBX only wants to update an individual key

Individual keys are requested like so (Notice it’s only processed 2 of the three keys)

[[email protected] root]$ gpg --keyserver hkps://keys.openpgp.org --refresh-keys 1013D73FECAC918A0A25823986CE877469D2EAD9 2016349F5BC6F49340FCCAF99F9169F4B33B4659
gpg: refreshing 2 keys from hkps://keys.openpgp.org
gpg: requesting key 69D2EAD9 from hkps server keys.openpgp.org
gpg: requesting key B33B4659 from hkps server keys.openpgp.org
gpg: key 69D2EAD9: no user ID
gpg: key B33B4659: no user ID
gpg: Total number processed: 2

Let’s find out what keys you actually have first:

[[email protected] root]$ gpg --list-keys --with-colons --fingerprint
tru::1:1555624212:0:3:1:5
pub:f:4096:1:86CE877469D2EAD9:1399270402:::m:::scESC:
fpr:::::::::1013D73FECAC918A0A25823986CE877469D2EAD9:
uid:f::::1462348010::6BF5E1714D6EE0E3E64FB65784D8310AD32E2C55::FreePBX Mirror 1 (Module Signing - 2014/2015) <[email protected]>:
sub:f:4096:1:3C355822CCEBF9CB:1399270402::::::e:
pub:u:4096:1:9F9169F4B33B4659:1398901041:::u:::scESC:
fpr:::::::::2016349F5BC6F49340FCCAF99F9169F4B33B4659:
uid:u::::1462388765::5EA8A2B8DAFF0B5B8270478B22680FBD5AC05C1A::FreePBX Module Signing (This is the master key to sign FreePBX Modules) <[email protected]>:
sub:u:4096:1:06C57CED5C2FE148:1398901041::::::e:
pub:f:4096:1:3DDB2122FE6D84F7:1462348340:::m:::scESC:
fpr:::::::::072410D159E9DA63A459AB203DDB2122FE6D84F7:
uid:f::::1462348340::CFC4B419F5A212E7C9EBD42E413C7CCADB048961::FreePBX Mirror 1 (Module Signing - 2016/2017) <[email protected]>:
sub:f:4096:1:36992456A6869B39:1462348340::::::e:

You want the fpr:::::::::1013D73FECAC918A0A25823986CE877469D2EAD9: line. That’s the key format that is passed back into refresh-keys. From this list you could also figure out who owns “C5C26167A09”.

The three included FreepBX keys are as follows:

  • 1013D73FECAC918A0A25823986CE877469D2EAD9
  • 2016349F5BC6F49340FCCAF99F9169F4B33B4659
  • 072410D159E9DA63A459AB203DDB2122FE6D84F7

Additionally (since you do ‘light’ programing) I would go into the GPG class and make sure this line is only sending non-local keys

before line 605 you can do dbug($refreshKeys) and then run fwconsole dbug and in another window do your reloading. Make sure the keys listed there don’t include the ones in the list below:

  • 1013D73FECAC918A0A25823986CE877469D2EAD9
  • 2016349F5BC6F49340FCCAF99F9169F4B33B4659
  • 072410D159E9DA63A459AB203DDB2122FE6D84F7
2 Likes

Additionally on my FreePBX system (installed from scratch) there are no additional keys. So it actually returns true here (https://github.com/FreePBX/framework/blob/release/14.0/amp_conf/htdocs/admin/libraries/BMO/GPG.class.php#L601) and never runs refresh-keys because the array of keys is empty because it detects they are all local in the BMO directory (https://github.com/FreePBX/framework/blob/release/14.0/amp_conf/htdocs/admin/libraries/BMO)

I’m 100% sure this is an issue with Ubuntu systems at this point. Try to determine the output from: https://github.com/FreePBX/framework/blob/release/14.0/amp_conf/htdocs/admin/libraries/BMO/GPG.class.php#L588

There’s probably a bug in there on ubuntu systems.

Yes, something extra here:

# sudo -u asterisk gpg --list-keys --with-colons --fingerprint
gpg: WARNING: unsafe permissions on homedir '/var/lib/asterisk/.gnupg'
tru::1:1569526665:0:3:1:5
pub:u:4096:1:9F9169F4B33B4659:1398901041:::u:::scESC::::::23:1569526672:1 http\x3a//159.69.208.88\x3a11371:
fpr:::::::::2016349F5BC6F49340FCCAF99F9169F4B33B4659:
uid:u::::1542492566::5EA8A2B8DAFF0B5B8270478B22680FBD5AC05C1A::FreePBX Module Signing (This is the master key to sign FreePBX Modules) <[email protected]>::::::::::0:
sub:u:4096:1:06C57CED5C2FE148:1398901041::::::e::::::23:
fpr:::::::::C5C26167A09555DB29DA4ECF06C57CED5C2FE148:
pub:f:4096:1:3DDB2122FE6D84F7:1462348340:::m:::scESC::::::23::0:
fpr:::::::::072410D159E9DA63A459AB203DDB2122FE6D84F7:
uid:f::::1462348340::CFC4B419F5A212E7C9EBD42E413C7CCADB048961::FreePBX Mirror 1 (Module Signing - 2016/2017) <[email protected]>::::::::::0:
sub:f:4096:1:36992456A6869B39:1462348340::::::e::::::23:
fpr:::::::::EB312FC936875A7BC236DE6A36992456A6869B39:
pub:f:4096:1:86CE877469D2EAD9:1399270402:::m:::scESC::::::23:1569526672:1 http\x3a//159.69.208.88\x3a11371:
fpr:::::::::1013D73FECAC918A0A25823986CE877469D2EAD9:
uid:f::::1462348010::6BF5E1714D6EE0E3E64FB65784D8310AD32E2C55::FreePBX Mirror 1 (Module Signing - 2014/2015) <[email protected]>::::::::::0:
sub:f:4096:1:3C355822CCEBF9CB:1399270402::::::e::::::23:
fpr:::::::::593E5D6A7107C285E698CB563C355822CCEBF9CB:

Updating the keyservers list in GPG.class.php allows the reload to proceed. It reaches out to the keyserver every time, as you figured, because of the extra key in the list. I don’t know how to determine where that is from.

Edit: ah, that key is the same one referenced in this ticket: https://issues.freepbx.org/browse/FREEPBX-20698

You should check the command itself and the code because it’s returning FreePBX fingerprints in a different format (that GPG still recognizes but FreePBX doesn’t)

I looked at that ticket and as suspected it’s something with ubuntu/debian.

Here’s the command that’s being run for him:

/usr/bin/gpg --homedir /var/lib/asterisk/.gnupg --no-permission-warning --keyserver-options auto-key-retrieve=true,timeout=10 --no-tty --status-fd 3 --keyserver pool.sks-keyservers.net --refresh-keys 593E5D6A7107C285E698CB563C355822CCEBF9CB C5C26167A09555DB29DA4ECF06C57CED5C2FE148 EB312FC936875A7BC236DE6A36992456A6869B39

It’s returning with this

[[email protected] BMO]$ gpg --keyserver hkps://keys.openpgp.org --refresh-keys --refresh-keys 593E5D6A7107C285E698CB563C355822CCEBF9CB C5C26167A09555DB29DA4ECF06C57CED5C2FE148 EB312FC936875A7BC236DE6A36992456A6869B39
gpg: refreshing 3 keys from hkps://keys.openpgp.org
gpg: requesting key 69D2EAD9 from hkps server keys.openpgp.org
gpg: requesting key B33B4659 from hkps server keys.openpgp.org
gpg: requesting key FE6D84F7 from hkps server keys.openpgp.org
gpg: key 69D2EAD9: no user ID
gpg: key B33B4659: no user ID
gpg: key FE6D84F7: no user ID
gpg: Total number processed: 3

The keys in the returned response are the freepbx keys. However they are in a different format than how they show up on CentOS. In fact it looks like Debian/Ubuntu return additional fingerprints for each single key (so each key has two fingerprints) This means on ubuntu systems the keys are being returned in a different format and thus aren’t able to be removed from the array. Thus they are passed back to the key server to be checked when they shouldn’t be.

It’d be an exercise for someone to figure out why fpr::::::::: gives the right fingerprint back on CentOS systems but not on other systems. Additionally someone should add a protected property in GPG to not check these fingerprints

  • 593E5D6A7107C285E698CB563C355822CCEBF9CB
  • C5C26167A09555DB29DA4ECF06C57CED5C2FE148
  • EB312FC936875A7BC236DE6A36992456A6869B39

Along with the originals

  • 1013D73FECAC918A0A25823986CE877469D2EAD9
  • 2016349F5BC6F49340FCCAF99F9169F4B33B4659
  • 072410D159E9DA63A459AB203DDB2122FE6D84F7

keys are also available locally typically

# ls -al /var/www/html/admin/libraries/BMO/*.key
-rw-rw-r--. 1 asterisk asterisk 3893 Apr 18  2019 /var/www/html/admin/libraries/BMO/3DDB2122FE6D84F7.key
-rw-rw-r--. 1 asterisk asterisk 3893 Apr 18  2019 /var/www/html/admin/libraries/BMO/86CE877469D2EAD9.key
-rw-rw-r--. 1 asterisk asterisk 5453 Apr 18  2019 /var/www/html/admin/libraries/BMO/9F9169F4B33B4659.key

On Debian, without the --with-colons I get output that makes sense:

$ sudo -u asterisk gpg -k 
gpg: WARNING: unsafe permissions on homedir '/var/lib/asterisk/.gnupg'
/var/lib/asterisk/.gnupg/pubring.kbx
------------------------------------
pub   rsa4096 2014-04-30 [SC]
      2016349F5BC6F49340FCCAF99F9169F4B33B4659
uid           [ultimate] FreePBX Module Signing (This is the master key to sign FreePBX Modules) <[email protected]>
sub   rsa4096 2014-04-30 [E]

pub   rsa4096 2016-05-04 [SC]
      072410D159E9DA63A459AB203DDB2122FE6D84F7
uid           [  full  ] FreePBX Mirror 1 (Module Signing - 2016/2017) <[email protected]>
sub   rsa4096 2016-05-04 [E]

pub   rsa4096 2014-05-05 [SC]
      1013D73FECAC918A0A25823986CE877469D2EAD9
uid           [  full  ] FreePBX Mirror 1 (Module Signing - 2014/2015) <[email protected]>
sub   rsa4096 2014-05-05 [E]

There you see three keys, three (known) fingerprints.

The man page for gpg says not to rely on that output for a script but only the output from --with-colons so I point this out only as an observation.

I have this exact same problem on Debian 9 and 10 with Fpbx 15. I tried the fix posted by tm1000 of updating GPG.class.php L52, which is exactly the same as Fpbx 14, but the GUI still hangs.

Running sudo -u asterisk gpg --keyserver hkps://keys.openpgp.org --refresh-keys also hangs

I think your problem is different, then.

I doubt it. I get exactly the same symptoms. Hangs on GUI but not fwconsole reload. However, I am using Fbpx v15 and PHP v7.3 which might be causing additional things.

These are completely fresh installs on D9 and D10. Same procedures that have worked in the past up until recently. I think that refresh key command was working a few days ago. Not sure though. I could have messed something up while troubleshooting this as well. But I am most certainly seeing the same issue as you.