FreePBX 17 GA Issues Encountered

and does 192.168.1.3 have a tftpd service running on UDP:69 ?

  1. Download the following run it on a command prompt on a windows machine, and make sure that your DHCP server is handing out tftp option 69

GitHub - CyberShadow/dhcptest: Cross-platform DHCP test client

  1. Open a command prompt under windows, turn OFF windows defender firewall, and at the command prompt issue “tftp -i x.x.x.x (where x.x.x.x is the IP of the tftp server) GET provisioningfilename” You can also just specify the filename of any file in the contents of your /tftpboot folder

#2 will tell you if your tftp server on your Debian install is actually handing out provisioning files

  1. Modify /etc/default/tftpd-hpa and in tftpoptions add --verbose 7 This will increase verbosity of the tftp server. Now power cycle the phone and do a tail /var/log/syslog on the debian tftp server. You should see the phone’s IP attempting to obtain it’s provisioning file from the server and getting it successfully.

That should get you going on troubleshooting this.

or if Windows is not part of your thinking, use nmap (netmap) from any host on the lan or lsof (list open files of a network flavor) on the PBX itself

nmap -an -sU 192.168.1.3

lsof -i:69

In the absence of tftpd , you will need to install a daemon, have inetd/xinetd control it and have it point to the base directory of where your provisioning files are (verbosity is of course good), but I would take the effort to change the protocol on the phones to use https as they would be more future proof and less likely to cost you money

1 Like

Thanks @tmittelstaedt .
I have ran your suggestions. I had mixed success. I have attached a screenshot. I need more handholding because I dont know what I’m looking for and what I am supposed to see. can you please show me more like if the command I’m typing in is correct, as shown below in the output?

Because I am not sure what I am supposed to expect, the output below are from the production FPBX16…to get to know what I should expect in FPBX17. Thanks for your assistance!

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe
dhcptest v0.9 - Created by Vladimir Panteleev
https://github.com/CyberShadow/dhcptest
Run with --help for a list of command-line options.

Listening for DHCP replies on port 68.
Type "d" to broadcast a DHCP discover packet, or "help" for details.
quit
Error on listening thread:

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 769
Fatal error: Overflow in integral conversion

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69
  op=BOOTREPLY chaddr=8E:33:85:76:69:D4 hops=0 xid=666C52C4 secs=0 flags=8000
  ciaddr=0.0.0.0 yiaddr=192.168.1.142 siaddr=192.168.1.1 giaddr=0.0.0.0 sname=192.168.1.3 file=
  6 options:
     53 (DHCP Message Type): offer
     54 (Server Identifier): 192.168.1.1
     51 (IP Address Lease Time): 86400 (1 day)
     58 (Renewal (T1) Time Value): 43200 (12 hours)
     59 (Rebinding (T2) Time Value): 75600 (21 hours)
      1 (Subnet Mask): 255.255.255.0

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69
  op=BOOTREPLY chaddr=86:7D:C1:39:00:21 hops=0 xid=8B5C63B1 secs=0 flags=8000
  ciaddr=0.0.0.0 yiaddr=192.168.1.223 siaddr=192.168.1.1 giaddr=0.0.0.0 sname=192.168.1.3 file=
  6 options:
     53 (DHCP Message Type): offer
     54 (Server Identifier): 192.168.1.1
     51 (IP Address Lease Time): 86400 (1 day)
     58 (Renewal (T1) Time Value): 43200 (12 hours)
     59 (Rebinding (T2) Time Value): 75600 (21 hours)
      1 (Subnet Mask): 255.255.255.0

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69
  op=BOOTREPLY chaddr=2E:60:80:9E:21:B8 hops=0 xid=430CB40B secs=0 flags=8000
  ciaddr=0.0.0.0 yiaddr=192.168.1.186 siaddr=192.168.1.1 giaddr=0.0.0.0 sname=192.168.1.3 file=
  6 options:
     53 (DHCP Message Type): offer
     54 (Server Identifier): 192.168.1.1
     51 (IP Address Lease Time): 86400 (1 day)
     58 (Renewal (T1) Time Value): 43200 (12 hours)
     59 (Rebinding (T2) Time Value): 75600 (21 hours)
      1 (Subnet Mask): 255.255.255.0

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69
  op=BOOTREPLY chaddr=02:69:0F:16:0B:5C hops=0 xid=B7FD7161 secs=0 flags=8000
  ciaddr=0.0.0.0 yiaddr=192.168.1.48 siaddr=192.168.1.1 giaddr=0.0.0.0 sname=192.168.1.3 file=
  6 options:
     53 (DHCP Message Type): offer
     54 (Server Identifier): 192.168.1.1
     51 (IP Address Lease Time): 86400 (1 day)
     58 (Renewal (T1) Time Value): 43200 (12 hours)
     59 (Rebinding (T2) Time Value): 75600 (21 hours)
      1 (Subnet Mask): 255.255.255.0

C:\Users\ronny.ko\Downloads>

Next,

on your point #2 and I type the command for tftp as follows:


there’s no reply.

@dicko,

I tried both commands under Ubuntu 22.04. For

ronny@containers:~$ nmap -an -sU 192.168.1.3
nmap: unrecognized option '-an'
See the output of nmap -h for a summary of options.

and nothing for

ronny@containers:~$ lsof -i:69
ronny@containers:~$ 

Both the Windows and Linux routes lead to nothing that I could understand to use.

Thanks for any clarifications.

I would need to install an ftpd server for ftpd to work.

No, that’s right – assuming you followed the instructions and disabled windows defender firewall, the output of those commands is showing the following:

C:\Users\ronny.ko\Downloads>dhcptest-0.9-win64.exe --quiet --query --request 69

  op=BOOTREPLY chaddr=8E:33:85:76:69:D4 hops=0 xid=666C52C4 secs=0 flags=8000

  ciaddr=0.0.0.0 yiaddr=192.168.1.142 siaddr=192.168.1.1 giaddr=0.0.0.0 sname=192.168.1.3 file=

The DHCP test output is that your DHCP server is handing out IP address 192.168.1.3 in option 69, and that option is reserved for use for the IP address of a tftp server.

Most phones once they get that option will then attempt to tftp a download file from the server specified - in this case 192.168.1.3

In the second window it is not showing that it’s downloading the config file from the tftp server - so - assuming you disabled windows defender firewall when you ran the tftp command - something is not right with the tftp server on 192.168.1.3 Is 192.168.1.3 the IP address of the Debian 12 system you installed FreePBX on? Because if it is, then the tftpd server is not running on it - or it’s running on it and not configured on it.

Thanks for your insightful reply.
As I mentioned in my reply, all this was done on the production FPBX16 in order to for me to understand what I was supposed to see or expect.

Having said that, there’s no way that the that the tftp server wasn’t running. The third-party windows machine (aka the “observer” sort of speak) had Windows defender disable for real-time virus and firewall was turned off.

I tested the phone by simply disconnecting it from the POE ethernet cable and reconnecting. When I see the screen of the S500, it was showing “retrieving config from tftp://192.168.1.3”…“retrieving background”. Mind you the phone wasn’t “hard reset”. Should I do that? I suspect that it WILL successfully be able to provision.

As to whether the tFTP server is running or not…again, since it’s FPBX16, it is running. when I ping 192.168.1.3 FPBX16 replies…so should tftp since it’s the running/production server running on v16.

What tests should I run on tftp on FPBX17?

FPBX16 and 17 installs are radically different because one is centos the other is debian. The tftp instructions I gave were for debian

If the phone is provisioning off the 192.168.1.3 tftp server on the centos box on FreePBX16, then when you tried doing a tftp get from 192.168.1.3 using the same filename that the phone is using, you should have been able to get that file. However, if you weren’t the only way to figure out what is going on is to enable verbose logging on the tftp server and I don’t know without looking how to enable that on centos/freepbx16.

tftp can be finicky and I’ve seen errors with it that make no sense at all. All I can tell you is that in testing with the windows tftp client and debian 12 as tftp server, I am able to obtain files using the tftp get, assuming windows defender is shut off. I won’t say though this would work with the centos tftp server unless I tested it.

You can setup the fpbx17 server on for example 192.168.1.6 and then try the same tftp get test at the command line with the windows box. You can up the verbosity of the tftpd server on the fpbx17 so that /var/log/syslog shows what is going on. You don’t need to mess with the dhcp server option to do this, the tftp client under windows does not require that

I don’t exactly understand how the routing is working between the voice vlan subnet and the default subnet that is when the phones provision themselves to a different vlan then how is the fpbx 16 system seeing the packets from the voice vlan? But that isn’t really an issue since when the phones are factory reset they are on the untagged network and I’m assuming the tftp server is also on the untagged network.

The only other thing I would suggest is are you absolutely sure that the s500 is in fact using tftp to provision and not http?

Thanks for your explanation.
It sounds like I’m going to have to do some digging on how and why tftp:

  1. on CentOS FPBX16 is working but I can’t seem to directly get files from it; while the S500 can
  2. on Debian 17, setup tftp to see if a. it works and b. why I can’t get it to work with the S500

Since you are telling me that tftp isn’t as good as http, may I ask if http is the first provisioning protocol followed by tftp? I don’t see that on the S500 screen. How do I go about to use http? May be that will work better going forward @dicko?

In the screenshot, it shows that the S500 is using tftp and not http protocol.
Screenshot 2024-09-18 at 7.38.36 PM

Regarding the routing to vlan, the provisioning of the S500 is a 2 step process from factory reset:

  • first provision, it uses a 192.168.1.x ip address. gets its vlan assignment
  • remove the ethernet cable (with POE) to soft reboot
  • second provision, S500 uses the vlan assigned ip address, then goes to 192.168.1.3 to get its provisioning config and then the background at tftp://192.168.1.3.
  • successfully finishes the provisioning and the phone can be used.

The fpbx16 sees the packets because the Dream Machine router routes them as I haven’t firewalled the vlans from default lan. It’s complex and it would be easier if the pbx server were on the vlan server but that’s another discussion. Having the phones on their own vlan solved the packet loss issue; leading to crystal clear sounds. Yes, SIP/pjsip are best when on their own Vlan and not sharing the default lan.

This is entirely dependent on the network and switching fabric and phones. For example on my test PBX which has around 20 extensions on it, 8 of them 100 miles away connected by an OpenSSL gateway-to-gateway VPN, all phones are on the default untagged networks and there is no packet loss and sound is clear on all extensions. In fact there are 3 separate routed networks involved.

At work on my production network that has 30 different subnets and 15 different remote sites on a private WAN, we use separate VLANS for SIP. However, none of the SIP VLANs are running any packet prioritization on the WAN (since the WAN is built on a non-deterministic metro ethernet fabric it’s not really possible to prioritize traffic) and yet there is no packet loss.

This is because all phones both on my test network and production network have jitter buffers. My production net is Cisco 8845 and 7821 phones the test network is a mixture of Cisco and Polycom phones plus a few other random phone manufacturers I experiment with from time to time. (In general, when someone posts here claiming it’s impossible to get a phone to work with FreePBX I like to prove them wrong, lol)

So the Dream Machine is routing between the untagged network and your VLAN, that is fine. As long as it is not doing any kind of packet filtering it should be OK. But for testing I would START with the Windows TFTP client on the SAME vlan as the PBX. Then I would move it to the phone VLAN for the second test. Note that it is NOT easy to put a Windows machine on a different VLAN because Microsoft left VLAN tagging to the Ethernet chipset vendor to implement in their drivers and not all of them have done so.

I have a laptop with a Intel(R) Ethernet Connection (6) I219-LM that allows vlan tagging from Windows but Microsot does not include the VLAN stuff in the driver it supplies. To load it you have to download:

https://www.intel.com/content/www/us/en/download/15084/intel-ethernet-adapter-complete-driver-pack.html?v=t

Then you follow this video:

https://www.intel.com/content/www/us/en/support/articles/000056184/ethernet-products/gigabit-ethernet-controllers-up-to-2-5gbe.html

  1. Open Windows Powershell as administrator.

  2. Import the PROSet Powershell* module using this command: Import-Module -Name “C:\Program Files\Intel\Wired Networking\IntelNetCmdlets\IntelNetCmdlets”

  3. Type “Get-IntelNetAdapter” to display the name of your network adapter.

  4. Type “Add-IntelNetVLAN” to initiate the creation of a VLAN.

  5. PowerShell will prompt for VLAN ID

For tagged VLAN, type the desired VLAN ID.

For example: VLAN ID = 9

To create an untagged VLAN, type VLAN ID = 0

I have NOT done this myself so I can’t tell you if it works. The last time I tried fooling around with defining VLANs under Windows for testing I was using a laptop with a Realtek ethernet chip in it and with that one, you have to download a driver from Realtek then they have a GUI program that does it - and when I tried the laptop blue-screened. So I now do any testing requiring VLANS with a laptop running Linux which is far easier to and and remove vlan tagging from an interface.

I wouldn’t worry about figuring out why you can’t use the Windows tftp client to get files from the CentOS FPBX16 system. Sometimes we just gotta let go of old tech.

It isn’t that tftp isn’t as good as http. The reason that http and https are used for phone provisioning is that unlike tftp, they work better over the Internet. A decade ago I managed a number of large customer networks running Cisco devices across the United States, and one of the first times I tried firmware updating a router that was on the Eastern Seaboard from the West Coast, I used tftp. The copy took 3 hours and 2/3 of the packets didn’t make it through and were resent. I was stunned that the firmware file arrived intact and passed it’s internal crc checksum but it did and I rebooted the router on it and it did come back up. I was crossing my fingers on that one for sure.

So many phones today are sold for use with Cloud PBXes (Sangoma has one) that the cloud providers use https provisioning. This allows quick provisioning from the cloud and prevents an attacker from sniffing the SIP username and password. But on a local LAN network like yours, tftp provisioning is perfectly fine and just as good as http provisioning.

As for which is used first that’s phone dependent you will need to read the manual for the s500. I can tell you on the Polycoms they use tftp option 150 which allows a freeform URL to be used, and their newest firmware defaults to whatever 150 says, but if 150 is not handed out they will fall back to option 69 and use ftp provisioning from whatever server is specified in 69 on the latest firmware while the earlier firmware defaults to tftp provisioning. But as you may know Polycom renamed itself to Poly then was acquired by HP.

It is not Debian 17 it is Debian 12 and FreePBX 17

On Debian if you run the FreePBX17 install script it installs tftpd and it configures it plus it loads all of the latest Sangoma phone firmware into /tftpboot One thing it does NOT do is turn on verbosity. To do that you would edit this file like this:

# /etc/default/tftpd-hpa

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="0.0.0.0:69"
TFTP_OPTIONS="--secure --verbosity 7"

The verbosity integer is not explained in the manual but 7 seems to be high enough to generate usable logging. You can try higher values if you want.

Then you can look in /var/log/syslog and see what is going on, what files the phones are requesting, etc.

And like I said I had no problems getting the Windows tftp client to obtain phone config files and firmware from tftpd on Debian 12 so I know those are compatible.

No, that’s exactly it. TFTP is highly insecure since it doesn’t have authentication, access control or encryption built in (to name a few) where as HTTP/HTTPS does. Using FTP/SFTP is better than using TFTP for something like this.

TFTP and HTTP/HTTPS work just as well as the other over the Internet. Not sure how you came to this conclusion. The reason cloud providers, etc provisioning over HTTPS is for the reasons stated above. Security, encryption and access control.

Yes Tom I pointed that out if you had read my post you would have seen that.

I came to this conclusion based on real world experience which I detailed in the post. TFTP is as you say, insecure so anyone on the Internet with access to your packet stream can obtain your SIP credentials so for that reason alone it shouldn’t be used on the Internet. But the big reason is that the Internet does not guarantee packet delivery and since TFTP is based on UDP it is up to the TFTP client to handle retransmission of UDP packets. Now maybe the Sangoma phones have a very good TFTP client that has a large retransmission window I know from experience the Cico router TFTP client is this way - but it’s entirely dependent on the skill in network programming that the firmware author of the phone has (and there’s some real embarassing VoIP firmware efforts out there - such as the Cisco 7960 phone firmware which can’t seem to handle TFTP config file pulls from a different subnet than the TFTP server)

Using FTP or HTTP you are using TCP packets and the tcp stack can dynamically reconfigure it’s sliding window to the lossiness and bandwidth available on the network path, whereas UDP leaves this up to the application, and I’m quite sure that the TFTP client in a phone does not do this so you are always going to have a faster transfer of the provisioning file over http than over tftp on a lossy network like an Internet path.

You ALSO can get a packet trashed on UDP if it is large since you are not guaranteed of an unobstructed 1500 MTU over an Internet path, and if your application does not implement PMTUD and uses 1500 byte sized packets, those will be trashed. And while the TFTP default is usually 500 byte packets, that’s just a default and the client is free to do whatever the heck it pleases.

Default your phone… Then go to Admin>SysAdmin>Provisioning Protocols and disable TFTP Server and then make sure you have either BOTH or HTTPS Only or HTTP Only enabled under HTTP Authentication then Save and Apply…

Now go to EPM>Brands>SANGOMA>D&P Series and select the template you are using for the phone in question… On the very first General tab that opens… Make sure you have HTTP or HTTPS selected under Provisioning Server Protocols… and make sure the correct Provisioning server address is selected right under that. Save, rebuild and apply…

Make sure all the correct information is setup for the redirect server in your portal.sangoma.com account for this deployment and this phone mac address… Specifically the correct IP and port for provisioning…

When you power on the freshly defaulted phone it should provision using HTTP or HTTPS now…

2 Likes

or…
nmap --script broadcast-dhcp-discover

:slight_smile:

1 Like

Thanks! I learn a new thing everyday

I have been able to get TFTP going on FPBX 17. The installation does not change the default tftp directory. So I had to manually change it. Instead, you need to change the default directory from:

/srv/tftp

TO:

/tftpboot

You will also have to bind the Nic to 0.0.0.0 so the tftp server responds.

restart the tftp or reboot the entire Debian server.

reference: TFTP in Debian

The other issue noted in the first post is still outstanding. While I can get my Sangoma S700 phones to now provision automatically via TFTP, the issue of routing calls does not work in the current version of FPBX17. This delays the movement to production. I’m happy to continue to test this as new version of FPBX17 come out. Certainly, the installation script is still a work in progress, isn’t it?

On that note, I would like to also note that while my TFTP server now works, not sure if provisioning updated configurations will be saved to /tftpboot or not. It should but I haven’t tested this yet.

binding to 0.0.0.0 is a horrible risk, actually using tftpd is always a horrible risk because of the T in the acronym.

You are correct, the tftpd-ha installation if you do it from apt-get under debian will not use /tftpboot they use the nonstandard tftp directory. However, when I last ran the FreePBX installation script, the script does install tftpd-ha and modifies it’s config file to have it use the correct directory as well as change the binding to 0.0.0.0. If that’s not happening now maybe they broke something in the FreePBX script.

Unless you have the tftpd port port-forwarded to the FreePBX server from the Internet, or your FreePBX server is on the Internet - it’s really not much more of a security risk to use tftpd to serve files over the internal LAN network particularly if you have modern switches. Remember that tftp is unicast traffic so some smart guy on your LAN network can’t sniff your switch traffic unless he can break into your switch, and reconfigure it to designate the port his machine is on a monitoring port.

The big issue with phone provisioning files is this - because many phones use their MAC address as their provisioning file, all an attacker on the LAN needs to do is ping the phone, get it’s MAC address from the arp table, construct a provisioning filename for that extension, and pull it off the tftp server. Or the http server. or FTP server. Then the attacker has the SIP extension and password. So the “fix” is to encrypt the file but that would mean public/private key encryption and a rather laborious process to distribute keys to all the phones and encrypt the provisioning files. I don’t know how many phones support that.

As for the outbound route thing - billsimon reponded to that and then the thread went down the phone provisioning/tftp rabbithole. My suggestion is you start a new posting on just that issue alone. This is one of the risks of making a “laundry list” post in any of these forums, is you end up with only 1 item in the list being examined.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.