Why Does Updating FreePBX Always Feel Like Defusing a Bomb?

FreePBX Installation / Upgrade
1

Just the mere thought of having to patch FreePBX from time to time using apt-get update && apt-get upgrade is enough to make my toenails curl.

Because every single time — and I mean EVERY. DAMN. TIME. — something goes wrong. Instead of the packages just updating cleanly and the system — specifically the web UI — continuing to run smoothly, some kind of crap alwayshappens.

This time? Ran apt-get update && apt-get upgrade… and once again, I’m hit with **“Detected Missing Dependency”**errors.
WHY? Why is it apparently too much to ask that a normal upgrade doesn’t immediately require dealing with dependency hell?

So, I followed the instruction from the error message and ran fwconsole ma install sysadmin — once again —
But this time? No success either:

Detected Missing Dependency of: firewall 17.0.1.19  
Found local Dependency of: firewall 17.0.1.30  
Detected Missing Dependency of: manager 17.0.1  
Found local Dependency of: manager 17.0.6  
Installing Missing Dependency of: manager 17.0.1  
Updating tables manager...

In ExceptionConverter.php line 117:  
  An exception occurred while executing a query: SQLSTATE[01000]: Warning: 1265 Data truncated for column 'name' at row 4  

In Exception.php line 28:  
  SQLSTATE[01000]: Warning: 1265 Data truncated for column 'name' at row 4  

In Connection.php line 32:  
  SQLSTATE[01000]: Warning: 1265 Data truncated for column 'name' at row 4

So apparently I now need to downgrade previously updated FreePBX modules before I can continue?
What the hell is this. Why is it seemingly impossible to align all modules and components of this software in a way that won’t let the whole phone system self-destruct unless the operator turns into some half-qualified developer just to fight their way through this mess?

And then, the classic:

Module: "Core", File: "/var/www/html/admin/modules/core/images/arrow_up_down.png altered"

Nobody cares whether some random image file has been tampered with…
Except maybe a few ultra-paranoid security nerds who spend their entire day tinkering with their FreePBX setup. Any normal user just wants to get rid of this stupid error. So instead of getting a clear fix, you’re forced to wade through the community just to come across a comment that drops the command fwconsole ma refreshsignatures like it’s some kind of secret handshake.

And — once again — it doesn’t work.

I just had to get that off my chest… might delete later.

7 Likes
2

Yes, Yes it does, but its not FreePBX’s fault. Applying patches in linux isn’t like windows, you can’t always just do an apt update; apt upgrade (or dnf upgrade, etc) and expect there to be no problems. Updates in linux affect many systems and things are constantly changing. Some times it will be fine, other times it will break something.

I’m not going to bleat on about why you should know how to fix dependencies, but you should it will come up again and again. Use fwconsole ma list and and other commands (enable, disable, download, upgrade etc) to fix module dependencies and try and keep them up to date going forward, running an update on a system that is many versions behind will almost always break something - but you can select the version, so do that, update incrementally and it will be fine.

It’s still not FreePBX’s fault :wink:

.. and today I had a customer with a trixbox install running FreePBX 2.4! how cool is that.

3

No, but honestly, it is usually an uneventful, successful process, in my estimate, on the order of 95% or better. Something is odd about @alex922 's experience. No wonder he’s upset.

1 Like
4

Not really, if you work on enough switches you’ll hit these sort of issues all the time. The biggest issue is with updating FreePBX modules, because you’ll come to a switch that hasn’t been updated in 3 years and I know its temping to just hit fwconsole ma upgradeall but I’m telling ya, it has a fair chance of blowing up.

The reason is fairly understandable. When module is updated it’ll bring with it an update script which is executed after its dropped on disk, and this is where it applies changes to things like database tables and files. The problem is, from a development point of view, trying to keep track of every variance between every version and the current version is not easy and mistakes are made.

If you keep the modules up to date, like weekly then its pretty much issue free. On a brighter note its never hard to fix, you just need to downgrade the affected modules, and upgrade them incrementally (fwconsole ma install somemodule:1.2.3) etc.

With system updates, it’s more complex because whilst apt is great its not foolproof and sometimes that’ll create package dependency issues that you’ll need to manually fix - again not hard to do you just need to see what needs what and then make the changes. Sometimes you’ll pull a kernel that doesn’t work with your hardware - that’s more fun but also easily fixable.

As a rule, assume something will break and be pleasantly surprised when it doesn’t - kinda like life really.

1 Like
5

It’s more like doing it in embedded Windows, although I’ve heard that embedded Windows is worse than embedded Linux. With standard windows, you generally only get one way of fulfilling a basic service, and all those are installed by default, making the basic install of Windows much bigger than the basic install of Linux.

6

Can you please elaborate on what was modified on this FreePBX prior to updating it?

7

I’ve had to install FreePBX from scratch 3 times in a row due to apt-upgrade so I refuse to run Linux upgrades anymore. It’s too risky in a production environment to have the system down for hours. It’s not like CentOS that some of us may have been used to. I never had issues like this doing YUM UPDATE. It is the Debian updates that will completely mess up the system. I’ve lost so much time. I just won’t do it anymore.

1 Like
9

Hey everyone – first of all, huge thanks to those of you who actually tried to extract something useful from my rant, despite the emotional outburst :slight_smile:

So… in this specific case, I updated from the version that was considered “latest” back in January to version 17.0.19.25.

  1. First, I install all outstanding module updates through the UI (Admin > Updates) for the currently installed version — just to make sure everything is up to date before I touch the system itself.
  2. Then I run apt-get update && apt-get upgrade. These commands themselves never throw any errors — everything looks clean. Then I reboot the server.

Now, after a reboot, a naive DAU user like me — would expect to simply open the dashboard again and, ifany prerequisites are needed, be guided through them in a straightforward and understandable way.

But nope. Instead, I get greeted — yet again — by the debugger UI, with some cryptic box in the top-left corner showing whatever it feels like.

Most of the time it’s something like:
“Unable to locate the FreePBX BMO Class ‘Userman’. A required module might be disabled or uninstalled. Recommended steps…”

I think it’s a terrible user experience to output a debugging screen instead of just silently doing whatever is obviously expected in that situation.

If the user is going to end up with two choices —
a) manually reinstall or update modules after an upgrade
b) deal with a broken system —
then why the hell doesn’t FreePBX just automatically do what needs to be done?

Why doesn’t the system repair itself after an apt-get upgrade, especially if it knows something’s wrong?

As a spoiled, naive user, I’d expect a smooth splash screen instead of a debugger — something like: “You’ve successfully updated to version XYZ. We’re now preparing a few things in the background so your system can return to normal.” Maybe even with some live logs showing what’s being fixed.

FreePBX built its reputation on being usable by idiots like me. But since the release of version 17, it really feels like we’ve regressed — from a remarkably idiot-proof professional-grade phone system to a developer-centric platform that assumes that users will figure things out on their own. And sure, expecting users to solve things on their own is totally fine in a community edition — no argument there. But my feelind is, that the usability of this system used to be miles ahead of what we’re seeing now.

Basic operations like a system update must be bulletproof. Especially when the UI (under Admin > Updates) tells you explicitly to run apt-get update && apt-get upgrade from the console.
But what it doesn’t say is what kind of chain reaction that command triggers — and what manual cleanup work you’re inevitably in for afterward.

It’s a total love-hate relationship. On one hand, I’m genuinely grateful for this insanely flexible and powerful telephony platform — which lets me implement setups that’d cost a fortune with a typical cloud telephony provider. But on the other hand, I’m fed up with how fragile the underlying architecture appears to me — and how it’s basically become the norm that, as a user, I end up dealing with debugger screens and console commands every time I just wanted to apply a simple patch.

How long the post-upgrade fixing takes varies from case to case. And it’s always a mystery box:

Will it be enough to reboot ans just run

fwconsole chown  
fwconsole ma upgradeall  
fwconsole reload or fwconsole restart

— or will this “quick update” instead unleash a cascade of weird errors that you can only solve by piecing together forum posts from the last several years?

Another example: In my case, after every upgrade, all PJSIP extensions and trunks show as “connected” and appear “Online” under Asterisk Info. But none of the extensions are actually reachable — calls fail with a busy tone. The fix? I have to go into Connectivity > Extensions, open each extension entry, and hit “Submit” — even without changing anything — just to make them work again.

I always assumed that a full server reboot would reload all dialplans, re-register extensions, and apply the configurations cleanly. But clearly, that’s not how it works.

2 Likes
10

IMO because the software is not adequately tested before being pushed out. Even issues that have been reported for YEARS remain issues.

Sangoma isn’t alone in doing this; there is a massive cost-shifting from software developers onto their customers. They even coined a name for it: “Agile software development”. Sounds so much better than “We didn’t pay to test it because we can make you install it and test it for us.”

4 Likes
11

That isn’t what Agile development actually is. Nothing publicly facing about this project suggest that there any sort of Agile based development being done. There are 3 C’s to Agile development: Collaboration, Communication and Coordination…believe me those aren’t being followed at all with this project.

12

Agile is just a management buzzword invented for software developer project managers to teach them how to manage large projects the way everyone else in business does, which is (apparently) something that none of them was taught in school when they were getting their comp-sci degrees.

Don’t get your panties in a twist on this - my daughter is currently working on her mechanical engineering degree and they STILL aren’t teaching that. I told her to take a minor in business - but she can’t - the engineering track major is way too heavy on the hard sciences part to leave room for anything outside of pure engineering courses.

And I have no doubt that comp sci is still the same way.

There are software devs who are graduating from school today who have never fired up a copy of Excel let alone Microsoft Project and don’t know what the term “Backward Design” means nor could they create even the simplest of spreadsheets with a backward designed project management design on it.

Trust me this is NOT a cost-shift thing as every business suit above the development group is pulling their hair out trying to figure out how to teach engineers how to plan anything.

The NASA US Space Shuttle exploded because of this kind of thing - Boeing lost a door off an airplane -in flight- because of this problem - it’s endemic to the discipline of engineering, IMHO because of the way engineering is taught in colleges.

Just my $0.02

1 Like
13

My understanding is that it treats software development as a continual process, not as being project based. The traditional, project based method, was “waterfall”.

My suspicion is that agile is used where software is a fashion item, and particularly for software products, which tend to have to continually change, to attract more revenue.

If you consider a house, the big project is from initial design until handover to the purchaser. It then enters a maintenance phase. There may be many small projects, e.g. redecorating, replacing a boiler, adding burglar alarms, and maybe some medium ones, like replacing the roof, but these are not part of the, initial, big project.

Maybe a big “system” is a better description of what FreePBX is now,

The other problem with management buzzwords, is that the original idea may be valid, but people adopt them thinking that they are ways of avoiding work, and understanding.

Another point that came up is defects that never get fixed That, my experience, is normal. Bug reports, software performance, reports, or whatever you want to call them, are inputs to management. Management then allocates limited resources to fixing those that it thinks will have a significant impact on its business, also with some consideration to the cost of fixing them. It’s quite possible for something to be recorded as being wrong, but to never get fixed.

Note that I have no insider knowledge of Sangoma management processes, I’ve been retired from software development for some years, and my employer never formally used the term Agile, although I think they had taken on some of its concepts.

14

The only real significant difference between the 2 is that Agile “allegedly” involves the customer during the entire project development cycle a chunk at a time vs handing the completed thing to the customer.

Agile vs. Waterfall: Differences and Benefits of Each | Indeed.com

The reality of course is that with pretty much ALL “installed software” aka “over the counter commercial software” the customer is not involved AT ALL so “Agile” is really, and truly a software development marketing buzzword since it’s impossible by definition.

The day Microsoft comes to me and says “we want your input during the development of our next version of Microsoft Office” THEN I’ll believe they are using Agile. Otherwise, it’s complete BS

And I can tell you they aren’t teaching either methodologies to engineering students still to this day. Sigh.

15

I feel your pain.

We’ve come to learn that it’s just something you have to be prepared for. It rarely goes without incident, so we just have to plan it in. It does mean we don’t upgrade as regularly as we’d otherwise want to though.

Hang in there, the system overall is worth it, despite your obvious frustration.

S

1 Like
16

Hi @alex922 ,
Could you please try install manager module latest version (manager 17.0.7) first then install sysadmin

17

Your point put another way - the people who design a thing generally do not use it; if they did, they wouldn’t have designed it how they did. Anyone who has worked on copiers, printers, cars, laptops, pre-built PCs, etc, etc, etc - really anything other than simple machines - intuitively knows this.

For examples, one of my sister’s headlight bulbs went out a couple years ago - she asked me to fix it for her. In order to change it, I had to take the wheel completely off that side and remove the body panels!!! Seriously people?!? How did that get approved??? :laughing: Then there is the fad of PC makers soldering RAM onto their motherboards… :man_facepalming: I guess that might seem like a good idea, if you’ve never found bad RAM in a PC or had a machine long enough to want to upgrade the memory in it. :rofl: I used to work at a GM dealership and actually wrote a program for the sales people to transcode digital music into a standard CBR MP3 as the radios would randomly spazz out when fed different file formats… customers would come & complain to the person who sold them a brand new car that their radio glitched out, so they had to pull off the highway, turn their car off & turn it back on just to fix the radio. :expressionless_face: <— I dare anyone to tell me with a straight face that whoever designed any of these things had to use / work on them… :joy: :rofl:

18

Hi @alex922. I had these issues with FreePBX 14, 15 and 16 (Dependency and update hell). But 2 things made my life infinitely better with this. FreePBX 17 on Deboan 12+ (Which it sounds like you are on), AS WELL as an update script a nice gentleman/user passed to me (I am not sure what happened to him, he has been absent for quite some time). Anyway, I installed Debian 12 and FreePBX 17 using the Install script with Debian, which for whatever reason seems to integrate things really well. I imported my config. Then I run ALL updates for Debian and the FreePBX 17 Module Updates in Unison with his original script. It’s been about 1 year now, and I have never had a failed update to date. It’s seems running the OS updates and the FreePBX modules update in Unison, seem to make all the difference. If you want, I can PM you the script. It started its life in FreePBX 16 on CentOS, but I modified the script for Debian/FreePBX 17, and its been working perfectly for updates for over 1 year now. PM me and I will pass it to you. I wish I could remember the orginal user’s name. I will search for him and post his username in here when I find it, so that he has credit for the script.

I found the original user who created the script: @sorvani (He was very helpful and supportive, as many here have been. Especially when I was first starting with FreePBX, which was awhile ago now.)

19

I definitely prefer the method of installing Debian-12 server, doing the full apt-update and apt-upgrade and then run the FreePBX-17 install script. After that point, using the GUI mostly keeps your modules correctly updated.

It is also important to keep up-to-date on module updates. I do it every Saturday. It seems that Sangoma is not always good about handling upgrade of a module if you have skipped too many of the intervening updates.

I periodically update Debian-12 and have never had a FreePBX issue afterwards.

If you are upgrading from one of the CentOS versions of FreePBX, usually you can use the FreePBX backup from that system and restore it to FreePBX-17 and it handles a lot of the heavy lifting. I prefer using the open-source only version of FreePBX because I don’t use commercial modules and that helps to bypass some of the pain associated with the full-blown FreePBX install.

By maintaining more granular control of the installation, I feel I sidestep most of the issues some seem to encounter.

1 Like
20

chuckles… on this past weekend, I had modules to update, carried that out, then debian told me freepbx needed updating from apt, then reloading couldnt find core,manual run modules update with fwconsole got core back.

I thought sangma said life would be simpler under debian… ummmmm… yeah.. nah.

1 Like
21

Nobody should do this! Yes apt update shows that there’s a newer version but once freepbx is installed it should be updated with fwconsole. Else you overwrite your modules and break your system.

@kgupta I think this is going to keep happening to people so it would be good to figure out a better way to do an “install once and then never mention it again” sort of package for initial installation of FreePBX.