I am having to pull a backup from my backup system. I am using one of the full system state image backups that I have because you cannot easily rollback Linux systems.
This is a major issue that not everyone is aware of, it should stay pinned longer than just 4 days.
2 Likes
Another option – instead of “rescue” mode – you might try booting up Debian 12 with a “live” CD.
^^^ this would get you closer to the binaries on the bookworm (12) system before you upgraded to trixie (13).
Well that seems like another similar but different matter… maybe split out into a new topic ?
Howdy! Welcome to the forums.
Generally, Linux distros are pretty good about not destroying your existing data during in-line changes, but its a tricky area and agree with you that regular backups are always advised.
Using the Discourse terms here: it is still “pinned” “globally” but no longer “banner”. And it is set to “un-pin” on October 1st. Also, users can “dismiss” the “pins” and/or “banners” from their own view by clicking the “X” in the upper-right corner of those notices.
I’m aware of how this works. Using the Discourse terms and the history of this topic you removed the “banner” two hours ago but you “unpinned” it an hour ago which is when I said it should stay “pinned” and then you “pinned” it again. If I thought the “banner” should have stayed, I would have suggested it.
@ncorbic @mwhite @penguinpbx @kgupta
All, you need to sit down and figure out some proper testing and processes here. This entire Trixie/stable repo thing broken not only existing installs but now it is breaking new installs. It seems there is a lack of understanding on how these things are done.
If a version specific ISO/installer is used then that install is locked to that version’s repo. Cloud providers offer version specific and in some cases, stable based options. For example, when I installed Debian 12 last year on Vultr all my repos where already locked to:
deb http://deb.debian.org/debian bookworm main contrib non-free
By running the FreePBX v17 install script back then the install script locked me to:
deb http://deb.debian.org/debian stable main contrib non-free
So Sangoma made a choice to put people on the floating stable repo. It was a bad choice but it was made. Now when people are trying to install a Bookworm specific ISO/installer the FreePBX v17 installer script is doing this:
if [ ! "$noaac" ] ; then
add-apt-repository -y -S "deb $DEBIAN_MIRROR bookworm main non-free non-free-firmware" >> "$log"
fi
This of course causes it to not do what is expected since it’s already set to that repo and thus the result is this [Install Error]
The only time the stable repo is set is when a netinstall or using specifically a Stable ISO release. However, when using the netinstall release it asks you at the start of the install to select the track you want to use, the current version track or the stable track.
I don’t understand why this wasn’t tested in various scenarios and this “fix” was just rushed out. The real fix should have been validating if Bookworm was already being used or not.
This is what should be done in the install script.
if ! grep -RqsE 'deb\s+http://deb\.debian\.org/debian\s+bookworm\b' /etc/apt/sources.list /etc/apt/sources.list.d/; then
echo "deb http://deb.debian.org/debian bookworm main contrib non-free" | sudo tee /etc/apt/sources.list.d/bookworm.list
fi
I’ll add a PR in the hour or so once I finish this scheduled project task I’m about to start. But guys, these sloppy mistakes need to really be addressed and stopped. We keep questioning the QA/QC process, we’re told the processes are in place and then things like this keep happening over and over again.
2 Likes
There might be a bug in Discourse – I removed the “Banner”, then you posted, then I thought maybe the topic got automagically “Unpinned” because it was both “Banner” and a “Pin”, so after you posted is when I “Unpinned” then “Pinned”. The log should show this, but sometimes posts happen at the same time, etc. (Corollary being that by re-pinning this again it would get bumped to the top for all users, which is probably a good thing – agreed with you there about keeping this top of mind.)
You might consider that changing the installer to use an apt-add-repository command is the better long-term fix – as alluded to by @billsimon:
…but happy to discuss further in your upcoming PR if that what’s you want to do @BlazeStudios.
Another option: we can continue to work it out here, but, this topic was started with the goal of helping users who are updating existing systems – as reflected by the title, which has seen only a few emoji-fications since its initial posting, and no edits of technical substance – and overall existing systems are generally a little harder to fix with a one-size-fits-all solution vs. new installs into greenfields.
That said the subsequent fix provided for by updated sysadmin does seem to be working for most existing installations if executed before apt upgrading any Debian packages to version 13:
Sure, it doesn’t change the fact that just doing it on its own (which is the current solution) doesn’t work as expected because the repo is already added with version specific ISO/installers. So I can change it to use apt-add-repository but the check is still needed because it throws an error that isn’t being handled properly.
Yes but as a result of this thread the install script was updated with the same “fix” and that fix is what is breaking new installs on version specific OS installs. Because if you didn’t fix the install script, every new install would have to run the fix in order to work moving forward.
So while the fix seems to fix existing installs, it seems no one actual tested using that fix with new installs but instead just added it to the install script thinking it would.
As of right now the install script tries to modify something that doesn’t need to be modified, when it doesn’t actual modify anything the srcentry value isn’t what the script is expecting (its None instead of an object) and because the script isn’t validating what value srcentry has just expects an object to be there, it causes the install script to choke out and die.
If proper testing was done on all of this it would have been caught but instead while pushing a fix for existing installs, the fix now breaks new installs. Not the trade off anyone wanted or was looking for.
Here’s the issue tracker ticket
PR is here and IT NEEDS TO BE TESTED FULLY AS OUTLINE OR MORE. before merging
This will now check if the stable repos are in use, including security repos. If it is, it will change it to Bookworm for both. It will then check if Bookworm repos are installed. If they do exist, nothing happens and the python bug is avoided. If they don’t, adds them.
Now I used apt-add-repository at @penguinpbx behest but it’s not really the best choice. It was something that was made for Ubuntu and was never fully back ported into Debian and therefore it has bugs. Those bugs are what causes the current installer to choke and die as the Python backend script expects an object value and doesn’t know what to do with any other value like None or NoneType when the update fails or doesn’t complete.
This is why I can’t stand binary updates, the way FreeBSD does it where you compile the stuff is far superior.
So, OK. For a newbie what are the current install steps for a brand new virgin Debian 12 distro then FPBX 17, that will avoid all the nonsense above?
-
Download debian 12 (from where, what ISO, someone above said it’s no longer available)
-
After Debina 12 is installed, modify apt so it will only update stuff in 12 and not update 12 to 13 (what files need to be changed for this? Many mentions of different things in the spaghetti above)
-
Run FPBX 17 install script (does the current script download a FreePBX that’s fixed?)
-
Run what against the FreePBX 17 install?
Will it all work?
What exactly does anyone mean when they say “I upgraded to debian 12 and my system broke” Are they talking “won’t boot” are they talking “boots but the PBX does not run” what, exactly? Can people stop saying completely unspecified scary stuff in these threads?
This is NOT 1962 where computer have 64k or ram and you get points for being concise. You are not a 14 year old moody teen who’s goal is to convey a 2 hour conversation in 4 words.
And if you update debian 12 to debian 13 and suddenly your entire server won’t boot - then explain how in the heck this is a FreePBX problem and why are you coming here at all? It seems to me first step would be to go to the DEBIAN forum and say “uh, dudes, your POS apt installer destroyed my system” Except that the concise crowd would probably say “d13 killz d12” and be mad they couldn’t shorten the word “killz”
FreeBSD offers binary package updates and installs just like Linux distros. Compiling each thing individually is a choice not a requirement.
It’s no longer the default download for Debian, Trixie is. The documentation still points to the default download page at debian.org, since Trixie is now the stable release Bookworm has been moved to the oldstable location. The documents need to reflect that.
The core issue of this bug is that the FreePBX installer was modifying the default repos to point them to the stable repo instead of keeping it locked to the version’s repos. So as of Aug 9th when Trixie was pushed to the stable repo (and Bookworm removed) anyone running apt update was updating their systems using Trixie repos. While there was a warning and an option not to continue doing so, some people just hit “Yes”.
What they mean is that they ran an apt update && apt upgrade and they ignored or didn’t understand the warnings that this would cause them to also update the OS to Trixie because they were on the stable repos. By updating to Trixie it broke FreePBX because it also meant things like PHP and other things were updated to versions FreePBX hasn’t be tested against. Trixie installs PHP 8.4 by default, FreePBX doesn’t run on PHP 8.4 so thus main cause of breakage.
FreeBSD has a ports directory which all you have to do is CD into what you want and run make install, the program is downloaded and patched and installed. That’s a lot different than doing a hand compilation of Asterisk or any other program on Linux. A big advantage is that orphaned GPL and BSD licensed programs that devs attempt to “withdraw” by removing them from Sourceforge or github or wherever or re-released as binary-only blobs with the source deleted, are copied to the FreeBSD private archives and are made available to Ports. There are thousands of FreeBSD systems with full sources on them for all their applications so if a dev does try any of these shenanigans, then they basically can’t.
This has saved my bacon many times. Not so much major apps but it always seems like there’s an app out there with a dependency on some little pee-ante bit of doggerel code written by someone who decides at a very inopportune moment to try to force everyone using it to update to “the latest version” I’m looking at you libusb, lol.
Anyway, getting back to it,
Looking at the info in the bug you filed the FreePBX installer is still doing this as of now. Reading through this thread we seem to have an argument being made by Sangoma/penguinpbx that /etc/apt/sources.list and friends are broken and to change stable to bookworm. We seem to have an argument being made by you that the installer is broken and it should be making that same change. What I don’t understand, though, is it appears that you and Sangoma are in agreement that this is the problem and that’s how to fix it - so why didn’t they just immediately apply the fix and close the ticket?
Instead they are dragging this out by arguing to fix it differently. So while they and you are discussing what the right color to paint the bike shed is, the bikes are getting stolen.
The very first post on this is 6 days old. The bug was IDed a few days after that, a fix was served to Sangoma on a silver platter yesterday - so we are going to now have a week of newbies trying out FreePBX for the first time and having it blow chunks on them.
If the water is coming through a hole in the dike it’s OK to put the little Dutch boy there and have him jam his finger in it while you run and get the cork and hammer to make it look pretty, is all I’m saying.
Thanks for the feedback @tmittelstaedt will try to keep the bikes secured 
– issue is that the fix was tested and did work on new installs, but then something else appears to have changed, so we are addressing this as well. Very likely this means a new ISO rolls out early this week with an updated shell script installer wrapped inside. But if users are still reporting issues with a) latest shell installer on vanilla Debian 12 install and/or b) the latest copy of the FreePBX 17 ISO from a few months ago – well then we need to sort that out.
The fix that Sangoma pushed adds the stable repo without doing any checks. In Debian when you run apt-add-repository and attempt to add a repository that already exists the Python script that handles apt-add-repository has a bug. It expects an object value to be returned, that only happens if the repository was addded if the repository wasn’t added for any reason, it returns a different value and the Python script doesn’t know what to do with it so it dies.
My addition to the fix does checks before attempting to do anything. It checks if the stable repos are installed, if they are change them to Bookworm. It then checks to see if the Bookworm repos are installed, if they aren’t add them. This avoids the Python bug in apt-add-repository.
Because it needs to be tested. Not everyone is going to use SNGDEB to install Debian. Cloud providers like Vultr, AWS, etc are going to install Debian 12 specific ISOs which will automatically install all the Bookworm repos. Therefore the install script doesn’t need to do a thing on those types of installs.
Now that Bookworm is oldstable, only the minimal netinstall ISO is available. If that is used now to install Debian 12 there is a pretty good chance that not all the needed repos are installed depending on the install options selected. So in these cases a check should be done because the install script may need to add repos.
As for SNGDEB, I have not idea what it does (I won’t use it) outside of give you some weirdly named options to install Debian 12 and then auto runs the same install script on reboot. It may or may not be installing all the needed repos during the OS install and the script needs to do it.
Tested against what, SNGDEB? Was it tested against the standard full Debian ISO like cloud providers offer? The official netinstall ISO (with and without the standard system option)?
If the only testing was against SNGDEB then Sangoma has already abandoned the promises of moving to Debian. The whole “the project will no longer publish an ISO” isn’t going very well at this point.
So again, what type of installs was this tested against?
Oh no, back the truck up on this one.
You go to : Launch of SNGDEB BETA ISO for Smoother FreePBX 17 Installations | FreePBX - Let Freedom Ring
And right there smack in the front it states:
" GPLv2.0-only license for the build scripts to reproduce your own ISO from A STOCK DEBIAN ISO as input!!
So, sorry but AWS’s hacked-up Debian 12 specific ISO does NOT qualify as a “stock Debian ISO”
The only OFFICIAL “stock Debian 12 ISOs” that exist are the ones at Index of /cdimage/archive/latest-oldstable/amd64/iso-cd
PERIOD!!!
Look this is the thing about documenting for newbies. Documentation MUST be EXPLICIT. If FreePBX wants to support installation on Cloud Provider’s versions of Debian 12 - then you DO NOT SAY statements such as “Cloud providers like Vultr, AWS, etc.” Instead you say “supported on the following Cloud Provider’s versions of Debian 12: AWS, Vultr” That is ANYTHING that is NOT explicitly stated as supported - IS NOT GUARANTEED.
I get the idea behind SNGDEB. Great, the project wants to return to distributing ISOs, fine. Like BlazeStudios, I won’t use SNGDEB -either- and NEVER will. The reason is quite simple - control.
With the “normal” install instructions - download stock Debian 12, then run the install script - when I download and install Debian 12, when I finish the Debian 12 portion of the install I have a KNOWN system. It’s exactly identical to every other Debian 12 system I might have doing anything else in my server room. I can install it virtually and clone off the virtual installation and use it repeatedly - for a great many other things that FreePBX.
Once the FreePBX install script finishes, since I can read the script before running it - I STILL have a Known System. I can see every system-level command the script is making every operating system level change. If the script does something to the OS that I don’t like that isn’t needed for FreePBX - I can revert it.
But, an ISO release of FreePBX - that includes the operating system - essentially turns the FreePBX system into a complete black box. There’s thousands of files in a Debian 12 OS and I can’t go through every one of them and see if Sangoma made some change or other.
Anyway, if the changes to the install script are broken on the SNGDEB ISO, that’s a SNGDEB problem. That is a COMPLETELY separate issue from whether or not the install script is broken on a “stock Debian 12”
The only official instructions at the moment that are NOT labeled “beta” are here:
GitHub - FreePBX/sng_freepbx_debian_install: FreePBX 17 Installation Script
And they say:
“…This script is to install FreePBX on the top of VANILLA Debian 12.x OS…”
Vanilla Debian 12 exists currently only at 1 location - the oldstable location.
Now I’m perfectly aware that there’s web pages like the following:
Getting Started | FreePBX - Let Freedom Ring
that say “…In the cloud, check with your hosting provider for their preferred, built-in Debian 12 image (if any)…”
But that is sloppy, hand-waving BS because it is directly contradicted by the documentation of the actual install script itself - which is linked to by that page - which states “vanilla Debian” A Cloud-provider’s build-in version of Debian 12 is not “stock Debian 12” or “vanilla Debian 12”
In fact, the entire word “vanilla” and “stock” has no place in this discussion anyway. Vanilla is a flavor for Ice Cream, and a type of sex. Stock is from auto racing meaning “stock car” that is unmodified automobile from factory assembly line - that is “dealer stock”
Neither really belongs in documentation or software installations because they are not explicit enough.
As I see it, the entire argument, hand-wringing on this comes from sloppy, imprecise language used in install instructions. Don’t you all see how confusing this is to a newbie to FreePBX? They go to one web page that tells them to use “their cloud providers install” well WTF does that mean anyway? What if they just go to their cloud provider and get a virtual machine and download Debian 12 and install it in that for the same reason I would - that they don’t trust that their cloud provider hasn’t inserted a whole host of BS into the “preferred ISO” behind their back? Then they go to another page that tells them to use “vanilla” Debian. Well, clearly, they probably realize this doesn’t involve Ice Cream or Sex, but it probably ISN’T their “cloud providers preferred ISO” And of course, they are going to see SNGDEB but is the explanation of that acronym anywhere? No, it is not. And, Sangoma Debian, SNGDEB, is beta anyway.
All this sloppy imprecise install language has finally been called out by stupidity in the PHP projects insistence on breaking every application on the Internet that uses PHP every single damn time they release a new version of PHP. It’s like their mission in life is to make people’s lives miserable who use PHP and stuff their religion on The Proper Way To Write PHP down everyone’s throats. But the answer to that is to do what every distro does nowadays - they lock specific PHP versions to specific distro versions. Fine and good. Clearly the PHP developers aren’t answering the clue phone and everyone has given up ringing it, so we all have to do this now.
But that means, since FreePBX is dependent on PHP, that FreePBX now has a very strong dependency on very specific versions of the OS. It would not matter if it was Debian or Ubuntu or FreeBSD or anything that they picked as the “blessed OS” That dependency exists - and all install documentation needs to explicitly reflect it.
(The entire point of picking Debian 12 in any case, IMHO, was because of the 600 pound gorilla named Canonical. It’s a way of tacitly using the most popular Linux Server distribution in the world as the host for FreePBX so that everyone out there paying for a maintenance copy of Ubuntu Pro can host their own FreePBX system, without just coming right out and saying “we give up just pay Canonical to set it up” The same thing was done by using Centos back in the day when Red Hat was the 600 pound Linux gorilla. But this is an Emperor’s New Clothes discussion)
Okay, we have 2 existing installs of FreePBX17 on Debian 12, install script was used last summer, and I have been updating regularly, UP UNTIL this issue. I am trying to follow the thread, but I am VERY unclear of a few things at this point. I want to make sure I DON’T end up on Debian 13, since it is not supported. Do I do this to prevent that from happening:
$ sudo fwconsole ma downloadinstall sysadmin --tag=17.0.2.35
$ sudo fwconsole sa disable-deb-update-v13
$ apt-add-repository -L | grep stable
Or do I do this? Or should I wait?
$ sudo sed -i ‘s/ stable / bookworm /g’ /etc/apt/sources.list.d/archive_uri-http_ftp_debian_org_debian-bookworm.list
$ sudo sed -i ‘s/ stable / bookworm /g’ /etc/apt/sources.list.d/archive_uri-http_ftp_debian_org_debian_-bookworm.list
$ sudo sed -i ‘s/ stable / bookworm /g’ /etc/apt/sources.list
Start there. That should be enough. 
For users not using newer sysadmin, the sed steps are an alternative.
What you DON’T want to do is press ‘y’ during apt update on any prompts asking you to change to version 13. 
You could – we are testing additional improvements, such as the patch from @BlazeStudios mentioned above. There will probably be more automation of the fix in the near future.
What is your recommendation? Do I benefit from waiting? If I do, I can hold on longer. Thanks for your response. Much appreciated.