Update modules, HD full, system down. :(

Good evening, all (well, at least for some it might be)…

Ran an all module update this evening, and apparently in the middle of it, the HD became 100% full, causing the update to fail. Once I figured out what was causing it, I was able to remove the excess logs and clear space, but by then the update had already been stopped and the system is now essentially down.

Running the fwconsole command merely responds with:
SELECT value FROM admin WHERE variable = 'version’

SQLSTATE[HY000]: General error: 1017 Can’t find file: ‘admin’ (errno: 2)::

My hope was to try installing all modules again (with a FORCE switch), but that’s all contingent upon “fwconsole”, which fails. I then thought, well maybe I can run the upgrade script, but that also failed because it essentially updates the modules via “fwconsole”.

Does anyone have any thoughts on this one?


Your database is corrupt probably because you ran out of disk space.

Any way to restore from a backup via CLI, when fwconsole doesn’t function, or would it just be easier to build a new machine? If I build a new machine, someone’s going to have to do a zendreset for me, as I can’t do them myself anymore (max reached, as we migrated from on-prem to virtualized, then amazon cloud, then google cloud, etc.). I could give them access to the existing box so they can kill it if they need that validation, but I’d have to have that done if I were going to build a new one. Hell, might be the time to upgrade to 14 then. :slight_smile:

As a side note, I’m thinking the Module Admin module should do a DF command (or something) and confirm at least 10% drive space available before allowing the upgrade. Just a thought…

That sounds like a good feature request (or even a bug-report, since the process failed for your installation). Use the Issues tab to request it - be sure to check if someone hasn’t already beat you to is (Andrew is pretty good about doing this for us “mere mortals”,

Is there any way to restore the config from the CLI (I have a backup from the night before)?

It depends on how your backup was done. In general, the “restore” function should reload the last backup, but you have to make sure the version numbers have to match for the restore to work.

Backups were done with the backup/restore module in FPBX. Versions should be the same, as the backup was done the night before the box died, but the box died in the middle of module updates (so it’s possible some of those version numbers were “updated”.

In looking over the CLI commands for 13, I haven’t found anything that indicates I could invoke the restore function with a call to that module (checked under module admin as well).

With the failure you had, there’s a 50/50 chance that nothing you do is going to get your system going again. because of the damage to the database, you might end up having to start over.

Well, now that FPBX14 uses Yum for those modules, that pre-checking is essentially built in. I certainly don’t mind making a feature request against 13, but since 14 now has it, I’m not sure the resources and time that would go into adding it to 13 would necessarily be worthwhile.

Granted, this is from the guy who just burned his system up over the very same problem, but I don’t get the sense it happens that often. :slight_smile:

I’m building a v14 image now to upload, and once I get that completed, I’ll rebuild everything fresh and then contact the licensing gods to get it re-activated (sigh).

Why not use a fresh installation with the same version you had and restore from backup?

Anyhow, storage alerts is there for a reason. It will bug you on the hour until you clean up some disk space.

Gee, why didn’t I think of that. :wink:

That’s what I ultimately did, but since it went ahead and unlicensed itself (which now means I have to have the licensing gods force a zend reset), I’m going ahead and building a fresh v14 box and will migrate the data over to that. So, we’re up and running, but of course none of my modules work since they’re now unlicensed, and now I’m building a fresh v14, so when I have them correct the licensing lockout, it’s against that box instead of the temp v13 box).