Free PBX ran out of disk space - How to extend the partition and file system

FreePBX Tips and Tricks
1

I’ve written this to hopefully save some time for you hard working, under-rated network techs, so read on :-

Symptoms – My New FreePBX 14.0 distro test installation – the internal phones suddenly are unable to make calls.

The Free PBX web interface showed some Daemons not starting and Asterisk module as errored, ie some critical errors

Logged into the console as root. Immediately a host of errors were shown, a quick scan through them came up with several that indicated a lack of disk space.

A quick df command showed that the mount point /dev/mapper/SangomaVG-root, on the root (/) was at 100%.

Back to the Web interface – Admins – System Admin – storage again showed root ‘/’ drive usage at 100%

After a lot of googling and some very helpful sites – this is how I created more space.

1) Increase size of the Virtual Hard Drive

Our FreePBX is a virtual machine on MS Hyper-V, so shutdown the machine and increased the size of the virtual hard drive for the virtual machine – added another 40 GB (Easy enough !)

2) Increase size of the partition

Using the ubiquitous and fantastic GParted partitioning tool, downloaded the GParted ISO, mounted this as VM media and booted into GParted (With a physical machine, create a bootable USB and boot into GParted).

Now it’s easy enough to select the partition with the SangomaVG mount point and drag the handle to extend the partition to fill the new space in the virtual drive.

Sooo after a reboot, hooray, I thought, job done !

BUT in the console, df to view the Filesystem – shows the Sangoma mount point as still at 100% and still at the old 50G size (should be 100G !). Now need to extend the File System to fill the partition.

3) Resizing the File System

Attempt 1 : Tried the usual way for a linux file system, resize2fs /dev/mapper/SangomaVG-root ( which is the file system name as shown by the df command), to expand the file system to fill the partition.

This FAILED with a message ‘Bad magic number in super-block

Attempt 2: command df –T shows that the file system is of type xfs

Instead of resize2fs, this would require the xfs_growfs command:
xfs_growfs /dev/mapper/SangomaVG-root -d to expand to fill the partition.

Although the command looked to run OK, this FAILED with message ‘data size unchanged’

Attempt 3: Using the linux partition editor - parted , then print commands, the partition shows flagged as lvm (logical volume, a bit like dynamic partitions, which apparently cannot be grown with xfs_growfs !

So more research:

Command vgs – shows the free space available for your file system, I had 42.6G spare (which would fill the partition)

Can now extend the xfs file system on the LVM flagged partition, with:

Lvextend –L +42.6G /dev/mapper/SangomaVG-root –r

df now shows spare space on the file system, usage at 54%

SUCCESS

Rebooted FreePBX and hey presto, daemons started and no Asterisk error

PHONES WORKING AGAIN !

6 Likes
2

Thanks for the write up, I’ll need this in the future! I’m on OpenVZ, it adjusts the container’s guestOS automatically, but down the road will be switching to either HyperV or ESXi. How has it performed on HyperV?

What’s taking up so much drive space? Do you have logging and debugging set to high, or call recordings?

Most of my Guests have 20-30GB and I rarely run into issues by adjusting log rotation settings. I offload historical data to an external SYSLOG server for long term retention on cheaper drives and set retention to save 2 day’s worth of logs on the server and compress.

vi /etc/logrotate.d/asterisk

rotate 2
compress

If there’s call recordings, I set a batch script to delete old call recordings:

       #!/bin/bash
       #This script will delete call recordings older than 25 days
       #this didn't work: find /var/spool/asterisk/monitor/ -name “*.wav” -mtime 25
       # Find all recordings older than 25 days and delete
       find /var/spool/asterisk/monitor -name "*.wav" -mtime +25 -delete
       exit

And run it every night with crond:

vi /etc/crontab

0 0 * * * root bash /root/25dayrec.sh

I’ve also converted wav files to mp3 to save on space, but drive space is easier to come by than CPU, I’ve run into issues with the conversion in my VM environment doing this, so I’d advise against it.

3

Thanks for the feedback. We tend to set fixed disk sizes on our Hyper-V VMs. We’ve had issues with dynamic in the past on general servers and I have read a couple of things about probs with FreePBX and dynamic expanding disks, but that might just be hear-say !
Huge amount of space is being used by asterisk logs. As this is a new install I guess the full debug is on by default. I like your idea of adjusting rotation to a couple of days and storing the logs elsewhere.

Cheers for the tip

1 Like
4

Thanks for sharing your experiences with configuring Freepbx VMs in Hyper-V, this will come in handy!

5

I’m not a linux specialist ,but i’ve tryed to make everything that describe step-by-step and all worked good, except only one little thing. I’ve stucked at the “Lvextend” command, you’ll need to use all symbols in lowercase, like - “lvextend”. I hope it’ll be help to anyone

6

Looks like Sangoma uses XFS on LVM, so for my new install of FreePBX 2020 which ran out of space:

  • I ran gparted iso to expand the partition
  • I ran lvextend –l +100%FREE /dev/mapper/SangomaVG-root –r to extend LVM to use the space
  • THEN I had to run xfs_growfs /dev/mapper/SangomaVG-root -d for XFS on LVM to use the newly available blocks.

Thanks again for your help, couldn’t have done it without you!

1 Like
7

I was able to do the same thing on my VM running on Hyper-V, but instead of booting to the gparted iso, I used fdisk, which was already installed. Increased the available disk space without shutting down the VM at all. Just be careful in fdisk and take a VM snapshot beforehand, in case you mess up

8

Since this is helping out others, you should write a How To Article
https://wiki.freepbx.org/display/FDT/How-to+articles

1 Like
9

Thank you!!!

10

yum|apt install logwatch then rtfm (and your emails :wink: )

1 Like