FreePBX | Register | Issues | Wiki | Portal | Support

Free PBX ran out of disk space - How to extend the partition and file system


(Crimsonbadger) #1

I’ve written this to hopefully save some time for you hard working, under-rated network techs, so read on :-

Symptoms – My New FreePBX 14.0 distro test installation – the internal phones suddenly are unable to make calls.

The Free PBX web interface showed some Daemons not starting and Asterisk module as errored, ie some critical errors

Logged into the console as root. Immediately a host of errors were shown, a quick scan through them came up with several that indicated a lack of disk space.

A quick df command showed that the mount point /dev/mapper/SangomaVG-root, on the root (/) was at 100%.

Back to the Web interface – Admins – System Admin – storage again showed root ‘/’ drive usage at 100%

After a lot of googling and some very helpful sites – this is how I created more space.

1) Increase size of the Virtual Hard Drive

Our FreePBX is a virtual machine on MS Hyper-V, so shutdown the machine and increased the size of the virtual hard drive for the virtual machine – added another 40 GB (Easy enough !)

2) Increase size of the partition

Using the ubiquitous and fantastic GParted partitioning tool, downloaded the GParted ISO, mounted this as VM media and booted into GParted (With a physical machine, create a bootable USB and boot into GParted).

Now it’s easy enough to select the partition with the SangomaVG mount point and drag the handle to extend the partition to fill the new space in the virtual drive.

Sooo after a reboot, hooray, I thought, job done !

BUT in the console, df to view the Filesystem – shows the Sangoma mount point as still at 100% and still at the old 50G size (should be 100G !). Now need to extend the File System to fill the partition.

3) Resizing the File System

Attempt 1 : Tried the usual way for a linux file system, resize2fs /dev/mapper/SangomaVG-root ( which is the file system name as shown by the df command), to expand the file system to fill the partition.

This FAILED with a message ‘Bad magic number in super-block

Attempt 2: command df –T shows that the file system is of type xfs

Instead of resize2fs, this would require the xfs_growfs command:
xfs_growfs /dev/mapper/SangomaVG-root -d to expand to fill the partition.

Although the command looked to run OK, this FAILED with message ‘data size unchanged’

Attempt 3: Using the linux partition editor - parted , then print commands, the partition shows flagged as lvm (logical volume, a bit like dynamic partitions, which apparently cannot be grown with xfs_growfs !

So more research:

Command vgs – shows the free space available for your file system, I had 42.6G spare (which would fill the partition)

Can now extend the xfs file system on the LVM flagged partition, with:

Lvextend –L +42.6G /dev/mapper/SangomaVG-root –r

df now shows spare space on the file system, usage at 54%

SUCCESS

Rebooted FreePBX and hey presto, daemons started and no Asterisk error

PHONES WORKING AGAIN !


(Wzkds) #2

Thanks for the write up, I’ll need this in the future! I’m on OpenVZ, it adjusts the container’s guestOS automatically, but down the road will be switching to either HyperV or ESXi. How has it performed on HyperV?

What’s taking up so much drive space? Do you have logging and debugging set to high, or call recordings?

Most of my Guests have 20-30GB and I rarely run into issues by adjusting log rotation settings. I offload historical data to an external SYSLOG server for long term retention on cheaper drives and set retention to save 2 day’s worth of logs on the server and compress.

vi /etc/logrotate.d/asterisk

rotate 2
compress

If there’s call recordings, I set a batch script to delete old call recordings:

       #!/bin/bash
       #This script will delete call recordings older than 25 days
       #this didn't work: find /var/spool/asterisk/monitor/ -name “*.wav” -mtime 25
       # Find all recordings older than 25 days and delete
       find /var/spool/asterisk/monitor -name "*.wav" -mtime +25 -delete
       exit

And run it every night with crond:

vi /etc/crontab

0 0 * * * root bash /root/25dayrec.sh

I’ve also converted wav files to mp3 to save on space, but drive space is easier to come by than CPU, I’ve run into issues with the conversion in my VM environment doing this, so I’d advise against it.


(Crimsonbadger) #3

Thanks for the feedback. We tend to set fixed disk sizes on our Hyper-V VMs. We’ve had issues with dynamic in the past on general servers and I have read a couple of things about probs with FreePBX and dynamic expanding disks, but that might just be hear-say !
Huge amount of space is being used by asterisk logs. As this is a new install I guess the full debug is on by default. I like your idea of adjusting rotation to a couple of days and storing the logs elsewhere.

Cheers for the tip


(Wzkds) #4

Thanks for sharing your experiences with configuring Freepbx VMs in Hyper-V, this will come in handy!