Disk usage help needed

FreePBX System Administration
1

I’ve looked over a couple of threads to do with disk usage but drawn a blank on what to do here as they don’t address what I’M seeing, but they gave me a clue for what to present here.

{Cleaning up space on a FreePBX system}
{Disk usage }

From @jmpg I found this command and output

[root@FreePBX /]# sudo du -h -d4 /boot /etc /home /opt /root /tmp /usr /var | sort -rh | head -n30
9.7G /var
5.6G /var/www/html/admin
5.6G /var/www/html
5.6G /var/www
5.5G /var/www/html/admin/modules
2.5G /var/lib
2.3G /usr
1.4G /var/log
1.2G /var/log/asterisk
922M /usr/lib
909M /var/lib/mysql
743M /home/asterisk
743M /home
542M /var/lib/php/session
542M /var/lib/php
514M /usr/lib64
480M /usr/lib/firmware
469M /var/lib/asterisk
449M /var/lib/mongodb
446M /usr/share
403M /var/lib/asterisk/sounds
397M /home/asterisk/.package_cache/npm/6.4.1
397M /home/asterisk/.package_cache/npm
397M /home/asterisk/.package_cache
384M /var/lib/mongodb/journal
300M /home/asterisk/.npm/_cacache
300M /home/asterisk/.npm
281M /home/asterisk/.npm/_cacache/content-v2
239M /usr/bin
238M /var/cache
[root@FreePBX /]#

That helps to explain a little why I’m seeing this chart in the dashboard

image
image526×499 21.5 KB

But I really can’t determine what I can remove from any folders, especially those with multiple G of contents. The only commercial module we use is Advanced Recovery

When I look in html/admin I don’t see anything of any particularly large size:
[root@FreePBX admin]# ls -lah
total 180K
drwxrwxr-x. 10 asterisk asterisk 4.0K Aug 19 2023 .
drwxrwxr-x. 7 asterisk asterisk 169 Dec 2 2022 ..
-rw-rw-r-- 1 asterisk asterisk 1.7K Apr 26 13:59 ajax.php
drwxrwxr-x 2 asterisk asterisk 38 Dec 2 2022 api
drwxrwxr-x. 7 asterisk asterisk 4.0K Sep 27 2024 assets
-rw-rw-r-- 1 asterisk asterisk 15K Apr 26 13:59 bootstrap.php
-rw-rw-r-- 1 asterisk asterisk 32K Apr 26 13:59 config.php
lrwxrwxrwx 1 asterisk asterisk 36 Aug 19 2023 cxpanel ->> var/www/html/admin/modules/cxpanel/
-rw-rw-r-- 1 asterisk asterisk 20K Apr 26 13:59 functions.inc.php
drwxrwxr-x 2 asterisk asterisk 113 Dec 2 2022 helpers
-rw-rw-r-- 1 asterisk asterisk 1.6K Apr 26 13:59 .htaccess
drwxrwxr-x 23 asterisk asterisk 303 Sep 7 2024 i18n
drwxrwxr-x 2 asterisk asterisk 4.0K Dec 2 2022 images
-rw-rw-r-- 1 asterisk asterisk 185 Apr 26 13:59 index.php
drwxrwxr-x. 10 asterisk asterisk 4.0K Dec 2 2022 libraries
-rw-rw-r-- 1 asterisk asterisk 398 Apr 26 13:59 module-builtin.xml
drwxrwxr-x. 121 asterisk asterisk 4.0K May 3 10:56 modules
-rw-rw-r-- 1 asterisk asterisk 56K Apr 26 13:59 page.modules.php
-rw-rw-r-- 1 asterisk asterisk 21 Apr 26 13:59 .user.ini
drwxrwxr-x. 4 asterisk asterisk 4.0K Jan 7 2023 views

Similar case with the html folder

[root@FreePBX admin]# cd /var/www/html
[root@FreePBX html]# ls -lah
total 20K
drwxrwxr-x. 7 asterisk asterisk 169 Dec 2 2022 .
drwxr-xr-x. 4 root root 33 Feb 21 2022 ..
drwxrwxr-x. 10 asterisk asterisk 4.0K Aug 19 2023 admin
drwxrwxr-x 2 asterisk asterisk 6 Feb 22 2022 .freepbx-known
-rw-rw-r-- 1 asterisk asterisk 16 Apr 26 13:59 .htaccess
-rw-rw-r-- 1 asterisk asterisk 453 Apr 26 13:59 index.php
drwxrwxr-x 2 asterisk asterisk 6 Sep 13 2022 restapi
drwxrwxr-x. 2 asterisk asterisk 4.0K Apr 26 14:00 restapps
-rw-rw-r-- 1 asterisk asterisk 361 Apr 26 13:59 robots.txt
lrwxrwxrwx 1 asterisk asterisk 38 Dec 2 2022 ucp → /var/www/html/admin/modules/ucp/htdocs
lrwxrwxrwx 1 asterisk asterisk 47 Dec 2 2022 wcb.php → /var/www/html/admin/modules/webcallback/wcb.php
drwxrwxr-x 3 asterisk asterisk 28 Feb 22 2022 .well-known

Does someone have an idea of what I should be doing here?

Thank you
Nathan.

2

The only thing strange in your disk storage usage is regarding path /var/www/html/.
I would dig deeper into that path to see were is the storage is going, using something like this command sudo du -h -d3 /var/www/html | sort -rh | head -n30

3

It looks like the output says the bulk of the usage is in _cache.

[root@FreePBX html]# sudo du -h -d3 /var/www/html | sort -rh | head -n30
5.6G /var/www/html/admin
5.6G /var/www/html
5.5G /var/www/html/admin/modules
4.0G /var/www/html/admin/modules/_cache
410M /var/www/html/admin/modules/zulu
359M /var/www/html/admin/modules/iotserver
98M /var/www/html/admin/modules/endpoint
91M /var/www/html/admin/modules/ucp
79M /var/www/html/admin/modules/sangomaconnect
78M /var/www/html/admin/modules/xmpp
73M /var/www/html/admin/modules/sangomartapi
64M /var/www/html/admin/libraries
62M /var/www/html/admin/libraries/Composer
40M /var/www/html/admin/modules/pm2
31M /var/www/html/admin/modules/vqplus
31M /var/www/html/admin/modules/faxpro
31M /var/www/html/admin/modules/cloudmigration
30M /var/www/html/admin/modules/api
26M /var/www/html/admin/modules/filestore
24M /var/www/html/admin/modules/contactmanager
14M /var/www/html/admin/modules/pms
13M /var/www/html/admin/assets
12M /var/www/html/admin/i18n
7.6M /var/www/html/admin/modules/recording_report
7.3M /var/www/html/admin/modules/calendar
5.9M /var/www/html/admin/modules/userman
5.6M /var/www/html/admin/assets/js
5.3M /var/www/html/admin/modules/firewall
4.2M /var/www/html/admin/modules/webrtc
3.8M /var/www/html/admin/modules/sysadmin
[root@FreePBX html]#

So I had a look in _cache and there are a whole lot of old files, far to much output to paste here. How do I clean this folder up?

[root@FreePBX _cache]# ls -lah
total 4.0G
drwxrwxr-x. 2 asterisk asterisk 28K May 3 10:56 .
drwxrwxr-x. 121 asterisk asterisk 4.0K May 3 10:56 ..
-rw-rw-r-- 1 asterisk asterisk 12K Dec 2 2022 accountcodepreserve-16.0.0.1.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 711K Feb 21 2022 adv_recovery-15.0.47.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 711K Mar 12 2022 adv_recovery-15.0.48.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 710K Mar 26 2022 adv_recovery-15.0.49.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 712K Jun 18 2022 adv_recovery-15.0.51.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 716K Aug 27 2022 adv_recovery-15.0.52.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 720K Nov 19 2022 adv_recovery-15.0.53.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 965K Dec 2 2022 adv_recovery-16.0.41.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 971K May 20 2023 adv_recovery-16.0.43.6.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 971K Jun 10 2023 adv_recovery-16.0.43.8.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 968K Sep 7 2024 adv_recovery-16.0.43.9.tgz.gpg
-rw-rw-r-- 1 asterisk asterisk 42K Dec 5 2022 allowlist-16.0.2.tgz.gpg

4

it looks like I can safely run
rm -rf /var/www/html/admin/modules/_cache/*

5

I can only address this in a general Posix way, as I’m an Asterisk, not a FreePBX person, but it seems clear that the biggest usage of those shown by your du is in /var/www/html/admin/modules, and below, but you haven’t run a directory listing on that. Of the two you have done them for, /var/lib/www/html/admin only adds about 100MB (with a large rounding error, probably -100MB +0MB), and the amount added by /var/www/html is lost in the rounding.

On further reading, you have run one, but not presented the details.

I’t likely that anything called _cache can be safely cleared, but I want to know for certain, before doing doing that, e.g. there might be an index database that needs to be rebuilt.

6

Yep, you can safely delete that _ cache folder contents… this seems to be that is a bug on FreePBX, as this folder should be cleaned from time to time…

1 Like
7

Thanks for the responses.
I’m going to leave the server alone for the weekend, and I’ll clear the folder out first thing Monday when I’m in the office, and will report back here to complete this thread.

1 Like
8

Is this a VM ? Are you able / willing to expand the drive ? I had the same problem on a system a few years ago where we set it up as a 30 Gig drive. But we also set up for recording calls and that ate up space real quick,. I had to learn how to expand the VM drive and then the linux drive on FreePBX. I might be able to find those instructions if you need ?

9

Hello, yes, VM. I tried the drive expansion on the secondary VM (the Advanced Recovery partner). I used it as a test to see how the process would work. This install doesn’t have the storage problem, but I’m not entirely sure the expansion went well. I added 5GB but the server only ended up seeing around 1GB. Not sure what happened there, so I rolled back. And I’m not sure that went well either because all the trunks ended up disabled.
The VM drive was someone else’s responsibility (I have no access to the virtual backend).
Once I was told I had the extra capacity allocated to the server:

pvresize /dev/sda2
lvextend -l +100%FREE /dev/mapper/SangomaVG-root
xfs_growfs /dev/mapper/SangomaVG-root

The commands appeared to work but the disk went from 15.23 GB to a little over 16GB.
Entirely possible I missed or mis-interpreted something, so I asked for a rollback.

10

It was XFS not EXT4 ?

11

‘Modern’ FreePBX installations - I took that to mean it would be XFS on a 16.x system.
But now you questioned it I just checked for a command that would tell me and the output was this:

/dev/mapper/SangomaVG-root xfs 15968256 9660280 6307976 61%

12

Got it - 16 (and 14 and 15 and …) use XFS, agreed – should not have assumed 17 ISO, which (mostly) uses EXT4.

Anyhow, before the pvresize did you run fdisk ?

Here’s some related documentation on the Sangoma KB wiki site: https://sangomakb.atlassian.net/wiki/x/JICi

13

This and runaway logs when modules break or licenses expire are usually the cause of storage issues.

14

This worked for version 16 of FreePBX, has not been tested on ver 17 yet.

How to EXPAND a Drive in Linux - FreePBX

This is similar to Windows and using Partition Wizard, but some extra steps.

FreePBX is installed as a virtual machine. It is a Linux based installation.

Linux Structure is :
“Volume Group” (which could span more than 1 drive.)
— “Logical Volume”
------- “FileSystem”

  1. Shut down FreePBX

  2. Find the VM Drive in the NAS POOL - Expand the drive to the desired size. (This is the equivalence of re-imaging and going from maybe a 30 Gig Drive to now having a 100Gig Drive)

  3. Expand the Drive Partition to access the entire space

— Note : This may be able to be done with just the VG commands from Linux, to expand the volume group by size. But I could not figure it out. This was the easiest way I got it done so far was with gParted.

Download the GParted Live-CD - http://gparted.sourceforge.net
Go to the VM and add Gparted Live CD as a CD ROM drive and set it to boot first.
Boot the VM to Gparted Live
Find the proper partition and expand it
Apply the change and shut down
Remove the CD ROM drive from the VM

  1. Boot the VM and log into the “root”

  2. Identify the Volume Group - type : vgs
    This should NOW show you have the extra drive space in the VM
    VG - VSize - VFree (The VFree space should be the difference now)

  3. Identify the Logical Volume- type : lvs
    – On FreePBX 16 it looks like /dev/SangomaVG/root

  4. Extend the Logical Volume- type : lvextend -l +100%FREE /dev/SangomaVG/root

  5. Extend the filesystem- type : xfs_growfs /dev/SangomaVG/root

  6. Verify the size of your extended partition.- type : df -h
    You will see the “available” space, and the “used” space.

  7. Done, Exit the root.

Reference Links :

How to resize a logical volume with 5 simple LVM commands

15

Thanks José, that command did work as simply as I’d hoped. I’m no Linux admin, so everything is a leap in the dark and makes me nervous. Fortunately I have the standby partner that I can treat as my ‘test’ box, which I’ve been making good use of lately. Shouldn’t do really but so long as the primary is up we’re good, and I’ve never had a service problem with that yet.

This is me administering Linux!
Michael A. Arouet on X: “Ok, now I have serious doubts about our future pensions https://t.co/or04Ro3hEL” / X

16

To wrap this thread up, thanks for help, it was needed.

What I’d failed to grasp having been led astray by our own storage admins (not their fault, just not their specific area) was that I needed to restart the FPBX VM. So when I first added the additional 5GB to the secondary FPBX VM it actually only ended up being 800MB. This was because there was already 800MB on unallocated space on the original disk, so the root partition took 100% of what was free, and that was 800MB. The server couldn’t see the 5GB that the storage admins had given it. So, issue ‘reboot’ on the FPBX CLI and go again. This was only after spending ages trying to figure out why 5GB had turned into 800MB. Led astray.

In the end the actual sequence looked like this:

lsblk to review before change to storage is allocated in VCentre

fdisk /dev/sda - enter fdisk partition manager

d - delete partition

2 - select part 2 to delete

n - new partition

p - primary type partition

2 - partition number 2

Enter, Enter - default to use all available space

t - type of partition

2 - select which partition to set type

8e - set type to LinuxLVM

p - verify partition info

w - write new config and exit (ignore error 16)

partprobe /dev/sda - re-read partition table

pvresize /dev/sda2 - update the partition

lvextend -l +100%FREE -r /dev/mapper/SangomaVG-root - extend the partition

Lsblk and df -h to review.

I’ve highlighted partprobe because without running that I was back to the 800MB problem even though I’d done a restart and lsblk showed there was an extra 5GB now available.

Thanks all.
Nathan.

1 Like
17

You’re welcome!
You’re post was also helpful to me, because it lead me to check in my FreePBX boxes that folder and encountered a lot of storage space occupied with old cache files.
As I told before, I think this should be cleaned as part of any FreePBX process/script. As it seems it does not exist, I added a cron job to run every day at 3 AM, which deletes any files on that folder that are older than 3 months:

0 3 * * * find /var/www/html/admin/modules/_cache -type f -mtime +90 -delete

2 Likes