Raid storage issues - storing to wrong partition?

I have a system set up with Raid that seems to be storing all our info to the wrong partition

image
image

cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sdb1[1] sda1[0]
      467680256 blocks super 1.2 [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

md127 : active raid1 sdb2[1] sda2[0]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

You mean / instead of /home ?
By default, FreePBX distro creates just 2 partitions, one is /boot and the other is an LVM for /
Have you selected a custom partition at install time?

I wasn’t involved in the creation of this system but I suppose that is an option. Is there a way to fix this with a system that is currently in use?

in descriptive terms, you can do the following:

  1. Create a new directory on the / partition called “home2”.
  2. Look in the /etc/fstab file and find the actual partition make of the /home directory.
  3. umount /home
  4. mount /dev/whatever_its_called /home2
  5. cd /home2
  6. find . | cpio -pdmuv /home

Step 6 copies all of your files from the old /home partition to the /home directory on the root drive.

  1. cd /home2

Yes, I know. Just being safe

  1. rm -fr *
  2. cd /var
  3. find . | cpio -pdmuv /home2

This will copy everything on the /var directory to your old home partition.

  1. vi /etc/fstab

Change the /home mount point to /var

  1. reboot -s

Reboot into single user mode.

  1. mv /var /var2 ; mkdir /var
  2. mount -u -a

This should mount your drives back onto their (new) mount points. All of the files in your old /var directory are now in /var2

  1. cd /var2
  2. find . | cpio -pdmv
  3. cd /
  4. rm -fr /var2

Reboot and hope I didn’t miss anything.

I moved the /var directory contents twice. Anything that was updated since the boot won’t be moved, so if you’re comfortable that the old /var2 directory is no longer needed, you can delete it and clear some space on your root directory.

1 Like

wow, that is impressive. Thank you so much. I’ll need to investigate how to do some of the steps but I really appreciate your help

The single-user reboot is there so you don’t auto mount all of the partitions. You want to make sure all of the files in /var are on the new partition when you remount the drives so that the logs work. Also, when mass deleting the /var directory, make sure you are deleting the old one (’/var2’)and not the actual ‘/var’ directory. That would smart.

a 2gb /var partition might be a problem unless you carefully massage your log rotation.

But initially you said

“I have a system set up with Raid that seems to be storing all our info to the wrong partition”

can you be more explicit?. Your /home would likely just have a small asterisk directory so not a huge problem. Anything about LVM in your case is a red-herring, dont worry about it.

your / partition is indeed quite full, lets see the issue of

du -hx --max-depth=1 /|sort -h
1 Like
du -hx --max-depth=1 /|sort -h
0       /media
0       /mnt
0       /opt
0       /srv
0       /uploads
108K    /root
564K    /tmp
67M     /etc
383M    /tftpboot
2.4G    /usr
31G     /var
34G     /

Can you post the output of df -h
Let’s see which Filesystems have been created.

Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/s7_freepbx-root   50G   34G   17G  67% /
devtmpfs                     3.8G     0  3.8G   0% /dev
tmpfs                        3.9G  440K  3.9G   1% /dev/shm
tmpfs                        3.9G  393M  3.5G  11% /run
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/s7_freepbx-home  388G  1.9G  387G   1% /home
/dev/md127                  1020M  131M  889M  13% /boot
tmpfs                        780M     0  780M   0% /run/user/0
tmpfs                        780M     0  780M   0% /run/user/995

Now

du -hx --max-depth=1 /var|sort -h

Rinse and repeat

0       /var/account
0       /var/adm
0       /var/crash
0       /var/empty
0       /var/ftp
0       /var/games
0       /var/gopher
0       /var/kerberos
0       /var/local
0       /var/nis
0       /var/opt
0       /var/preserve
0       /var/yp
4.0K    /var/tmp
8.0K    /var/db
118M    /var/cache
937M    /var/log
1.2G    /var/www
2.0G    /var/lib
27G     /var/spool
31G     /var

. . ./var/spool

My guess, either backups or recordings

Sorry if this has been already answered, but this doesn’t seem to be a FreePBX distro in terms of partitioning. Did you select a custom partition scheme when installing?

You are correct - backups and recordings both seem to be saving to this smaller sized partition

If my memory serves me correctly this was a raid install using the distro for sure but… I was using a supermicro server which would not properly recognize the raid. I worked with a sangoma tech from Australia maybe… who helped with using the custom partitioning scheme to make it work. There is certainly a chance I just didn’t follow his directions very well.

I would definately follw @cynjut s recipe

It is not a problem per-se. You can fix this with some Linux commands, some reboots and ideally if you have a spare disk you can use. The idea behind this is to make two partitions, one for /boot and another for LVM, where two LVM partitions will reside, root and swap.
How comfortable are you with editing partitions, moving Filesystems and so on?

Sadly I wouldn’t call it a strength. I’m wondering if this might be a good time for some paid sangoma support. Not sure if they take on tasks like this. With this being an active server for a large customer I’m wary. @cynjut left amazing instructions but one misstep and I would not really know how to rescue it.

Well in this case, not sure if remote support can help you because you need to unmount active partitions and for that your server must be booted from a live cd, and if you are going with the route of adding a spare disk for the temporary move, then you need to put that disk physically inside the server. But you can always try asking for their help.