Thursday, June 18, 2015

Converting RAID1 array to RAID5 using the mdadm grow command

I have finally decided to upgrade the storage in the home theatre PC, by adding a third 3TB hard drive. The storage was set up previously as RAID 1, using the software mdadm solution for the two 3TB disks. By adding a third drive and changing to a raid 5 format, the storage would increase from 3TB (about 2.7TiB) to 6TB (about 5.4TiB).

There are a few ways to do this:
1. Copy the data to a spare drive, delete the RAID 1 array, create the RAID 5 array with the three disks, copy the data back to it.

2. Back up the data to a spare drive, remove one disk from the RAID 1 array, use that disk and the new disk to make a 2 disk RAID 5 array, copy data over, remove the RAID 1 array and add that disk to the RAID 5 array so it is 3 disks.

3. Back up the data to a spare drive, Use the mdadm --grow command to change the RAID level from RAID 1 to RAID 5, add the third drive to the RAID 5 array and let it rebuild.

Initially I was going to try option 2, such as described here. But noticing one of the comments (by the wonderfully named Svenne Krap) describing that you can just change the level of the array using the mdadm grow command, I thought it would warrant further investigation. I couldn't find many other mentions of it elsewhere on the internet, so I thought I'd document what I'd done so it might help someone else.

So to try it out, I set up a virtual machine on another PC, and created three separate drives. I used mdadm to create a RAID 1 array with two of them, and then converted that array to RAID 5. It worked! I then added the third drive, and after a little while it rebuilt into a full three disk array.

So the time came to do it for real. I first wanted to back up the array - and found an old 2TB drive lying around. Fortunately, the array was only just over two thirds full, so it all managed to squeeze on to the 2TB drive after I deleted a few old TV shows. That's still nearly 600 TV shows and movies left - I have no idea when I'll get around to watching them all, but it's nice to have.

So the data was safe. I could do this switchover with a little less stress.

One thing I noticed with the options was chunk size. Different sizes give different performance. Since this box would be primarily a media box, reading and writing large video files, I went for a larger chunk size of 512KB. When creating the array in the virtual machine initially, it created only 4KB chunks as the size of the array was not a multiple of 512KB. So first up, I resized the RAID 1 array so it was. Basically I divided the size reported by the cat /proc/mdstat command, by 512. It gave a number ending in .625, meaning there weren't an exact number of 512KB chunks. By multiplying that answer (minus .625) by 512, it gave a size that could work.

So I resized the array with the following command:

sudo mdadm -G /dev/md0 --size=2930266112

I then let mdadm know about the new drive, by adding it. The new drive is /dev/sde:

sudo mdadm /dev/md0 --add /dev/sde

Now there was a third drive added as a spare to the array, but not actually in use by it yet.

Next, the big one, change the array from RAID 1 to RAID 5 (still only 2 drives):

sudo mdadm /dev/md0 --grow --level=5 --chunk=512

The array is now a RAID 5 array, with two disks. The chunk size of 512KB was also set with that command. Time to let it use the third drive to create the full, three disk RAID 5 array:

sudo mdadm --grow /dev/md0 --raid-devices 3

This kicks off the rebuilding process, which takes many hours. You can track the progress with the cat /proc/mdstat command. It ran pretty slowly on my system, at around 30,000KB/sec, until it got to the empty part of the array, when speed nearly tripled.

Later that day, the process was finished. Running sudo mdadm --detail /dev/md0 gave the new array size of about 5.4TiB - something I was a little concerned about while the rebuild was underway, because it was only showing the old size of less than 3TB. I thought I might have to resize the array afterwards, but it all came out good.

Resizing LVM

Because I was running LVM on top of the raid array, the size of the LVM volume was still unchanged - I had to increase that to let it use the extra space that was now available. Going by the size reported by the mdadm --detail command run previoulsy, there were approximately 5588GB in the array. LVM works with three 'layers' - the physical volume, basically the hard drive or RAID array; volume groups, which consist of a number of logical volumes. See here for more information. The first step is to resize the physical volume to match the size of the array:

sudo pvresize --setphysicalvolumesize 5588G /dev/md0

Next, I could extend the logical volumes to use up more of that space. I have two main logical volumes on this volume group: one called 'tv' that holds TV shows, movies, and music from MythTV, and another called 'sysbackup' that holds backups of data. I wanted to enlarge both of these - the 'tv' one, and also 'sysbackup' because I wanted to use it with Crashplan as a backup destination for some other PCs in the house.

I wanted to increase the 'sysbackup' volume to 1.2TB, so I used the following command:

sudo lvextend -L1.2T /dev/raid1/sysbackup

A couple of things to note here: the '-L1.2T' instructs it to make the total size of the volume 1.2 Terabytes. Also, the name of the volume group is still /dev/raid1 - I haven't changed the name of that, even though it is now RAID 5. It can be done with the vgrename command, but it would also mean changing mount points in the /etc/fstab file.

Next was the resizing of the 'tv' volume. I wanted to increase it by 1.6TB, so I used the following command:

sudo lvextend -L+1.6T /dev/raid1/tv

Notice the '-L+1.6T' - the plus sign commands it to expand the volume by the specified amount. Unfortunately, it reported that there wasn't 1.6TB spare, and gave a number of available extents that were free. So I tried a different approach, specifying size in extents rather than TB:

sudo lvextend -l+418800 /dev/raid1/tv

The lower-case 'l' instructs it to use extents rather than GB, TB or whatever. This worked, and I now had two newly-resized logical volumes.

Resizing the file systems

But there was one final step - the logical volumes were now bigger, but the file systems did not know about it yet. Fortunately it isn't too difficult.

The sysbackup volume was formatted with the ext4 file system. To resize that, it first had to be unmounted:

sudo umount /mnt/sysbackup

Then a check of the file system was done:

sudo e2fsck -f /dev/raid1/sysbackup

Then finally, the file system was resized to fill the available space:

sudo resize2fs /dev/raid1/sysbackup

Finally, the file system can be re-mounted. Since it is specified in the /etc/fstab file, running

sudo mount -a

did the trick. Next was the 'tv' volume. This volume was formatted with the XFS file system, and it can be resized while still mounted. The command for that was:

sudo xfs_growfs /dev/raid1/tv

Conclusion

The advantage of using the mdadm --grow command is that it is still the same array according to mdadm, just a different level. Running

sudo mdadm --detail --scan

shows that the array still has the same identifiers as when it was a RAID 1 array, so no further work is required. I ran

sudo update-initramfs -u

just in case, so it would pick up any changes in the array (and not give it a name like /dev/md127 after a reboot). I restarted the PC, and everything came back up, with the roomy new array showing up as it should. Done!

7 comments:

  1. Thanks a lot for this. It was such a time saver.
    I initially thought there was no choice but recreating the array and re-copying all the data.

    With your steps I was able to convert my /home mountpoint from raid1(2 disks) to raid5(3 disks), and go from 2TB to 4TB, all with a single system reboot to pop in the new HDD (which could have been avoided if I had a hot swappable bay).

    The whole process was very seamless. I did not even have to unmount /home to do this. Gotta love Linux.

    ReplyDelete
  2. First, thanks for the awesome instructs they (mostly) worked perfectly!

    I'm mainly commenting for anyone who might stumble onto this great tutorial and who's looking to go from RAID0 to RAID5.

    I say it mostly worked because I confused RAID1 and RAID0, so when I tried to follow your instructions, it wouldn't let me add the 3rd device because I was using RAID0 (mdadm --add just threw an error).

    It turns out that to go from RAID0 to RAID5 using mdadm you have to convert the array to RAID5 first, then add the device, and then finally grow the array.

    The downside to this is that you have to rebuild the array twice: once when you convert it to a 2 disk RAID5 (not sure why it allows that?) and again when you grow the array to include the 3rd disk.

    Same end result and same commands, really. Just takes double the time due to needing 2 reconstructions.

    ReplyDelete
  3. Following up on my above comment, to add that you can avoid the seemingly notorious /dev/md127 issue by updating your mdadm.conf file under /etc/mdadm/ using the following command:

    mdadm --detail --scan >> /etc/mdadm/mdadm.conf

    The mdadm.conf file appears to permanently "reserve" a specific /dev/mdX device for a specific array based on its UUID. That way you can still just use the /dev/mdX notation in /etc/fstab

    mdadm.conf also includes details about the array's level and I'm not sure what consequences, if any, there are for not updating this file after modifying or expanding an array.

    ReplyDelete
  4. Hi, I tried following this tutorial, and after following the first part of the tutorial, my raid array is still at the same size as it was in a raid 1 config.

    I didn't set the size of the array since I wasn't changing the block size.
    Commands I ran:
    sudo mdadm /dev/md0 --add /dev/sdd
    sudo mdadm /dev/md0 --grow --level=5
    sudo mdadm --grow /dev/md0 --raid-devices 3

    I am using 3x4tb drives, and I'm only able to use 4tb of space even though the --detail shows that the array size is 8tb

    running mdadm --grow /dev/md0 --size=max doesn't change it above 4tb.
    doing the same thing but replacing "max" with the array size from --detail says there's no space left on the device and resize2fs tells me that it's too large for 32 bits
    Any suggestions?

    ReplyDelete
  5. Hi, unsure of why it didn't show the increased size. When I did it, the new size didn't appear until it had finished rebuilding. Has that finished yet?

    ReplyDelete
  6. I've seen a bug report on redhat for --grow with -size and raid bigger than 2TB (https://bugzilla.redhat.com/show_bug.cgi?id=1164494)
    I don't know if it is the case, but I also had problems reducing size in order to change chunk...

    Since I had full backups, I've forced it a bit...

    # mdadm /dev/md127 --grow --level=5 --chunk=512
    mdadm: New chunk size (512K) does not evenly divide device size (3906887488k)
    mdadm: After shrinking any filesystem, "mdadm --grow /dev/md127 --size 3906887168"
    mdadm: will shrink the array so the given chunk size would work.
    unfreeze

    # resize2fs /dev/md127 3906887168k
    resize2fs 1.43.3 (04-Sep-2016)
    Resizing the filesystem on /dev/md127 to 976721792 (4k) blocks.
    The filesystem on /dev/md127 is now 976721792 (4k) blocks long.

    # mdadm -G /dev/md127 --size 3906887168
    mdadm: component size of /dev/md127 unchanged at 3906887488K

    # cat /sys/block/md127/md/component_size
    3906887488

    # echo "3906887168" > /sys/block/md127/md/component_size

    # cat /sys/block/md127/md/component_size
    3906887168

    # mdadm /dev/md127 --grow --level=5 --chunk=512
    mdadm: level of /dev/md127 changed to raid5
    chunk size for /dev/md127 set to 65536
    unfreeze

    #

    ReplyDelete
  7. Awesome thanks. That worked lovely

    ReplyDelete