I have finally decided to upgrade the storage in the home theatre PC, by adding a third 3TB hard drive. The storage was set up previously as RAID 1, using the software mdadm solution for the two 3TB disks. By adding a third drive and changing to a raid 5 format, the storage would increase from 3TB (about 2.7TiB) to 6TB (about 5.4TiB).
There are a few ways to do this:
1. Copy the data to a spare drive, delete the RAID 1 array, create the RAID 5 array with the three disks, copy the data back to it.
2. Back up the data to a spare drive, remove one disk from the RAID 1 array, use that disk and the new disk to make a 2 disk RAID 5 array, copy data over, remove the RAID 1 array and add that disk to the RAID 5 array so it is 3 disks.
3. Back up the data to a spare drive, Use the mdadm --grow command to change the RAID level from RAID 1 to RAID 5, add the third drive to the RAID 5 array and let it rebuild.
Initially I was going to try option 2, such as described here. But noticing one of the comments (by the wonderfully named Svenne Krap) describing that you can just change the level of the array using the mdadm grow command, I thought it would warrant further investigation. I couldn't find many other mentions of it elsewhere on the internet, so I thought I'd document what I'd done so it might help someone else.
So to try it out, I set up a virtual machine on another PC, and created three separate drives. I used mdadm to create a RAID 1 array with two of them, and then converted that array to RAID 5. It worked! I then added the third drive, and after a little while it rebuilt into a full three disk array.
So the time came to do it for real. I first wanted to back up the array - and found an old 2TB drive lying around. Fortunately, the array was only just over two thirds full, so it all managed to squeeze on to the 2TB drive after I deleted a few old TV shows. That's still nearly 600 TV shows and movies left - I have no idea when I'll get around to watching them all, but it's nice to have.
So the data was safe. I could do this switchover with a little less stress.
One thing I noticed with the options was chunk size. Different sizes give different performance. Since this box would be primarily a media box, reading and writing large video files, I went for a larger chunk size of 512KB. When creating the array in the virtual machine initially, it created only 4KB chunks as the size of the array was not a multiple of 512KB. So first up, I resized the RAID 1 array so it was. Basically I divided the size reported by the cat /proc/mdstat command, by 512. It gave a number ending in .625, meaning there weren't an exact number of 512KB chunks. By multiplying that answer (minus .625) by 512, it gave a size that could work.
So I resized the array with the following command:
sudo mdadm -G /dev/md0 --size=2930266112
I then let mdadm know about the new drive, by adding it. The new drive is /dev/sde:
sudo mdadm /dev/md0 --add /dev/sde
Now there was a third drive added as a spare to the array, but not actually in use by it yet.
Next, the big one, change the array from RAID 1 to RAID 5 (still only 2 drives):
sudo mdadm /dev/md0 --grow --level=5 --chunk=512
The array is now a RAID 5 array, with two disks. The chunk size of 512KB was also set with that command. Time to let it use the third drive to create the full, three disk RAID 5 array:
sudo mdadm --grow /dev/md0 --raid-devices 3
This kicks off the rebuilding process, which takes many hours. You can track the progress with the cat /proc/mdstat command. It ran pretty slowly on my system, at around 30,000KB/sec, until it got to the empty part of the array, when speed nearly tripled.
Later that day, the process was finished. Running sudo mdadm --detail /dev/md0 gave the new array size of about 5.4TiB - something I was a little concerned about while the rebuild was underway, because it was only showing the old size of less than 3TB. I thought I might have to resize the array afterwards, but it all came out good.
Resizing LVM
Because I was running LVM on top of the raid array, the size of the LVM volume was still unchanged - I had to increase that to let it use the extra space that was now available. Going by the size reported by the mdadm --detail command run previoulsy, there were approximately 5588GB in the array. LVM works with three 'layers' - the physical volume, basically the hard drive or RAID array; volume groups, which consist of a number of logical volumes. See here for more information. The first step is to resize the physical volume to match the size of the array:
sudo pvresize --setphysicalvolumesize 5588G /dev/md0
Next, I could extend the logical volumes to use up more of that space. I have two main logical volumes on this volume group: one called 'tv' that holds TV shows, movies, and music from MythTV, and another called 'sysbackup' that holds backups of data. I wanted to enlarge both of these - the 'tv' one, and also 'sysbackup' because I wanted to use it with Crashplan as a backup destination for some other PCs in the house.
I wanted to increase the 'sysbackup' volume to 1.2TB, so I used the following command:
sudo lvextend -L1.2T /dev/raid1/sysbackup
A couple of things to note here: the '-L1.2T' instructs it to make the total size of the volume 1.2 Terabytes. Also, the name of the volume group is still /dev/raid1 - I haven't changed the name of that, even though it is now RAID 5. It can be done with the vgrename command, but it would also mean changing mount points in the /etc/fstab file.
Next was the resizing of the 'tv' volume. I wanted to increase it by 1.6TB, so I used the following command:
sudo lvextend -L+1.6T /dev/raid1/tv
Notice the '-L+1.6T' - the plus sign commands it to expand the volume by the specified amount. Unfortunately, it reported that there wasn't 1.6TB spare, and gave a number of available extents that were free. So I tried a different approach, specifying size in extents rather than TB:
sudo lvextend -l+418800 /dev/raid1/tv
The lower-case 'l' instructs it to use extents rather than GB, TB or whatever. This worked, and I now had two newly-resized logical volumes.
Resizing the file systems
But there was one final step - the logical volumes were now bigger, but the file systems did not know about it yet. Fortunately it isn't too difficult.
The sysbackup volume was formatted with the ext4 file system. To resize that, it first had to be unmounted:
sudo umount /mnt/sysbackup
Then a check of the file system was done:
sudo e2fsck -f /dev/raid1/sysbackup
Then finally, the file system was resized to fill the available space:
sudo resize2fs /dev/raid1/sysbackup
Finally, the file system can be re-mounted. Since it is specified in the /etc/fstab file, running
sudo mount -a
did the trick. Next was the 'tv' volume. This volume was formatted with the XFS file system, and it can be resized while still mounted. The command for that was:
sudo xfs_growfs /dev/raid1/tv
Conclusion
The advantage of using the mdadm --grow command is that it is still the same array according to mdadm, just a different level. Running
sudo mdadm --detail --scan
shows that the array still has the same identifiers as when it was a RAID 1 array, so no further work is required. I ran
sudo update-initramfs -u
just in case, so it would pick up any changes in the array (and not give it a name like /dev/md127 after a reboot). I restarted the PC, and everything came back up, with the roomy new array showing up as it should. Done!
Showing posts with label lvm. Show all posts
Showing posts with label lvm. Show all posts
Thursday, June 18, 2015
Monday, November 26, 2012
Adding a larger drive to a software RAID array - mdadm and lvm
The MythTV box I have in the lounge previously had two storage hard drives, in a RAID 1 configuration to prevent data loss in case of a drive failure. The drives were a 3TB Hitachi and a 2TB Samsung. I figured the Samsung drive was getting on a bit now, and it was time a new drive was installed. Might as well make it a 3TB model as well, to take advantage of all the space available on the other one that was sitting unused.
A 3TB Western Digital Red drive was picked up. I chose this as it is designed for use in a NAS environment: always on. It also has low power consumption and a good warranty. I considered a Seagate Barracuda 3TB - they were cheap, performance would be better than the Red, but they are only designed for desktop use, around 8 hours a day powered on. Warranty was pretty short as well.
Removing and replacing the old drive
The drives were configured in a software RAID 1 array, using mdadm, with lvm on top of that. This makes the array portable, and not dependent on a particular hardware controller.
The commands here were adapted from the excellent instructions found here at howtoforge.com.
Fortunately I had enough space on another PC that I was able to back up the contents of the array before starting any of this.
To remove the old drive, which on this machine was /dev/sdc, the following command was issued to mark the drive as failed, in the array /dev/md0:
sudo mdadm --manage /dev/md0 --fail /dev/sdc
Next step is to remove the drive from the array:
sudo mdadm --manage /dev/md0 --remove /dev/sdc
Then, the system could be shut down and the drive removed and replaced with the new one. After powering the system back up, the following command adds the new drive to the array:
sudo mdadm --manage /dev/md0 --add /dev/sdc
The array will then start synchronising the data, copying it to the new drive, which could take a few hours. Note that no partitioning was done on the disk, as I am just using the whole drive in the array.
While the sync is in progress, you can check how it is progressing via:
cat /proc/mdstat
It will show a percentage of completion as well as an estimated time remaining. Once it is done, the array is ready for use! I left the array like this for a day or so, just to make sure everything was working alright.
Expanding the array to fill the space available - the mdadm part
Once the synchronisation has completed, the size of the array was still only 2TB, since that is the largest a RAID 1 array could go when it consists of a 3TB and a 2TB drive. We need to tell mdadm to expand the array to fill the available space. More information on this can be found here.
This is where things got complicated for me. It is to do with the superblock format version used in the array. More detail can be found at this page of the Linux RAID wiki.
To sum up, the array I had was created with the version 0.90 superblock. The version was found by entering
sudo mdadm --detail /dev/md0
The problem, potentially, was that if I grew the array to larger than 2TB it may not work. As quoted by the wiki link above:
Now, Mythbuntu 12.04 runs the 3.2 kernel, so according to that it should be OK supporting up to 4TB. But I wasn't 100% sure on that, and couldn't find any references elsewhere about it. I decided the safest way to go about this was to convert the array to a later version of the superblock, that didn't have that size limitation. Besides, it would save time in the future in case of trying to repeat this with a drive larger than 4TB.
Following the suggestion of the wiki, I decided to update to a version 1.0 superblock, as it would store the superblock information in the same place as the 0.90.
Note: if you are trying this yourself, and the array is already version 1.0 or later, then the command to grow it is just as below (may not want to do it with 0.90 superblock and larger than 2TB):
mdadm --grow /dev/md0 --size=max
Since I was going to change the superblock version, it involved stopping the array and recreating it with the later version.
Once again, to check the details of the array at the moment:
sudo mdadm --detail /dev/md0
Now, since the array is in use by MythTV, I thought it safest to stop the program:
sudo service mythtv-backend stop
Also, I unmounted where the array was mounted:
sudo umount /var/lib/mythtv
Since the data is in an LVM volume on top of the array, I deactivated that as well (the volume is named raid1 in this instance):
sudo lvchange -a n raid1
The array is now ready to be stopped:
sudo mdadm --stop /dev/md0
Now it can be re-created, specifying the metadata (superblock) version, the RAID level, and the number and names of the drives used:
sudo mdadm --create /dev/md0 --metadata=1.0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
The array will now start resynchronising. This took a number of hours for me, as there were around 770GB of recordings there. The RAID wiki link included --assume-clean in the above command, which would have skipped the resync. I elected to leave it out, for safety's sake.
Progress can be monitored with:
cat /proc/mdstat
The lvm volume can be restarted:
sudo lvchange -a y raid1
and the unmounted volumes can be re-mounted:
sudo mount -a
Check if they are all there with the
mount
command. The mythtv service can also be restarted:
sudo service mythtv-backend start
When the array is recreated, the UUID value of the array will be different. You can get the new value with:
sudo mdadm --detail /dev/md0
Edit the /etc/mdadm/mdadm.conf file, and change the UUID value in it to the new value. This will enable the array to be found on next boot.
Another thing to do before rebooting is to run
sudo update-initramfs -u
I didn't do this at first, and after rebooting, the array showed up named /dev/md127 rather than /dev/md0. Running the above command and rebooting again fixed it for me.
Expanding the array to fill the space available - the lvm part
Quite a long-winded process, isn't it? Using the lvm command to show the lvm physical volumes:
sudo pvdisplay
showed the array was still 1.82TiB (2TB). It needed to be extended. The following command will fill the volume to the available space:
sudo pvresize -v /dev/md0
To check the results, again run:
sudo pvdisplay
Now, running:
sudo vgdisplay
gave the following results for me:
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 4.00 MiB
Total PE 715397
Alloc PE / Size 466125 / 1.78 TiB
Free PE / Size 249272 / 973.72 GiB
VG UUID gvfheX-ifvl-yW9h-v4L2-eyzs-95fe-sng2oN
A 3TB Western Digital Red drive was picked up. I chose this as it is designed for use in a NAS environment: always on. It also has low power consumption and a good warranty. I considered a Seagate Barracuda 3TB - they were cheap, performance would be better than the Red, but they are only designed for desktop use, around 8 hours a day powered on. Warranty was pretty short as well.
Removing and replacing the old drive
The drives were configured in a software RAID 1 array, using mdadm, with lvm on top of that. This makes the array portable, and not dependent on a particular hardware controller.
The commands here were adapted from the excellent instructions found here at howtoforge.com.
Fortunately I had enough space on another PC that I was able to back up the contents of the array before starting any of this.
To remove the old drive, which on this machine was /dev/sdc, the following command was issued to mark the drive as failed, in the array /dev/md0:
sudo mdadm --manage /dev/md0 --fail /dev/sdc
Next step is to remove the drive from the array:
sudo mdadm --manage /dev/md0 --remove /dev/sdc
Then, the system could be shut down and the drive removed and replaced with the new one. After powering the system back up, the following command adds the new drive to the array:
sudo mdadm --manage /dev/md0 --add /dev/sdc
The array will then start synchronising the data, copying it to the new drive, which could take a few hours. Note that no partitioning was done on the disk, as I am just using the whole drive in the array.
While the sync is in progress, you can check how it is progressing via:
cat /proc/mdstat
It will show a percentage of completion as well as an estimated time remaining. Once it is done, the array is ready for use! I left the array like this for a day or so, just to make sure everything was working alright.
Expanding the array to fill the space available - the mdadm part
Once the synchronisation has completed, the size of the array was still only 2TB, since that is the largest a RAID 1 array could go when it consists of a 3TB and a 2TB drive. We need to tell mdadm to expand the array to fill the available space. More information on this can be found here.
This is where things got complicated for me. It is to do with the superblock format version used in the array. More detail can be found at this page of the Linux RAID wiki.
To sum up, the array I had was created with the version 0.90 superblock. The version was found by entering
sudo mdadm --detail /dev/md0
The problem, potentially, was that if I grew the array to larger than 2TB it may not work. As quoted by the wiki link above:
The version-0.90 superblock limits the number of component devices within an array to 28, and limits each component device to a maximum size of 2TB on kernel version [earlier than] 3.1 and 4TB on kernel version 3.1 [or later].
Now, Mythbuntu 12.04 runs the 3.2 kernel, so according to that it should be OK supporting up to 4TB. But I wasn't 100% sure on that, and couldn't find any references elsewhere about it. I decided the safest way to go about this was to convert the array to a later version of the superblock, that didn't have that size limitation. Besides, it would save time in the future in case of trying to repeat this with a drive larger than 4TB.
Following the suggestion of the wiki, I decided to update to a version 1.0 superblock, as it would store the superblock information in the same place as the 0.90.
Note: if you are trying this yourself, and the array is already version 1.0 or later, then the command to grow it is just as below (may not want to do it with 0.90 superblock and larger than 2TB):
mdadm --grow /dev/md0 --size=max
Since I was going to change the superblock version, it involved stopping the array and recreating it with the later version.
Once again, to check the details of the array at the moment:
sudo mdadm --detail /dev/md0
Now, since the array is in use by MythTV, I thought it safest to stop the program:
sudo service mythtv-backend stop
Also, I unmounted where the array was mounted:
sudo umount /var/lib/mythtv
Since the data is in an LVM volume on top of the array, I deactivated that as well (the volume is named raid1 in this instance):
sudo lvchange -a n raid1
The array is now ready to be stopped:
sudo mdadm --stop /dev/md0
Now it can be re-created, specifying the metadata (superblock) version, the RAID level, and the number and names of the drives used:
sudo mdadm --create /dev/md0 --metadata=1.0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
The array will now start resynchronising. This took a number of hours for me, as there were around 770GB of recordings there. The RAID wiki link included --assume-clean in the above command, which would have skipped the resync. I elected to leave it out, for safety's sake.
Progress can be monitored with:
cat /proc/mdstat
The lvm volume can be restarted:
sudo lvchange -a y raid1
and the unmounted volumes can be re-mounted:
sudo mount -a
Check if they are all there with the
mount
command. The mythtv service can also be restarted:
sudo service mythtv-backend start
When the array is recreated, the UUID value of the array will be different. You can get the new value with:
sudo mdadm --detail /dev/md0
Edit the /etc/mdadm/mdadm.conf file, and change the UUID value in it to the new value. This will enable the array to be found on next boot.
Another thing to do before rebooting is to run
sudo update-initramfs -u
I didn't do this at first, and after rebooting, the array showed up named /dev/md127 rather than /dev/md0. Running the above command and rebooting again fixed it for me.
Expanding the array to fill the space available - the lvm part
Quite a long-winded process, isn't it? Using the lvm command to show the lvm physical volumes:
sudo pvdisplay
showed the array was still 1.82TiB (2TB). It needed to be extended. The following command will fill the volume to the available space:
sudo pvresize -v /dev/md0
To check the results, again run:
sudo pvdisplay
Now, running:
sudo vgdisplay
gave the following results for me:
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 4.00 MiB
Total PE 715397
Alloc PE / Size 466125 / 1.78 TiB
Free PE / Size 249272 / 973.72 GiB
VG UUID gvfheX-ifvl-yW9h-v4L2-eyzs-95fe-sng2oN
Running:
sudo lvdisplay
gives the following result:
--- Logical volume ---
LV Name /dev/raid1/tv
VG Name raid1
LV UUID Dokbch-ZJkg-QmRW-d9vR-wfM8-BFxb-3Z0krs
LV Write Access read/write
LV Status available
# open 1
LV Size 1.70 TiB
Current LE 445645
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
I have a couple of smaller logical volumes also in this volume group, that I have not shown. That's why there's a bit of a difference between the Alloc PE value in the volume group, and the Current LE value in the logical volume. As you can see from the lines shown in bold text, the volume group raid1 has 249272 physical extents (PE) free, and the logical volume /dev/raid1/tv is currently sized 445645. To use all space, I made the size 249272+445645, which is 694917.
The command to resize a logical volume is lvresize. Logical.
sudo lvresize -l 694917 /dev/raid1/tv
Alternatively, if you want to avoid all the maths, an alternative command is
sudo lvresize -l +100%FREE /dev/raid1/tv
That command just tells lvm to use 100% of the free space. I didn't try it myself (I only found it after running the command before it).
Now, after that has been run, to check the results, enter:
sudo lvdisplay
The results:
--- Logical volume ---
LV Name /dev/raid1/tv
VG Name raid1
LV UUID Dokbch-ZJkg-QmRW-d9vR-wfM8-BFxb-3Z0krs
LV Write Access read/write
LV Status available
# open 1
LV Size 2.65 TiB
Current LE 694917
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
and
sudo vgdisplay
gives:
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 6
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 4.00 MiB
Total PE 715397
Alloc PE / Size 715397 / 2.73 TiB
Free PE / Size 0 / 0
VG UUID gvfheX-ifvl-yW9h-v4L2-eyzs-95fe-sng2oN
No free space shown; the lvm volume group is using the whole mdadm array, which in turn is using the whole of the two disks.
The final step for me was to grow the partition that is on the logical volume. I had formatted it with XFS, as it is good with large video files. XFS allows increasing size on a mounted partition, so the command used was:
sudo xfs_growfs -d /var/lib/mythtv
Finally, it is complete!
Subscribe to:
Posts (Atom)