TL;DR: XFS filesystem corrupted, ran xfs_repair on unmounted system via a live USB to fix it.
A couple of weeks back one of the drives in my RAID 5 array started going bad, with SMART errors appearing. It so happened to be the drive that I had bought to replace another drive that went bad (with the same error) back at the start of 2017. Anyway, warranty replacement drive went in, re-added it to the array and all seemed fine.
Then one Friday night, I went to watch a movie that had been recorded on MythTV previously. Switched on the TV, changed to MythTV, all of the recorded programs were showing as not found. Uh-oh. Rebooted the frontend, issue still there. Time to check the backend. Logged in to it, ran dmesg, up come a heap of XFS errors. That's not good. Better restart it and see if that fixes things. Nope, now it won't boot. It gets to the recovery console and I can't do much. Checking the SMART status of the drives, they seem fine, but the RAID array is not shown. It also seems to be missing a drive, but not the one I replaced. Scratching my head, wondering what's wrong, a niggling thought that I'd have to do a fresh re-install because Mythbuntu hosed itself. I'd been thinking about it for a while, maybe setting it up in a virtual machine, changing over to ZFS, but I wanted to do that in my own time, rather than having my hand forced. I went to bed to think about it.
The next day, I thought I'd try booting off a live USB of Mythbuntu. Fortunately, that worked, and I could see the array. It was missing one of the drives - so I added it back in. Many hours of rebuilding later, it was back up. While it was rebuilding, I looked in to repairing XFS filesystems.
Once the array was up, I unmounted the volume that contained the recordings, and ran xfs_repair /dev/raid1/tv. It found quite a few problems, and rectified quite a few. With fingers crossed, I tried rebooting back into the original system...success! With a sense of relief and amazement the desktop appeared. Upon checking the list of recordings, quite a few were still missing, mostly from the last few days. I hadn't checked up on it for a little while, so whatever issue there was, was screwing up the recordings. Deleting them from within MythTV, to clean up the database, fixed that. I also noticed that a number of older recordings were also showing as missing. Probably just a byproduct of what was happening. No big deal, they are just TV shows after all.
The system has been running OK for the last few days, so hopefully the issue is resolved. I'm still at a loss as to what actually went wrong though. Might have to dig through some log files.
Showing posts with label MythTV. Show all posts
Showing posts with label MythTV. Show all posts
Thursday, May 11, 2017
Thursday, June 18, 2015
Converting RAID1 array to RAID5 using the mdadm grow command
I have finally decided to upgrade the storage in the home theatre PC, by adding a third 3TB hard drive. The storage was set up previously as RAID 1, using the software mdadm solution for the two 3TB disks. By adding a third drive and changing to a raid 5 format, the storage would increase from 3TB (about 2.7TiB) to 6TB (about 5.4TiB).
There are a few ways to do this:
1. Copy the data to a spare drive, delete the RAID 1 array, create the RAID 5 array with the three disks, copy the data back to it.
2. Back up the data to a spare drive, remove one disk from the RAID 1 array, use that disk and the new disk to make a 2 disk RAID 5 array, copy data over, remove the RAID 1 array and add that disk to the RAID 5 array so it is 3 disks.
3. Back up the data to a spare drive, Use the mdadm --grow command to change the RAID level from RAID 1 to RAID 5, add the third drive to the RAID 5 array and let it rebuild.
Initially I was going to try option 2, such as described here. But noticing one of the comments (by the wonderfully named Svenne Krap) describing that you can just change the level of the array using the mdadm grow command, I thought it would warrant further investigation. I couldn't find many other mentions of it elsewhere on the internet, so I thought I'd document what I'd done so it might help someone else.
So to try it out, I set up a virtual machine on another PC, and created three separate drives. I used mdadm to create a RAID 1 array with two of them, and then converted that array to RAID 5. It worked! I then added the third drive, and after a little while it rebuilt into a full three disk array.
So the time came to do it for real. I first wanted to back up the array - and found an old 2TB drive lying around. Fortunately, the array was only just over two thirds full, so it all managed to squeeze on to the 2TB drive after I deleted a few old TV shows. That's still nearly 600 TV shows and movies left - I have no idea when I'll get around to watching them all, but it's nice to have.
So the data was safe. I could do this switchover with a little less stress.
One thing I noticed with the options was chunk size. Different sizes give different performance. Since this box would be primarily a media box, reading and writing large video files, I went for a larger chunk size of 512KB. When creating the array in the virtual machine initially, it created only 4KB chunks as the size of the array was not a multiple of 512KB. So first up, I resized the RAID 1 array so it was. Basically I divided the size reported by the cat /proc/mdstat command, by 512. It gave a number ending in .625, meaning there weren't an exact number of 512KB chunks. By multiplying that answer (minus .625) by 512, it gave a size that could work.
So I resized the array with the following command:
sudo mdadm -G /dev/md0 --size=2930266112
I then let mdadm know about the new drive, by adding it. The new drive is /dev/sde:
sudo mdadm /dev/md0 --add /dev/sde
Now there was a third drive added as a spare to the array, but not actually in use by it yet.
Next, the big one, change the array from RAID 1 to RAID 5 (still only 2 drives):
sudo mdadm /dev/md0 --grow --level=5 --chunk=512
The array is now a RAID 5 array, with two disks. The chunk size of 512KB was also set with that command. Time to let it use the third drive to create the full, three disk RAID 5 array:
sudo mdadm --grow /dev/md0 --raid-devices 3
This kicks off the rebuilding process, which takes many hours. You can track the progress with the cat /proc/mdstat command. It ran pretty slowly on my system, at around 30,000KB/sec, until it got to the empty part of the array, when speed nearly tripled.
Later that day, the process was finished. Running sudo mdadm --detail /dev/md0 gave the new array size of about 5.4TiB - something I was a little concerned about while the rebuild was underway, because it was only showing the old size of less than 3TB. I thought I might have to resize the array afterwards, but it all came out good.
Resizing LVM
Because I was running LVM on top of the raid array, the size of the LVM volume was still unchanged - I had to increase that to let it use the extra space that was now available. Going by the size reported by the mdadm --detail command run previoulsy, there were approximately 5588GB in the array. LVM works with three 'layers' - the physical volume, basically the hard drive or RAID array; volume groups, which consist of a number of logical volumes. See here for more information. The first step is to resize the physical volume to match the size of the array:
sudo pvresize --setphysicalvolumesize 5588G /dev/md0
Next, I could extend the logical volumes to use up more of that space. I have two main logical volumes on this volume group: one called 'tv' that holds TV shows, movies, and music from MythTV, and another called 'sysbackup' that holds backups of data. I wanted to enlarge both of these - the 'tv' one, and also 'sysbackup' because I wanted to use it with Crashplan as a backup destination for some other PCs in the house.
I wanted to increase the 'sysbackup' volume to 1.2TB, so I used the following command:
sudo lvextend -L1.2T /dev/raid1/sysbackup
A couple of things to note here: the '-L1.2T' instructs it to make the total size of the volume 1.2 Terabytes. Also, the name of the volume group is still /dev/raid1 - I haven't changed the name of that, even though it is now RAID 5. It can be done with the vgrename command, but it would also mean changing mount points in the /etc/fstab file.
Next was the resizing of the 'tv' volume. I wanted to increase it by 1.6TB, so I used the following command:
sudo lvextend -L+1.6T /dev/raid1/tv
Notice the '-L+1.6T' - the plus sign commands it to expand the volume by the specified amount. Unfortunately, it reported that there wasn't 1.6TB spare, and gave a number of available extents that were free. So I tried a different approach, specifying size in extents rather than TB:
sudo lvextend -l+418800 /dev/raid1/tv
The lower-case 'l' instructs it to use extents rather than GB, TB or whatever. This worked, and I now had two newly-resized logical volumes.
Resizing the file systems
But there was one final step - the logical volumes were now bigger, but the file systems did not know about it yet. Fortunately it isn't too difficult.
The sysbackup volume was formatted with the ext4 file system. To resize that, it first had to be unmounted:
sudo umount /mnt/sysbackup
Then a check of the file system was done:
sudo e2fsck -f /dev/raid1/sysbackup
Then finally, the file system was resized to fill the available space:
sudo resize2fs /dev/raid1/sysbackup
Finally, the file system can be re-mounted. Since it is specified in the /etc/fstab file, running
sudo mount -a
did the trick. Next was the 'tv' volume. This volume was formatted with the XFS file system, and it can be resized while still mounted. The command for that was:
sudo xfs_growfs /dev/raid1/tv
Conclusion
The advantage of using the mdadm --grow command is that it is still the same array according to mdadm, just a different level. Running
sudo mdadm --detail --scan
shows that the array still has the same identifiers as when it was a RAID 1 array, so no further work is required. I ran
sudo update-initramfs -u
just in case, so it would pick up any changes in the array (and not give it a name like /dev/md127 after a reboot). I restarted the PC, and everything came back up, with the roomy new array showing up as it should. Done!
There are a few ways to do this:
1. Copy the data to a spare drive, delete the RAID 1 array, create the RAID 5 array with the three disks, copy the data back to it.
2. Back up the data to a spare drive, remove one disk from the RAID 1 array, use that disk and the new disk to make a 2 disk RAID 5 array, copy data over, remove the RAID 1 array and add that disk to the RAID 5 array so it is 3 disks.
3. Back up the data to a spare drive, Use the mdadm --grow command to change the RAID level from RAID 1 to RAID 5, add the third drive to the RAID 5 array and let it rebuild.
Initially I was going to try option 2, such as described here. But noticing one of the comments (by the wonderfully named Svenne Krap) describing that you can just change the level of the array using the mdadm grow command, I thought it would warrant further investigation. I couldn't find many other mentions of it elsewhere on the internet, so I thought I'd document what I'd done so it might help someone else.
So to try it out, I set up a virtual machine on another PC, and created three separate drives. I used mdadm to create a RAID 1 array with two of them, and then converted that array to RAID 5. It worked! I then added the third drive, and after a little while it rebuilt into a full three disk array.
So the time came to do it for real. I first wanted to back up the array - and found an old 2TB drive lying around. Fortunately, the array was only just over two thirds full, so it all managed to squeeze on to the 2TB drive after I deleted a few old TV shows. That's still nearly 600 TV shows and movies left - I have no idea when I'll get around to watching them all, but it's nice to have.
So the data was safe. I could do this switchover with a little less stress.
One thing I noticed with the options was chunk size. Different sizes give different performance. Since this box would be primarily a media box, reading and writing large video files, I went for a larger chunk size of 512KB. When creating the array in the virtual machine initially, it created only 4KB chunks as the size of the array was not a multiple of 512KB. So first up, I resized the RAID 1 array so it was. Basically I divided the size reported by the cat /proc/mdstat command, by 512. It gave a number ending in .625, meaning there weren't an exact number of 512KB chunks. By multiplying that answer (minus .625) by 512, it gave a size that could work.
So I resized the array with the following command:
sudo mdadm -G /dev/md0 --size=2930266112
I then let mdadm know about the new drive, by adding it. The new drive is /dev/sde:
sudo mdadm /dev/md0 --add /dev/sde
Now there was a third drive added as a spare to the array, but not actually in use by it yet.
Next, the big one, change the array from RAID 1 to RAID 5 (still only 2 drives):
sudo mdadm /dev/md0 --grow --level=5 --chunk=512
The array is now a RAID 5 array, with two disks. The chunk size of 512KB was also set with that command. Time to let it use the third drive to create the full, three disk RAID 5 array:
sudo mdadm --grow /dev/md0 --raid-devices 3
This kicks off the rebuilding process, which takes many hours. You can track the progress with the cat /proc/mdstat command. It ran pretty slowly on my system, at around 30,000KB/sec, until it got to the empty part of the array, when speed nearly tripled.
Later that day, the process was finished. Running sudo mdadm --detail /dev/md0 gave the new array size of about 5.4TiB - something I was a little concerned about while the rebuild was underway, because it was only showing the old size of less than 3TB. I thought I might have to resize the array afterwards, but it all came out good.
Resizing LVM
Because I was running LVM on top of the raid array, the size of the LVM volume was still unchanged - I had to increase that to let it use the extra space that was now available. Going by the size reported by the mdadm --detail command run previoulsy, there were approximately 5588GB in the array. LVM works with three 'layers' - the physical volume, basically the hard drive or RAID array; volume groups, which consist of a number of logical volumes. See here for more information. The first step is to resize the physical volume to match the size of the array:
sudo pvresize --setphysicalvolumesize 5588G /dev/md0
Next, I could extend the logical volumes to use up more of that space. I have two main logical volumes on this volume group: one called 'tv' that holds TV shows, movies, and music from MythTV, and another called 'sysbackup' that holds backups of data. I wanted to enlarge both of these - the 'tv' one, and also 'sysbackup' because I wanted to use it with Crashplan as a backup destination for some other PCs in the house.
I wanted to increase the 'sysbackup' volume to 1.2TB, so I used the following command:
sudo lvextend -L1.2T /dev/raid1/sysbackup
A couple of things to note here: the '-L1.2T' instructs it to make the total size of the volume 1.2 Terabytes. Also, the name of the volume group is still /dev/raid1 - I haven't changed the name of that, even though it is now RAID 5. It can be done with the vgrename command, but it would also mean changing mount points in the /etc/fstab file.
Next was the resizing of the 'tv' volume. I wanted to increase it by 1.6TB, so I used the following command:
sudo lvextend -L+1.6T /dev/raid1/tv
Notice the '-L+1.6T' - the plus sign commands it to expand the volume by the specified amount. Unfortunately, it reported that there wasn't 1.6TB spare, and gave a number of available extents that were free. So I tried a different approach, specifying size in extents rather than TB:
sudo lvextend -l+418800 /dev/raid1/tv
The lower-case 'l' instructs it to use extents rather than GB, TB or whatever. This worked, and I now had two newly-resized logical volumes.
Resizing the file systems
But there was one final step - the logical volumes were now bigger, but the file systems did not know about it yet. Fortunately it isn't too difficult.
The sysbackup volume was formatted with the ext4 file system. To resize that, it first had to be unmounted:
sudo umount /mnt/sysbackup
Then a check of the file system was done:
sudo e2fsck -f /dev/raid1/sysbackup
Then finally, the file system was resized to fill the available space:
sudo resize2fs /dev/raid1/sysbackup
Finally, the file system can be re-mounted. Since it is specified in the /etc/fstab file, running
sudo mount -a
did the trick. Next was the 'tv' volume. This volume was formatted with the XFS file system, and it can be resized while still mounted. The command for that was:
sudo xfs_growfs /dev/raid1/tv
Conclusion
The advantage of using the mdadm --grow command is that it is still the same array according to mdadm, just a different level. Running
sudo mdadm --detail --scan
shows that the array still has the same identifiers as when it was a RAID 1 array, so no further work is required. I ran
sudo update-initramfs -u
just in case, so it would pick up any changes in the array (and not give it a name like /dev/md127 after a reboot). I restarted the PC, and everything came back up, with the roomy new array showing up as it should. Done!
Monday, March 16, 2015
HTPC update - separating backend from frontend
After using the Atom-based HTPC for a while, I noticed it start to struggle in high load situations - like playing back HD TV while recording multiple streams, commercial flagging, trying picture-in -picture, running disk checks. I decided to upgrade the box to something a bit more powerful, so there’s a bit more headroom.
So a while back I picked up an AsRock H77-M motherboard, with a cheap Pentium G2020 CPU. 4GB of RAM was sitting around idle after a previous desktop PC upgrade, so in that went. Two, 3GB drives were installed, still in a RAID-1 configuration. The OS was installed on an SSD that was left over. Graphics were handled by an Nvidia GT630 card, made by ASUS. I chose that one because of the passive cooler, and the fact that it is quite power efficient for what it does. A TP-Link wireless card was put in so it could see the network.
For the operating system, I went with Mythbuntu 12.04 again. Installation went pretty straightforward. No real problems, TV listing data was handled by the excellent shepherd - that just does its own thing, causing very little trouble.
Playback worked beautifully on this machine - it handled everything I could throw at it. However, my curiosity was getting the better of me, so I dug out the old Atom box and reinstalled mythbuntu, this time running just as a frontend. That made quite a bit of difference - without all the overhead of doing all the database tasks, flagging commercials and the like, it ran beautifully.
Since I had recently had some network cabling installed in the house, I tried running the new box as a backend only, up in the study, with just the Atom frontend in the lounge. Without any worries about wifi dropping out, the nice fast, stable wired network connection supplied data from the backend nicely.
It ran like that for a while, but then I started thinking that there are now three points of failure instead of one. Where previously there was just the Pentium backend/frontend to troubleshoot, there was now also the frontend box, plus the router that could cause problems. Eventually I put the frontend box back away.
I’m not sure what made me do it, but soon after getting a new TV (LCD! HDMI inputs! No more fooling around with xorg.conf and custom modelines and VGA to Component transcoders!) I wanted to clear out a bit of clutter from the lounge room. Moving the big box meant I could get rid of a whole heap of things from the lounge - the UPS, the transcoder (not needed any more), the aerial connection with splitter and the external TV tuners. All of the bulky component cables are no longer needed, either, replaced by a single HDMI cable.
So up to the study the box went, along with all the tuners and UPS, but they have been tucked away behind the desk, out of sight. The lounge room now just has a power cable to the frontend box, a network cable, an optical cable to the receiver, and the HDMI cable to the TV.
For now, I’m just rebooting the router every few weeks, or whenever there is some sort of issue like PCs not being able to access the internet. A lot of data is now going across it with TV viewing.
The backend box was able to have the graphics card removed from it, as it is now running headless. I only connect to the machine via ssh. That has saved a bit of power consumption. Another change made to the backend box has been to add a second SSD, to run the OS off a RAID-1 array. This way, a drive failure will not stop the machine. It will run in a degraded state until a new one can be installed.
I did the same with the frontend box - a couple of old laptop drives were pressed into service for that. Both are pretty elderly, but since they are in RAID-1, unless they both die at the same time, it should be safe. And really, it’s only a frontend, it wouldn’t be too much of a hassle to rebuild it from scratch if needed. I’m considering replacing those drives with a couple of USB sticks, to further reduce power consumption.
For the backend box, disk usage is sitting at around 65%. I’ve been copying some of my DVD library to the server, just to avoid dealing with physical discs. I don’t actually have a DVD drive in the frontend, anyway. If space gets tight, I might need to expand storage. I’m toying with getting a third 3TB drive, and converting the system to either RAID-5, or going a bit radical and putting ZFS on Linux back on it. My only concern with that is that it can have issues with fragmentation, and my PC doesn’t use ECC RAM. A lot of memory is also used for caching purposes with ZFS.
One thing I am wondering about with my current XFS file system is to do with fragmentation. I currently have a cron script to defragment it nearly every morning. Not sure what the deal is with RAID-5, and how it deals with fragmentation. Might just have to try it and see, when I get around to it. Might just go with ext4 for the file system. Outright performance isn’t so much of an issue for this machine - video recording and playback isn’t pushing it too much.
Monday, November 26, 2012
Adding a larger drive to a RAID array - ZFS
If you have waded through my previous post about adding a larger drive to an lvm volume running on an mdadm RAID array, you'll have seen it's a bit of a fiddly, complex process. I have been fiddling around a bit with ZFS on Linux on my old desktop machine (Core 2 Quad Q9300, 8GB RAM). I had a couple of old 1TB drives, and added the 2TB drive that I removed from the home theatre PC, mentioned in the previous post. Actually, I had put the 3TB drive in this old box initially, intending it to replace the HTPC. But it's a bit power hungry, using around 90W at idle, and I am having some niggling playback issues with it. But that's another story. So the 3TB WD Red drive gets replaced with the old Samsung 2TB.
With the three drives, I tried creating a ZFS raidz array, basically equivalent to RAID 5. It runs pretty well. Swapping the drives over (from 3TB to 2TB) gave me an opportunity to do the ZFS version of a drive swap.
It was a bit of an anticlimax, really. There was one command, the syntax from the man page is below:
zpool replace [-f] pool device [new_device]
In my case it was something like
sudo zpool replace tank /dev/disk/by-id/scsi-SATA...(the drive removed) /dev/disk/by-id/scsi-SATA....(the 2TB drive)
That started off the resilvering process (resynchronising the data across the drives). If I then were to replace each other drive, one at a time, with the above process, it would resize the array to match the new drive sizes.
The beauty of ZFS is it simplifies things so much. It is the RAID management and file system all in one.
I would have liked to run it in the HTPC, but ZFS is quite resource-hungry, in terms of processing power and memory. Since my HTPC just has a little Atom 330 chip and is maxed out with 4GB of memory (of which only 3GB is visible), I would be staying with mdadm and lvm. There is virtually no noticeable performance penalty with it. Oh well, going through all those steps is not exactly a frequent task.
With the three drives, I tried creating a ZFS raidz array, basically equivalent to RAID 5. It runs pretty well. Swapping the drives over (from 3TB to 2TB) gave me an opportunity to do the ZFS version of a drive swap.
It was a bit of an anticlimax, really. There was one command, the syntax from the man page is below:
zpool replace [-f] pool device [new_device]
In my case it was something like
sudo zpool replace tank /dev/disk/by-id/scsi-SATA...(the drive removed) /dev/disk/by-id/scsi-SATA....(the 2TB drive)
That started off the resilvering process (resynchronising the data across the drives). If I then were to replace each other drive, one at a time, with the above process, it would resize the array to match the new drive sizes.
The beauty of ZFS is it simplifies things so much. It is the RAID management and file system all in one.
I would have liked to run it in the HTPC, but ZFS is quite resource-hungry, in terms of processing power and memory. Since my HTPC just has a little Atom 330 chip and is maxed out with 4GB of memory (of which only 3GB is visible), I would be staying with mdadm and lvm. There is virtually no noticeable performance penalty with it. Oh well, going through all those steps is not exactly a frequent task.
Adding a larger drive to a software RAID array - mdadm and lvm
The MythTV box I have in the lounge previously had two storage hard drives, in a RAID 1 configuration to prevent data loss in case of a drive failure. The drives were a 3TB Hitachi and a 2TB Samsung. I figured the Samsung drive was getting on a bit now, and it was time a new drive was installed. Might as well make it a 3TB model as well, to take advantage of all the space available on the other one that was sitting unused.
A 3TB Western Digital Red drive was picked up. I chose this as it is designed for use in a NAS environment: always on. It also has low power consumption and a good warranty. I considered a Seagate Barracuda 3TB - they were cheap, performance would be better than the Red, but they are only designed for desktop use, around 8 hours a day powered on. Warranty was pretty short as well.
Removing and replacing the old drive
The drives were configured in a software RAID 1 array, using mdadm, with lvm on top of that. This makes the array portable, and not dependent on a particular hardware controller.
The commands here were adapted from the excellent instructions found here at howtoforge.com.
Fortunately I had enough space on another PC that I was able to back up the contents of the array before starting any of this.
To remove the old drive, which on this machine was /dev/sdc, the following command was issued to mark the drive as failed, in the array /dev/md0:
sudo mdadm --manage /dev/md0 --fail /dev/sdc
Next step is to remove the drive from the array:
sudo mdadm --manage /dev/md0 --remove /dev/sdc
Then, the system could be shut down and the drive removed and replaced with the new one. After powering the system back up, the following command adds the new drive to the array:
sudo mdadm --manage /dev/md0 --add /dev/sdc
The array will then start synchronising the data, copying it to the new drive, which could take a few hours. Note that no partitioning was done on the disk, as I am just using the whole drive in the array.
While the sync is in progress, you can check how it is progressing via:
cat /proc/mdstat
It will show a percentage of completion as well as an estimated time remaining. Once it is done, the array is ready for use! I left the array like this for a day or so, just to make sure everything was working alright.
Expanding the array to fill the space available - the mdadm part
Once the synchronisation has completed, the size of the array was still only 2TB, since that is the largest a RAID 1 array could go when it consists of a 3TB and a 2TB drive. We need to tell mdadm to expand the array to fill the available space. More information on this can be found here.
This is where things got complicated for me. It is to do with the superblock format version used in the array. More detail can be found at this page of the Linux RAID wiki.
To sum up, the array I had was created with the version 0.90 superblock. The version was found by entering
sudo mdadm --detail /dev/md0
The problem, potentially, was that if I grew the array to larger than 2TB it may not work. As quoted by the wiki link above:
Now, Mythbuntu 12.04 runs the 3.2 kernel, so according to that it should be OK supporting up to 4TB. But I wasn't 100% sure on that, and couldn't find any references elsewhere about it. I decided the safest way to go about this was to convert the array to a later version of the superblock, that didn't have that size limitation. Besides, it would save time in the future in case of trying to repeat this with a drive larger than 4TB.
Following the suggestion of the wiki, I decided to update to a version 1.0 superblock, as it would store the superblock information in the same place as the 0.90.
Note: if you are trying this yourself, and the array is already version 1.0 or later, then the command to grow it is just as below (may not want to do it with 0.90 superblock and larger than 2TB):
mdadm --grow /dev/md0 --size=max
Since I was going to change the superblock version, it involved stopping the array and recreating it with the later version.
Once again, to check the details of the array at the moment:
sudo mdadm --detail /dev/md0
Now, since the array is in use by MythTV, I thought it safest to stop the program:
sudo service mythtv-backend stop
Also, I unmounted where the array was mounted:
sudo umount /var/lib/mythtv
Since the data is in an LVM volume on top of the array, I deactivated that as well (the volume is named raid1 in this instance):
sudo lvchange -a n raid1
The array is now ready to be stopped:
sudo mdadm --stop /dev/md0
Now it can be re-created, specifying the metadata (superblock) version, the RAID level, and the number and names of the drives used:
sudo mdadm --create /dev/md0 --metadata=1.0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
The array will now start resynchronising. This took a number of hours for me, as there were around 770GB of recordings there. The RAID wiki link included --assume-clean in the above command, which would have skipped the resync. I elected to leave it out, for safety's sake.
Progress can be monitored with:
cat /proc/mdstat
The lvm volume can be restarted:
sudo lvchange -a y raid1
and the unmounted volumes can be re-mounted:
sudo mount -a
Check if they are all there with the
mount
command. The mythtv service can also be restarted:
sudo service mythtv-backend start
When the array is recreated, the UUID value of the array will be different. You can get the new value with:
sudo mdadm --detail /dev/md0
Edit the /etc/mdadm/mdadm.conf file, and change the UUID value in it to the new value. This will enable the array to be found on next boot.
Another thing to do before rebooting is to run
sudo update-initramfs -u
I didn't do this at first, and after rebooting, the array showed up named /dev/md127 rather than /dev/md0. Running the above command and rebooting again fixed it for me.
Expanding the array to fill the space available - the lvm part
Quite a long-winded process, isn't it? Using the lvm command to show the lvm physical volumes:
sudo pvdisplay
showed the array was still 1.82TiB (2TB). It needed to be extended. The following command will fill the volume to the available space:
sudo pvresize -v /dev/md0
To check the results, again run:
sudo pvdisplay
Now, running:
sudo vgdisplay
gave the following results for me:
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 4.00 MiB
Total PE 715397
Alloc PE / Size 466125 / 1.78 TiB
Free PE / Size 249272 / 973.72 GiB
VG UUID gvfheX-ifvl-yW9h-v4L2-eyzs-95fe-sng2oN
A 3TB Western Digital Red drive was picked up. I chose this as it is designed for use in a NAS environment: always on. It also has low power consumption and a good warranty. I considered a Seagate Barracuda 3TB - they were cheap, performance would be better than the Red, but they are only designed for desktop use, around 8 hours a day powered on. Warranty was pretty short as well.
Removing and replacing the old drive
The drives were configured in a software RAID 1 array, using mdadm, with lvm on top of that. This makes the array portable, and not dependent on a particular hardware controller.
The commands here were adapted from the excellent instructions found here at howtoforge.com.
Fortunately I had enough space on another PC that I was able to back up the contents of the array before starting any of this.
To remove the old drive, which on this machine was /dev/sdc, the following command was issued to mark the drive as failed, in the array /dev/md0:
sudo mdadm --manage /dev/md0 --fail /dev/sdc
Next step is to remove the drive from the array:
sudo mdadm --manage /dev/md0 --remove /dev/sdc
Then, the system could be shut down and the drive removed and replaced with the new one. After powering the system back up, the following command adds the new drive to the array:
sudo mdadm --manage /dev/md0 --add /dev/sdc
The array will then start synchronising the data, copying it to the new drive, which could take a few hours. Note that no partitioning was done on the disk, as I am just using the whole drive in the array.
While the sync is in progress, you can check how it is progressing via:
cat /proc/mdstat
It will show a percentage of completion as well as an estimated time remaining. Once it is done, the array is ready for use! I left the array like this for a day or so, just to make sure everything was working alright.
Expanding the array to fill the space available - the mdadm part
Once the synchronisation has completed, the size of the array was still only 2TB, since that is the largest a RAID 1 array could go when it consists of a 3TB and a 2TB drive. We need to tell mdadm to expand the array to fill the available space. More information on this can be found here.
This is where things got complicated for me. It is to do with the superblock format version used in the array. More detail can be found at this page of the Linux RAID wiki.
To sum up, the array I had was created with the version 0.90 superblock. The version was found by entering
sudo mdadm --detail /dev/md0
The problem, potentially, was that if I grew the array to larger than 2TB it may not work. As quoted by the wiki link above:
The version-0.90 superblock limits the number of component devices within an array to 28, and limits each component device to a maximum size of 2TB on kernel version [earlier than] 3.1 and 4TB on kernel version 3.1 [or later].
Now, Mythbuntu 12.04 runs the 3.2 kernel, so according to that it should be OK supporting up to 4TB. But I wasn't 100% sure on that, and couldn't find any references elsewhere about it. I decided the safest way to go about this was to convert the array to a later version of the superblock, that didn't have that size limitation. Besides, it would save time in the future in case of trying to repeat this with a drive larger than 4TB.
Following the suggestion of the wiki, I decided to update to a version 1.0 superblock, as it would store the superblock information in the same place as the 0.90.
Note: if you are trying this yourself, and the array is already version 1.0 or later, then the command to grow it is just as below (may not want to do it with 0.90 superblock and larger than 2TB):
mdadm --grow /dev/md0 --size=max
Since I was going to change the superblock version, it involved stopping the array and recreating it with the later version.
Once again, to check the details of the array at the moment:
sudo mdadm --detail /dev/md0
Now, since the array is in use by MythTV, I thought it safest to stop the program:
sudo service mythtv-backend stop
Also, I unmounted where the array was mounted:
sudo umount /var/lib/mythtv
Since the data is in an LVM volume on top of the array, I deactivated that as well (the volume is named raid1 in this instance):
sudo lvchange -a n raid1
The array is now ready to be stopped:
sudo mdadm --stop /dev/md0
Now it can be re-created, specifying the metadata (superblock) version, the RAID level, and the number and names of the drives used:
sudo mdadm --create /dev/md0 --metadata=1.0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
The array will now start resynchronising. This took a number of hours for me, as there were around 770GB of recordings there. The RAID wiki link included --assume-clean in the above command, which would have skipped the resync. I elected to leave it out, for safety's sake.
Progress can be monitored with:
cat /proc/mdstat
The lvm volume can be restarted:
sudo lvchange -a y raid1
and the unmounted volumes can be re-mounted:
sudo mount -a
Check if they are all there with the
mount
command. The mythtv service can also be restarted:
sudo service mythtv-backend start
When the array is recreated, the UUID value of the array will be different. You can get the new value with:
sudo mdadm --detail /dev/md0
Edit the /etc/mdadm/mdadm.conf file, and change the UUID value in it to the new value. This will enable the array to be found on next boot.
Another thing to do before rebooting is to run
sudo update-initramfs -u
I didn't do this at first, and after rebooting, the array showed up named /dev/md127 rather than /dev/md0. Running the above command and rebooting again fixed it for me.
Expanding the array to fill the space available - the lvm part
Quite a long-winded process, isn't it? Using the lvm command to show the lvm physical volumes:
sudo pvdisplay
showed the array was still 1.82TiB (2TB). It needed to be extended. The following command will fill the volume to the available space:
sudo pvresize -v /dev/md0
To check the results, again run:
sudo pvdisplay
Now, running:
sudo vgdisplay
gave the following results for me:
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 4.00 MiB
Total PE 715397
Alloc PE / Size 466125 / 1.78 TiB
Free PE / Size 249272 / 973.72 GiB
VG UUID gvfheX-ifvl-yW9h-v4L2-eyzs-95fe-sng2oN
Running:
sudo lvdisplay
gives the following result:
--- Logical volume ---
LV Name /dev/raid1/tv
VG Name raid1
LV UUID Dokbch-ZJkg-QmRW-d9vR-wfM8-BFxb-3Z0krs
LV Write Access read/write
LV Status available
# open 1
LV Size 1.70 TiB
Current LE 445645
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
I have a couple of smaller logical volumes also in this volume group, that I have not shown. That's why there's a bit of a difference between the Alloc PE value in the volume group, and the Current LE value in the logical volume. As you can see from the lines shown in bold text, the volume group raid1 has 249272 physical extents (PE) free, and the logical volume /dev/raid1/tv is currently sized 445645. To use all space, I made the size 249272+445645, which is 694917.
The command to resize a logical volume is lvresize. Logical.
sudo lvresize -l 694917 /dev/raid1/tv
Alternatively, if you want to avoid all the maths, an alternative command is
sudo lvresize -l +100%FREE /dev/raid1/tv
That command just tells lvm to use 100% of the free space. I didn't try it myself (I only found it after running the command before it).
Now, after that has been run, to check the results, enter:
sudo lvdisplay
The results:
--- Logical volume ---
LV Name /dev/raid1/tv
VG Name raid1
LV UUID Dokbch-ZJkg-QmRW-d9vR-wfM8-BFxb-3Z0krs
LV Write Access read/write
LV Status available
# open 1
LV Size 2.65 TiB
Current LE 694917
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
and
sudo vgdisplay
gives:
--- Volume group ---
VG Name raid1
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 6
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 2.73 TiB
PE Size 4.00 MiB
Total PE 715397
Alloc PE / Size 715397 / 2.73 TiB
Free PE / Size 0 / 0
VG UUID gvfheX-ifvl-yW9h-v4L2-eyzs-95fe-sng2oN
No free space shown; the lvm volume group is using the whole mdadm array, which in turn is using the whole of the two disks.
The final step for me was to grow the partition that is on the logical volume. I had formatted it with XFS, as it is good with large video files. XFS allows increasing size on a mounted partition, so the command used was:
sudo xfs_growfs -d /var/lib/mythtv
Finally, it is complete!
Tuesday, August 21, 2012
Upgrading MythTV to Mythbuntu 12.04 64-bit
As described in the previous post, the time had come to upgrade my MythTV installation. It had been installed using Mythbuntu 10.10, 32-bit. I wanted to move to the most recent version, currently Mythbuntu 12.04. There were two options available to me: upgrading the current installation, or a new install of the latest version, and copying the previous MythTV database over to it. The first option was not too appealing. Doing a distribution upgrade is a fingers-crossed, hope-this-works process a good part of the time - although I have had pretty good fortune with it on my desktop PC. If I were to go this route, I couldn't just upgrade from 10.10 -> 12.04. It would have to be 10.10 -> 11.04 -> 11.10 -> 12.04. Three complete system upgrades in succession, which would likely leave a lot of cruft left over in the installation. Lots of points of failure. The final nail in the coffin was that I wanted to move to a 64-bit installation, to take advantage of the full 4GB of memory that was installed in the system.
So a clean install it was. To prepare, I copied a number of setup files to a safe place that wouldn't be overwritten during the install. Files like:
Not all the files would end up being needed, but I felt safer having them handy, just in case.
The installation process started from a USB stick loaded with the Mythbuntu 12.04 installer, since I don't have an optical drive in it. No room, with three drives already there. I also hooked up a monitor instead of a TV, because that needs extra setup to output to a component connection.
Since there was some left over free space on the SSD, I installed to that, leaving the 10.10 install intact for now. It went remarkably smoothly. One thing I noted was that at the beginning, there is an option to update the installer. I clicked that, and the system seemed to do nothing for quite a while. Opening up a terminal window and running "top" showed that it was in fact doing something, so I waited until it finished. Once done, the installation process could begin. It would have been nice for it to have some sort of indication that it was doing something, though.
Partway through the process, it prompted for what sort of remote control is being used. I selected "Windows Media Center remote", and the thing was set up automatically. Brilliant! Didn't need that LIRC setup file listed above.
To get the system outputting to my TV properly, I just copied the old xorg.conf file to the /etc/X11 directory (I renamed the existing file, in case I needed it again) and rebooted with a sense of hope and anxiety. A minute or so later, the screen came to life! It worked!
The big one, that I was dreading, was getting the RAID array to be seen by the new install. I still consider myself quite the mdadm novice, so I was pleasantly surprised to find that the RAID array was picked up by the new install quite easily - after installing mdadm and lvm2. This command:
sudo mdadm --assemble --scan
was all that was needed for it to scan for any drives, re-assemble the array, and write a new mdadm.conf file. Easy!
Next was seeing if LVM could find the volumes created. Running
sudo lvm pvscan
sudo lvm vgscan
sudo lvm lvscan
detected all the physical volumes, volume groups, and logical volumes, respectively. Mounting the logical volume that held the recordings was successful - I could see them all. Adding the device entry to /etc/fstab seemed to work, eventually. On the first reboot it came up with an error message, mentioning there were serious problems with the mount. Not sure what it was, but a reboot afterwards seemed to make it happy. I had installed xfsprogs, to manage and defragment XFS filesystems; maybe that helped.
Going through some of the MythTV screens brought up some error messages that it couldn't write to the drive - it turned out to be a permissions issue for the files and directories on the array. Entering
sudo chown -R mythtv:mythtv /var/lib/mythtv
changed the ownership back to the mythtv user, and it seemed happy again.
Bringing the previous database over was a painless process as well. I followed the instructions here and here and the new database came over without drama.
So far it all looks good. I may have a tweak of the VDPAU settings to see if playback can be improved any - it looks fine, but the settings have just been carried over from the previous 0.23 version. Things may have changed in the current 0.25 release.
Now that the system is updated to 12.04, which is a Long Term Support (LTS) release of Ubuntu, I won't be caught out with an obsolete version after 18 months like I was with 10.10.
As a side note, one of the main reasons for moving to 64-bit was because the Zotac IONITX-A-U motherboard, with an Atom 330, was only showing 3GB of RAM installed, even though I had two, 2GB sticks in there. I thought this was a 32 bit software issue. After installing the 64-bit OS, the free memory remained the same. Even in the BIOS, it only reported 3072MB. Strange. A bit of searching around revealed this thread on Zotac's supprt forums. As it turns out, if 4GB is installed, 512MB is reserved for system use, and then there is the amount used by the onboard Nvidia ION graphics - which I had set to the maximum allowed 512MB. So only 3GB visible. It's OK, it still runs fine, but it would have been nice to be able to use a little extra.
So a clean install it was. To prepare, I copied a number of setup files to a safe place that wouldn't be overwritten during the install. Files like:
- mdadm.conf, from /etc/mdadm, which had the RAID configuration information.
- xorg.conf, from /etc/X11, which had the custom modeline used to output to my (now ancient) CRT TV.
- ssmtp.conf, from /etc/ssmtp, for email notification in the case of errors.
- apcupsd.conf, from /etc/apcupsd, the setup of the APC UPS monitoring software.
- hardware.conf, from /etc/lirc, used by LIRC to set up the Media Centre remote control.
- the MythTV database password, from /etc/mythtv/config.xml
Not all the files would end up being needed, but I felt safer having them handy, just in case.
The installation process started from a USB stick loaded with the Mythbuntu 12.04 installer, since I don't have an optical drive in it. No room, with three drives already there. I also hooked up a monitor instead of a TV, because that needs extra setup to output to a component connection.
Since there was some left over free space on the SSD, I installed to that, leaving the 10.10 install intact for now. It went remarkably smoothly. One thing I noted was that at the beginning, there is an option to update the installer. I clicked that, and the system seemed to do nothing for quite a while. Opening up a terminal window and running "top" showed that it was in fact doing something, so I waited until it finished. Once done, the installation process could begin. It would have been nice for it to have some sort of indication that it was doing something, though.
Partway through the process, it prompted for what sort of remote control is being used. I selected "Windows Media Center remote", and the thing was set up automatically. Brilliant! Didn't need that LIRC setup file listed above.
To get the system outputting to my TV properly, I just copied the old xorg.conf file to the /etc/X11 directory (I renamed the existing file, in case I needed it again) and rebooted with a sense of hope and anxiety. A minute or so later, the screen came to life! It worked!
The big one, that I was dreading, was getting the RAID array to be seen by the new install. I still consider myself quite the mdadm novice, so I was pleasantly surprised to find that the RAID array was picked up by the new install quite easily - after installing mdadm and lvm2. This command:
sudo mdadm --assemble --scan
was all that was needed for it to scan for any drives, re-assemble the array, and write a new mdadm.conf file. Easy!
Next was seeing if LVM could find the volumes created. Running
sudo lvm pvscan
sudo lvm vgscan
sudo lvm lvscan
detected all the physical volumes, volume groups, and logical volumes, respectively. Mounting the logical volume that held the recordings was successful - I could see them all. Adding the device entry to /etc/fstab seemed to work, eventually. On the first reboot it came up with an error message, mentioning there were serious problems with the mount. Not sure what it was, but a reboot afterwards seemed to make it happy. I had installed xfsprogs, to manage and defragment XFS filesystems; maybe that helped.
Going through some of the MythTV screens brought up some error messages that it couldn't write to the drive - it turned out to be a permissions issue for the files and directories on the array. Entering
sudo chown -R mythtv:mythtv /var/lib/mythtv
changed the ownership back to the mythtv user, and it seemed happy again.
Bringing the previous database over was a painless process as well. I followed the instructions here and here and the new database came over without drama.
So far it all looks good. I may have a tweak of the VDPAU settings to see if playback can be improved any - it looks fine, but the settings have just been carried over from the previous 0.23 version. Things may have changed in the current 0.25 release.
Now that the system is updated to 12.04, which is a Long Term Support (LTS) release of Ubuntu, I won't be caught out with an obsolete version after 18 months like I was with 10.10.
As a side note, one of the main reasons for moving to 64-bit was because the Zotac IONITX-A-U motherboard, with an Atom 330, was only showing 3GB of RAM installed, even though I had two, 2GB sticks in there. I thought this was a 32 bit software issue. After installing the 64-bit OS, the free memory remained the same. Even in the BIOS, it only reported 3072MB. Strange. A bit of searching around revealed this thread on Zotac's supprt forums. As it turns out, if 4GB is installed, 512MB is reserved for system use, and then there is the amount used by the onboard Nvidia ION graphics - which I had set to the maximum allowed 512MB. So only 3GB visible. It's OK, it still runs fine, but it would have been nice to be able to use a little extra.
My MythTV system - the story so far
For the past 18 moths or so I've been running a MythTV system in the lounge, which has been an excellent performer. It has liberated us from TV advertisements and from having to be home in time to watch a particular show. It has been slowly enhanced and upgraded over that time to be quite a capable machine, exceeding what could be purchased from a store.
Following on from one of the earlier build posts, one thing done was purchasing a larger wireless antenna. For under $20, it allowed the antenna to be moved to a position with better reception to the wireless router, and with a stronger signal. There have been very few connection dropouts since putting that in. The eventual goal is to have a hard-wired ethernet connection, but that will be some time down the track if I ever get the house wired up for it.
Another issue was with a hard drive upgrade. Originally with the 2TB Samsung disk, I upgraded it to a Hitachi 5K3000 3TB drive. It ran well for about three months, but then I came down to the lounge one morning and the system wasn't happy. There were drive errors of one kind or another. I grabbed the 2TB Samsung drive again (which had been serving a happy retirement in my desktop PC), and luckily was able to copy all the recordings over to it. It was fortunate that I hadn't filled the extra space either.
So the system was back to its original storage drive, until the warranty fix came back from Hitachi. About three weeks later, a replacecment drive showed up - this time it was a 3TB 7K3000, the next model up. Also, since this happened after the Thailand floods, one of those drives was now worth nearly $400, quite a bit different from the ~$160 I paid.
Now I had a refurbished 3TB drive and a 2TB drive with a year's heavy usage already on it. This made me a little nervous. It was time to move to a RAID1 system. I used the linux mdadm software RAID, with some LVM volumes on top of it. It is a while back now, but the install went OK I can't quite remember the tutorial I used, but these two have quite a lot of information on doing so. The performance hit from running RAID and LVM was not noticeable on the system, either. If desired, you can also set up email notifications if a drive fails or some other problem arises with the array - see here, here and here. One thing to note if you use gmail to send the notifications: if your password has a "#"in it, ssmtp will read the rest of the line as a comment and you'll get all sorts of strange errors. Oh yes, it's best to set up a secondary gmail account to do this, with nothing important in it, since you have to keep the password in a text file.
Since the system worked well, I didn't bother with updating it. It wasn't open to the wider internet (to access via MythWeb). But it was time for a change, and Mythbuntu 10.10 was no longer supported. It was time for an upgrade to a newer version - which will be detailed in another post.
Following on from one of the earlier build posts, one thing done was purchasing a larger wireless antenna. For under $20, it allowed the antenna to be moved to a position with better reception to the wireless router, and with a stronger signal. There have been very few connection dropouts since putting that in. The eventual goal is to have a hard-wired ethernet connection, but that will be some time down the track if I ever get the house wired up for it.
Another issue was with a hard drive upgrade. Originally with the 2TB Samsung disk, I upgraded it to a Hitachi 5K3000 3TB drive. It ran well for about three months, but then I came down to the lounge one morning and the system wasn't happy. There were drive errors of one kind or another. I grabbed the 2TB Samsung drive again (which had been serving a happy retirement in my desktop PC), and luckily was able to copy all the recordings over to it. It was fortunate that I hadn't filled the extra space either.
So the system was back to its original storage drive, until the warranty fix came back from Hitachi. About three weeks later, a replacecment drive showed up - this time it was a 3TB 7K3000, the next model up. Also, since this happened after the Thailand floods, one of those drives was now worth nearly $400, quite a bit different from the ~$160 I paid.
Now I had a refurbished 3TB drive and a 2TB drive with a year's heavy usage already on it. This made me a little nervous. It was time to move to a RAID1 system. I used the linux mdadm software RAID, with some LVM volumes on top of it. It is a while back now, but the install went OK I can't quite remember the tutorial I used, but these two have quite a lot of information on doing so. The performance hit from running RAID and LVM was not noticeable on the system, either. If desired, you can also set up email notifications if a drive fails or some other problem arises with the array - see here, here and here. One thing to note if you use gmail to send the notifications: if your password has a "#"in it, ssmtp will read the rest of the line as a comment and you'll get all sorts of strange errors. Oh yes, it's best to set up a secondary gmail account to do this, with nothing important in it, since you have to keep the password in a text file.
Since the system worked well, I didn't bother with updating it. It wasn't open to the wider internet (to access via MythWeb). But it was time for a change, and Mythbuntu 10.10 was no longer supported. It was time for an upgrade to a newer version - which will be detailed in another post.
Thursday, March 17, 2011
Using a solid state drive in a MythTV box - does it make a difference?
The other day I finally got around to installing a solid-state hard drive (a 40GB Intel X25-V) into my MythTV box, to see what sort of difference it would make in day-to-day running. Of course, I only have the root partition on it, no recordings. I still have the 2TB drive for them. What prompted the move was the occasional sluggishness to respond when going through menus, and the odd pause or stutter when hitting the "i" or Info button on the remote. It would also sometimes get a brief pause when it popped up the notice that a commercial break was about to start.
I figured that it was down to too much being asked of the main storage drive - asking it to respond snappily when it is recording four or five episodes and playing back another is just too much. What if there was a second drive that could take care of the actual running of the machine, just leaving the big drive as a bit bucket to store media on?
Taking the Intel SSD from my main desktop (I had bought a 60GB OCZ Vertex to replace it), there was a spot ready for it in the Lian Li PC-Q07 case - right under the main drive. It wouldn't get a whole lot of air, so I'm glad it's a cool-running SSD. I copied the root partition from the 2TB drive onto it, as well as creating a swap partition. Partition sizes were kept down - 16GB for the root partition, 3GB for swap, leaving more than half free for the drive to use for wear-levelling. Overkill, but Mythbuntu doesn't need much room for its base installation. Eventually I got it working and booting from the ssd (after much swearing - how I did it may be covered in a future post) and I could check out the results.
And has it made a difference? A small one, but it's noticeable. Going through menu screens is a little quicker, more responsive. During playback, hitting the info button brings up the playback time without any stutters. Ending playback brings you back to the menu quicker. It's made the interface seem that little more polished. I'd say that it's not the fact there's a solid-state drive that has made the difference, more the fact that there is a second disk that's able to take some of the load off the storage one. I am not able to test it with another type of drive (frankly I can't be stuffed), but there are advantages in power consumption in going for an SSD.
I figured that it was down to too much being asked of the main storage drive - asking it to respond snappily when it is recording four or five episodes and playing back another is just too much. What if there was a second drive that could take care of the actual running of the machine, just leaving the big drive as a bit bucket to store media on?
Taking the Intel SSD from my main desktop (I had bought a 60GB OCZ Vertex to replace it), there was a spot ready for it in the Lian Li PC-Q07 case - right under the main drive. It wouldn't get a whole lot of air, so I'm glad it's a cool-running SSD. I copied the root partition from the 2TB drive onto it, as well as creating a swap partition. Partition sizes were kept down - 16GB for the root partition, 3GB for swap, leaving more than half free for the drive to use for wear-levelling. Overkill, but Mythbuntu doesn't need much room for its base installation. Eventually I got it working and booting from the ssd (after much swearing - how I did it may be covered in a future post) and I could check out the results.
And has it made a difference? A small one, but it's noticeable. Going through menu screens is a little quicker, more responsive. During playback, hitting the info button brings up the playback time without any stutters. Ending playback brings you back to the menu quicker. It's made the interface seem that little more polished. I'd say that it's not the fact there's a solid-state drive that has made the difference, more the fact that there is a second disk that's able to take some of the load off the storage one. I am not able to test it with another type of drive (frankly I can't be stuffed), but there are advantages in power consumption in going for an SSD.
Tuesday, January 11, 2011
A MythTV box for the lounge room: Part 3: The build
In part 2 of this series, I went over the decision process for choosing the hardware used in the MythTV box. To sum up, I went with a Zotac IONITX-A, due to the low-power Intel Atom/Nvidia ION CPU/graphics combination, inbuilt wireless, and an included laptop-style power supply. I could also use some of the DDR2 RAM sticks that I already had.
Other items bought for the box included two more ASUS U3100 mini USB TV tuners. These supplemented the one I already had (sadly I couldn't use the PCI tuner card I had in my main PC, due to the board not having any PCI slots). I picked up a 2 terabyte Samsung ecogreen hard drive as the system and storage disk. These have had good reports of being low noise, low power consumption items.
The remote control chosen was a Hauppauge unit, as described in an earlier post.
The choice of case was driven by stock availability more than anything - initially I had chosen a Mini-ITX case from Aywun (known as Apex in some countries) but none would be arriving in stock until mid January - so I still would have been waiting. Since I wanted this to be completed as a Christmas holiday project, I looked elsewhere and found the Lian Li PC-Q07 - a rather minimal-looking aluminium case. It was a tad larger than the spot I had originally planned to put the box, but I was able to do some re-arranging inside the cabinet (like removing the video recorder - won't be needing that relic any more!) and it fits nicely.
Observations from assembly
One thing noticed when putting the various parts together is that the TV tuners would not fit together when pugged into adjacent USB slots at the back of the motherboard. Their chunky plastic bodies meant you could not plug much into a neighbouring slot. This was solved by picking up some USB extension cables and hanging the tuners off those. Using the cables also meant that I could run the tuners out the back of the cabinet into free air, rather than let them warm up the inside of the cabinet. They do get surprisingly warm during use.
To connect them to the aerial, I purchased a three-way digital TV splitter. I was concerned about losing too much signal with three plugged in, but in use the strength has been around 66-70%, enough to give trouble-free viewing.
The PC-Q07 case has nowhere inside it to allow for additional fan mounting, but it was designed to take full-sized PC components, like normal ATX power supplies, full-sized DVD drives and the like. Since there was no need for a PSU to be put in it (the motherboard I chose had its own laptop-style power brick that could sit out behind the cabinet), I rigged up a 92 mm fan in the gap where a power supply would normally go. It is set up so it is drawing in air from behind the case, supplying a stream of fresh air to the other, smaller fan on the CPU/GPU, as well as the hard drive. I just used one of the power supply mounting holes - it seems to be holding in place just fine. To stop air short-circuiting too much straight out the back, I cut up a small piece of cardboard and used that to blank off the remainder of the gap in the power supply hole.
To get a better supply of air to the box, I borrowed a holesaw, attached it to my drill, and cut a couple of extra air holes in the back of the TV cabinet. This also provided some extra paths to lead cables from.
The infrared sensor from the remote was on a fairly long cable, so I was able to lead that out the back of the cabinet to a spot that is higher up, easier for the remote to reach.
Finally, I've left the wireless receiver for the keyboard and mouse plugged into the back for convenience - I can just bring them down from the other room if some typing or clicking is needed.
Findings from the first weeks of use
First of all, and the biggest relief, is that the Wife Acceptance Factor (WAF) is very high - she loves it. We both love having the power to watch what we want, when we want, being able to skip the ads if we've recorded a show earlier. Playback is flawless - any glitches we have aren't any worse than what we previously encountered with our HD set top box. It can handle 1080i playback perfectly smoothly. The only stutters are related to the hard drive when it is working particularly hard, like the other night when we found ourselves in the situation of recording four shows simultaneously (two were on different sub-channels on the same network, so one tuner could record them both) while watching a recording of a previous program. You could see the hard drive light blinking away pretty furiously. Oh yes, it would have also been flagging commercials at the same time. All these things can be tweaked and tuned, I've set it to only run one job at a time to give the little atom CPU some breathing room. Commercial flagging has been set to begin as soon as recording starts - I changed that from the default of waiting until a show has finished recording. So far, it hasn't slowed things down noticeably.
For the playback settings in MythTV, I have chosen the VDPAU Normal profile. I haven't tried the High Quality setting yet, the GPU may struggle with that, and Normal looks just fine to me.
Mythbuntu:
I decided to used the MythTV-oriented version of Ubuntu for this project, Mythbuntu. This runs the XFCE desktop, using less resources than Gnome would. Setup was quite simple, and I was pleased to find that Mythweb was installed by default - something I had never bothered with before on my previous desktop PC Myth install.
I would recommend this to anyone else considering a dedicated HTPC build. It is quite stripped back, however, so it is not much use for other purposes (although there is a facility to enable it to be reconfigured as a regular desktop through the Mythbuntu control panel).
CPU Use:
One CPU-hungry process I have noted is upconverting the audio stream to 5.1 Dolby Digital. Remotely logging in and checking CPU use during playback shows that with regular stereo passthrough to the optical output, usage is around 12-14%. With upconversion enabled at best quality, CPU use shoots up to around 63%.
Memory:
Memory-wise, I have followed recommendations posted elsewhere on the web and run two sticks of RAM, enabling dual-channel operation. This provides better bandwidth for the GPU in particular, improving playback performance.
I have two, one-gigabyte sticks of DDR2-800 ram installed, with the graphics set in BIOS to use the maximum 512MB. This has been quite sufficient, and I dare say that 1 GB would be enough if it weren't for the onboard graphics using some of the memory.
Power consumption:
I have a power meter (discussed in an earlier post on my other blog) connected to the wall outlet, measuring the power use of the PC, as well as the VGA-component transcoder. When idle, the unit consumes 44 Watts. Watching live TV, it uses 47 Watts. When 5.1 upconversion was enabled, it used 49 Watts, reflecting the extra CPU usage.
These figures are quite a bit higher than what I've seen elsewhere on the web - I saw a picture of a system playing a blu-ray disc using 30W. And this machine doesn't have an optical drive installed. All I can think of is the extra power draw of the TV tuners, the fan, and the fact that linux may not make the best use of power saving features of the CPU compared with windows 7. The other factor could be the power meter being out, something I'm finding quite likely. It does give pretty high voltage readings, which would as a result give an inflated Watt reading.
Final thoughts:
I'm glad the system build went off without a hitch. Performance is fine, although had I gone with a core i3 system it could have handled many more concurrent jobs like commercial flagging without breaking a sweat. Power use would have been higher, too, as too the cost.
The wireless signal is not the best, with the antenna located at the back of the motherboard hidden away at the back of the TV unit. I may try an external antenna, to let it get a better signal than the 40-ish percent it now gets. It makes copying movie files pretty slow, the way it is at the moment.
I'm wondering whether to put in a solid-state hard drive to have the operating system partition located on, to free up the workload of the storage drive. There is sometimes a delay of a few seconds when starting playback of a show. This isn't a major priority - it will probably inherit the Intel X25-V from my main system whenever I upgrade that.
Other items bought for the box included two more ASUS U3100 mini USB TV tuners. These supplemented the one I already had (sadly I couldn't use the PCI tuner card I had in my main PC, due to the board not having any PCI slots). I picked up a 2 terabyte Samsung ecogreen hard drive as the system and storage disk. These have had good reports of being low noise, low power consumption items.
The remote control chosen was a Hauppauge unit, as described in an earlier post.
The choice of case was driven by stock availability more than anything - initially I had chosen a Mini-ITX case from Aywun (known as Apex in some countries) but none would be arriving in stock until mid January - so I still would have been waiting. Since I wanted this to be completed as a Christmas holiday project, I looked elsewhere and found the Lian Li PC-Q07 - a rather minimal-looking aluminium case. It was a tad larger than the spot I had originally planned to put the box, but I was able to do some re-arranging inside the cabinet (like removing the video recorder - won't be needing that relic any more!) and it fits nicely.
Observations from assembly
One thing noticed when putting the various parts together is that the TV tuners would not fit together when pugged into adjacent USB slots at the back of the motherboard. Their chunky plastic bodies meant you could not plug much into a neighbouring slot. This was solved by picking up some USB extension cables and hanging the tuners off those. Using the cables also meant that I could run the tuners out the back of the cabinet into free air, rather than let them warm up the inside of the cabinet. They do get surprisingly warm during use.
To connect them to the aerial, I purchased a three-way digital TV splitter. I was concerned about losing too much signal with three plugged in, but in use the strength has been around 66-70%, enough to give trouble-free viewing.
The PC-Q07 case has nowhere inside it to allow for additional fan mounting, but it was designed to take full-sized PC components, like normal ATX power supplies, full-sized DVD drives and the like. Since there was no need for a PSU to be put in it (the motherboard I chose had its own laptop-style power brick that could sit out behind the cabinet), I rigged up a 92 mm fan in the gap where a power supply would normally go. It is set up so it is drawing in air from behind the case, supplying a stream of fresh air to the other, smaller fan on the CPU/GPU, as well as the hard drive. I just used one of the power supply mounting holes - it seems to be holding in place just fine. To stop air short-circuiting too much straight out the back, I cut up a small piece of cardboard and used that to blank off the remainder of the gap in the power supply hole.
To get a better supply of air to the box, I borrowed a holesaw, attached it to my drill, and cut a couple of extra air holes in the back of the TV cabinet. This also provided some extra paths to lead cables from.
The infrared sensor from the remote was on a fairly long cable, so I was able to lead that out the back of the cabinet to a spot that is higher up, easier for the remote to reach.
Finally, I've left the wireless receiver for the keyboard and mouse plugged into the back for convenience - I can just bring them down from the other room if some typing or clicking is needed.
Findings from the first weeks of use
First of all, and the biggest relief, is that the Wife Acceptance Factor (WAF) is very high - she loves it. We both love having the power to watch what we want, when we want, being able to skip the ads if we've recorded a show earlier. Playback is flawless - any glitches we have aren't any worse than what we previously encountered with our HD set top box. It can handle 1080i playback perfectly smoothly. The only stutters are related to the hard drive when it is working particularly hard, like the other night when we found ourselves in the situation of recording four shows simultaneously (two were on different sub-channels on the same network, so one tuner could record them both) while watching a recording of a previous program. You could see the hard drive light blinking away pretty furiously. Oh yes, it would have also been flagging commercials at the same time. All these things can be tweaked and tuned, I've set it to only run one job at a time to give the little atom CPU some breathing room. Commercial flagging has been set to begin as soon as recording starts - I changed that from the default of waiting until a show has finished recording. So far, it hasn't slowed things down noticeably.
For the playback settings in MythTV, I have chosen the VDPAU Normal profile. I haven't tried the High Quality setting yet, the GPU may struggle with that, and Normal looks just fine to me.
Mythbuntu:
I decided to used the MythTV-oriented version of Ubuntu for this project, Mythbuntu. This runs the XFCE desktop, using less resources than Gnome would. Setup was quite simple, and I was pleased to find that Mythweb was installed by default - something I had never bothered with before on my previous desktop PC Myth install.
I would recommend this to anyone else considering a dedicated HTPC build. It is quite stripped back, however, so it is not much use for other purposes (although there is a facility to enable it to be reconfigured as a regular desktop through the Mythbuntu control panel).
CPU Use:
One CPU-hungry process I have noted is upconverting the audio stream to 5.1 Dolby Digital. Remotely logging in and checking CPU use during playback shows that with regular stereo passthrough to the optical output, usage is around 12-14%. With upconversion enabled at best quality, CPU use shoots up to around 63%.
Memory:
Memory-wise, I have followed recommendations posted elsewhere on the web and run two sticks of RAM, enabling dual-channel operation. This provides better bandwidth for the GPU in particular, improving playback performance.
I have two, one-gigabyte sticks of DDR2-800 ram installed, with the graphics set in BIOS to use the maximum 512MB. This has been quite sufficient, and I dare say that 1 GB would be enough if it weren't for the onboard graphics using some of the memory.
Power consumption:
I have a power meter (discussed in an earlier post on my other blog) connected to the wall outlet, measuring the power use of the PC, as well as the VGA-component transcoder. When idle, the unit consumes 44 Watts. Watching live TV, it uses 47 Watts. When 5.1 upconversion was enabled, it used 49 Watts, reflecting the extra CPU usage.
These figures are quite a bit higher than what I've seen elsewhere on the web - I saw a picture of a system playing a blu-ray disc using 30W. And this machine doesn't have an optical drive installed. All I can think of is the extra power draw of the TV tuners, the fan, and the fact that linux may not make the best use of power saving features of the CPU compared with windows 7. The other factor could be the power meter being out, something I'm finding quite likely. It does give pretty high voltage readings, which would as a result give an inflated Watt reading.
Final thoughts:
I'm glad the system build went off without a hitch. Performance is fine, although had I gone with a core i3 system it could have handled many more concurrent jobs like commercial flagging without breaking a sweat. Power use would have been higher, too, as too the cost.
The wireless signal is not the best, with the antenna located at the back of the motherboard hidden away at the back of the TV unit. I may try an external antenna, to let it get a better signal than the 40-ish percent it now gets. It makes copying movie files pretty slow, the way it is at the moment.
I'm wondering whether to put in a solid-state hard drive to have the operating system partition located on, to free up the workload of the storage drive. There is sometimes a delay of a few seconds when starting playback of a show. This isn't a major priority - it will probably inherit the Intel X25-V from my main system whenever I upgrade that.
Tuesday, January 4, 2011
A MythTV box for the lounge room: Part 2: Choosing the box to build
Making sure that I could get something displaying on my TV was the main stumbling block with the build of the MythTV PC. Since it was overcome in my last post, I was now free to get on with speccing up a machine. The major requirements for this were:
Option 1 - Mini-ITX, using an Intel Atom/Nvidia ION board
Motherboard: Zotac IONITX-A
CPU: Intel Atom 330 Dual core 1.6GHz
Graphics: Onboard Nvidia 9400
DDR2 ram slots
Onboard Wireless-N adapter
90W Power supply included (laptop-style)
Case: Lian Li PC-Q07
Pros:
Lowest power usage of all options
Everything integerated on the board
Power supply included, and can be located away from PC case - less heat
Small size will fit into TV cabinet
Can use existing DDR2 RAM I already have
Cons:
Not useful for much else if this project doesn't work out
Not the cheapest option
Option 2 - Separate CPU/Motherboard, budget option
Motherboard: Gigabyte G41M-combo
CPU: Intel Pentium E5400 2.7GHz
Graphics: Nvidia GT210
DDR2 ram slots
Case: Antec NSK3480 (with PSU)
Pros:
Better performance than Atom CPU
Can be used as general purpose PC later
Cheaper than option 1
Can use existing DDR2 RAM I already have
Cons:
More power use than option 1
Will need an S/PDIF bracket for optical out
Takes up more room with full-size case
Need separate Wifi card
Option 3 - Separate CPU/Motherboard - higher end
Motherboard: Gigabyte GA-H55N-USB3 Mini-ITX
CPU: Intel Core i3-540
Graphics: Nvidia GT210
DDR3 RAM slots
Case: Lian Li PC-Q07
PSU: Corsair VX450
Pros:
Best performance of all, no problems handling all jobs, even concurrent ones
Good power efficiency
Same small form factor as ION
Cons:
More power use than option 1 (maybe not much more)
Need separate Wifi card
Lots more expensive than other options
Newer CPUS now out (Sandy Bridge)
Of all these options, I went with Option 1 - the Atom/ION combination. It had the best compromise of the three, with the lowest power usage. That is important to me, as I wanted to reduce the ongoing costs as much as possible. That's what eliminated option 2 as well, along with the fact that a larger case would have to sit behind the TV cabinet, in amongst the already messy tangle of cables. Option 3 was the most attractive performance-wise, but was really getting quite expensive compared to the others, and getting a little hard to justify the price for something that is just for watching TV, after all.
In the next part, I will go over the build of the box, setting it up and what my experiences have been with it.
- It had to use a minimal amount of power, as it would likely be running 24/7 (the nvidia binary driver means suspend doesn't usually work).
- It had to be able to handle 1080i playback of recorded or live TV, flawlessly
- It needed to have an S/PDIF output, either optical or coaxial to connect to my existing receiver
- It needed a wireless network connection, as it was at the opposite end of the house to where my router was.
- The overall cost had to be similar to the price of buying a DVR + IceTV subscription
Option 1 - Mini-ITX, using an Intel Atom/Nvidia ION board
Motherboard: Zotac IONITX-A
CPU: Intel Atom 330 Dual core 1.6GHz
Graphics: Onboard Nvidia 9400
DDR2 ram slots
Onboard Wireless-N adapter
90W Power supply included (laptop-style)
Case: Lian Li PC-Q07
Pros:
Lowest power usage of all options
Everything integerated on the board
Power supply included, and can be located away from PC case - less heat
Small size will fit into TV cabinet
Can use existing DDR2 RAM I already have
Cons:
Not useful for much else if this project doesn't work out
Not the cheapest option
Option 2 - Separate CPU/Motherboard, budget option
Motherboard: Gigabyte G41M-combo
CPU: Intel Pentium E5400 2.7GHz
Graphics: Nvidia GT210
DDR2 ram slots
Case: Antec NSK3480 (with PSU)
Pros:
Better performance than Atom CPU
Can be used as general purpose PC later
Cheaper than option 1
Can use existing DDR2 RAM I already have
Cons:
More power use than option 1
Will need an S/PDIF bracket for optical out
Takes up more room with full-size case
Need separate Wifi card
Option 3 - Separate CPU/Motherboard - higher end
Motherboard: Gigabyte GA-H55N-USB3 Mini-ITX
CPU: Intel Core i3-540
Graphics: Nvidia GT210
DDR3 RAM slots
Case: Lian Li PC-Q07
PSU: Corsair VX450
Pros:
Best performance of all, no problems handling all jobs, even concurrent ones
Good power efficiency
Same small form factor as ION
Cons:
More power use than option 1 (maybe not much more)
Need separate Wifi card
Lots more expensive than other options
Newer CPUS now out (Sandy Bridge)
Of all these options, I went with Option 1 - the Atom/ION combination. It had the best compromise of the three, with the lowest power usage. That is important to me, as I wanted to reduce the ongoing costs as much as possible. That's what eliminated option 2 as well, along with the fact that a larger case would have to sit behind the TV cabinet, in amongst the already messy tangle of cables. Option 3 was the most attractive performance-wise, but was really getting quite expensive compared to the others, and getting a little hard to justify the price for something that is just for watching TV, after all.
In the next part, I will go over the build of the box, setting it up and what my experiences have been with it.
Tuesday, December 28, 2010
A MythTV box for the lounge room - Part 1: getting the display to work
The first hurdle:
After a number of years trying to get a picture on my extremely fussy lounge room TV from the computer, I finally bit the bullet and ordered a VGA to component transcoder - an Audio Authority 9A60A. This is because I am still using a Panasonic TX-76PW200A CRT TV, with only component inputs. Also, since it is not in the USA, the refresh rate is 50Hz,rather than the 60Hz that the nvidia drivers support out of the box. I mentioned in an earlier post that I was going to have a task getting a working resolution to the TV, but eventually something was found. Initially I used the program Powerstrip, with my desktop system set up in the lounge. A couple of settings worked, but with pretty bad overscan. Every time I tried tweaking it a little, the display just went out of range and I was greeted with a blank screen. Disheartened, I packed up the PC and contemplated putting the transcoder on ebay.
After a couple of days' break, i thought about trying again, this time with my laptop that runs linux and has a Mobility radeon graphics chipset. Using the xrandr command, I was able to feed it a modeline that I had found on another forum. An owner of the same TV I had, had managed to get a good picture up after a bit of experimenting. Lo and behold, I was able to get a picture up!
From memory, the xrandr commands used for the laptop were:
That created a new mode for the display to use.
This assigns the mode to the display device - in this case, it was to the VGA external output for the monitor.
And that final command sent the 1920x1080 resolution to the VGA output - where the ubuntu desktop came up on the TV, much to my relief!
Knowing that the TV was capable of accepting the resolution, and that the transcoder was able to output the correct timings to the TV, it was back to the main box to see how the output worked from an Nvidia card.
The Nvidia binary driver unfortunately does not accept the xrandr commands I had used with the radeon drivers on the laptop, so another method was going to have to be used. After some inital lack of success with manually editing the xorg.conf file, I had a go using nvidia's utility designed for adding things to the file, nvidia-xconfig. Using the command:
added a mode to the xorg.conf file. I opened that file up in a text editor and had a look where it turned up. When I found it, I added the following line straght below it:
I restarted the machine to let nvidia-settings find the modeline. From there, it was then selectable from the resolution dropdown box, where I could choose it. It worked again!
The final step was getting it to work when the TV was the only display hooked up. To summarise, the xorg.conf file needed to be edited to remove all references to the PC monitor I had connected as well, so the TV was the only device it detected. Have a look at the xorg.conf file here, for your reference.
One other thing I had to do was set the PC to log in automatically - since this is to be a Home Theatre PC, there will not be a keyboard connected to allow typing in a password. It's not what I was after anyway. This option can be found under System -> Administration -> Login Screen
In a later entry I'll go over the setup of the actual system, and what I've learned in its first week of use.
After a number of years trying to get a picture on my extremely fussy lounge room TV from the computer, I finally bit the bullet and ordered a VGA to component transcoder - an Audio Authority 9A60A. This is because I am still using a Panasonic TX-76PW200A CRT TV, with only component inputs. Also, since it is not in the USA, the refresh rate is 50Hz,rather than the 60Hz that the nvidia drivers support out of the box. I mentioned in an earlier post that I was going to have a task getting a working resolution to the TV, but eventually something was found. Initially I used the program Powerstrip, with my desktop system set up in the lounge. A couple of settings worked, but with pretty bad overscan. Every time I tried tweaking it a little, the display just went out of range and I was greeted with a blank screen. Disheartened, I packed up the PC and contemplated putting the transcoder on ebay.
After a couple of days' break, i thought about trying again, this time with my laptop that runs linux and has a Mobility radeon graphics chipset. Using the xrandr command, I was able to feed it a modeline that I had found on another forum. An owner of the same TV I had, had managed to get a good picture up after a bit of experimenting. Lo and behold, I was able to get a picture up!
From memory, the xrandr commands used for the laptop were:
xrandr --newmode "1920x1080i_50" 78.300 1920 2464 2520 2784 1080 1102 1117 1125 interlace +hsync +vsync
That created a new mode for the display to use.
xrandr --addmode VGA-0 1920x1080i_50
This assigns the mode to the display device - in this case, it was to the VGA external output for the monitor.
xrandr --output VGA-0 --mode 1920x1080i_50
And that final command sent the 1920x1080 resolution to the VGA output - where the ubuntu desktop came up on the TV, much to my relief!
Knowing that the TV was capable of accepting the resolution, and that the transcoder was able to output the correct timings to the TV, it was back to the main box to see how the output worked from an Nvidia card.
The Nvidia binary driver unfortunately does not accept the xrandr commands I had used with the radeon drivers on the laptop, so another method was going to have to be used. After some inital lack of success with manually editing the xorg.conf file, I had a go using nvidia's utility designed for adding things to the file, nvidia-xconfig. Using the command:
sudo nvidia-xconfig --mode=1920x1080i_50
added a mode to the xorg.conf file. I opened that file up in a text editor and had a look where it turned up. When I found it, I added the following line straght below it:
Modeline "1920x1080i_50" 78.300 1920 2464 2520 2784 1080 1102 1117 1125 interlace +hsync +vsync
I restarted the machine to let nvidia-settings find the modeline. From there, it was then selectable from the resolution dropdown box, where I could choose it. It worked again!
The final step was getting it to work when the TV was the only display hooked up. To summarise, the xorg.conf file needed to be edited to remove all references to the PC monitor I had connected as well, so the TV was the only device it detected. Have a look at the xorg.conf file here, for your reference.
One other thing I had to do was set the PC to log in automatically - since this is to be a Home Theatre PC, there will not be a keyboard connected to allow typing in a password. It's not what I was after anyway. This option can be found under System -> Administration -> Login Screen
In a later entry I'll go over the setup of the actual system, and what I've learned in its first week of use.
Sunday, November 28, 2010
Getting a media centre remote working with LIRC, in Ubuntu 10.10
Recently I bought a new remote control for my PC, to replace the one that came with my TV tuner card. I picked up a Hauppauge Media Center Remote Control Kit, model 335, which has more logically labelled buttons than what I had. It looks a bit nicer, too, an important consideration as I am attempting to finally build a Home Theatre PC to go in the lounge room.
While the previous Technisat remote worked fine in Ubuntu 10.10, I read of some other people having problems with getting remotes to work - namely the Media Centre-compatible versions. When I plugged in my new one, I also noticed the problem.
A bit of trawling around on the internet uncovered a thread at the ever-reliable Ubuntu forums. It turns out that the newer kernel includes a module for the media centre remote (mceusb), that clashes with the lirc_mceusb module you want to use, with LIRC. When the two modules are loaded, neither works.
What has to be done, is make sure only one of them gets loaded. The user uniden9 posted an excellent how-to in that thread linked above, describing two methods that are available - disabling the old lirc one, or disabling the newer, kernel built-in module. Based on other responses in the thread, most people had more luck with the second method - disabling the newer one. I guess the MythTV key bindings etc. would work better with the old version.
I have included the relevant quote from uniden9's post here, in case there is a problem accessing the forums:
I used this method as well, and now have the media centre remote working just fine on my system. I'm now just waiting on the arrival of a VGA to Component Transcoder, that will hopefully allow my PC to connect with my older-style TV that has only component inputs. As an aside, although I have a graphics card that has a component-out attachment, it only puts out a 1080i, 60Hz signal. My TV, being made for the Australian, PAL-based market, requires a 1080i, 50Hz input signal. Fingers are crossed that the transcoder will allow it to work - I'll be posting an update when it arrives and I can try it out.
While the previous Technisat remote worked fine in Ubuntu 10.10, I read of some other people having problems with getting remotes to work - namely the Media Centre-compatible versions. When I plugged in my new one, I also noticed the problem.
A bit of trawling around on the internet uncovered a thread at the ever-reliable Ubuntu forums. It turns out that the newer kernel includes a module for the media centre remote (mceusb), that clashes with the lirc_mceusb module you want to use, with LIRC. When the two modules are loaded, neither works.
What has to be done, is make sure only one of them gets loaded. The user uniden9 posted an excellent how-to in that thread linked above, describing two methods that are available - disabling the old lirc one, or disabling the newer, kernel built-in module. Based on other responses in the thread, most people had more luck with the second method - disabling the newer one. I guess the MythTV key bindings etc. would work better with the old version.
I have included the relevant quote from uniden9's post here, in case there is a problem accessing the forums:
2. Blacklist the kernel source drivers and use the old dkms built drivers.
-Well you already have lirc-sources-modules install and probably had this working on 10.04 or older version of ubuntu.
A. Edit /etc/modprobe.d/blacklist.conf , create it if it doesn't exist.
Add>
blacklist mceusb
line to the file and save it.
B. Update initramfs, #sudo update-initramfs -u -k all
C. reboot
D. Reconfigure lirc if its not already working.
#sudo dpkg-reconfigure lirc
On the Remote, Select "Windows Media Center Transceiver/Remote (All)
E. Thats it.
This is the method I chose after wrestling with button repeats on option 1. I use XBMC and could not figure out how to disable Ubuntu treating the remote as a keyboard device.
I used this method as well, and now have the media centre remote working just fine on my system. I'm now just waiting on the arrival of a VGA to Component Transcoder, that will hopefully allow my PC to connect with my older-style TV that has only component inputs. As an aside, although I have a graphics card that has a component-out attachment, it only puts out a 1080i, 60Hz signal. My TV, being made for the Australian, PAL-based market, requires a 1080i, 50Hz input signal. Fingers are crossed that the transcoder will allow it to work - I'll be posting an update when it arrives and I can try it out.
Saturday, October 16, 2010
Upgrade to Maverick Meerkat - version 10.10 worked!
I am writing this from my newly-upgraded Ubuntu 10.10 system! This has been the second time I upgraded the installation, rather than start afresh. I was a little nervous, to be honest, but backups were made before the upgrade so there was an escape plan.
There were only a few issues that were pretty easily fixed, and I may, just may, have fixed the hanging on boot problem I had with version 10.04.
One problem I had was the Hard drive temperature reporting in Conky - it stopped displaying a temperature. This was because the hddtemp command could only be run as root - Conky doesn't run with those privileges, and I didn't want to start doing so. A quick search revealed a single line command to allow hddtemp to be run as a regular user:
After doing that, the disk temperature re-appeared and all was good.
Fixing the boot screen
This was a problem that I had in the previous version, 10.04, as well as in 10.10. It is to do with running the Nvidia proprietary drivers, and switching to different video modes. Basically, there are some steps to try at this site, which I did. It involves setting some video modes in the boot loader. I initially tried 1680x1050 (about the best that was supported by my card, even though it runs at 1920x1080 once at the desktop) and got a comically large ubuntu logo during boot. A later change to 1280x1024 yielded a much better result.
So far, the boot and shutdown screens look much better, and I haven't yet had it hang on me. I am hesitant to call it truly fixed until a few more weeks have passed.
Edit January 7, 2011: If you do these changes and then switch back to the open source driver (nouveau), you have to undo the changes to be able to set the proper resolution for your screen(s).
There were only a few issues that were pretty easily fixed, and I may, just may, have fixed the hanging on boot problem I had with version 10.04.
One problem I had was the Hard drive temperature reporting in Conky - it stopped displaying a temperature. This was because the hddtemp command could only be run as root - Conky doesn't run with those privileges, and I didn't want to start doing so. A quick search revealed a single line command to allow hddtemp to be run as a regular user:
sudo chmod u+s /usr/sbin/hddtemp
After doing that, the disk temperature re-appeared and all was good.
Fixing the boot screen
This was a problem that I had in the previous version, 10.04, as well as in 10.10. It is to do with running the Nvidia proprietary drivers, and switching to different video modes. Basically, there are some steps to try at this site, which I did. It involves setting some video modes in the boot loader. I initially tried 1680x1050 (about the best that was supported by my card, even though it runs at 1920x1080 once at the desktop) and got a comically large ubuntu logo during boot. A later change to 1280x1024 yielded a much better result.
So far, the boot and shutdown screens look much better, and I haven't yet had it hang on me. I am hesitant to call it truly fixed until a few more weeks have passed.
Edit January 7, 2011: If you do these changes and then switch back to the open source driver (nouveau), you have to undo the changes to be able to set the proper resolution for your screen(s).
Thursday, July 1, 2010
Should you use the XFS file system?
One of the good things, and also one of the confusing things about Linux, is the amount of choice you have. Even down to what sort of file system you want to use for storing your data. By default, the Ubuntu installer will use ext4, a good, general purpose file system that will suit most people's needs just fine.
There are other options though, depending on your usage patterns. If you have a lot of large files stored, such as lots of video or movies, maybe even if you record TV with MythTV like I do, a good alternative is XFS. Its design means it is particularly good at handling large files, such as High Definition TV recording, giving good performance in reading and writing them.
You can specify a partition to be in XFS format at the time of installation, or later on by using GParted. Once it is created, there are some further tweaks you can do to make its performance better, and also to maintain its performance over time.
1. Add the following options to the fstab entry for the partition (located in /etc/fstab):
UUID=xxxx /var/lib/mythtv xfs defaults,noatime,allocsize=512m,logbufs=8 0 2
noatime refers to stopping the file system logging each time the file is accessed. Normally it is not needed, and this option reduces the number of writes done to disk, speeding things up.
allocsize=512m lets XFS set aside 512 megabytes of space at a time when it writes. This reduces the chance of the file being split up into many small chunks, or fragmenting. Especially good for recording TV.
logbufs=8 refers to the log buffers held in RAM. Values can be from 2 to 8 buffers, which are 32K in size. Since most PCs have RAM to spare these days, there is no problem maxing this out.
Sometimes a bit of periodic maintenance is needed with XFS, and this can make people used to NTFS on Windows feel a little more at home. You sometime have to defragment the file system, and there is a utility that can do it. It isn't installed by default - you have to install the xfsdump package to get the folllowing commands.
To check the fragmentation level of a drive, for example located at /dev/sda6:
sudo xfs_db -c frag -r /dev/sda6
The result will look something like so:
actual 51270, ideal 174, fragmentation factor 99.66%
That is an actual result I got from the first time I installed these utilities, previously having no knowledge of XFS maintenance. Pretty nasty. Basically, the 174 files on the partition were spread over 51270 separate pieces. To defragment, run the following command:
sudo xfs_fsr -v /dev/sda6
Let it run for a while. the -v option lets it show the progress. After it finishes, try checking the fragmentation level again:
sudo xfs_db -c frag -r /dev/sda6
actual 176, ideal 174, fragmentation factor 1.14%
Much better! I have the xfs_fsr command scheduled to run daily with a cron job, at a time when it is unlikely to be recording, to keep things in good shape.
More in depth information on XFS can be found at the MythTV wiki. Some more detailed performance optimisation info can be found here and here.
There are other options though, depending on your usage patterns. If you have a lot of large files stored, such as lots of video or movies, maybe even if you record TV with MythTV like I do, a good alternative is XFS. Its design means it is particularly good at handling large files, such as High Definition TV recording, giving good performance in reading and writing them.
You can specify a partition to be in XFS format at the time of installation, or later on by using GParted. Once it is created, there are some further tweaks you can do to make its performance better, and also to maintain its performance over time.
1. Add the following options to the fstab entry for the partition (located in /etc/fstab):
UUID=xxxx /var/lib/mythtv xfs defaults,noatime,allocsize=512m,logbufs=8 0 2
noatime refers to stopping the file system logging each time the file is accessed. Normally it is not needed, and this option reduces the number of writes done to disk, speeding things up.
allocsize=512m lets XFS set aside 512 megabytes of space at a time when it writes. This reduces the chance of the file being split up into many small chunks, or fragmenting. Especially good for recording TV.
logbufs=8 refers to the log buffers held in RAM. Values can be from 2 to 8 buffers, which are 32K in size. Since most PCs have RAM to spare these days, there is no problem maxing this out.
Sometimes a bit of periodic maintenance is needed with XFS, and this can make people used to NTFS on Windows feel a little more at home. You sometime have to defragment the file system, and there is a utility that can do it. It isn't installed by default - you have to install the xfsdump package to get the folllowing commands.
To check the fragmentation level of a drive, for example located at /dev/sda6:
sudo xfs_db -c frag -r /dev/sda6
The result will look something like so:
actual 51270, ideal 174, fragmentation factor 99.66%
That is an actual result I got from the first time I installed these utilities, previously having no knowledge of XFS maintenance. Pretty nasty. Basically, the 174 files on the partition were spread over 51270 separate pieces. To defragment, run the following command:
sudo xfs_fsr -v /dev/sda6
Let it run for a while. the -v option lets it show the progress. After it finishes, try checking the fragmentation level again:
sudo xfs_db -c frag -r /dev/sda6
actual 176, ideal 174, fragmentation factor 1.14%
Much better! I have the xfs_fsr command scheduled to run daily with a cron job, at a time when it is unlikely to be recording, to keep things in good shape.
More in depth information on XFS can be found at the MythTV wiki. Some more detailed performance optimisation info can be found here and here.
Monday, June 28, 2010
Most painless Ubuntu upgrade ever
Wow. That's all I can say. Today I decided to upgrade to Ubuntu 10.04, but rather than doing a clean install like I have done with previous versions, I thought I'd have a go at the upgrade. To prepare, I copied the /boot, root and /home partitions of the existing 9.10 install to some free space on another hard drive. Just as a precaution in case it all went pear-shaped. I ran the update manager, chose the upgrade option, and away it went - downloading all the new packages, applying the updates. Only a couple of dialog boxes popped up asking about Grub, and what it should do with it.
The process finished without incident, I rebooted, and surprisingly, it started, allowed me to log in, and it got to the desktop! All previous programs and settings there, all programs updated. I am pleasantly surprised - even MythTV updated without a hitch, for the first time. I'm sure there will be the odd hiccup along the way that I'll discover in time, but so far so good.
Oh yes, regarding Windows 7 - I've removed it from my machine. Partly because I was barely using it, apart from watching full-screen Flash videos in decent quality, partly because I have obtained an old laptop from work, and installed 7 on that.
(Originally posted on May 2, 2010 on my other blog)
The process finished without incident, I rebooted, and surprisingly, it started, allowed me to log in, and it got to the desktop! All previous programs and settings there, all programs updated. I am pleasantly surprised - even MythTV updated without a hitch, for the first time. I'm sure there will be the odd hiccup along the way that I'll discover in time, but so far so good.
Oh yes, regarding Windows 7 - I've removed it from my machine. Partly because I was barely using it, apart from watching full-screen Flash videos in decent quality, partly because I have obtained an old laptop from work, and installed 7 on that.
(Originally posted on May 2, 2010 on my other blog)
New keyboard! And more on Windows 7
This post is basically inspired solely by the fact that I have bought a new keyboard and need an excuse to do a lot of typing on it. After finally having enough of interference, non-responsiveness and battery changes, I ditched by former wireless keyboard and mouse combo and bought a couple of new ones - a Logitech M500 corded mouse and an Illuminated keyboard. It has a nice, laptop-style key feel with just a little more travel, with the added bonus of keys that light up for use in a dark room.
I love it. Ubuntu picked it up without a problem, all keys working just fine. Windows detected it as well, but after installing drivers for it, insisted on a reboot. Some things never change. I didn't bother installing the setpoint software that is included - no need for it in my opinion.
As a progress update on my dual-boot exploits, there have been good points and bad about windows.
The good:
Sleep actually works. It worked even better when I downloaded a patch that stopped it crashing on resume when the hard drive didn't wake up in time. In Ubuntu, I only ever got to a blank screen with a blinking cursor when I tried the suspend feature.
Full-screen flash videos work with hardware acceleration. Only 5-10% CPU utilisation, compared with almost maxing-out a core under linux. This is due to differing stages of development of Adobe's flash player.
General polish and feel of the desktop. You can tell a lot of work has gone into this. The help functionality is excellent as well - far better than Ubuntu's vague documentation.
Backup works great - nice and straightforward, asking to also create a startup disc for system recovery. In Ubuntu, I am using backintime. While it is great for making backups, restoring from them is not so straightforward. Must look into an image based program.
Homegroups work great - it found the other Windows 7 PC on the network with no problems and is effortlessly sharing files between them. I can even use the printer connected to the other machine - a Canon that has stuff-all driver support under linux.
The bad:
While sleep works, the simpler task of powering off the screen after a set period seems more difficult for it. It doesn't always do it. Then again, a number of XP systems at my work have problems with that as well - and they are factory Dell and HP boxen.
The media centre application isn't quite up there with MythTV, one of linux's killer apps, in my opinion. Guide data is only available from the networks' broadcast guide, and I can't find a way to get it to set up two tuners. Actually, finding any documentation on it is somewhat difficult.
All the rebooting needed. It's still there, and it gets old pretty quick.
I think I'll be dual-booting for a while longer, especially since MythTV will not be replaced any time soon. It has been interesting though, and I am learning the interface and all. I have noticed a lot of fixes since the Release Candidate that I was running last year on the other PC.
(Originally posted on January 29, 2010 on my other blog)
I love it. Ubuntu picked it up without a problem, all keys working just fine. Windows detected it as well, but after installing drivers for it, insisted on a reboot. Some things never change. I didn't bother installing the setpoint software that is included - no need for it in my opinion.
As a progress update on my dual-boot exploits, there have been good points and bad about windows.
The good:
The bad:
I think I'll be dual-booting for a while longer, especially since MythTV will not be replaced any time soon. It has been interesting though, and I am learning the interface and all. I have noticed a lot of fixes since the Release Candidate that I was running last year on the other PC.
(Originally posted on January 29, 2010 on my other blog)
The Dark Side is looming...
Well. I'm having a crisis of faith. Looking at going back to the dark side. Windows 7 Home Premium, 3 user pack for ~$210 depending on where you shop. With the Release Candidate soon to expire in the next few months on the wife's PC, I have to either put XP back on it, or move to Windows 7. The 3 user pack is a good deal, normally it's about $160 for a single user, but it is going to be very short-lived. It only ran a couple of months even in the USA. There's a sense of urgency to this.
My wife couldn't care less what OS is running (as long as it's windows), so there'd be no problems with putting XP on, in her view. Realistically, I just need to get a decent backup program for it, because the Win7RC one is pretty good, while the standard XP one is rubbish by comparison. The only issue with the RC is while it is scheduled to run every Saturday at 3am, it seems to insist that the next scheduled time is December 30, 1899. Back in the days of steam-powered computers. I assume it is fixed in the release version of Windows 7.
Over the last few days and weeks, running linux has started to wear me down a bit. It is such a battle sometimes, to do stuff that would be simple in Windows or even OSX. Yesterday I wanted to install Google SketchUp – no native linux client, so installed through Wine. Go to run it, crashes straight away. Then it's off trawling forums and web searching for fixes, workarounds, all of that. Then there's flash player. Still buggy, still get sound lagging behind so speaking is out of sync. There's always more work, a lot of the time. F-Spot is a piece of crap, crashing all the time. Picasa is a half-arsed effort, just a windows program running under wine, really. Just doesn't look right.
But still, there are aspects of linux that I love. MythTV is the killer app, in my opinion. There is nothing in the proprietary software world that is so capable and unrestricted, with nothing locked down by media companies. I would miss that the most if I went back to windows. Being able to scroll anywhere without first clicking in the window is something I wish windows had, when I'm using it at work.
Anyway, I'll keep thinking about it, and pick it up on boxing day if I decide to. Otherwise, I could put that money towards hardware. I need a new screen, which I'll get regardless, but that $200 could go towards a solid state hard drive. Mmmm, fast.
(Originally posted December 25, 2009 on my other blog)
My wife couldn't care less what OS is running (as long as it's windows), so there'd be no problems with putting XP on, in her view. Realistically, I just need to get a decent backup program for it, because the Win7RC one is pretty good, while the standard XP one is rubbish by comparison. The only issue with the RC is while it is scheduled to run every Saturday at 3am, it seems to insist that the next scheduled time is December 30, 1899. Back in the days of steam-powered computers. I assume it is fixed in the release version of Windows 7.
Over the last few days and weeks, running linux has started to wear me down a bit. It is such a battle sometimes, to do stuff that would be simple in Windows or even OSX. Yesterday I wanted to install Google SketchUp – no native linux client, so installed through Wine. Go to run it, crashes straight away. Then it's off trawling forums and web searching for fixes, workarounds, all of that. Then there's flash player. Still buggy, still get sound lagging behind so speaking is out of sync. There's always more work, a lot of the time. F-Spot is a piece of crap, crashing all the time. Picasa is a half-arsed effort, just a windows program running under wine, really. Just doesn't look right.
But still, there are aspects of linux that I love. MythTV is the killer app, in my opinion. There is nothing in the proprietary software world that is so capable and unrestricted, with nothing locked down by media companies. I would miss that the most if I went back to windows. Being able to scroll anywhere without first clicking in the window is something I wish windows had, when I'm using it at work.
Anyway, I'll keep thinking about it, and pick it up on boxing day if I decide to. Otherwise, I could put that money towards hardware. I need a new screen, which I'll get regardless, but that $200 could go towards a solid state hard drive. Mmmm, fast.
(Originally posted December 25, 2009 on my other blog)
Subscribe to:
Posts (Atom)