Previously in my linux box I was running an Nvidia 8600GT graphics card, along with the Nvidia proprietary driver. This driver has the VDPAU extensions, that enable some of the video calculations to be offloaded to the graphics card, giving smoother playback. Since this works in the GeForce 8-series and later cards, it worked on mine.
As it turns out, it didn't work terribly well. When I enabled visual effects (via the System -> Preferences -> Appearance -> Visual effects option), all the standard Compiz effects like wobbly windows and transparency worked, but when viewing TV, either live or recorded in MythTV, playback was slightly choppy, with some tearing of the picture. Since I valued smooth video over wobbly windows, I disabled the video effects.
Looking around on some forums I noticed some saying that their playback was OK, even with Compiz enabled. I finally tried biting the bullet and getting a quicker video card (A couple of recent game purchases had nothing to do with it, honest...). I was concerned about the power usage of a quicker card, so I kept an eye out for power usage figures. Eventually I came across the excellent list of specs at TechARP, where the Nvidia GTS250 was listed. It showed a fairly high max usage of 145 Watts, but also some good performance figures compared with the 8600GT I had. A few reviews showed that recently, a new revision of the board was out - a "green" version. Clocked slightly lower with the GPU and memory speeds, it also only had one additional power connector compared with previous versions that had two. They also showed its power usage was reduced.
I managed to find an Inno3D 1GB GTS250 Green at a pretty good price, plugged it in the PC, and fired it up, with one eye on my watt meter that the PC is plugged into. To my surprise, power usage was about the same, if not less, than with the 8600GT. The shrinkage in process technology really made a difference. The only downside was the fan noise - it's easily the loudest component in the PC now, and of course louder than the previous card, which was a passively-cooled model from Gigabyte.
Best of all, I booted into linux, enabled the Compiz visual effects, and fired up MythTV to watch some video. It played back beautifully. I think that the previous card was only marginal in performance, and with its memory being fairly slow DDR2, didn't have quite enough grunt to run MythTV with Compiz. As mentioned earlier, other people have reported success with a 7600GT and a 9500m GS, and another person had success running an 8800GT, with two monitors at 1920x1080.
I believe when they were released, the 8-series cards drew some flak for using more power but not having much more performance than the previous 7-series. Maybe there is some truth to that. This new card has power to spare, with the side benefit of good gaming performance. For a HTPC, I'd look into a different, quieter cooler for it - something like an Arctic Cooling Accelero Twin Turbo Pro.
I should add that I was tempted to buy an ATI card - the 5 series cards are quite power efficient with better performance than what Nvidia has to offer, and if I was running a straight Windows system I'd have no hesitation in picking one up. Unfortunately, ATI's drivers for linux are not as well developed, particularly their equivalent to the VDPAU functionality, so I was scared off buying one for that reason.