I am unsure from your description whether you are actually seeing tearing; normally, tearing looks like the picture is split horizontally into two or three sections, with the parts not matching up. This happens because the top part of the picture and the bottom part are not coming from the same frame in the video. Wikipedia has a good simulated example of screen tearing. Fine horizontal lines sounds different, more like combing artifacts from poor de-interlacing, although maybe it is just hard to describe.
You don't mention any details of the video you are trying to watch, such as its resolution, encoding, and if it is interlaced or progressive, and whether you have de-interlacing turned on in VLC. It might also be worth checking if the video processing is being offloaded to the GPU, or whether the CPU is doing all the work. Have you confirmed that you can successfully view the same video on different hardware, or on the MacBook using OS X?
You say you've tried with compositing disabled on XFCE, but it probably wouldn't hurt to try adding:
Section "Extensions"
Option "Composite" "Disable"
EndSection
to your /etc/X11/xorg.conf just to be sure. Also, one other suggestion not mentioned so far is to add:
Option "UseEvents" "True"
to the Device or Screen sections of your xorg.conf.
UPDATE:
The attached screenshot definitely shows tearing! If the player is otherwise keeping up with decoding the video (ie, no stuttering slow downs, just the tearing), then it probably doesn't matter if it's the CPU or GPU doing the decoding. VLC has an option on the Video settings page in Preferences to turn on and off "Accelerated video output"; if it's on, it ought to be using the GPU to decode. If you toggle that setting in VLC and run top
in a terminal whilst playing the video, you should see a difference in CPU usage between the two settings. You could also try turning off any de-interlacing in VLC to see if that makes a difference.
I have two other suggestions:
I don't see anything obviously wrong with your xorg.conf, but you could double-check that both sync to vblank settings (on X Server XVideo Settings and OpenGL Settings) are stored correctly in .nvidia-settings-rc. Some people run nvidia-settings as root or using gksudo (so it can update /etc/X11/xorg.conf), but this can result in the settings being stored in /root/.nvidia-settings-rc rather than ~/.nvidia-settings-rc. It ought to be picking up your own copy, but to avoid any doubt, you should check both vblank settings when running nvidia-settings as you; don't forget to hit "Save Current Configuration" on the bottom page of settings. If you run:
nvidia-settings -q=XVideoTextureSyncToVBlank -q=SyncToVBlank
then you should see both settings are set to 1. You should probably also check that however you auto-loading the settings is working by rebooting and running the above command again.
The other thing you could try would be to upgrade/downgrade your nvidia drivers. I have experienced tearing with some minor driver version updates before; there are definitely "good" and "bad" versions of the nvidia drivers and the latest are not always the best. You can try upgrading using the X-swat PPA:
https://launchpad.net/~ubuntu-x-swat/+archive/x-updates
And you can try downgrading to an earlier version available for 12.04:
https://launchpad.net/ubuntu/precise/amd64/nvidia-current
or
https://launchpad.net/ubuntu/precise/i386/nvidia-current
depending on whether you have amd64 or i386 installed. Click on the version number to the right that you want to try, download the .deb file and then install with:
sudo dpkg --install <name of file>.deb
For example, due to an unrelated nvidia bug, I had to download this:
http://launchpadlibrarian.net/90395807/nvidia-current_290.10-0ubuntu2_amd64.deb
from
https://launchpad.net/ubuntu/precise/amd64/nvidia-current/290.10-0ubuntu2
and installed it using:
sudo dpkg --install nvidia-current_290.10-0ubuntu2_amd64.deb
One other tiny point: one other answer mentions changing the refresh rate from Auto to 60Hz, and you say that it is already set to 60Hz. However, your attached xorg.conf has nvidia-auto-select in metamodes (in the Screen section), which suggests it is not fixed. However, your Monitor definition only offers 60.0, so maybe this is inconsequential. It might be interesting to back up and move your xorg.conf, and re-run nvidia-xconfig and nvidia-settings to see if you get a different config.
Option "TripleBuffer" "1"
in your xorg.conf file?xorg.conf
.