I had a problem with X.org using way too much shared memory with the Nouveau driver.
The shared memory was taking up all the physical memory and swap space making the system unusable.
Switching to the proprietary driver (or switching to the Intel card) seems to have resolved the issue.
