If you're using an integrated Intel GPU, immediately switch to modesetting X driver. Intel DDX leaks surfaces and that is attributed to process which allocated them.

Modesetting is faster too.