Why don't you manipulate the vertices close to the mouse/ brush on the CPU? This should be really fast in your case. If you want you can still calculate a heightmap from your vertices for whatever reason.
"Falls das Resultat nicht einfach nur dermassen gut aussieht, sollten Sie nochmal von vorn anfangen..." - Manual
