i think you can't solve that with vertex shaders since you know nothing about the neighboring geometry there.

this article seems to describe some quake3 specific stuff (i only took a quick glance).