You are right, but it has little to do with drawing lines about a pretty clear definition.

Funny thing is that Wilders, Verdonk and Fortuyn all are pretty careful about what they say and never will say or have said things that are racist ideas, nor do they really believe in such ultra-right (nazi?) nonsense if you ask me. They are/were pretty nationalistic though, but that's obviously not the same as being racist.
It always sounds a bit wrong the way people often describe these kind of politics, but in reality there is not all that much wrong with 'defending' a nation's own internal culture/structure/politics and so on by making immigration laws.
The thing I am worried about is that a lot of the free speech surrounding this topic is pretty damn Islamophobic in nature, which makes little sense in my opinion as most Muslims in our country are actually modern, very moderate when it comes to there religion and all in all just great people. The psychological feeling of 'living next to terrorists' often gets abused when the topic of immigration is concerned.