Yes, you should, if you really want to mess it up... Honestly, set is a macro designed for easily setting flags of an entity. Don't use it on pointers.

One thing caught my eye: You said there were three entities non-speakers and on speaker. Your code doesn't represent this. You just check the distance and assign a pointer corresponding to the distance. As you only defined two pointers only the last two entites checked will be pointed to anyways. If you're lucky and the distance of one is > 100 and the other ones is < 100 then you will have to pointers, one pointing to an entity farer away, the other one pointing to the nearer one. If you're not so lucky the distance will be < 100 or < 100 for all entites and only one pointer will get assigned at all.

Keep in mind a pointer can only point to one certain address and will not be able to point to a group of entities.