well you can always analise the 4d dimension right? the time...

So, you can run/apply a probability prediction of who is who. istead of random, at least you could get more accurate result.

So you know that if you mix 2 points one coming from above to down, and one from the left to the right..and they cross to a single point.. the next value from that cross, is what? probably the dots would keep their direction, so the above will keep goin down, and the left will keep going right, and not, the up will go right and the left to right will go down.. understand me?

So you have the var delta_x and delta_y that holds the position frame -1 frames, so you get a direction vector, and you then predict based on that. like(if the direction vector is the same then id is the same, else switch).