Just some words more
Claiming :

We are moving...
We are going to opposite direction...

You assume the existence of the " Empty box " (The space) to which you refer yours and the other object's speed
Claiming

I am moving.....

We normaly mean ....relative to the ground
It easy to replace earth (ground) with a generic absolute coord system
It is what Newton did.
On the contrary it is hard to eliminate the "Absolute 3d Coord system "
Suppose for a while that universe disappears
Our intuition assume that as long as you we define a 3d ccord system then we can define a speed
But it is not Einstein's opinion
According to the theory of relativity,If matter does not exist also space and time do not exist