That's indeed difficult. Blocks are meant to stay at the same position, and the renderer uses this to its advantage.

For your purpose, if you want to move or manipulate blocks, better already place entities in the level instead of blocks.