You can improve on that by putting the objects on a grid and letting the broadphase only check for potential collisions within a cell of the grid (and keeping in mind that an object can span multiple cells)
I actually tried that some time ago. It worked pretty well (sorting objects into the grid, moving them) but there were some bugs and the whole thing was dirty a.f. since it was my first try to code something like that. In the end I threw that part of the code away. However, I gained a lot of insight and my next approach should turn out a lot better (and probably maintainable so I don't have to trash it again )
)
