The selection box can be drawn using draw_line, draw_quad or something similar.

To check if any entity is within the box, you could cycle through all visible entities (with ent_next() + CLIPPED flag or ent_pvs()) and convert their world coordinates to screen coordinates (vec_to_screen()). Each entities screen coords have to be compared with the screen coords of the selection box.