Do you move them by clicking to a location (like in an RTS)
Or by moving the player directly (like in a 3rd person game)?

In the case of an RTS, the position within the formation
are calculated first, and the Units move toward their target-points.
In the case the targetpoints are in an inpassable area, you need
a function to look for a point closest to the targetpoint, wich
can be accessed. (you could try that out by looking around
in a "circle", until the new targetpoint is on a passable terrain),
and set this as new targetpoint.

--

In the case of a 3rd person game, its more tricky,
as the units will move constantly relative to the players
position, and their favorable formation position.

I would simply first develop a normal obstacle avoidance
(such as using traces to the front, and turn the unit when
a fronttrace gets too close to a wall)
Such that the unit first tries to walk to the position in
the formation, and in the second step rotates to avoid
getting too close to a wall)

After a timeinterval (like 2 seconds) the unit tries to turn and
walk towards the formation-position again.

This way the units will head towards the formation position,
but avoid obstacles as long as needed.
But on a free terrain, they will exacly follow the formation then.

Good would be for the unit to constantly turn towards the
formation-target (every frame), but once the
avoidance-trace hiths a wall in a savety-distance, the
unit will stop turning towards the formation.target every frame,
and rather rotate sidewasy, until the trace does not hit a
wall. (it continious walking, as long as it does not get
too far away from the player)
The unit checks then every 2 seconds if a trace towards
the formation-target is possibe.
If yes, it switches back to the normal state.

Instead of turning before an obstacle, you can also
let the units step sideways, and keep turning towards to
formation-target.
You should use traces to the side then as well.
(its really mainly testing out wich works best)