Well let me tell you first of all I don't know the first thing about realistic behavior physics in this area, but if you are going for something that "looks good" and will behave well besides I think I can help you out, and trust you to fine tune it afterward.
Your idea intrigues me, and I personally wouldn't mind creating something like this myself if it actually fit into a project that I'm doing, unfortunately this is not the case.
I'm going to say I've borrowed the ideas in this post and they are not my own, but the fine details are simply guesswork of how the models that I have seen have worked (no models I've seen have been applied in a game though, so you should be the first that I've seen of that
). I must ask you what type of animal you are trying to model, since birds, fish and land animals will all move differently in each model. Regardless the following rough models should work though with some adjustments to each.
1. Leader based "flocking", this is commonly done in squad based games to some extent, but for animals these are seen among geese or ducks flying in a V in the sky.
-Given one leader, every entity relates itself to the leader, it stays close in proximity (small distance between it and the leader compared to other objects)
-It's orientation is the same or has slight deviations to the leader (more or less they are all moving in the same direction) unless their is an obstacle in their path, they will deviate from the formation and return as soon as possible once the obstacle is avoided.
-Given the type of animal and its behavior, "formation" is used loosely, for example a duck in formation would probably specify himself as 3 feet to the right and 5 feet behind the leader at practically all times and in the exact same orientation (direction moving/facing) and speed. Where a wolf of a pack would orient himself to a leader as no closer than 3 feet of the leader and closer then 100 feet from the leader, and simply going to the same destination, while arriving there within 1 minute of the leader.
-The benefits of this model is it is easy to assign sub-leaders and re-assign units within the formation so multiple groups can go to different locations, flanks and searching parties are possible with this method.
*If your going to ask someone to help, or a model, see IntenseAI, the project has done very well with this type of AI.
2. Animals with no large goal in mind would generally be considered part of an larger entity type behavior, schools of fish swim together, while sheep simply graze together in the same area. This is a bit simpler to do, but generally nothing complex or precise is being done by the group of animals.
-This introduces the idea of the "Entity" the entity can grow, shrink and move as necessary to the goals of the whole (a threatened entity may shrink in size to protect its members however it might grow if they feel safe and are looking for food)
-Units or animals of the entity don't really have to move in a given direction but instead just have to stay within it's limits to be part of the entity (usually for safety)
-Animals generally wander in this method, and moves slower then leader based movement. Although once an animal has drifted from the Entities boundaries all it simply needs to do is move towards it's center (avoiding obstacles) to get back to the entity.
If you are making different styles of animals and looking to diversify them I would suggest leader based movement for predators and Entity based movement for prey.
If your looking for predator based movement, all that is really needed is to set a far destination with random deviations in it's path every once in a while as it "wanders" to find it's prey, unless your going for pack based tactics after prey is found a simple chase and run, while avoiding obstacles would be sufficient.
