If you want to track a hand, you can do the following, if you have a color image: try to make a model of skin color. I used a Bayesian Histogram (one for skin color, one for non-skin color) to approximate the probability, if a pixel covers skin or not.

The you get a skin probability image, in which you can segment with a classic approach the areas: ideally, you get the area around the head and two areas for the arm/hand. Approximate which area the hands are and feed that position (and hand size) into a tracking algorithm, to predict the positions in the next frame, that helps you as a new cue to extract the hands in the next frame.