The rendertargets are only rendered into when you use the COLOR1..COLOR3 semantics in a shader in that particular view. When you don't use them, the render targets are supposed to preserve their content until it's rendered into them again.