This could explain the different behavior. The engine uses the August 2009 SDK. I don't know if shader compiling with a different version causes a problem, but better use the August 2009 fxc.exe to get consistent behavior. Or compile fxo files with the engine - fxc.exe won't work for library shaders anyway.