If it's never visible you shouldn't use a skybox. Why wasting the memory?

Always learn from history, to be sure you make the same mistakes again...