"What's being pulled from the broker" - is just a 'single-point snapshot", not really representative of 'live trading costs' - both forward (for spreads) and backward (swaps).
With backtesting, you generally would want to minimize impact of 'other variables' on performance of the 'signal'.
So, unless your strategy is very short-term or very long-term, it makes sense to "hold other things equal".