Does sierra chart bitmex connection use rest or websocket?
Neither. Bitmex set up a DTC service
just for Sierra Chart users. It comes complete with live streaming of market data and market depth, historical intraday, live trading, simulated trading, etc.
I heard that bitmex have order rejection problem.
Do you have any experience about that?
I don't use Bitmex, but Sierra Chart has declared the Bitmex interface to be stable. You can read about setting up the Bitmex/SC connection here