You still need to encode and decode the jpeg, and that is quite costly. A jpeg really doesn't give you anything over a raw buffer in terms of how fast you can push it to the GPU to get it rendered, or how fast you can get it from the GPU to the socket, because in both cases you have to transform the image.

Really, use h.264 or any other video format which uses keyframes, as that will drastically improve quality over jpeg and decrease bandwidth need. Plus, your jpeg is not immune to packet loss, whereas h.264 is (although with reduced quality as a result).