I have had the same fractals in my shadow renderer,

try to make each trace step have some random offset, and
do more traces, than there are actual pixels, this will smooth
these fractals.

rtsmaker.com