Replace

vec_set(a,dummy.x);
vec_sub(a,my.x);
vec_to_angle(my.pan,a);

with

vec_diff(a,dummy.x,my.x);
vec_to_angle(b,a);
my.pan += ang(b.x-my.pan)*0.5*time_step;

Be aware that you will need a second vector ("b"). If the entity still turns to fast, try different values than 0.5 or use clamp:

vec_diff(a,dummy.x,my.x);
vec_to_angle(b,a);
my.pan += clamp(ang(b.x-my.pan)*0.5,-5,5)*time_step;

and play with the limit of 5.