I've forwareded the right ports in my router and also tried connecting directly without the router. So my guess is that the modem is the problem.

@Dark_samurai
Ye, good idea tomorrow I'll phone my ISP, it's to late now, maybe they know something. For now, hopefully google knows