Ok, so after googling I found this awesome answer on stackoverflow
C++/OpenGL convert world coords to screen(2D) coordsIt explains that you need to:
- transform into clip-space
- then transform from clip-space to normalized device coordinate space (NDC space)
- and then transform from this [-1, 1] (NDC) space to window-relative coordinates!
And from there I've got this code
vec4 clipSpacePos = projectionMatrix * (viewMatrix * vec4(point3D, 1.0));
vec3 ndcSpacePos = clipSpacePos.xyz / clipSpacePos.w;
vec2 windowSpacePos = ((ndcSpacePos.xy + 1.0) / 2.0) * viewSize + viewOffset; // OpenGL bottom-left of the window
vec2 windowSpacePos = vec2(((ndcSpacePos.x + 1.0) / 2.0) * viewSize.x + viewOffset.x, ((1.0 - ndcSpacePos.y) / 2.0) * viewSize.y + viewOffset.y); // top-left
Which I've later on converted into hlsl
float4 clipSpacePos = mul(float4(vecLightPos[i].xyz, 1.0f), matViewProj);
float3 ndcSpacePos = clipSpacePos.xyz / clipSpacePos.w;
float2 windowSpacePos = float2(((ndcSpacePos.x + 1.0) / 2.0) * vecViewPort.x, ((1.0 - ndcSpacePos.y) / 2.0) * vecViewPort.y);
And at the end it worked like a charm!
Thank you all for replies! Next step is getting fog of war/lights working!