How to downsample correctly and what is HDRR?

Posted By: Slin

How to downsample correctly and what is HDRR? - 05/27/13 09:30

For some basic bloom and in the future also tone mapping with auto exposure, I need to downsample a render target.
How would I do this correctly? Sure, I could render into a new render target and downsample all the way down in just one shader by taking every pixel of the texture to downsample building the new one and average them and that might be a good idea on modern hardware, especially in combination with the slow DirectX 9 API, but if I want to go all the way down to 1*1, it is clearly too much and it seems extremely inefective in terms of parallelism.
So what I want to do in my case is just basic downsampling dividing height and width of the next target by two, going down as many stages as needed.
If I do this, each pixel of the new texture should be the average of four pixels of the previous texture. Again, I could just do texture lookups on those four pixels and average them and on AMD hardware there even is fetch4 which might help to optimize it (although, I am not sure if may only works with depth maps). However, every "normal" hardware supports linear interpolation on texture lookups, so I guess I can just sample into the middle of those four pixels and the result will be the average, is that correct?
Does this in gamestudio only mean offsetting the texture coordinate by half a pixel of the new texture (which should be one pixel of the previous texture) and sample there?

If I want to upsample, do I just have to sample using the same texcoords as for the pixel I am rendering and interpolation does the rest and there is no difference if I do this in several passes or just in one, or just use the texture as if it already had the correct resolution?


I want to do high dynamic range rendering. Does this just mean to do every the same as always, but instead of rendering into RGB888 targets, I will render into floating point render targets?
And then if I want I can apply tone mapping of some kind to such a floating point texture and everything will be the way it is supposed to be?
I guess the tonemapping is done in linear gamma space?


My results with gamma correction lost a lot of contrast compared to my results without, which might look more realistic, but it also lacks a lot of contrast. Is this normal, or am I doing something wrong? Any ideas on how to get some contrast back?
I am currently using an ambient value of 0.05 and it is still brighter than a value of 0.2 without gamme correction. I am also having a lot of banding artifacts around light sources because of the low render target resolution with gamma correction, are there any solutions for this other than a higher precision render target?

Thanks tongue
Posted By: Kartoffel

Re: How to downsample correctly and what is HDRR? - 05/27/13 12:52

regarding the downsampling:
I'm not really sure... I'm using 13 samples per pixel for the first downsample step (to prevent ugly
aliasing) and for the other downsample passes I use less samples because it's already blurred then.

When upsampling you have to be careful about the resolutions. if for example:
screen_size.y (1080) / 16 = 67.5 -> you have to compensate the difference that the .5 causes.

EDIT: didn't see that you want to downsample to 1x1 smirk
What I did: render the average of multiple samples into a 1x1 texture. I'm pretty sure most games do something similar.

About gamma correction: I'm not doing any gamma-correction (I'm not sure if it's THAT important),
however gamma correct lighting is important.

Quote:
I want to do high dynamic range rendering. Does this just mean to do every the same as always,
but instead of rendering into RGB888 targets, I will render into floating point render targets?

Well... yes thats all you need to make HDRR possible.
However, I use bloom (and a little color modification) to illustrate colors which are out of range,
my tonemapping just adds some sharpness.

It looks like this at the moment:

(the bright spots are ~30 times brighter than the maximum)
Posted By: oliver2s

Re: How to downsample correctly and what is HDRR? - 05/27/13 16:39

Sorry for offtopic, but is this shader available somewhere, Kartoffel?
Posted By: Kartoffel

Re: How to downsample correctly and what is HDRR? - 05/27/13 16:46

I'm really sorry but no, it's not.

(Also it's not one simple shader but a whole pipeline. Even if I would give it away, people would have huge problems using / implementing it)
Posted By: HeelX

Re: How to downsample correctly and what is HDRR? - 05/27/13 17:45

Originally Posted By: Slin
So what I want to do in my case is just basic downsampling dividing height and width of the next target by two, going down as many stages as needed.
If I do this, each pixel of the new texture should be the average of four pixels of the previous texture. (..) so I guess I can just sample into the middle of those four pixels and the result will be the average, is that correct?
Yes, this is correct and that is how bilinear interpolation works. When you do this repeatedly down to a 1:1 pixel image, you create a Gaussian pyramid, that encodes in each pixel the local average that corresponds to a pixel neighborhood of a lower level of the pyramid (lower levels = bigger images with high frequency details).

Originally Posted By: Slin
Does this in gamestudio only mean offsetting the texture coordinate by half a pixel of the new texture (which should be one pixel of the previous texture) and sample there?
As I understand, you don't need to do this, because the center of a pixel in the lower texture is in correspondence with the center of the according 2x2 source pixels:



When your lower image is the destination of your shader, it can be the case, though, that the output UV space is not in correspondence with the input UV space. In my SSAO solution, I feed hires textures into the SSAO stage for which the output target is half the size, so I had to multiply the incoming texcoords with 2 to project the coords back to the input UV space; the same when you scale it up again: when the incoming image is half the size of the outgoing image, I had to divide the coords by 2. I don't know if this is a bug or intended, but you have to do this. After that, the coords are aligned as it is shown in my picture above.

Originally Posted By: Slin
Does this just mean to do every the same as always, but instead of rendering into RGB888 targets, I will render into floating point render targets?
Yes, because averaging pixels values which are originally bucket values between 0...255, you would throw them back into a bucket-like image.

I don't know much about HDR processing, but since you always loose some luminance during a straight forward gaussian pyramid on RGB images (even with high precision floating point targets), you should normalize the image accordingly when going up the pyramid again - and then the bucket procedure you did before with 8888 targets will lead to some artifacts I guess (maybe these are those bands you see.. I don't know).

If you want to stay with 8888 targets and want to keep quality, I suggest you change to a colorspace, in which the luminance is on a single channel, whereas the other channels encode the rest. HSV is a popular color space which is also easy to implement, but it also has its flaws. In my experience with image processing I had most significant good results with CIE Lab color space, in which L encodes the luminance and A and B the color. For an implementation I would copy the corresponding source code from OpenCV, or you can use my version (see my B.Sc. thesis page 53):



Since a whole pyramid is costly, I would try to avoid FP targets under any circumstance, but that is just a gut feeling... wink
Posted By: Slin

Re: How to downsample correctly and what is HDRR? - 05/27/13 19:43

Thank you very much HeelX and Kartoffel, that basically confirms that I am doing it correctly laugh

Kartoffel:
Quote:

When upsampling you have to be careful about the resolutions. if for example:
screen_size.y (1080) / 16 = 67.5 -> you have to compensate the difference that the .5 causes.

do you mean downsampling? because there I am going with ceil(size/factor), which slightly destroys the average by not sampling exactly between the pixels, but that shouldnīt be a problem.

Quote:

What I did: render the average of multiple samples into a 1x1 texture. I'm pretty sure most games do something similar.

I plan to go all the way down, which arenīt as many stages as one expects tongue

Quote:

About gamma correction: I'm not doing any gamma-correction (I'm not sure if it's THAT important),
however gamma correct lighting is important.

isnīt that kinda the same? the diffuse textures for example are already gamma corrected, so I have to tell the graphics API to transform them to linear color space where I apply the lighting and the result is then again gamma corrected, transforming it into gamma space.

Quote:

Well... yes thats all you need to make HDRR possible.
However, I use bloom (and a little color modification) to illustrate colors which are out of range,
my tonemapping just adds some sharpness.

What do you use for tone mapping? Are you applying the bloom before or after the tone mapping? (http://frictionalgames.blogspot.de/2012/09/tech-feature-hdr-lightning.html states that the bloom should be applied before the tone mapping)



HeelX:
Quote:

As I understand, you don't need to do this, because the center of a pixel in the lower texture is in correspondence with the center of the according 2x2 source pixels:

Does this mean that gamestudio already does the half pixel offset?

Quote:

I don't know much about HDR processing, but since you always loose some luminance during a straight forward gaussian pyramid on RGB images (even with high precision floating point targets), you should normalize the image accordingly when going up the pyramid again - and then the bucket procedure you did before with 8888 targets will lead to some artifacts I guess (maybe these are those bands you see.. I don't know).

Those banding artifacts are easily solved with fp targets and are caused by a low precision of the really dark parts in linear rgb space which are streched over a bigger range after gamma correction.
But does this mean, that Iīll have to do some kind of upsampling, just like I did for the downsampling?
Currently I completely removed any upsampling and blurring and just use the result of blurring a low res target, but the result is flickering, as if Iīd do the downsampling wrong and the applied bloom is quite blocky. Any ideas on this?
How could I effectively blur my bloom texture in you opinion?

Quote:

Since a whole pyramid is costly, I would try to avoid FP targets under any circumstance, but that is just a gut feeling...

It seems like everyone uses them a lot and from my experience so far this is not an issue at all, at least as long as my bottleneck is a stupid cpu stall I canīt find a solution for.


Some more stuff:
I am not using gamestudio for this, but a custom OpenGL renderer, but thought I might use this to get some discussion into this forum, as I am still visiting on an hourly basis and not much is happening frown

Here are two more very usefull links about tone mapping, HDRR and some more stuff (like the idea of not applying SSAO to the direct lighting term (but other than that their SSAO solution is really dirty, but does its job on the screenshots)):
http://de.slideshare.net/ozlael/hable-john-uncharted2-hdr-lighting (lighting in uncharted 2 with lots of details on gamma and tone mapping)
http://filmicgames.com/archives/75 (overview of the tone mapping from the other link, by the same guy)

Today I experimented with gamma and tone mapping and somehow I am not that happy with my results. Ignoring gamma and without tone mapping, there seemed to be much more contrast in the image which in my opinion looked better. On the other hand, tweaking the exposure and ambient and using the uncharted 2 tone mapping, I got at least kinda close to what it looked like before, only the bloom still doesnīt have the power it had before and I am not exactly sure what kind of value would make sense for the whitepoint.
Here are some screenshots:

(no gamma, no tone mapping, the bloom is obviously a bit overdone :P)


(correct gamma handling, no tone mapping, fp targets)


(correct gamma, uncharted 2 tone mapping, exposure of 5, using ldr targets with 24bit colors)


(correct gamma, uncharted 2 tone mapping, exposure of 5, using hdr fp targets, tweaked and overdone bloom :P, lower ambient)

I tried to find similar screenshots, but camera and sun positions change a bit and I also tweaked different setting trying to make things look better (or worse :P)


Edit: Setting the whitepoint to something closer to the actual highest brightness, I can use a lower exposure and it looks better in my opinion :

(exposure=1.0)


(exposure=2.0)
Posted By: HeelX

Re: How to downsample correctly and what is HDRR? - 05/27/13 20:53

Quote:
Does this mean that gamestudio already does the half pixel offset?
As far as I know, yes.

Originally Posted By: HeelX
When you do this repeatedly down to a 1:1 pixel image, you create a Gaussian pyramid
I have to correct myself, a real Gaussian Pyramid is created by convolving an image with a 5x5 Gaussian bell and removing each second row and column:



example image:



After two iterations:



Nevertheless, taking the average of 2x2 neighbouring pixels by 1 bilinear lookup is a cheap but incorrect approximation. Though, I guess you get approximate results to the gaussian approach, if you apply a 3x3 gaussian bell afterwards, in order to get the average of 6x6 source pixels. Whereas you need in the original approach 10 lookups per stage, you need with this one only 7. This is not much, but hey smile

Quote:
How could I effectively blur my bloom texture in you opinion?
I can not tell you how to do HDR since I never wrote a shader like that on my own. I assume that you take the low frequency information of the downsampled image take it's luminancy value as hint how the light and dark parts are distributed over the screen, so indeed you have to upscale downsampled images to match them with their next-level hires-counterparts.

With the original approach, you can do this by first filling all new even rows and columns filled with zeros, followed by a convolution by the same 5x5 Gaussian bell multiplied by 4 to approximate the missing pixel values. After upscaling the above image back to original size:



But I guess this is a somehow inconvinient implementation, although I saw it somewhere in the internet already for CG (search for GPU implementation of Gaussian / Laplace pyramid or similar). An alternative could be a rotated poisson filter disc with 12 samples or so, while the disc is always 5x5 pixel big or so in screen space, maybe with applied jittering. I saw such an approach for dynamically sized DOF and I guess it should work for your case, too.
Posted By: Hummel

Re: How to downsample correctly and what is HDRR? - 05/28/13 02:24

Quote:
Nevertheless, taking the average of 2x2 neighbouring pixels by 1 bilinear lookup is a cheap but incorrect approximation.
If you just want the average it's perfectly fine.
It's common praxis to create a full mip-map pyramide to average the luminance for tonemapping. The required memory consumption converges to 1+1/3 times the original amount (determined by the original texture).

Also, bloom is a cheap approximation to light scattering in the lenses of the camera or alternatively the human eye. If you want to do it somewhat physically plausible you have to add the blur before tonemapping to the _full_ HDR scene, without any bright pass - the amount of scattering is not influenced by the amount of photons, of course.

Regarding gamma correction you have to differentiate between perceptual and photometrical linearity. If you output a gamma-corrected mathematical linear gradient it will not appear linear to you which is due to the gradient being kinda photometrical linear and the human perception of brightness is in it's nature logarithmic. The original gradient without gamma correction appears much more perceptual linear.
The conversion from and to CIE Lab is rather costly (I would sooner suggest YCrCb), but if you are working with LDR RTs you can also convert back to gamma space befor writing to the target and back to linear when you are reading from the target in a later pass. Doesn't work with alpha blending, though. Also you can't use the bilinear fetch anymore.

Originally Posted By: HeelX
[...]to divide the coords by 2. I don't know if this is a bug or intended, but you have to do this.
Year, this is horribly annoying.
Posted By: Slin

Re: How to downsample correctly and what is HDRR? - 05/28/13 11:32

Quote:
But I guess this is a somehow inconvinient implementation, although I saw it somewhere in the internet already for CG (search for GPU implementation of Gaussian / Laplace pyramid or similar). An alternative could be a rotated poisson filter disc with 12 samples or so, while the disc is always 5x5 pixel big or so in screen space, maybe with applied jittering. I saw such an approach for dynamically sized DOF and I guess it should work for your case, too.

I just need a cheap and good looking blur wink
For now, I am going with downsampling to 1/8th resolution, applying a box blur with a kernel size of 5 and doing the same again on 1/4th resolution to get rid of blocky artifacts and that is blended over the final image. I should maybe use a gaussian blur to have less of a darkening effect, on the other hand a "stronger" box blur looks usually pretty good.
I also found some link where they added the results of the different blur kernels to get rid of the darkening, but in my opinion that was a bit overdone tongue


Quote:
Also, bloom is a cheap approximation to light scattering in the lenses of the camera or alternatively the human eye. If you want to do it somewhat physically plausible you have to add the blur before tonemapping to the _full_ HDR scene, without any bright pass - the amount of scattering is not influenced by the amount of photons, of course.

Okay, but then things will look very blurry/foggy unless I keep the bloom at a minimum, but then I donīt really need it at all and I am a big fan of bright parts covering everything on the screen with their bloom halos tongue


(bloom everywhere, very low white point and low exposure :P)

I am fine with my solution for now laugh.
Next up is SDSM and ESM for better shadows laugh.
© 2024 lite-C Forums