HDRI or Image based lighting

Hi all,

I just finished wathching the PS3 presentation and I'm truly impressed. I was a bit dissapointed by the new generation when I first saw all the xbox 360 stuff, but the PS3 gave me new hope. Anyways, it seems that HDRI lighting is becoming very important in the coming generation of computer games. Rightfully so, I might add, because HDRI is one of the best and 'cheapest' lighting methods I can think of. The ps3 presentation made me remember an idea I had some time ago. I don't know how real HDRI could be implemented in 3DGS(I know how HDRI works, but only in render engines), but I do have an idea for a sort of fake HDRI, that should be relatively easy to implement in 3DGS.

My idea was to convert a HDRI map to a standard six panel cube HDRI map. You could also use 3DGS to make that cube map, by making a standard cubemap(the 6x 90 degrees screenshots technique) and then bring it in photoshop to make it useable as a HDRI map. (If we're going to develop this type of HDRI for 3DGS, I'll make a short tutorial for making the map.)
So, when you've got your HDRI cube map ready, you'd have code the shader. Basically it could work as a normal environment reflection shader(as in the wiki and templates) with a couple of alterations. Instead of simply blending the two textures(skin and reflection) the normal way, it should be blended like using the 'soft light' blend option in photoshop. All colours under RGB 128,128,128 should should darken the skin and all above should lighten. Furthermore, the colour of the 'reflection map' should be multiplied with the colour of the skin.

This is the basic principle that should work for a lot of situations. There are two things that could be added:
- the glow shader(from the shader forum) using Matt's renderview Dll. A soft bloom would help the effect tremendously.
- Realtime environment mapping, also using Matt's renderview Dll and the realtime environment mapping script made by... Matt, Ventilator or Steempipe? This way you could have realtime HDRI.

Were you to add these last two effects to the shader, you would have a true HDRI lighting system. I do think it would make the effect a bit heavy to handle, but on the other hand, you could remove dynamic lights(just use shadowmaps) and you don't need to use it on all entities(for example, you could just use it for the player).

I'm posting this idea, because I think it could be a great shader and a great addition to 3DGS, and I myself have no experience whatsoever with shaderprogramming.

So what do you think?


BTW. I'm not really a 'guest'. I have been working with 3DGS for four years now, I just lost all my login info.

Very,very good idea...

Now we just need some people who can code... Matt, Ventilator, Steempipe, Ello..? Anyone? I mean common', how awesome would it be to have HDRI in 3DGS? It'd put it on the same page as the highend next-gen engines (well not really, but it'd still be an awesome feature).

HDR lighting is not yet possible in A6, because it requires floating point textures. I suppose you could fake it, but what's the point?

For the same reason people use environment cube reflection mapping: If you can't have the real thing, go for the second best. Also, I would imagine that this method wouldn't require too much coding.
I've just now came up with an even easier way to do it, and all I would need is the ability to use the alpha channel of the environment cube map. Sounds like a small alteration of the environment cube reflection shader, but as said before, I don't know much about shader programming.

Can you explain this concept more clearly? I'm having trouble understanding exactly what using this 'environent map' ill accomplish.

HDRI has a much broader range of lighting values so the solution you described would fake it for a nice image-based lighting, but not be true HDRI lighting.
See http://debevec.org

Ah, great to see there is some interest.

First off, I'll try to explain the idea I had a bit better.


I wanted to start with a model with a skin containing an alpha channel. The alpha channel determine how 'reflective' each part of the model would be(Just as you would do with normal environment cube reflection mapping).
Then you'd have the cube map, which would also have to contain an alpha channel. This alpha channel would determine the intensity of the reflection. For example the brighter the light in the cube map, the more opaq(or white) that part of the alpha channel would be.
So the alpha channel in the models texture would determine how much CAN be reflected and the alpha channel in the cube map would determine WHAT and HOW MUCH would be reflected.

This is all just me thinking as low tech as possible.

But while I was surfing a couple of other forums I came across a thread discussing whether or not KillZone 2 could really be rendered by the PS3. One of the people trying to prove it could, came with this link:

http://www.ati.com/developer/shaderx/ShaderX_DiffuseCubeMapping.pdf

For what I can gather this would be even easier and even better than my technique. There's a link to a demo which uses this method and it looks pretty good. Furthermore they say that there's virtually no performance impact.

I have read this article, and this doesnt appear to be related to high dynamic range illumination, and seem to be a cheap global illumination, but i cant imagine that the lights could be easily accumulated using the current shader or redner to target features. In reality, how often do you really need more than 3 or 4 lights at a time? in that case, you can handle each light in a normal per-pixel fashion. The point of HDR is that by using floating point textures, you can have arbitrary brightness effects, which take the place of lens flare/bloom, etc, for a more realistic approximation of biological vision. I suppose this method could be used to good effect in certain circumstances, but i cant imagine an entire game lit this way all the time. Also, i dont see how this lighting method can achieve a blinn/phong specular reflection.

I'm aware that this is not true high dynamic range lighting. Basically it's just image based lighting. It doesn't have a high colour range, but with the method discribed it would be dynamic. And ofcourse you couldn't light an entire game this way(well you could, but why would you), but I think dynamic objects such as the player could be lit very well by image based lighting(let's call it that). It seems they're using this in the Source Engine as well, which does a pretty nice job(I know, don't compare engines).
Specular effects could be made by upping the brightness and contrast of the cubemap, either statically in photoshop or dynamically in the engine. But I don't know how good that effect would be. A screen glow would definitely help.

Basically it would make for a very good GI-like look, comparable to the standard Brazil render in Max, or Final Gather in Maya.

Btw. are you interested in developing such a shader or..?

So after doing some more research I came across the 'Color op' page in the Wiki and there I found the following blend option:

Result = Texture1 + Texture2 - 0.5
ColorOp[n] = AddSigned;

Looking at the example and assuming that texture 1 should be mirrord(in the wiki, right?) I think this is what I need. If I just go for the static or even the dynamic reflection, I could use this blendmode with the standard environment cube reflection shader from the templates.

Like this:

bmap bcube=<cube+6.tga>;

function mtl_envmap_view()
{
mat_set(mtl.matrix,matViewInv);
mtl.matrix41=0;
mtl.matrix42=0;
mtl.matrix43=0;
}

function mtl_envmap_init()
{
//...maybe you should copy the mat_model properties here...
bmap_to_cubemap(mtl.skin1);
mtl.event=mtl_envmap_view;
mtl.enable_view=on;
}

material mtl_envmap
{
skin1=bcube;
event=mtl_envmap_init;

effect=
"
texture entSkin1;
texture mtlSkin1;
matrix matMtl;
technique envmap
{
pass p0
{
texture[0]=<entSkin1>;
texture[1]=<mtlSkin1>;

zWriteEnable=true;
alphaBlendEnable=false;

colorArg1[0]=Texture;
colorOp[0]=AddSigned;///////////////just change this from modulate 2x tot AddSigned
colorArg2[0]=Diffuse;

colorArg1[1]=Texture;
colorOp[1]=blendCurrentAlpha;
addressU[1]=Clamp;
addressV[1]=Clamp;
texCoordIndex[1]=cameraSpaceReflectionVector;
textureTransformFlags[1]=Count3;
textureTransform[1]=<matMtl>; // transform camera space back to world space
}
}
";
}

The cube map would need some tweaking and colour adjustment in photoshop, but it should give a pretty good result.

If I could connect this to a normal mapping shader then it would be perfect.