Post-processing effects test (SSAO/HDR/Bloom)

[myconid]

I'm not sure I understand this perfectly:

   float sign = v_tangent.w;
mat3 tbn = transpose(mat3(v_tangent.xyz, v_bitangent * sign, vec3(h)));
eyeDir = normalize(tbn * v_eyeVec);
float dist = length(v_eyeVec);

Eyevec is the normalized (camera - position) vector, but I'm not sure why you multiply the bitangent with v_tangent.w... And I'm getting a weird EyeDir vector, do you have any screenshot of that? Also, you seem to transpose the matrix and I'm not sure why. Is "length" returning the squared length or the Euclidian?

You have two vectors, the normal and tangent, which are at 90 degrees. You want to use those vectors to create a coordinate space (tangent space), and to do that you need a third vector, the bitangent, which is orthogonal to those vectors. But, in which direction does the bitangent go? That's what sign determines.

Eyevec should be unnormalised, it's the difference between the camera and the vertex. Eyedir is normalised, and it's the direction from the camera to the vertex. I can get a screenshot, but of what exactly?

The transpose can be get rid of like I said earlier:

transpose(M) * V = V * M

Length is just a measure of distance (i.e. unnormalised) from the vertex to the camera (and it's actually a bit redundant, but nevermind). You shouldn't need/use that if you are unrolling the loop!

[wraitii]

Thanks for answering. As you can see from my edit above, I've got why it didn't work...

Edit: need to change v_tangent = tangent to v_tangent = a_tangent too.

EditN°2: thought that may break other things...

EditN°3: so there actually is something with the matrix calculation…

Edited July 6, 2012 by wraitii

[myconid]

Complete edit: got it!

Live 92 of model_common.vs, change

	v_normal = normal;

to

	 v_normal = a_normal;

.

Not so sure about that. a_normal is in object space, so the lighting (and other things) won't work.

Does this mean your problem is with the instancing transform matrix?

[wraitii]

Yeah, I realized that later, but I posted too early in excitement. So yes, I believe my problem is with the instancing matrix. Only it seems to work for the position.

[myconid]

Now that I think about it, why are we multiplying a normal with the instance matrix, which I assume contains the modelview matrix? Surely we'd need a normal matrix for that?

You could try calculating a normal matrix from the instance matrix and check if that works for you. It should be the transpose of the inverse of the instance matrix. You may need to change the GLSL version to 140 (temporarily) to check if this helps.

[wraitii]

I'll try that. I have no idea what the instancing matrix is... On the ARB shader, the calculations works and the normals don't look completely wrong, but they might be.

I also have the modelview matrix accessible, I'll try and compare the two.

[myconid]

Ok, I've tried it and "it works". By "it works" I mean it looks like it always does, for me.

  #if USE_INSTANCING
	vec4 position = instancingTransform * vec4(a_vertex, 1.0);
	//vec3 normal = mat3(instancingTransform) * a_normal; <-- replacing this
	mat3 nm = mat3(transpose(inverse(instancingTransform)));
	vec3 normal = nm * a_normal;
	#if (USE_NORMAL_MAP || USE_PARALLAX_MAP)
	  vec4 tangent = vec4(nm * a_tangent.xyz, a_tangent.w);
	#endif
  #else

Tangents etc should also use this transform, btw.

Edited July 6, 2012 by myconid

[wraitii]

Looks like OS X doesn't support GLSL versions higher than 1.2... I'd have to inverse the matrix by hand then.

[myconid]

Looks like OS X doesn't support GLSL versions higher than 1.2... I'd have to inverse the matrix by hand then.

Matrix inverse is a really slow operation that should be done on the CPU side (and I think a function to inverse a matrix should already exist somewhere).

If you just want it to test, then look online for some code. No point in writing your own, as we really shouldn't be doing an inverse in the shader.

[myconid]

So I've been looking at the math... The top 3x3 portion of the instance matrix is the rotation bit (we don't care about translation for normals), which makes it orthonormal by default. The inverse of an orthonormal matrix is equal to its transpose. The transpose of the transpose is equal to the original matrix. Thus, what I suggested above does nothing at all. Sorry.

[wraitii]

I've read about some "scaling" stuffs... Is there no problem with that?

I've also found out about "gl_NormalMatrix" but it seems kind of unreliable (makes the normals change when I rotate the camera, I have no idea wether this is normal or not.)

Edit: anyway, it still crashes with InstancingTransform, which is probably not normal.

Edited July 6, 2012 by wraitii

[myconid]

I've read about some "scaling" stuffs... Is there no problem with that?

I've also found out about "gl_NormalMatrix" but it seems kind of unreliable (makes the normals change when I rotate the camera, I have no idea wether this is normal or not.)

There's no scaling, I believe. I think gl_NormalMatrix isn't used (and it's deprecated), as the engine is passing the transforms in manually.

Edit: anyway, it still crashes with InstancingTransform, which is probably not normal.

What? It crashes? I thought it just looked wrong.

[wraitii]

Nah, it just looks wrong. Seriously wrong, but using "crashes" was more of a metaphor than literal statement.

Though I do get segmentation faults by using the GLSL shaders and adding the temple to the map if I haven't priorly loaded a map, which is kind of weird.

I'll try by manually multiplying the two.

Interestingly, I also get the blinking effect if I set the normal to be any component of "mat3(InstancingTransform)".

And it does not blink if I set my normal to be

(instancingTransform * vec4(a_normal, 0.0)).rgb;

Though the the effect is completely messed up. I'll try also changing the tangents/everything.

Edit: looks like that one works, but for a weird stretching effect over some textures... I'll look into it.

Edited July 6, 2012 by wraitii

[myconid]

Though I do get segmentation faults by using the GLSL shaders and adding the temple to the map if I haven't priorly loaded a map, which is kind of weird.

Replace model_common.xml with this, it might help:

<?xml version="1.0" encoding="utf-8"?>
<program type="glsl">
    <vertex file="glsl/model_common.vs">
	    <stream name="pos"/>
	    <stream name="normal"/>
	    <stream name="uv0"/>
<stream name="uv1" if="USE_AO"/>
	    <attrib name="a_vertex" semantics="gl_Vertex"/>
	    <attrib name="a_normal" semantics="gl_Normal"/>
	    <attrib name="a_uv0" semantics="gl_MultiTexCoord0"/>
<attrib name="a_uv1" semantics="gl_MultiTexCoord1" if="USE_AO"/>
	    <attrib name="a_skinJoints" semantics="CustomAttribute0" if="USE_GPU_SKINNING"/>
	    <attrib name="a_skinWeights" semantics="CustomAttribute1" if="USE_GPU_SKINNING"/>
<attrib name="a_tangent" semantics="CustomAttribute2" if="USE_INSTANCING"/>
    </vertex>
    <fragment file="glsl/model_common.fs"/>
</program>

Interestingly, I also get the blinking effect if I set the normal to be any component of "mat3(InstancingTransform)".

And it does not blink if I set my normal to be

(instancingTransform * vec4(a_normal, 0.0)).rgb;

Though the the effect is completely messed up. I'll try also changing the tangents/everything.

That won't work. You're translating the normals, which makes no sense. And of course it's blinking when you set the normal to a random value.

I'm starting to think the problem is on the CPU side, not in the shader. Maybe the matrix that gets passed in contains invalid values (e.g. infinity) or something like that.

[wraitii]

I dunno, it seems to work really well using this

vec3 normal;
normal = ((instancingTransform) * vec4(a_normal, 0.0)).rgb;
//vec3 normal = nm * a_normal;
	#if (USE_NORMAL_MAP || USE_PARALLAX_MAP)
	  vec4 tangent = vec4(((instancingTransform) * vec4(a_tangent.xyz, 0.0)).rgb, a_tangent.w);

I do have weird black lines.

Edit: lines caused by the AO, actually. The UV sets are not working as intended, I think.

Edited July 6, 2012 by wraitii

[myconid]

If you move the building to a different part of the terrain, doesn't it mess up the lighting?

[wraitii]

Well, if it does, it's quite well camouflaged. It seems to me like it's working on any part of the terrain, on any rotation. It doesn't mess up buildings that use these textures. Shadows, specular, diffuse lighting and parallax are working (I'd upload a picture but my internet is messy right now).

I have no idea why it works, but it works.

[myconid]

If that's the case, see if doing the same thing for the tangents fixes the black lines.

[wraitii]

I must stop editing, as you can't read everything I posted ... The black lines actually come from the AO. I thin the AO textures uses another UV set, and i'm not completely sure it works... Removing that and adapting the texture, I have everything working properly.

If you try it and it works too, I think it'd be the best solution.

Edited July 6, 2012 by wraitii

[myconid]

It doesn't work for me, I'm afraid. Which is actually a good thing, because it's mathematically incorrect.

I'm starting to get the impression that there's something broken with your GLSL setup, not with the code. If one thing is failing, it's probably the engine's fault, but if everything is failing...

[wraitii]

I should probably redownload your fork and start over, to be sure.

That or OS X GLSL implementation is really messed up.

Anyway, I should have just commited a fix for the GLSL version of the water shader.

Edited July 6, 2012 by wraitii

[wraitii]

From a clean download of your modelMapping branch, I can confirm that parallax only works when using the fix I put 3/4 posts higher.

I wonder if it comes from my computer or if I for some reason get a wrong matrix.

[myconid]

My code didn't touch that matrix, so I'd say this is a pre-existing bug (if it's even a bug).

We know the transform matrix works correctly for the vertices. We also know that the normals, if left untransformed, are accessed correctly.

So I can only conclude that unless your computer can't multiply, your problem must be elsewhere.

[wraitii]

Yeah, that's the most logical answer.

BTW, using my loops and if implementations, I have managed to get parallax running with the ARB shader, using the same wrong mathematic as in the GLSL shader.

I'm not sure how to unroll the loop, however, if I want to not use the Nvidia only extension.

Edit: well I think it's the same hack anyway... Lemme get the maths straight for a second:

InstancingTransform is a 4x4 matrix. The top left 3x3 matrix is the rotation matrix, right? That's what we want to use to create the normal matrix, right?

If I understood this all correctly, the normal matrix is the transpose of the inverse of InstancingTransform. However, since the 3x3 matrix is a rotation, it's orthogonal, and thus inverse = transpose. Thus, the normal matrix would be the instancingTransform, right? Or are the other parameters important still?

Edited July 6, 2012 by wraitii

[myconid]

BTW, using my loops and if implementations, I have managed to get parallax running with the ARB shader, using the same wrong mathematic as in the GLSL shader.

I'm not sure how to unroll the loop, however, if I want to not use the Nvidia only extension.

The loop always executes a fixed number of times. After the loop "exits", it continues to get executed but the calculations' results are discarded. For example:

	float h = texture2D(normTex, coord).a;
	float sign = v_tangent.w;
	mat3 tbn = mat3(v_tangent.xyz, v_bitangent * sign, v_normal);

	vec3 eyeDir = normalize(v_eyeVec * tbn);
	float dist = length(v_eyeVec);

	float s;
	vec2 move;
	vec2 nil = vec2(0.0);
	float height = 1.0;
	float scale = 0.0075;
	s = 1.0 / 8.0;
	move = vec2(-eyeDir.x, eyeDir.y) * scale / (eyeDir.z * 8.0);

	{
        vec2 temp = (h < height) ? move : nil;
		height -= s;
		coord += temp;
		h = texture2D(normTex, coord).a;
	}
	..... x8

Edit: well I think it's the same hack anyway... Lemme get the maths straight for a second:

InstancingTransform is a 4x4 matrix. The top left 3x3 matrix is the rotation matrix, right? That's what we want to use to create the normal matrix, right?

If I understood this all correctly, the normal matrix is the transpose of the inverse of InstancingTransform. However, since the 3x3 matrix is a rotation, it's orthogonal, and thus inverse = transpose. Thus, the normal matrix would be the instancingTransform, right? Or are the other parameters important still?

Yup, that's correct.

Edited July 6, 2012 by myconid