A "psychic" shader mod; development begins...

[DanW58]

A Psychic Shader Hack

Hi.  I recently wrote a crazy shader hack that tries to make out the artistic intent to represent metal, and where it decides that metal was intended, changes the diffuse and specular colors to actually best represent metal.  The hack works about 50% of the time...  It misses very few metals;  but it converts many colors NOT intended as metals into metals, foremost among them human skin (which non-the-less becomes a believable representation of sun-tanned skin, as opposed to metallic).
If you are interested in this topic, wish to look at screen shots, or try that shader, please check out this post,

and the posts before --or follow the instruction on-- this linked post:

 

For some background on what I was doing there:

There are two types of light reflectors:  metallic and non-metallic.
Metallic reflections are colored by the metal's color.  Gold is actually yellow, for example.  The color we call "gold" is where chroma is yellow, but usually implies it is more specular than diffuse.  A good first approximation for gold, using the standard texture set, is as 90% yellow (240, 220, 80) in RGB, say, for the specular texture, and black for diffuse.  Unfortunately, most artists have no idea about material optics, and make the diffuse texture yellow, then try to add brilliance by making the specular texture light grey or white.  The result is a fairly good representation of yellow plastic, and that's what it looks like in-game;  not even yellow paint.  Why?  Because gray or white specular with a saturated color in diffuse is the closest approximation for non-metals;  read below.
Non-metallic reflections are NOT colored by the material.  If you find this hard to believe, stop reading this and look around your kitchen for something non-metal but glossy finished and colored.  The fridge won't help convince you, because it is white.  Look for a can or box painted red or blue, or even having many colors, but where the paint is glossy enough to see reflections;  or a glossy colored plastic item that you can look at stuff reflecting off it.  Found something?  Now look for something with multiple colors and orient the painted or plastic item to look at the shallow reflection of the multi-colored thing.  The reflection is probably dimmed, depending on the paint or plastic's dielectric constant and the angle of reflection, but ignoring the dimming, you'll see the colors of the reflected scene represented in the reflection.  (This is not true for metallic reflections: If you have a gold item with a flat side, look at a the same multi-colored scene, and you'll see that blue items look rather black on reflection.)
Metallic specular reflections do not vary in intensity with the angle of reflection.
Non-metal specular reflections DO vary in intensity with the angle of reflection and the material's index of refraction, following Fresnel law.
Paints generally consist of a matte (diffuse) color layer under or within a layer of transparent material.  Some of the incoming light reflects specularly, the intensity of this reflection being (1.0-refraction), in other words, the light splits into a part reflecting off the surface, and a part entering the transparent layer.  The part of the light that enters the transparent material eventually meets the underlying or suspended diffuse pigment, becomes colored by it, and eventually refracts back out of the transparent layer.
So, in a simplified model, we are looking at a non-colored but dimmed specular reflection mixed with a dimmed diffuse color, both dimmings being complementary and depending on the reflection angle.
Now, to have a reflection (and non-reflection) factors depending on reflection angle, these factors have to be computed for every point on a surface in real time;  they cannot come from a texture.  The truest way to represent non-metals, therefore, would be for the artists to have a color channel to express the dielectric constant or the index of refraction of a material;  but most artists are not science oriented people, and would probably hate to have to specify dielectric constants;  but this starts to get off topic and into politics.  The fact is, non-metals reflect without coloring the reflections, and so game assets where specular texture is some shade of grey while diffuse is colored, will look like a non-metal.

And so my shader hack tried to identify where a metal look was actually intended, and delivered it.

 

FAQ:
Q: What about non-colored metals? Can they be told from non-metals?
A: Yes.  Reflections off a chromed surface are non-colored, just like dielectric reflections, but they do not change in intensity with angle.  Non-metals, on the other hand, reflect the least when looking face on, but most when looking at them at a shallow angle.

Q: Is Fresnel easy to compute?
A: No. Fresnel is a long formula spanning the whole width of a page. In fact, it is two formulas: one for longitudinal and one for transversal polarizations.  Furthermore, Fresnel factors do not only depend on refractive index (or dielectric constant) and angle, AND polarization... they also depend on wavelength (color) of the light.  What you find in game engines are ridiculous oversimplifications of Fresnel.  20 years ago I implemented Fresnel in a shader for glass windows.  I tried to find a half-decent approximate function.  I used a tool called DataFit that takes in many points and tries to come up with math formulas that fit the data statistically. Even then it was to no avail.  After about two fruitless weeks of math work, I finally gave up and implemented the whole Fresnel formula from the book into the shader, and it was done per-color channel, so it gave slightly different fresnel factors for red, green and blue.  The result was absolutely spectacular, however;  people could not believe such a photo-realistic representation of glass was even possible.  But the computation was so long that this glass shader had to be its own shader and do nothing else.  

Q: Does 0ad have Fresnel?
A: Yes, to an extent...  The Pyrogenesis engine has a simplified implementation of Fresnel which, I believe, is only used for water refraction and reflection. As far as I know, it is not available at the materials level, such as to compute paint, plant leaf or human skin specularities.

Q: What's the difference between a plastic and a paint?
A: In a high gloss paint, such as car paint, the diffuse color pigments are below a glossy, transparent lacker layer; so the fresnel reflection tends to be pure, and only the refracted portion of the incident light comes back out colored.  Plastics are also made of a transparent, dielectric medium and diffuse pigments, however in plastics the two are mixed together prior to injection molding, rather than layered, such that many grains of pigment break through the surface. The optical model is for a fresnel reflection tainted (mixed) with diffuse color, where the portion of diffuse color is not under Fresnel (angle) modulation.  Then, in addition, there is the re-emerging refracted light, also coming out colored, but modulated by fresnel refraction.  To simplify all this:  Plastic representation can be computed as a glossy paint representation, but where the whole result (diffuse AND specular) is then dimmed by a factor, say 10% (mul by 0.9), and then added a 10% standard diffuse color computation; --where the 10% represents the fraction of surface are where pigments are exposed, at the microscopic level.  The result is stunning, however.  When I implemented plastic shading, I applied this to a Harley model I had made earlier, and I applied glossy blue paint to the gas tank, and glossy blue plastic to the tool box under the seat (same base color, same dielectric constant, same shininess power), and in-game you could easily tell that the paint was paint, and the plastic was plastic !!!  ... though of course you could not tell HOW you could tell ... ;-)

Q: You've said elsewhere that white diffuse and white specular don't make sense together;  but then how do I represent a fridge?
A: Let's look first at how a fridge works (on the outside):  We are talking about a glossy paint with a white base.  So, at any given angle, after calculating refracted portion from Fresnel, call it RefractionFactor, we calculate ReflectionFactor = 1.0-RefractionFactor, and the result for any pixel on the screen would be,
fridgePixel = RefractionFactor*RefractionFactor*DiffuseColor + ReflectionFactor*EnvironmentMapFetch(ReflectionVector);
Why did I multiply by RefractionFactor twice?  Because it affects refraction of light into the material, as well as coming out of it.  But even this is a gross simplification.
Anyways, as the the angle becomes more shallow, the white base gets darker, dimmed, as the reflections get brighter. This is VERY different from having white diffuse light from all over the fridge, PLUS mirror-like reflection from all over the fridge as well...  A case of twice as much energy bouncing off it as the energy coming in...  If fridges were like that, we should place them around solar energy farms !  XD  But so you simply CANNOT represent a fridge without Fresnel and without a way to specify dielectric constant.

Q: I'm an artist; how can I specify paint, such as on a ceramic surface?
A: If the texture stack allows specification of dielectric constant, you just need a base color in the diffuse texture, and some dielectric constant for the laquer medium (anywhere from 1.5 to about 7.0, but more realistically between 2.0 and 5.0), leaving the specular texture black;  and of course you might also want to specify an appropriate power for shininess, to indicate if this is a blurry, eggshell or chrystal polished surface finish.  If you don't have a way to specify dielectric constant, you'll just have to make specular texture some shade of grey... but it will look like a very dull and boring plastic;  not like any real paint.  If you really want paint to look like paint, you need a channel for dielectric constant in the texture stack, so that specularity changes with angle;  so ask for it;  or stage a demonstration  ;-)  Joking aside, I think most game project devs around the world are reluctant to introduce fancy channels in the texture stack because they fear the artists won't use them (or worse, mis-use them);  so if you are an artist and you want this, then be sure to be heard.

Q: So, to represent gold I should use 0,0,0 diffuse and 240,200,80 specular?
A: That's virtual gold, with a perfect surface.  Real gold is more complicated...  In real metals there are usually surface imperfections at the microscopic level, craters, canyons, open caves, such that some of the incident light comes out after bouncing several times, not just once, which is what diffuse reflectance tries to model.  In other words, diffuse should not be exactly zero, or black, for real metal.  However, since metal reflections are colored, and this diffuse component results from multiple light bounces within metal cavities, the diffuse color of a metal is colored multiple times, and ends up being a power of its specular color.  If we use 0~1 notation for color channels, assume gold's specular color is (0.9, 0.8, 0.3) --and this depends on what it is alloyed with, the karats, etc.  After two bounces, the coloration becomes the square of that, namely (0.81, 0.64, 0.09).  After a third bounce, the coloration is the cube of the specular color, thus (0.729, 0.512, 0.027).  If we assume most of the diffuse light comes from between two and three bounces, we could average the square and the cube of the specular color, thus (0.77, 0.57, 0.33) for our raw diffuse basis.  However, most of the surface is polished, single bounce;  and only a small fraction of the surface has pot-holes.  Assuming a 10% blemished surface (for first Century technology), we could multiply diffuse base by 0.1, and specular by 0.9.  Doing so we obtain the following colors for gold: diffuse = (0.077, 0.057, 0.033), specular = (0.81, 0.72, 0.27).  These numbers will produce stunningly more realistic gold looks than the popular black+yellow.

Q: I fixed my (whatever material) representation, but still doesn't look realistic in-game... What's next?
A: For specular reflections to look realistic, several things have to happen:  Proper material representation is one of them.  Secondly, there needs to be environment mapping in the shader.  Presently I'm not sure there is, but I haven't specifically looked for it.  If the only specularity visible is the sun's reflection, this won't work.  Wherever in a gold bracelett the sun is not glittering off, there should still be a reflection of a blue sky or a cloud... --not just nothing.  Third: There needs to be environment cube maps with LOD levels with progressive blurring, and the environmental reflections of shiny objects should have their environment mapping LOD bias modulated to account for the material's shininess power, to agree with the blurring of sunlight reflecting off it.  If environment mapped reflections are sharp (non-LOD-biased) while specular highlights are blurred, the result is a visual cacophony.  So art AND technology need to cooperate.

Q: Funny that your hack tries to fix metal representation, which CAN be made explicit through the textures;  and not non-metals, which can't...

A: Funny you should ask.  Indeed, metal representation should not need any special help;  it is easy;  just the right diffuse and specular colors will do the trick.  Unfortunately, most metals are poorly represented, and not just in 0ad but in all games, universally.  It's a disaster that should not be.  In any case, this new shader will try to address non-metals as well, and even implement a bit of Fresnel.

 

Back to the shader hack:

My shader hack tried to second-guess where metallic look was intended (but poorly implemented), and changed the implementation on the fly, pixel by pixel, real time.  Like I said, it worked about 50%, and it looks pretty cool, but it's admittedly too invasive.  Additionally, it has the political problem of discouraging good artistic practice by changing what the artist does as it sees fit, often for the better;  but threatening to change a good material into a bad one.

Here's a plan for my next shader hack. The plan is to make this "metal intent detection" hack powerful and more general, and yet benevolent.  How?  By first detecting CORRECTLY represented materials and NOT changing them!;  not touching them at all;  and only THEN dealing with all the impossible materials and second-guessing what they intended to represent.  That way, this new shader would be able to handle new and better material representations without altering them, and ONLY fix what REALLY needs to be fixed.  It is a tall order, however...  For this to work, it is not enough to identify correct representations for metals, but also "most correct possible" representations of non-metals given a texture stack that doesn't include such information as dielectric constant. With such a limited texture stack, high gloss car paint cannot be distinctly specified from cheap paint or from plastic of the same color;  but such distinctions may be hopelessly attempted by an artist via the brightness of the specular texture, which makes the result look like dull plastic wrapped in glossy film, and yet it is the best thing the poor artist can do.  So, not only we need to detect correctly represented materials before fixing all the rest, but we must add in-between them a category of best attempt material representations possible, and strive to deliver the actual intention with minimal intervention.  The most incorrect and ludicrous diffuse and specular color combinations should be saved for last.

Here is a basic detection flow based only on diffuse (diff) and specular (spec) RGB colors.  Take it as a work in progress.  In shaders you try to avoid conditionals, whereas this pseudo-code below is a deep, nasty conditionals rats-nest;  but so this is going to be a bunch of "boolean floats" computed from input data, and a bunch of boolean floats controlling operations by multiplying their data pipes by 1.0 or 0.0;  but it helps in visualizing things to write this out as a nested conditional.  In other words, the pseudocode below is a visualization exercise;  nothing more.

But another foggy visualization in my mind is as a 6-dimensional cube, where the six dimensions are the H, S and V (hue, saturation, intensity) for diffuse and specular.  This is a VERY sparse matrix when it comes to representations of "Real World" materials, but actual representations from texture artists could fall anywhere.  Within this hyper-cube there are zones corresponding to real material types;  and these zones can be large or small, long or short, wide or narrow, axially aligned or oblique, straight or curved.  An ideal materials-correcting shader would find the location of an incoming color set for a pixel, find the nearest real material location to it in any direction in this 6D space, and move it there instead.  But this is easier said than done.  Anyways, the process of writing this pseudocode will hopefully shed light on the problem.

Most easily identified as correctly represented material colors combos are detected first:

 

// Clairvoyant shader, reads artists' hearts and minds, and produces the
//materials they actually intended to portray :)  Shader mod pseudo-code:

diff = convert_to_HSV( diffuse_tex_color );
spec = convert_to_HSV( specular_tex_color );

if( diff.v < 0.05 ) //black diffuse
{
    if( spec.v < 0.05 )      //black specular
    	material = BLACK_HOLE;
    else if( spec.v < 0.15 ) //dark specular
    	material = CHARCOAL;
    else                     //not so dark specular
        material = METAL_MIRROR;
}
else if( diff.v < 0.15 ) //dark diffuse
{
    if( spec.v < 0.15 )  //dark specular
        material = CHARCOAL;
}
else
{
    if( spec.v < 0.1 )
        material = MATTE_MATERIAL;
    //almost nothing to do
}
else if( diff.v < 0.1 )
(
    //VALID REPRESENTATION OF A METAL MIRROR
    disable_fresnel();
    set_shininess_power( 1000 ); //a very high shininess
}
/*
 NOTE: It may be worth considering to just ignore overly dark input, as it
 does not make too big a difference to the visual experience, so there's not
 much of a point making guesses and corrections.
 */
else if( diff.v < spec.v ) //specular brighter than diffuse
{
    if( diff.s >= spec.s ) //if diff sat equal or higer than spec sat,
    {
        if( spec.s > 0.1 ) //if specular has some saturation
        {
            if( diff.s + spec.s < 0.1  //if both are greyish
            || diff.h ~= spec.h )   //or if their hues match
            {
                //CLEARLY AND CORRECTLY REPRESENTED METAL!!! YEY!!
                //nothing to do vis-a-vis the colors; congrats!
                set_shininess_power( incoming * 2 );
                disable_fresnel();
            }
            else //saturated colors in diff and spec that don't match in hue
                spec = BLACK; //convert this abomination into a matte color.
        }
    else if( diff.s > spec.s ) //high achromatic specularity
    {
        //probably a high-gloss paint was intended
        set_shininess_power( incoming * 2 );
        set_dielectric_constant_to( 5.0 * spec.v );
        enable_fresnel();
    }
    else //spec more saturated than diff; aberration!
    {
       //a desaturated diffuse color could conceivably be an intent
       //to depict salt or dust on a metal surface. The correct way
       //would be to darken the specular wherever a bird dropping
       //covers the diffuse.  We'll try to complete such an intent
       //by darkening spec in proportion to the loss of saturation
       //in the diffuse
       disable_fresnel();
       spec *= ( diff.s / spec.s );
       set_shininess_power( 10 ); //a very low shininess
   ....... to be continued, or trashed, we shall see .......
       
       
 switch( material )
 {
 case CHARCOAL:
    spec.v += diff.v;
    spec.s = 0;
    diff.v = 0;
    disable_fresnel();
    set_shininess_power( 22 ); //a very low shininess
    break;
 case BLACK_HOLE:
    disable_fresnel();
    break;
 case MATTE_MATERIAL:
    disable_fresnel();
    break;
 case :
    break;
 default: 
 }

 

Edited February 12, 2021 by DanW58

[hyperion]

Roughly it's metal and non metal materials, the rest shouldn't be of much concern in a real time engine, right? So why not simply provide a map where the artist can explicitly state what are metals and what are not. Wouldn't that make it easier for shader and artist alike with no possibility of misidentification?

 

Btw, I quickly had a look at the patch in the post you linked but need some time to go through more maps and civs to give a somewhat educated answer, haven't ignored you

[Panther]

It's nice seeing there are still people in this community that spend much time and effort in improving this game. Keep up the good work.

[DanW58]

1 hour ago, hyperion said:

Roughly it's metal and non metal materials, the rest shouldn't be of much concern in a real time engine, right? So why not simply provide a map where the artist can explicitly state what are metals and what are not. Wouldn't that make it easier for shader and artist alike with no possibility of misidentification?

 

Btw, I quickly had a look at the patch in the post you linked but need some time to go through more maps and civs to give a somewhat educated answer, haven't ignored you

Actually, metals and non-metals account for all materials ...     But I think I know what you mean.  However, I do need to identify ALL materials that are well represented so as to avoid messing with them;  and whatever is left should be MIS-representations.  And I don't mean this IN the shader program;  I mean at the pseudo-code stage.  Hopefully, 90% of the pseudo-code will generate no real code.

Indeed, having a metal/non-metal boolean channel in the textures might help things.  If we did that, we'd have all four channels of the specular to play with, to specify non-metal characteristics.

Well, okay, suppose we use a texture for specular that has a single-bit alpha, and make that alpha channel specify metal vs non-metal:

Then, we can use RGB in the spec channel normally if it is a metal;  but if it is a non-metal we can use G for dielectric constant and R for surface impurity (where high impurity results in plastic, as opposed to high gloss paint).  That leaves B not used, but I think I do have a good use for it.

Then the diffuse texture could have a ... (7 or 8 bit alpha?, the good DDS, I forget) and use this alpha channel for AO... darn, I forgot again you use a second UV map for AO.

What a great idea!, by the way.

Okay, so we're laughing.

The diffuse alpha channel could still be used for one very important thing:  a detail texture modulator.  Food for thought.  Even if detail textures won't be implemented till some point in the future, it might help to keep them in mind and make room for them.

The only issue I see with this kind of texture packing, and with the use of a boolean channel, in general, is that it's going to run into trouble due to filtering.  It can cause artifacts galore when you filter a boolean.  I think the unwraps need to leave a bit more space between islands.  I used to leave about 15 texels of space, and then I'd grab all textures in the set and use GIMP with a mask to blur the textures so as to smoothly bridge colors between neighboring islands.

For testing the patch, it might help save time to just test extremes, such as the Acropolis at Night map, where darkness reigns supreme.

54 minutes ago, Panther said:

It's nice seeing there are still people in this community that spend much time and effort in improving this game. Keep up the good work.

Thanks!  I just do this because I enjoy tackling hard problems;  and I got the time because Elon won't hire me because I'm Canadian.

Edited February 12, 2021 by DanW58

[Loki1950]

I remember that glass shader @DanW58 it truly was awesome especially for the cockpits of the ships you designed even remember the flame wars about it the people chiming in did not even use blender then.

Enjoy the Choice

[DanW58]

2 minutes ago, Loki1950 said:

I remember that glass shader @DanW58 it truly was awesome especially for the cockpits of the ships you designed even remember the flame wars about it the people chiming in did not even use blender then.

Enjoy the Choice

You have better memory than me, Loki.  Then again, there were so many flame wars back then that remembering each particular one is difficult

[DanW58]

@hyperion

Wait!  The texture set MUST include shininess power.  THIS is THE purpose for that diffuse Alpha channel.  And it HAS to be 7+ bits;  precision is crucial.  This is good for believable depictions of metal and wood.  Nothing looks more believable in a metal than subtle splashes of shininess power variations.  If the splashes add power they look like oil marks;  if they subtract power they look like sugar marks from a coffee mug that once sat there.

Wood is also best represented by copying the diffuse texture onto the shininess with low contrast, which gives the wood "depth", in the sense that the grain of the wood then becomes more or less visible depending on how the light reflections play.  There's no texture channel more valuable to realism of materials than the shininess, in my experience;  not even normal map.  Having spec power come from a setting, like most people do, is a huge wasted opportunity.  Wood goes from cartoonish to photo-realistic, with shininess variations in the grains.

For modulation of a detail texture we can use that blue channel left over in the specular texture when the material is a non-metal.  Which means we have to forgo detail textures for metal objects, which is fine since metal objects are typically small;  they don't need extra detail.

This will be an EXCELLENT packing.

Edited February 12, 2021 by DanW58

[gameboy]

@DanW58 Let me see the new adjustments you've made. It's going to be a whole new look.

Edited February 13, 2021 by gameboy

[Stan`]

@vladislavbelov thoughts?

 

[DanW58]

I'm sorry gentlemen;  I distracted myself from this development to revisit the previous shader work;  I was unhappy about the lack of environment mapping, so I decided to try and fake one, by adding ambient light to specular reflections.

By itself this would break things, because polished materials would reflect the entire sky in all directions and just look like lamps.  For this to even appear to work, I needed some kind of specular occlusion computation, namely how features nearby occlude reflection rays, preventing them from seeing the sky.  But specular occlusion is hard.

But I managed to fake it a little bit;  see the screenshot, look at the two metal woks where the fires are, and you'll see inside them circles of light... That's circles of reflected sky.  If you move around, these circles keep facing you.  EUREKA!

The problem now is the queen's dress, which has too much specularity, and therefore shows sharply edged dark zones of specular occlusion;  but that's an art issue.

My occlusion threshold is for now a sharp step() function;  but it should be a smoothstep(), with the width between the two threshods increasing with the inverse of material shininess.

The relevant shader code goes like this for now:

      float ao = texture2D(aoTex, v_tex2).r;
      vec3 ao_color = mix(vec3(0.5), vec3(ao,ao,ao), effectSettings.w);
      float reflection_unobstructed = step( 1.0-ao, dot(normalize(v_eyeVec),normal) );
      vec3 spec_from_sun = sunColor * pow(max(0.0, dot(normal, v_half)), specPow) * 0.25 * sqrt(specPow);
      specular.rgb = specCol * reflection_unobstructed * (spec_from_sun + ambient);

The computation of 'reflection_unobstructed' merits some explanation.

The ambient occlusion (ao) carries some cumulative information about over-all occlusion of light rays from all directions.  Of course, we cannot tell in detail what ray is or isn't occluded by the tip of a spear nearby, but we can make a rough assumption about a "cone of visibility".  Assume for now that this cone is oriented as the normal to the surface;  we won't depend on this assumption;  just an arbitrary setup.  Now, we could ask what is the maximum angle to the surface that could be reflected, assuming the occluding environment is circular.  This boils down to computing the half angle of a cone whose solid angle represents the ao value.

The formula for solid angle of a cone is, from the half-angle, is:   SA = 2*pi*(1-cos(HA))

The units are steradians and radians, respectively.

Now, there are 6.28 steradians in a semisphere;  so if we use "semispheres" (or "hemispheres"?) as our units,  SA = 1 - cos( HA )

Moving terms:  cos( HA ) = 1 - SA

So the half angle is the arccos...  HA = arccos( 1 - SA )

Now, the solid angle measured in hemispheres is nothing but the ao value, as ao = 1 when a point on a surface is totally unoccluded and therefore gets a whole hemisphere of light.  So,

                              HA = arccos( 1.0 - ao )

So then we would compute the angle of reflection to the normal, and compare it with this angle... but wait!:  To calculate angle of reflection to normal we'd have to compute dot of normal and reflection, and then take the arccos of that...  Why not save the arccos computations and simply compare cos to cos?

The cos of an angle up to 90 degrees decreases with angle, so comparing the cos of the angles, versus the angles, simply reverses the logic.

 

But now, can we make a further assumption about the actual orientation of this "cone of occlusion"?

Well, imagine a point that is so occluded that the ao value is 0.01, --almost black--, now the question is, does this very narrow cone of visibility include the eye vector?

 

Think about it.

 

You'd probably say "fat chance!", but you'd be terribly wrong in your skepticism...

If the vis cone did NOT include the eye vector, however narrow it may be, we would not be processing it, because it would be occluded!!!

So, we know for a fact that the eye vector is within the visibility cone.  It *could be* at a point near the edge of the cone, and the cone could be assuming any rotation all around the eye vector to possibly include rays at twice the half angle;  or the vis cone *could be* dead center aligned with the view vector.  We don't know which.

But now we know enough to shamelessly assume that if view dot normal is less than 1-ao a reflected ray has less than 100% chances to reach the sky.

 

 

Edited February 13, 2021 by DanW58

[DanW58]

So, I changed the step() function to a smoothstep(), but not to account for material shininess, but rather to account for uncertainty.  I'm smoothly interpolating between comparing view dot normal to the square root of 1-ao, to comparing to 1.ao squared.  Just arbitrary thresholds for now.  I'm not attaching the shader yet, as this is still evolving.

In the meantime, I took several screenshots to show you how the specular light and dark zones in the queen's dress change with viewing angle.

EDIT:

Another art problem I notice:  Even if the queen's dress being so specular would make sense, reflections off the lower part of it should be occluded by the ground.  This tells me that the AO baking was done "in a vacuum" --i.e. without a prop to simulate occlusion by the ground plane.

 

Edited February 13, 2021 by DanW58

[Stan`]

We can't use a prop to simulate the ground plane because it wouldn't work on uneven terrain...

[DanW58]

Well, this is part of the general truth that ao bakings are static and don't work exactly right in every situation, so you try to average all situations in how you set it up, to minimize discrepancies.  In this case, terrains that are vertical, or up-side down on top of the model are not due consideration;  it is only slanting terrains, and the average of all possible slanting terrains is a horizontal one.  But having NO terrain is a much worse setup for ao baking than having a horizontal one.

Edited February 13, 2021 by DanW58

[Stan`]

Well maybe it's possible for some but for big buildings it's not really.

[DanW58]

How about making a prop that's half way between a flat plane and a mountaintop?  A hill prop.  It wold at least provide SOME occlusion from below, as opposed to nothing.

EDIT:  But actually much closer to a flat plane than to a hill top, reason being that even at the top of the hill, sky scatter light does not come from below the horizon, anyways, unless it's Mirror Mountain...

Edited February 13, 2021 by DanW58

[DanW58]

Frankly, the BEST policy would be to have TWO ao bakings per model, with and without a ground prop.  That way the shader could use an AverageGroundColor uniform, and color the occlusion difference with it, giving a first order approximation of second reflection by the biggest second reflection source of all, the ground...

EDIT:  But if we must have only one AO baking, it should have a flat ground prop.  Yes, even if the terrain doesn't match it doesn't matter, because what the prop emulates is the sky limited by the horizon.  Frankly, ao baking tools are too stupid.  They should all have the option to bake from a hemisphere of light, instead of forcing people to use props, and then wasting half of the baking time computing useless rays from below the ground...  Idiots...

 

Edited February 13, 2021 by DanW58

[DanW58]

Okay, this is not fully debugged, but worth sharing anyways.  I got the problem of missing ground occlusion info in the AO as far as specular occlusion is concerned by comparing the Y value of the reflection vector with zero, so, if the eye view vector reflects even slightly downards it should reflect ground.  The only problem is it doesn't work consistently for some reason...  It works with queen's dress;  but it doesn't with the sides of the chairs they sit on, which have become metallic;  don't tell anybody...  See the screenshot below.

I'm including the shaders for anyone wishing to try it, but keep in mind that the color of the ground is green in the shader;  so if you use it on ground of any other color it will give weird results.  For this to work, ground color has to come from a uniform.

 

foreground_overlay.fs foreground_overlay.vs

[DanW58]

Shader's minorly updated, attached.

There's a problem somewhere that I think it's something with the normals in the model.

I get a very strange artifact when looking from the left.  By the way, I switched from Oceanside to Belgian Bog, to have more sunshine for testing.

Yeah, if you look at the boxy seats, from the left side I get these square shadows that don't make sense, and they move and flicker as I turn the camera.

 

model_common.fs model_common.vs

Edited February 13, 2021 by DanW58

[DanW58]

Nah,  I found what the problem was, vis a vis that weird artifact.

//the code as it was:
  vec3 color = (texdiffuse * sundiffuse + specular.rgb) * getShadow();

//the fixed code:
  vec3 color = (texdiffuse * sundiffuse) * getShadow() + specular.rgb;

Specularly reflected light is not affected by shadows in the real world.

It's like something casting a shadow on a mirror;  it doesn't prevent it from reflecting.  In fact, you can't even see a shadow on a mirror unless there's some dirt on it.

Only model_common.fs just changed;  I'm including all 3 shaders because the changes go together.

@hyperionYou should give this a try too.  You might fall head over heels in love with this shader;  it adds a lot of life to the scene;  more than you can tell from just looking at screenshots.  (Except that women's arms look fatally sunburnt.)

model_common.fs model_common.vs

terrain_common.fs

Edited February 13, 2021 by DanW58

[DanW58]

@hyperionor anyone that knows, could it be that many models don't have a specular texture, but that the engine is defaulting specular to white in the absence of a specular texture?  I just can't understand why there's so much specularity in everything.  The default color for specular, if any, should be black;  NOT white!

(The Vegastrike engine used to do exactly that, at one time, and any spaceship without a specular texture looked like a floating mirror from hell.)

EDIT:  Nevermind;  I think I get it:  Some units have textures;  the others have a setting for specular color.  I overrode it with black in the shader.  Still, most things seem to have ridiculously high specularity coming from the textures...

Edited February 13, 2021 by DanW58

[DanW58]

   EUREKA !!! 

I got the metal detection algorithm to NOT detect human skin as metallic.

Women's arms no longer look fatally sun-burnt.

In fact, there's now far fewer false metal detections than before.

EDIT:  Added more screenshots with random civs, different maps.  Note how just a handful of items look metallic.  Sometimes a couple of small pots.  Particularly in the last screenshot below, notice the shields inside the roofed patio;  they correctly reflect sky from our general direction, in spite of being in a dark place.  YEY!!!

model_common.fs model_common.vs terrain_common.fs

Edited February 13, 2021 by DanW58

[hyperion]

Dan,

what I meant is except for metal all others are similar enough to not care. I mean photorealistic rendering can't be done as a 1-2 frames per hour don't allow for a fun game. So we end up with compromises anyway.

As for implementation, as it's 0ad we definitely need a 3rd distinct uv for the metal bit map! More seriously, clever encoding shouldn't be a concern. What is important is to make importing and exporting from lets say blender straight forward. So I downloaded a pbr texture for use in blender. It comes with the following maps: albedo, ao, height, metallic, normal, and roughness. Sure you could substitute metallic and roughness with specular, diffuse and glossiness. I guess blender also supports the latter somehow but the former seems like it's more intuitive for an artist as it's closer to how we perceive materials.

I'm not fond of the approach of detecting material. Let's say the algorithm works for the current assets. Now I would like to add a new model and the algorithm misdetects  metal. So how should I know what to change to get the heuristic working. To me it seems this approach is a pain in the neck for artists. I might be wrong ofcourse.

The screenshots at least demonstrate that 0ad may profit from a more realistic render.

[DanW58]

Yeah, I'm a bottom up guy;  I like to establish what is needed before I worry how to obtain it, and so I think of the best (most complete and efficient) texture stack at the graphics level, and how to persuade blender to deliver it is an afterthought to me.

But I've been there;  I was working on getting Blender to look on screen the same as the engine I was using at the time, while using the same textures;  and it wasn't easy.  It should be much easier today with material nodes.  The old pipeline was utterly inscrutable.

Part of the trick of getting a good stack is to take bit depths of the format into account.  I remember there was one DDS format that had a nice, 8 bit alpha channel, ideal for precision sensitive things like specPower or AO.

14 hours ago, hyperion said:

what I meant is except for metal all others are similar enough to not care. I mean photorealistic rendering can't be done as a 1-2 frames per hour don't allow for a fun game. So we end up with compromises anyway.

You don't have to worry about performance with my work;  it is always my top concern together with realism.  The shaders I was writing for Vegastrike were about 4 times as big as the current shaders are here, and were running at full frame-rate ... 20 years ago, those were the days of 150 nm transistors;  we're at 7, going to 5 nm soon.

Photorealistic rendering does not require raytracing.  I had photorealistic shaders in real time that were in many ways superior to raytracers.  In really MANY ways.  Loki1950 will tell you.  Not that raytracers couldn't be better, but there's people behind everything, and many of them... er, MOST of them are not perfectionists, and even hate perfectionists.  Happens in every organization.  They treat perfectionists like they are trash.  So you got dozens of raytracers that look the same, make the same mistakes, or that add ridiculous amounts of noise for no good reason.  My experience is that if you add a little bit more realism every day, or every other day, after a few months you got an award-winning shader;  and it doesn't have to be too big or slow;  on most days, the work is not about adding code but merely improving the existing code.

But I think I know what you mean by "all other materials", probably you refer to say matte materials, such as cloth or carpeting, for example;  where there is not much specularity to speak of, and therefore it is moot whether it is metal or non-metal.

Totally agreed also about risking the shader ruining a good material.  That's precisely why I started to work on a new "psychic" shader that would guarantee not to intervene where a material representation is valid.  The current metal detection shader does not offer such a guarantee.  It cannot, as it is rather hackish.

However, I do have to say that this metal detection shader does not mess with materials that have zero specularity.  It looks at the specular texture for hints of metallicity;  if there is no specularity to analyze, then it does nothing.  And you might argue that specularity may come not from a texture but from an xml file;  and it would have been true an hour ago;  but I just found that, and disabled it;  if specularity doesn't come from a texture, now I'm setting it to zero.  Why?  Because uniform specular color is for the birds.  And I don't mean for the chickens in the game...  So, indeed, all it is doing is converting some non-metals to metals, and leaving the rest of the materials alone.

Another important thing I can assure you about this shader is that it is targeted and deliberate.  You can change the #if 1 near the bottom to #if 0, and set which float variable you want to watch in grey-scale.  Set it to "is_metal", and you'll see a black world with a bunch of white spots.  The white spots are what is detected as metal.  You will very rarely see any shade of gray.  Some metal detections may be incorrect, I do admit;  but you don't need to fear that everything in a scene is being slightly changed, unless you see large areas of grey, but you won't.

I'm working on cleaning up this metal detection shader.  My portions of the code grew monolithic;  I'm trying to fit it now into subroutines so that it is easier to understand.

I'm also intending to add a bit of Fresnel for non-metal objects, particularly human skin and plant leafs;  probably will have it working within this weekend.  I want to see them both sporting a natural shimmer.

PS, Would be nice to have a per-map, or map location oriented "GroundColor" uniform, for specular reflections occluded by the ground, so that the shader can "reflect" this color, instead of the sky-box.  One way to approach this would be for any object on a terrain, to copy the color of the ground at the instancing location onto a variable, then pass it on to the shader.  I can probably code it myself;  just running it by you in case you have better ideas.  Heck, I also need an index of refraction uniform;  this would be per-material...  In lieu of a texture channel, of course.

One thing this game is lacking is flowers.  I mean, the berries look like flowers, but what about wild flowers?  Flowers give a special opportunity for nifty shader hacks, as many flowers (red and yellow, primarily) can absorb blue light and re-radiate it at a lower wavelength.  This excludes blue, violet and white flowers;  mostly red, orange and yellow flowers have this ability to *emmit* light in proportion to the amount of blue light they receive.  Modelling this should result in very "vivid" flowers...

 

Edited February 14, 2021 by DanW58

[DanW58]

Anyways, I'm by no means insisting that this metal detecting shader be adopted;  but I'm continuing to work with it, as I'm in the process of solving some problems that are better solved with a working shader than with a dozen psychic shaders in the sky.  I continue the clean-up work, and many arbitrary factors and formulas are screaming at me to be either justified or fixed.  Some of these problems I solved 20 years ago, but I no longer have the papers or the glsl code.

 

Problem Number 1:

One issue that I seem to be the only guy in the world to ever have pondered, is the fact that the Sun is NOT a point-source;  it has a size.  I don't mean the real size, but the apparent size:  its diameter, in degrees.  If we were on Mercury, it would be 1.4 degrees.  From Venus, (if you could see through the darned clouds), it would be 0.8 deg.  From Mars it looks a tiny 0.4 degrees diameter.  From Earth: 0.53 degrees.

This should be taken into account in Phong specular shading.  The Phong light spot distribution (which is a hack;  it is NOT physics-based) is (R dot V)^n, where R is the reflection vector, V is the view vector, the dot operation yields the cosine of the angle between them, and n is the specular power of the surface, where 5 is very rough, 50 is kind of egg-shell, and 500 is pretty smooth.  Given an angle between between our eye vector and the ray of light reflecting from the spot we are looking at, if the alignment is perfect, the cosine of 0 degrees is 1, and so we see maximal reflection.  If the angle is not zero, but it is small, say 1 degree, the cosine of 1 degree is 0.9998477, a very small decrement from 1.0,  but if the specular power of the material finish is 1000 (very polished), the reflection at 1 degree will fall by 14% from the 0 degree spot-on.  But with a perfect mirror, --a spec-power of infinity--, a 1 degree deviation (or any deviation at all) causes the reflected light to fall to zero.  But that is assuming a point-source light...

If what is reflecting off a surface is not a point source, however, the minimum specular highlight spot size is the size of our light source.  This can be modeled by limiting our specular power to the power that would produce that same spot size from a point source.  But this limiting should be smooth;  not sharp...

 

Problem Number 2:

This is horrendous graphical mistake I keep seeing again and again:  As the specular power of a surface finish varies, specular spotlights change in size, and that is correct;  but the intensity of the light should vary with the inverse of the spotlight's size in terms of solid angle.  If the reflected light is not so modulated, it means that a rough surface reflects more light (more Watts or Candelas) than a smooth surface, all other things being the same, which is absurd.  As the specular spots get bigger, they should get dimmer;  as they get smaller, they should get brighter.

But the question will immediately come up:  "Won't that cause saturation for small spot-lights?"

The answer is yes, of course it may.  So what?  That's not the problem of the optics algorithm;  it is a hardware limitation, and there are many ways to deal with it...  You can take a saturated, super-bright reflection of the sun off a sword, and spread it around the screen like a lens-flare;  you can dim the whole screen to simulate your own temporary blindness...  Whatever we do, it is an after-processing;  it is not the business of the rendering pipeline to take care of hardware limitations.  Our light model is linear, as it should be, and as physics-based as we can get away with.   If a light value is 100 times greater than the screen can display, so be it!  Looking at the reflection of the sun off a chromed, smooth surface is not much less bright than looking at the sun directly.  Of course it cannot be represented.  Again, so what?!

So the question is, how much should we set the brightness multiplier as specular power goes up?  And also, at what specular power should the brightness multiplier be 1?

 

Research:

The two problems have a common underlying problem, namely, finding formulas that relate specular powers to spot sizes, where the latter need to be expressed in terms of conical half-angle and solid angle.

If we define the "size" of a specular highlight as  the angle at which the reflection's intensity falls by 50%, then for spec power = 1, using Phong shading, (R dot V)^n, the angle is where R dot V falls to 0.5.  R dot V is the cosine of the angle, so the angle is,

          SpotlightRadius(power=1) = arccos( 0.5 ) = 60 degrees.

Note that the distribution is equivalent to diffuse shading, except that diffuse shading falls to half intensity at 60 degrees from the spot where the surface normal and the light vector align, whereas a specular power of 1 spotlight falls to half intensity at 60 degrees from the mid vector between the light and view vectors, to the surface normal.  But the overall distributions are equivalent.  We can right away answer one of our questions above, and say that,

                 Specular power of 1.0 should have a light adjustment multiplier of 1.0

How this multiplier should increase with specular power is yet to be found...

But so, to continue, what should be our formula for spotlight radius as a function of specular power?  For a perfectly reflective material,

        Ispec/Iincident = (R dot V)^n

 If we care about the 50% fall-off point, we can write,

        0.5 = (R dot V)^n

        (R dot V) = 0.5^(1/n)

So our spot size, in radians radius terms

                SpotRadius = arccos( 0.5^(1/n) )

We are making progress!

Now, specular power to solid angle:

Measured in steradians, the formula for solid angle from cone half-angle (radial angle) is,

        omega = 2pi * (1 - cos(tetha))

But there are 2pi steradians in a hemisphere, so, measured in hemispheres, the formula becomes,

        omega = 1 - cos(tetha)

If we substitute our spot radius formula above, we get

        omega = 1 - cos( arccos( 0.5^(1/n) ) )

which simplifies to,

                SpotSizeInHemispheres = 1 - 0.5^(1/n)

where n is the specular power.  Now we are REALLY making progress...

Our adjustment factor for specular spotlights should be inversely proportional to the solid angle of the spots, so,

        AdjFactor = k * 1 / ( 1 - 0.5^(1/n) )

with a k to be determined such that AdjFactor is 1 when spec power is 1.  What does our right-hand side yield at power 1?

1/1 = 1.  0.5^1 = 0.5.  1 - 0.5 = 0.5.  1/0.5 = 2.  So k needs to be 0.5

So, our final formula is,

                BrightnessAdjustmentFactor = 0.5 / ( 1 - 0.5^(1/n) )

where n is specular power.

Almost done.  One final magic ring we need to find is what is the shininess equivalent for the Sun's apparent size.

We know that its apparent diameter is 0.53 degrees.  So, its apparent radius is 0.265 degrees.

Multiply by pi/180 and...

                SunApparentRadius = 0.004625 radians

Good to know, but we need a formula to translate that into a shininess equivalent.

Well, we just need to flip our second formula around.  We said,

        SpotRadius(radians) = arccos( 0.5^(1/n) )

so,

        cos( SpotRadius ) = 0.5^(1/n)

        ln( cos(SpotRadius) ) = ln( 0.5^(1/n) )

        ln( cos(SpotRadius) ) = (1/n) * ln( 0.5 )

                n = ln( 0.5 ) / ln( cos(SpotRadius) )

Plugging in our value,

        ln( 0.5 ) = -0.69314718

        cos( 0.004625 radians ) = 0.99998930471

        ln( cos( 0.004625 radians ) ) = -1.0695347 E-5

Finally,

                SunSizeSpecPwrEquiv = 64808  ...  (make that 65k )

So, we really don't need to be concerned, except for ridiculously high spec power surfaces;  but it's good to know, finally.

 

EDIT:  Just so you know, when I worked on this, decades ago, I obviously made a huge math error somewhere, and I ended up with a Sun size derived specular power limitation to about 70 I think it was.  I smooth-limited incoming specular power by computing n = n*70/(n+70).  I knew it was wrong;  the spotlights on flat surfaces were huge.  What was cool about it was the perfect circular shape of those highlights.  It was like looking at a reflection of the Sun, literally;  except that it was so big it looked like I was looking at this reflection through a telescope.

EDIT2:  One thing to notice here is the absurd non-linearity of the relevance of spec power;  maybe we should consider encoding the inverse of the square root of spec power, instead.  This way we have a way to express infinite (perfect surface;  what's wrong with that?) by writing 0.  We can express the maximum shimmery surface as 1, to get power 1.  At 0.5 we get 4.  At 0.1 we get 100.  We could even encode fourth root of 1/n. 

 

Edited February 14, 2021 by DanW58

[DanW58]

@hyperion  Another way to go about it, that you might care to consider, is to incorporate this shader now (if it works with all existing assets), but to also include a new shader with NO metal detection, and encourage artists to target the new shader.  Different texture stack, channels for spec power, index of refraction and detail texture modulation, etc.  So this shader and the new one would be totally incompatible, textures-wise, and even uniforms-wise.  This path removes the concern about people getting unexpected results with new models.  New models would NOT target the metal detection shader.  It would be against the law.

I could come up with the new shader in a couple of days;  I got most of the code already.  So, even if there are no assets using them, people can start targetting it before version 25.

In this case, I'd cancel the "psychic" shader project.

 

Channels needed for a Good shader:

Specular texture: rgba with 8-bit alpha

1. specular red
2. specular green
3. specular blue
4. specular_power  (1 to infinity, encoded as fourth root of 1/spec_power)is_metal

Diffuse texture: rgba with 1-bit alpha  (rgb encoding <metals> / <non-metals>)

1. diffuse red /  purity_of_surface ((0.9 means 10% of surface is diffuse particles exposed, for plastics))
2. diffuse green / index_of_refraction (0~4 range) ((1~4, really, but reserve 0 for metals))
3. diffuse blue / detail_texture_modulator
4. is_metal

Normalmap:  rgb(a)

1. u
2. v
3. w
4. optional height, for parallax

Getting Blender to produce them should be no issue.

Note that I've inverted the first and second textures, specular first, with the diffuse becoming optional...  For metals, the first texture alone would suffice, as diffuse can be calculated from specular color in the shader.  Artists who want to depict dirt or rust on the metal, they can provide the diffuse texture, of course;  but in the absence of diffuse the shader would treat specular as metal color and auto the diffuse.  Also, when providing diffuse texture for a metal, it could be understood to blend with autogenerated metallic diffuse;  so you only need to paint a bit of rust here, a bit of green mold there, on a black background, and the shader will replace black with metal diffuse color.

 

EDIT:  Here's how some of the cleaned-up code is evolving;  not compiling yet, far from it.  I have the functions evolving in the places where they will be called;  I'll move them to before main() after...

  vec3 incident_specular_light( vec3 suncolor, vec3 groundcolor, float matspecpwr, float brightadj )
  {
    vec3 spec_from_sun = sunColor * pow(max(0.0, dot(normal, v_half)), matspecpwr) * brightadj;
    vec3 specAmbient = mix( ambient, ambient * ground_color, reflection_hits_the_ground(eyeVec, normal, specPow);
    return spec_from_sun + specAmbient;
  }
  vec3 SchlickApproximateReflectionCoefficient( float ray_dot_normal, float n_from, float n_to )
  {
    float R0 = (n_to - n_from) / (n_from + n_to);
    R0 = R0 * R0;
    float angle_part = pow( 1.0 - ray_dot_normal, 5.0 );
    float RC = R0 + (1.0 - R0) * angle_part;
    return vec3(RC, RC, RC);
  }
  vec3 specularly_reflected_light( float spec_path_open, vec3 mspeccol, vec3 fresnelcolor, float is_metal, vec3 incident_light )
  {
    vec3 final_specular_color = mix(fresnelcolor, mspeccol, is_metal);
    return spec_path_open * final_specular_color * incident_light;
  }
  vec3 diffusely_reflected_light(
    vec3 fresnelcolor, vec3 suncolor, vec3 AO, vec3 ambientcolor, vec3 groundcolor, float normal_hits_ground, float ray_dot_normal )
  {
    vec3 FresnelRefractFactor = vec3(1.0) - fresnelcolor;
    incident_direct = suncolor * getShadow() * FresnelRefractFactor * FresnelRefractFactor; //refract in, then out
    incident_ambient = AO * mix( ambientcolor*up_bias, groundcolor, normal_hits_ground );
    return  texdiffuse * (incident_ambient + ray_dot_normal*incident_direct);
  }
  vec3 color = diffusely_reflected_light();
  color += specularly_reflected_light();
  #if USE_SPECULAR_MAP && USE_SELF_LIGHT
    color = mix(texdiffuse, color, specular.a); //WTF?...
  #endif
  color = applyFog(color);
..................

 

Edited February 14, 2021 by DanW58