[A "psychic" shader mod; development begins...]

7 minutes ago, Stan` said:

GPU are actually very under used. The game is mostly cpu bound. The switching is mostly done by the driver.

The point of those materials is to allow artists to have models with only a few textures eg only diffuse, or diffuse + spec, diffuse +spec + ao, diffuse +spec + ao + parallax, diffuse +ao etc.  If you load a material requiring more texture you'll get warnings.

I can incorporate all of that on this shader I'm working on ... easily.  In fact, I've mostly done so already, without knowing it was needed.  I just saw so many confusing, inscrutable conditionals in the shader that I assumed anything was possible vis a vis textures being or not being there.  I have defaults everywhere that I can turn into smarter guesses in no time at all.

Having separate shaders for all these texture combinations is a nightmare.  Means that every time you tweak one shader you have to tweak them all !!!  That's shooting yourself on the foot with a silver bullet.

[A "psychic" shader mod; development begins...]

That's where all the GPU time is spent, then.  I remember in the Vegastrike engine we had two shaders, basically, one for solids and one for transparencies, and we segregated their use, to have only a single switch.  So, the transparent shader was running first, while painting back to front;  and once all transparent objects were disposed of, we switched to the solid shader and proceeded with  solid objects from front to back.

I would suggest you try to do the same.  I have no idea what all this profusion of shaders is for;  I can't tell from their names;  but I can tell you up-front they are NOT justified.  I'm sure they implement simple functionalities that could easily be added to the main shaders, and be disposed of.

[A "psychic" shader mod; development begins...]

OUCH!  That's one hell of a lot of shaders...  Are they actually used?

EDIT:  Heck, now I see why it is so hard to work with this.  The sides of the blocky chairs where Mr. Faro and his wife seat are downright white in specular, like they should be perfect mirrors.  Not to speak of all the splashes of white on the walls.  This is a Metallic Universe.

Maybe I should just detect desaturation in specular and change it to black.

EDIT2:  IT WORKS !!!

Somehow, AO doesn't seem to be working, tho....

[A "psychic" shader mod; development begins...]

I just decided to show specular color on-screen, unlit, just raw specular color (NOT the terrain, as that is a separate shader, but applying to all models).

As you can see, the obelisks, the lions, camel, cherry bushes, windows, and some of the people are black in specular, as they should be.

Mostly everything else is 50% gray for no reason.

Some of the hieroglyphics on the walls have colored specular, which now explains why I had such a hard time trying not to detect them as metal;   they ARE metal.

 

[A "psychic" shader mod; development begins...]

57 minutes ago, Stan` said:

Ah yeah I see. In general the specular was made for metallic parts but I can see how that went wrong for some stuff.

Maybe we should grab a big bucket of black paint and go on a specular texture fixing campaign...

It would be too much to ask of a shader to know when to ignore specular.

EDIT:  This is absurd.  Changing to Ptol, the females turn pitch-black, head and all;  the camel looks fine, unlike horses, but the mount on the camel looks black-hole black...  What on Earth is going on?!

[A "psychic" shader mod; development begins...]

But it could be that many of the assets do NOT have specular textures, but specularity is specified elsewhere for the whole unit.  I tried to disable that in the shader;  namely I try to reject (set to black) anything that doesn't come from an actual texture, but maybe I failed in the attempt.

[A "psychic" shader mod; development begins...]

22 minutes ago, Stan` said:

I suppose we could use it for shields.

There is a HUGE performance cost in changing shaders while rendering a frame;  it's better to do everything with one shader, if at all possible, or have a separate one for water, but that's it.  Well, maybe 3;  but not a shader for each item or material.

That's why it's important to have a texture stack that can cover every material.  It would be too easy to call a metal shader for all metals, and a non-metal shader for all non-metals;  but it's not a good idea performance-wise.

With the current stack, if the best possible representation was used for every asset, it would be quite easy for the shader to deduce how to shade.

Namely, if both the diffuse and specular are colored, have the same hue, then it is a metal.

If the diffuse is colored, but the specular is gray-scale, it is a non-metal.

Then I could make ad-hoc adjustments to spec-power and refractive index tied to specular intensity.

But I can't;  it's too chaotic.

[A "psychic" shader mod; development begins...]

Though walls and skin and things do have a bit of specularity, of the fresnel type rather, and they are VERY low in specular power;  you'd have to specify a specular power below 10 for them;  otherwise they look ridiculous.  ((Or they WILL look ridiculous when environment mapping is implemented;  right now the ridiculous look is not revealed yet... But once you see the environment sharply reflected on walls as if they were mirrors, or as if they were packaged in shrink-wrap clear plastic, then you'd see what I mean.))

[A "psychic" shader mod; development begins...]

1 minute ago, Stan` said:

I'm not sure I get you. Are we talking about the new bronze shields speculars that were made yellow so they could keep that tint or do you also talk about b&w ones ?

Oh, not at all!  If you have been making shields yellow in specular, that is a Good Thing !!!  Maybe that's why they look so good.

I meant assets that should NOT have any specularity at all, like walls and columns and roofs and clothes and human skin.

[A "psychic" shader mod; development begins...]

Well, I don't have a mac or a windows machine, and I'm poor as @#$%, so there's not a chance I'll be able to cover those fronts;  I can only ensure it works in Lubuntu;  someone else will have to help me with those other targets.

With my old game project we did have trouble serving mac users;  but we had two mac users in the dev team, so they jumped quickly to help people in distress.

The thing about players manually using svn or not is not whether it is easy or difficult.  People always have HUGE resistance to doing anything they haven't done before;  THAT is the issue.  I'm no exception:  If you asked me now to learn git I'd probably go bonkers.  I'd probably learn it in 10 minutes, once I got to it;  but it would take me 10 days to look at the manual.

I'll take a look at that metal shader.  Thanks.

EDIT:  Okay, that was a start;  what happened to it?  I think it metallized everything, tho.  But it may be the fact that all the assets contain too much specularity;  --same problem I have.

(( @Stan`   The Art Rule should be:  "Refrain from using any color but black for specular, unless you really know what the @#$% you are doing." ))

[A "psychic" shader mod; development begins...]

Embedded svn, I would think.  I've no experience with git, but my understanding is that only svn handles binary files as well as text files.   You don't have to work on it;  I'd be willing to put time in it, if there is concensus.

[A "psychic" shader mod; development begins...]

Okay, if you prefer periodic releases, that's all there is to it.  But just to clarify, my point is not of continuous release being "better" in every conceivable metric;  it is easier to deal with;  that's my point.  Struggling towards a release is elliminated.  Makes YOUR life easier;   was my whole point.

EDIT:

Besides, the auto-update feature does NOT have to mean daily or weekly updates;  it could be seasonal updates;  it is whatever you decide.  What it removes is the need to put together these huge zip files and whatnot.

EDIT2:

In fact, you could have a semi-zip, with all the biggest, most compressible and stable files in it, that installs and then does svn up under the hood.  for all intents and purposes, this zip file does not need to be updated, because it updates itself.  But you can still change the name of it, and the revision number, "for marketing purposes"...

[A "psychic" shader mod; development begins...]

Well, okay, BAR have the same problem we do:  lack of ambient shadowing.

Or we could do what I suggested in the last edit, namely hold back on releasing game-play affecting changes.

We can release art updates and graphics updates on a daily basis without touching anything that might affect balance.  Just save such updates for a major new release.

 

[A "psychic" shader mod; development begins...]

50 minutes ago, Stan` said:

Could be they have normal maps / spec maps, or use player color, or something else.

Everything looks very green?

The reason everything looks so green is I made the ground color green, in the shader, and we are looking at a downward slant, so the reflections off things all hit a green ground.  Not that most things should have any specular color at all but black, but unfortunately everything has specularity.  For now I just have to change the color in the shader to gray and problem fixed.  I'll do it now and attach a new screenshot.  Eventually I will need to get the color of the ground into the shader from the C++ side of things.  I don't think meshes have vertex colors, so that will problably involve sampling the terrain texture around a building at the moment of creation.  I'll cross that bridge when I get there.

As for the rest, it could be; not sure how to check;  investigating...  Maybe it's an all female swat team...  Their heads look normal;  it's only from the neck down.

Another solution to the game-play changing is to NOT change the game-play.  Hold back from releasing game-play affecting features.  You can still update the graphics, models and whatnot on a daily basis.

Adding a shot WITHOUT metal detection at the bottom, for comparison.

[A "psychic" shader mod; development begins...]

4 minutes ago, wraitii said:

You're really not addressing my point. The technical side is trivial to an extent. The fact is that the gameplay would change, and that's not a good player experience. Changing the revision while joining a game, so you don't even know what you're playing? Absurd.

Well, having choices is not absurd.  If most players agree that some revision is "THE official revision for online play", they will be sure to be familiar with it.  But if someone is hosting a game with bleeding edge revision, if you are not familiar with it you just don't join.

 

Anyways, here's a first screenshot of the new shader.

Look at those big vasses inside the building.  You should all see it while turning the camera around...  It is a quite believable depiction of metallic specularities occluded by the mess of columns.  How does it know the specularities should be occluded?  From the AO,  and with some math tricks...

What I don't understand, off the bat, is why some of the people look good, and some look black.  Most curious of all is my cavalry, human looking ok, horse looking blacker than a primordial black hole.

[A "psychic" shader mod; development begins...]

34 minutes ago, wraitii said:

Further, for players, continuous balance changes won't be very good. There are advantages to periodic release - 'marketing', things like that.
'Real' companies do a bit of both, but video games that I know tend to work with semi-regular "patches", not actual continuous releasing. There's likely a good reason for that.

Well, for players there could be revision synchronizatin verification protocol under the hood, and a recommended official revision for online gaming.  Maybe the panel I was describing could include a list of revisions, say last 10, and people agree what revision they are going to play with.

You could make it that whoever hosts a game decides the version.  When players join, their version is automatically reverted or upgraded to the required one.

Frequent patches could work, but I'm afraid they'd be more difficult to produce than simply updating the install SVN.

Patches take manual labor.

With an Install SVN you don't have to do ANYTHING;  just update the art or binaries as they come;  and let the players decide if they want the latest update or play at some older, stable version.  I suspect they'd do both.  Try the latest when playing alone;  play a recommended version online.

As far as marketing is concerned, everybody tends to assume that what everybody else does is the best.

You got Tesla selling cars online, and the customers love it, as they always hated the dealership experience.

But until Tesla did something different NOBODY questioned the dealership model;  and even now none of the big companies has actually switched away from the dealership model.  They will watch in disbelief all the way to their demise, it seems.  I'd be weary of copying what anybody else does in marketing, as the public hate marketing for a reason.

 

[A "psychic" shader mod; development begins...]

The new, totally reorganized shader is now compiling.  There are new bugs of course.

Surprise was, everything looked metallic at first;  so I disabled the metal detection, which is now a subroutine;  and everything still looked metallic...  WTF?!

So then I remembered that I had made Index of Refraction 2.0 for initial testing of Fresnel.

So I made it 0.0, instead.

Then everything looked like a perfect mirror...

Of course, my default medium refractivity is 1.0, which is air's refractivity;  so having objects with a refractivity of 0.0 is like swimming underwater and looking up...  The waves look mirror-like.

Refraction from low IOR to higher IOR is one thing;  but going the other way, dielectric reflectivity is actually waaay higher.

So I fixed it again, made it 1.0 to match air, and now everything looks more normal.

Except people and elephants are now pitch-black...

[A "psychic" shader mod; development begins...]

1 hour ago, Stan` said:

I really liked the idea of an autoupdater. Not sure how the official maintainers would look at it, since it could be a security concern.

Yes, that IS a concern, though much less so today with SSH than it was before.  But indeed, if a user's machine was compromised to access another server, from then on the compromised server could mirror the true server 99% of the time, but occasionally slip some piece of malaware.  The code that logs in perhaps should be checksum-checked by other parts of the code.  I'm not sure;  I'm no security expert.

P.S.  The graphics on that Bar screenshot look HORRIBLE.  Its players must be masochists.

EDIT:  But yeah, to say it again, this auto-updater would be a bit of work to implement, but from then on it will

1. Make players happy they don't have to download big files every time there's an update
2. Remove the huge work of preparing releases every season or so.
3. Remove the need for feature freezes and obsessive bug testing
4. Make the development cycle from a huge loop down to a tiny one
5. Remove the fear of a bug making it out, by making it easy to revert and report
6. Tremendously reduce bandwidth, as only the first download is extensive
7. Better hook players, with the possibility of a better game every day

 

[A "psychic" shader mod; development begins...]

6 hours ago, badosu said:

Bar has an autoupdater that does it on every new commit, it's pretty great for alpha development.

I don't know who Bar is, but I just imagined there could be an updates panel accessible through the menu or via an icon or button where the user can update manually or set auto-updates, and even revert and report bugs.  This would all use svn under the hood, but the user would never need to install svn or learn how to use it.  So the Install repo would be as I was describing earlier, and the game would ship as a standard zip or whatever, but it would install svn (after checking it's not installed, first) and do an svn up at the end of the install.  This also cures the problem of having compressible items uncompressed;  all would be maximally compressed in the zips, and the first svn up should take milliseconds to complete.

The screen could be titled UPDATES.  It would show your revision and what revisions are available, and whether they are stable or experimental.  There would be a green UP button for next stable and a smaller red UP button for upgrading to bleeding edge, and an orange perhaps down button for reverting to previous.  Hitting this button would revert to the previous revision the user had, and open a screen for bug-reporting.

Under the green and red arrows there could be a Manual/Auto sliding switch.

@Nescio  I'm not sure if you were talking to me.  The repo I was suggesting for end-users, players, would not be the 35 gig development version;  it would be the 2 gig version, and there'd be no compiling involved, as it would be the binaries that are updated;  not the sources.  SVN has this ability to govern all sorts of files, not just sources, unlike some other version control systems.

EDIT:

Perhaps even better would be for the installer to carry say half the assets, the most compressible AND stable ones, but not everything;  --i.e. NOT be a full install, just so that it doesn't have to have a version;  it's "the installer".  As otherwise it would be hard to explain to people that it doesn't matter what version you download.

[A "psychic" shader mod; development begins...]

Stan, I'm the LAST PERSON on this planet to ever give up a discussion;  just so you know.  I was born this way.

My best friend and I, when we were kids, one time we got into an argument that was vast, involved physics, philosophy, ethics, epistemology and the meaning of life;  and we argued for 3 days almost continuously;  and finally we arrived at our basic difference of opinion:

He believed that once we humans come to discover and know everything, the laws of the universe would be a bunch of equations explaining everything.

I believed that once we humans come to discover and know everything, the laws of the universe would be a single equation explaining everything.

That was when we realized that our basic difference was a matter of opinion that could not be resolved.

And we agreed to disagree, and we did so wholeheartedly.

[A "psychic" shader mod; development begins...]

33 minutes ago, Stan` said:

Also I wonder (because I've been too busy to test) how your shader works for objectcolor.xml, basic_glow basic_spec basic_trans_wind etc

Sorry, Stan, you got me;  I've no idea what any of that means.

As for assuming that my shader would cause you extra work,  I trust your thoughts were in error;  but then the easiest way to clarify errors is to discuss.  Clamming up, plotting in secret, or "agree to disagree" the moment any difference comes up, that's not constructive.  It hurts me MUCH more to feel that people are plotting against me in hush voices than if they question what I say.

By the way, this will be eye-candy for anyone that can read glsl...

  vec3 incident_specular_light( vec3 diffuse_col, vec3 ao, float matspecpwr )
  {
    vec3 normal = normalize(v_normal);
    vec3 eyevec = normalize(v_eyeVec);
    vec3 hafvec = normalize(v_half);
    vec3 refl_view = -reflect( eyevec, normal ) );
    float spec_pwr = max(1.0, matspecpwr);
    float unblocked_specular = reflection_unobstructed(eyevec, normal, ao);
    float point_is_in_shadow = getShadow();
    float blocker_in_sun = 0.5 + (dot(refl_view_,sunDir) * 0.5);      //just a likelihood...
    float Phong_coefficient = pow( max(0.0, dot(normal, hafvec)), spec_pwr );
    float brightness_adj = 0.5 / ( 1.0 - pow(0.5, (1.0/spec_pwr) );   //sharper reflections look brighter
    float reflecting_ground = vector_hits_the_ground(refl_view, spec_pwr);
    vec3 light_on_blocker = mix( ambientcolor*ao*ao, sunColor, blocker_in_sun*blocker_in_sun );
    vec3 spec_blocker_color = diffuse_col * light_on_blocker; //assume blocker's same material as this
    vec3 spec_from_sun = mix( sunColor * brightness_adj, spec_blocker_color, point_is_in_shadow );
    vec3 spec_from_env = mix( spec_blocker_color, ambient, unblocked_specular );
    spec_from_env = mix( spec_from_env, ambient * groundColor, reflecting_ground );
    return mix( spec_from_env, spec_from_sun, Phong_coefficient);
  }

...but it is probably full of errors;  this is fresh code, never compiled.

Note that it is not mixing concerns, such as material reflectivity;  its ONLY concern is what light is incoming and ready to be reflected specularly.  Note the stuff about "blocker", which is any features of the same model likely blocking specular reflections (based on AO, assuming circular distribution).  Depending on whether the reflected ray runs same way as sunlight, or the opposite way, I compute a fuzzy boolean about whether the specular occluder is in facing the sun or whether what's reflecting is facing away from the sun, to paint its reflection accordingly.

I should be calling this shader "Sherlock".

[A "psychic" shader mod; development begins...]

3 hours ago, Samulis said:

Returning to the topic at hand, I think what you are doing looks great so far, especially if combined with the aforementioned new textures. I think the proper solution here would be to fully implement texture maps/keys for metallic as you suggested earlier, rather than trying to auto-detect. That'd probably take an enormous effort, but it is the sort of thing which needs to be done honestly. In the sound department, for A24 we mostly just tried to patch problems up with 'meatball surgery', but for A25 we're already at work on deeper, more fundamental changes to make the sound assets and their implementation consistent and "correct".

Keep in mind 0 AD has been in development for well over a decade. There are brilliant, clever implementations of things, and sloppy messy hacks right next to each other in every corner of the game. New features and changes are not checked by a singular Producer or Director, but by the team as a whole- an experience which can seem at times like an inquisition, when the actual intention is to ensure there are no weaknesses in the proposal and that it will provide a suitable foundation for future development. The deeper the change, the more things are at risk, and as a result the game tends to move more slowly the deeper you want to go. It is easy for me to replace a few bad or poorly made sound effects from a decade ago, but when we get into stuff like changing (or fixing!) the way the sound engine works, it takes a lot more planning and discussion to make sure things work and what we leave behind will be workable for many years to come.

In my opinion: Having said that, it also means it is fundamentally necessary that any code changes really need to be done the "right" way. Layers of hacks upon hacks is what has caused numerous issues on the project, so I believe it's essential whatever is decided must be done fully and in such a way that it doesn't impede future work. This means in a perfect world, changes shouldn't compensate for, override, assume, or guess about assets or previous work because that creates a feedback cycle of spaghetti going back into spaghetti. Ideally every change should be at the root of the issue, not treating or concealing the symptoms. If I understand your proposal and the architectures involved correctly, that means implementing this requires fixing textures so they give you explicit information on what is metallic, and integrating that in the workflow so all textures present and future have that.

I hate hacks as much as the next guy;  but sometimes they are necessary.  What I suggested to hyperion was, and is, adopting the latest metal detection shader (together with the old patch for ambient occlusion, etc, which he rejected for no good reason in my opinion), JUST FOR EXISTING ASSETS, and then violently encourage artists to target the next shader, with a more comprehensive texture set, and without any hacks.  That way it would NOT be necessary to change all the art assets at once.  You could change one at a time, and indicate that it is to use the new shader.  It is the least problematic path moving forward;  it would put all the eye-candy of the metal shader out in the public distribution, causing an uproar of happiness out there;  and it doesn't have ANY negative consequences.  It is a WIN-WIN-WIN, as your beloved former President would say.

But it had to be rejected.  Why?  Whatever...   If I offer my opinion on the reasons, I will be told not to take things personally.  It really isn't personal.  It is cultural.  I'm going to leave it at that.

And I understand about the fear of a chance in a million that something that wasn't tested might break.  But then I question, do you want to be like Boeing or like SpaceX?  Do you want to be like GM or like Tesla?  This ridiculously absurd dread of shipping a single bug is dysfunctional.  Just say "sorry", and release a patch!!

EDIT:

And since you are a sound guy, I have a request:  Please implement 3D positional sound as in Starcraft.  In Starcraft, if a battle started somewhere in the map off-screen, you could tell from the reverberated sound the direction to go, without even looking at the mini-map.

[A "psychic" shader mod; development begins...]

Ahhh, good! 

[A "psychic" shader mod; development begins...]

Thanks for the story;  helps to know.

As far as the different distros, does each of them need a different binary?  I only ever used Ubuntu, besides Windows;  I know nothing else.

If that's the case, I'm sure there must be a way to persuade SVN to do a file selection.

For example,  I know there's a way for a repo to reference another repo.  I was doing that with the Boost libraries.

So, you could have a private shared Installation repo with 99% of the files;  then a dozen tiny public repos for the varius distros, each containing just the binaries, but referencing internally the shared repo.

 

By the way, last time I was discussing this with people it was mentioned a plan to separate the engine and game repos.

That is good, but doesn't remove the convenience of separating the Art repo also.

An Art repo would contain all the files needed to produce each game item, including high resolution versions of textures, XCF's, all the maps used to create cube-maps,  high detail 3D models used to produce normal maps, Blender files, etc;  which you don't want to include in the game install.

So artists would have access to the art repo, and work there, with all their materials and props and multi-layer XCF's.  When they finish working on some item, they move the final product to the game repo.

Just as people working on the engine, when done, move validated sources to the game repo.

The game repo itself is where most of the javascript work would happen, as well as game-testing, and compiling for various platforms.

Final product from that, excluding sources and including binaries, could then move to the network of Install repos.

 

You guys should do this.  Bit of work at the beginning, but it will make your life much easier in the long term;  telling you from experience, because I did it.

 

[A "psychic" shader mod; development begins...]

15 minutes ago, Stan` said:

but for people on macOS it's usually too hard to compile as

No;  you misunderstand me.  My players did NOT have to compile.  The Installation repo had the binaries.  It was 'svn up' and play.

For mac and windows I forget what we were doing, as other people took care of that;  but I think there was no compiling involved either.

The bandwidth argument holds no water, as it is only the FIRST time they check out that they have to get all the dds textures;  from then on each (daily even) release they will only need bandwidth for anything that is new or changed.

This solution takes less than 1% the bandwidth of traditional releases, where at each release people have to download everything again, even though less than 10% of it changed.

I is a HUGE bandwidth saver;  and it removes the headache of struggling towards a release, making sure there are no bugs, etc.  If you release a bug you get immediate feedback and can retract it.  You get used to releasing daily.

EDIT:

Frankly, I don't understand why every operating system and every piece of software doesn't work this way.  It is a testament to the clinical lack of imagination, creativity and innovation that somehow afflicts the entire world, --Elon Musk being the sole exception.  I toyed with the idea of writing my own operating system, a Linux fork perhaps, that takes over Internet communications;  makes them look normal, but uses SVN under the hood.