A "psychic" shader mod; development begins...

[wraitii]

3 minutes ago, Samulis said:

Granted, we're talking about projects made by teams 5-10x larger than the active dev team on 0 A.D., who are also full-time paid employees of a company, but I digress.

More like x50-x1000, truthfully. In terms of "Full Time Equivalent", I would wager 0 A.D. averages under one FTE/day. Forgotten Empires, the makers of Age2, are 51 people at the moment. I'd wager the team making Age 4 is even larger. The gap isn't large, the gap is huge.

3 minutes ago, Samulis said:

Keep in mind 0 AD has been in development for well over a decade.

Two decades, in fact

 

[DanW58]

3 hours ago, Samulis said:

Returning to the topic at hand, I think what you are doing looks great so far, especially if combined with the aforementioned new textures. I think the proper solution here would be to fully implement texture maps/keys for metallic as you suggested earlier, rather than trying to auto-detect. That'd probably take an enormous effort, but it is the sort of thing which needs to be done honestly. In the sound department, for A24 we mostly just tried to patch problems up with 'meatball surgery', but for A25 we're already at work on deeper, more fundamental changes to make the sound assets and their implementation consistent and "correct".

Keep in mind 0 AD has been in development for well over a decade. There are brilliant, clever implementations of things, and sloppy messy hacks right next to each other in every corner of the game. New features and changes are not checked by a singular Producer or Director, but by the team as a whole- an experience which can seem at times like an inquisition, when the actual intention is to ensure there are no weaknesses in the proposal and that it will provide a suitable foundation for future development. The deeper the change, the more things are at risk, and as a result the game tends to move more slowly the deeper you want to go. It is easy for me to replace a few bad or poorly made sound effects from a decade ago, but when we get into stuff like changing (or fixing!) the way the sound engine works, it takes a lot more planning and discussion to make sure things work and what we leave behind will be workable for many years to come.

In my opinion: Having said that, it also means it is fundamentally necessary that any code changes really need to be done the "right" way. Layers of hacks upon hacks is what has caused numerous issues on the project, so I believe it's essential whatever is decided must be done fully and in such a way that it doesn't impede future work. This means in a perfect world, changes shouldn't compensate for, override, assume, or guess about assets or previous work because that creates a feedback cycle of spaghetti going back into spaghetti. Ideally every change should be at the root of the issue, not treating or concealing the symptoms. If I understand your proposal and the architectures involved correctly, that means implementing this requires fixing textures so they give you explicit information on what is metallic, and integrating that in the workflow so all textures present and future have that.

I hate hacks as much as the next guy;  but sometimes they are necessary.  What I suggested to hyperion was, and is, adopting the latest metal detection shader (together with the old patch for ambient occlusion, etc, which he rejected for no good reason in my opinion), JUST FOR EXISTING ASSETS, and then violently encourage artists to target the next shader, with a more comprehensive texture set, and without any hacks.  That way it would NOT be necessary to change all the art assets at once.  You could change one at a time, and indicate that it is to use the new shader.  It is the least problematic path moving forward;  it would put all the eye-candy of the metal shader out in the public distribution, causing an uproar of happiness out there;  and it doesn't have ANY negative consequences.  It is a WIN-WIN-WIN, as your beloved former President would say.

But it had to be rejected.  Why?  Whatever...   If I offer my opinion on the reasons, I will be told not to take things personally.  It really isn't personal.  It is cultural.  I'm going to leave it at that.

And I understand about the fear of a chance in a million that something that wasn't tested might break.  But then I question, do you want to be like Boeing or like SpaceX?  Do you want to be like GM or like Tesla?  This ridiculously absurd dread of shipping a single bug is dysfunctional.  Just say "sorry", and release a patch!!

EDIT:

And since you are a sound guy, I have a request:  Please implement 3D positional sound as in Starcraft.  In Starcraft, if a battle started somewhere in the map off-screen, you could tell from the reverberated sound the direction to go, without even looking at the mini-map.

Edited February 15, 2021 by DanW58

[Stan`]

6 minutes ago, DanW58 said:

But it had to be rejected.  Why?  Whatever...   If I offer my opinion on the reasons, I will be told not to take things personally.  It really isn't personal.  It is cultural.  I'm going to leave it at that.

Once again it wasn't rejected. I emitted a concern with the amount of work it requires. I'd actually would like to have @vladislavbelov's opinion on it as he probably has plans to change some things about materials too and maybe there is some common ground to be found.

I'm gonna emit a lot of concerns because I have been there a while and I've seen a lot of projects left alone. When I commit to something this big I like to have other team members back me up. Because at the end of the day I might be the one doing it instead of something else.

Having a new material to migrate to sounds good for a smoother transition.

Also I wonder (because I've been too busy to test) how your shader works for objectcolor.xml, basic_glow basic_spec basic_trans_wind etc

 

[DanW58]

33 minutes ago, Stan` said:

Also I wonder (because I've been too busy to test) how your shader works for objectcolor.xml, basic_glow basic_spec basic_trans_wind etc

Sorry, Stan, you got me;  I've no idea what any of that means.

As for assuming that my shader would cause you extra work,  I trust your thoughts were in error;  but then the easiest way to clarify errors is to discuss.  Clamming up, plotting in secret, or "agree to disagree" the moment any difference comes up, that's not constructive.  It hurts me MUCH more to feel that people are plotting against me in hush voices than if they question what I say.

By the way, this will be eye-candy for anyone that can read glsl...

  vec3 incident_specular_light( vec3 diffuse_col, vec3 ao, float matspecpwr )
  {
    vec3 normal = normalize(v_normal);
    vec3 eyevec = normalize(v_eyeVec);
    vec3 hafvec = normalize(v_half);
    vec3 refl_view = -reflect( eyevec, normal ) );
    float spec_pwr = max(1.0, matspecpwr);
    float unblocked_specular = reflection_unobstructed(eyevec, normal, ao);
    float point_is_in_shadow = getShadow();
    float blocker_in_sun = 0.5 + (dot(refl_view_,sunDir) * 0.5);      //just a likelihood...
    float Phong_coefficient = pow( max(0.0, dot(normal, hafvec)), spec_pwr );
    float brightness_adj = 0.5 / ( 1.0 - pow(0.5, (1.0/spec_pwr) );   //sharper reflections look brighter
    float reflecting_ground = vector_hits_the_ground(refl_view, spec_pwr);
    vec3 light_on_blocker = mix( ambientcolor*ao*ao, sunColor, blocker_in_sun*blocker_in_sun );
    vec3 spec_blocker_color = diffuse_col * light_on_blocker; //assume blocker's same material as this
    vec3 spec_from_sun = mix( sunColor * brightness_adj, spec_blocker_color, point_is_in_shadow );
    vec3 spec_from_env = mix( spec_blocker_color, ambient, unblocked_specular );
    spec_from_env = mix( spec_from_env, ambient * groundColor, reflecting_ground );
    return mix( spec_from_env, spec_from_sun, Phong_coefficient);
  }

...but it is probably full of errors;  this is fresh code, never compiled.

Note that it is not mixing concerns, such as material reflectivity;  its ONLY concern is what light is incoming and ready to be reflected specularly.  Note the stuff about "blocker", which is any features of the same model likely blocking specular reflections (based on AO, assuming circular distribution).  Depending on whether the reflected ray runs same way as sunlight, or the opposite way, I compute a fuzzy boolean about whether the specular occluder is in facing the sun or whether what's reflecting is facing away from the sun, to paint its reflection accordingly.

I should be calling this shader "Sherlock".

Edited February 15, 2021 by DanW58

[Stan`]

12 minutes ago, DanW58 said:

Sorry, Stan, you got me;  I've no idea what any of that means.

Those are other materials used by the game. I was merely wondering how they looked next to your new ones.

Quote

As for assuming that my shader would cause you extra work,  I trust your thoughts were in error;  but then the easiest way to clarify errors is to discuss.  Clamming up, plotting in secret, or "agree to disagree" the moment any difference comes up, that's not constructive.  It hurts me MUCH more to feel that people are plotting against me in hush voices than if they question what I say.

I am all for discussions! I believe they are the entire point of these forums. I said agree to disagree because I got the feeling the discussion was over and you were giving up.

Just so we're clear, nobody is plotting against you. If we make you feel this way, if I make you feel that way, it's because of a miscommunication through these forums.

 

[DanW58]

Stan, I'm the LAST PERSON on this planet to ever give up a discussion;  just so you know.  I was born this way.

My best friend and I, when we were kids, one time we got into an argument that was vast, involved physics, philosophy, ethics, epistemology and the meaning of life;  and we argued for 3 days almost continuously;  and finally we arrived at our basic difference of opinion:

He believed that once we humans come to discover and know everything, the laws of the universe would be a bunch of equations explaining everything.

I believed that once we humans come to discover and know everything, the laws of the universe would be a single equation explaining everything.

That was when we realized that our basic difference was a matter of opinion that could not be resolved.

And we agreed to disagree, and we did so wholeheartedly.

[Genava55]

8 hours ago, Stan` said:

During which I spent my time working on a lot of 3D models, a lot of which were just thrown in the garbage can because they were not good enough.

I know it is not only this example, but I still want to say I am sorry for your Gallic tavern, I really like the model though.

[Stan`]

12 minutes ago, Genava55 said:

I know it is not only this example, but I still want to say I am sorry for your Gallic tavern, I really like the model though.

Actually this one made it so it's fine. It's just retired

[Grapjas]

8 hours ago, Stan` said:

We have too many distros on Linux/BSD to provide binaries Ubuntu, Debian, Gentoo, Arch Linux, Mageia, Haiku, FreeBSD,OpenBSD, NetBSD to just name a few.

In December 2018 we released, and things seemed to get back to normal. However most of the team was weared down because of the GDPR compliance the long time without commit, and the rushed re-release due to segfaults.

Then, people also got very busy with life, leaving only a handful of developpers, most of which could not tackle the challenges at the time, either by lack of knowledge, time, or both.  Time passed by with little activity on the programming side, while the art part got a lot of changes.  At each big holidays we were waiting for someone to finally get the time to tackle the spidermonkey upgrade, which was the big roadblock, along with macOS issues (hidpi, sdl2 etc) and the occasional library upgrade e.g. NVTT and each time, he either ran out of time, or had to tackle other problems and could not deliver. this went on for at least a year.

In July last year, I burnt out. I had a big breakdown and left 0 A.D. for almost a month. After this sad episode, I decided that I wanted to give it one last shot, or let the game die on its own. So I took even more things on my shoulders than I ever did before, trying to micromanage everyone, get people in touch, fix some issues on my own, talk to maintainers, get help from other projects, look for new contributors. and generally try to move the project forward, even though forward was as blurry as "get a release done".

Around september wraitii got back and the heavy task of finishing spidermonkey was given to him.  In December  he was almost done, and since that made a release possible, all the energy was shifted towards fixing the game breaking bugs, and releasing. I think we're at the end of the tunnel here.

 

 

 

Ya'll do a @#$% ton of work and barely get recognition from it, it's almost a crime. It impressed me while being on the dev IRC chat how many work you all are doing behind the scene, while still helping out noobs like me with small problems. I saw alot of tickets being made/solved/adjusted. So thanks devs for all the hard work so far and getting this next release ready.

 

[Samulis]

4 hours ago, DanW58 said:

And since you are a sound guy, I have a request:  Please implement 3D positional sound as in Starcraft.  In Starcraft, if a battle started somewhere in the map off-screen, you could tell from the reverberated sound the direction to go, without even looking at the mini-map.

Thanks for the suggestion! This is something I'd like to see too; there are already some existing concessions for off-screen sounds, but they're pretty limited (essentially just, things that sound in the ~200px to the horizontal sides of the screen get played a little quieter).

The sound engine we're using is extremely limited: no reverb, no filters, no equalization, no effects of any kind, so everything would need to be done with sound groups and some hackery. All we can do is pretty much play sounds at different volumes and pitches (speeds), at a particular spot on the screen. So, to implement off-screen cues like this we'd need to add a new set of sounds with the reverb et al baked in and code something to determine if a sound is on or off the screen and play the right variant. This is a recipe for a lot of work (and duplicated assets), but we might be able to find a way with maybe just a generic distant battle sound.

Our main focus presently is on cleaning up and standardizing the sound assets and their definitions; more or less 'triage' medicine. Most of the sound stuff specifically is from over 9 years ago and the people who did most of it have long left the project I believe, so it's more archaeology than sound design some days. This stuff was coded and designed when I was still scoring Flash games for a living! XD

Once the cleanup work is done (I'm hoping for A25), we can start to actually implement new ideas or tweaks.

[badosu]

13 hours ago, DanW58 said:

but once people tasted the honey of being able to update their game every day, they loved it, and never complained again.

Bar has an autoupdater that does it on every new commit, it's pretty great for alpha development.

I think having an 'alpha' lobby together with an autoupdater would do wonders for the dev-feedback cycle, does not even need to be per commit, nightly or weekly would suffice.

Could keep the 'stable' one and balance the load to whichever players prefer the most. I think the result could be surprising and exciting :-).

Edited February 16, 2021 by badosu

[Nescio]

The development version is actually playable and doing a svn up (or git pull) is often sufficient: the game only needs to be rebuilt if C++ files are changed, which are only a minority of the commits.

There are already autobuilds for Windows users. I don't know how difficult it would be to set up autobuilds for macOS too.

For Linux (excluding Android), compiling the game the first time is rather time-consuming, though not too difficult: https://trac.wildfiregames.com/wiki/BuildInstructions is quite clear. (It's a bit unfortunate `--with-system-mozjs` doesn't work, that would save a lot of space and time.) Afterwards rebuilding usually takes a few minutes; occassions where you have to do a clean and compile from nothing are actually quite rare. I guess it's similar on BSD.

Doing weekly (e.g. every Saturday) unstable releases of the development version is possible in principle, provided someone (not a team member) takes responsibility for this and stays around in the long term.

0 A.D. should adopt a different version numbering then, instead of 0.0.25, go to 25.0.0 (start), 25.0.1 (after the first week), 25.0.2 (after the second week), etc., though that's a minor point.

 

Having said that, making it easier for users to update the game does not mean they actually will: there is no way to force people to do a svn up or git pull.

Moreover, as already pointed out before, the development version is much larger than the stable releases, which are stripped down: on my end (Fedora 33) the former is c. 35 GB, the latter only c. 2 GB. While not a problem for me, it is for many others around the world.

[DanW58]

6 hours ago, badosu said:

Bar has an autoupdater that does it on every new commit, it's pretty great for alpha development.

I don't know who Bar is, but I just imagined there could be an updates panel accessible through the menu or via an icon or button where the user can update manually or set auto-updates, and even revert and report bugs.  This would all use svn under the hood, but the user would never need to install svn or learn how to use it.  So the Install repo would be as I was describing earlier, and the game would ship as a standard zip or whatever, but it would install svn (after checking it's not installed, first) and do an svn up at the end of the install.  This also cures the problem of having compressible items uncompressed;  all would be maximally compressed in the zips, and the first svn up should take milliseconds to complete.

The screen could be titled UPDATES.  It would show your revision and what revisions are available, and whether they are stable or experimental.  There would be a green UP button for next stable and a smaller red UP button for upgrading to bleeding edge, and an orange perhaps down button for reverting to previous.  Hitting this button would revert to the previous revision the user had, and open a screen for bug-reporting.

Under the green and red arrows there could be a Manual/Auto sliding switch.

@Nescio  I'm not sure if you were talking to me.  The repo I was suggesting for end-users, players, would not be the 35 gig development version;  it would be the 2 gig version, and there'd be no compiling involved, as it would be the binaries that are updated;  not the sources.  SVN has this ability to govern all sorts of files, not just sources, unlike some other version control systems.

EDIT:

Perhaps even better would be for the installer to carry say half the assets, the most compressible AND stable ones, but not everything;  --i.e. NOT be a full install, just so that it doesn't have to have a version;  it's "the installer".  As otherwise it would be hard to explain to people that it doesn't matter what version you download.

Edited February 16, 2021 by DanW58

[Stan`]

18 minutes ago, DanW58 said:

I don't know who Bar is

https://github.com/Balanced-Annihilation-Reloaded

19 minutes ago, DanW58 said:

@Nescio  I'm not sure if you were talking to me.  The repo I was suggesting for end-users, players, would not be the 35 gig development version;  it would be the 2 gig version, and there'd be no compiling involved, as it would be the binaries that are updated;  not the sources.  SVN has this ability to govern all sorts of files, not just sources, unlike some other version control systems.

The issue is binary patching archives I suppose we don't even need to version anything.

 

I really liked the idea of an autoupdater. Not sure how the official maintainers would look at it, since it could be a security concern.

[DanW58]

1 hour ago, Stan` said:

I really liked the idea of an autoupdater. Not sure how the official maintainers would look at it, since it could be a security concern.

Yes, that IS a concern, though much less so today with SSH than it was before.  But indeed, if a user's machine was compromised to access another server, from then on the compromised server could mirror the true server 99% of the time, but occasionally slip some piece of malaware.  The code that logs in perhaps should be checksum-checked by other parts of the code.  I'm not sure;  I'm no security expert.

P.S.  The graphics on that Bar screenshot look HORRIBLE.  Its players must be masochists.

EDIT:  But yeah, to say it again, this auto-updater would be a bit of work to implement, but from then on it will

1. Make players happy they don't have to download big files every time there's an update
2. Remove the huge work of preparing releases every season or so.
3. Remove the need for feature freezes and obsessive bug testing
4. Make the development cycle from a huge loop down to a tiny one
5. Remove the fear of a bug making it out, by making it easy to revert and report
6. Tremendously reduce bandwidth, as only the first download is extensive
7. Better hook players, with the possibility of a better game every day

 

Edited February 16, 2021 by DanW58

[wraitii]

I'm not convinced that we need to, or indeed should, switch to a "continuous release" model, to be honest. Yes, it has pros, namely more testing more often, but it'll make the game both less buggy and less stable. Further, for players, continuous balance changes won't be very good. There are advantages to periodic release - 'marketing', things like that.
'Real' companies do a bit of both, but video games that I know tend to work with semi-regular "patches", not actual continuous releasing. There's likely a good reason for that.

It would also be rougher on modders, who would need to maintain their mod continuously to avoid issues (though of course, it somewhat makes it easier).

I believe shorter release cycles (4-6 months) with an "svn" alongside, is overall a rather good model.

[DanW58]

The new, totally reorganized shader is now compiling.  There are new bugs of course.

Surprise was, everything looked metallic at first;  so I disabled the metal detection, which is now a subroutine;  and everything still looked metallic...  WTF?!

So then I remembered that I had made Index of Refraction 2.0 for initial testing of Fresnel.

So I made it 0.0, instead.

Then everything looked like a perfect mirror...

Of course, my default medium refractivity is 1.0, which is air's refractivity;  so having objects with a refractivity of 0.0 is like swimming underwater and looking up...  The waves look mirror-like.

Refraction from low IOR to higher IOR is one thing;  but going the other way, dielectric reflectivity is actually waaay higher.

So I fixed it again, made it 1.0 to match air, and now everything looks more normal.

Except people and elephants are now pitch-black...

Edited February 16, 2021 by DanW58

[DanW58]

34 minutes ago, wraitii said:

Further, for players, continuous balance changes won't be very good. There are advantages to periodic release - 'marketing', things like that.
'Real' companies do a bit of both, but video games that I know tend to work with semi-regular "patches", not actual continuous releasing. There's likely a good reason for that.

Well, for players there could be revision synchronizatin verification protocol under the hood, and a recommended official revision for online gaming.  Maybe the panel I was describing could include a list of revisions, say last 10, and people agree what revision they are going to play with.

You could make it that whoever hosts a game decides the version.  When players join, their version is automatically reverted or upgraded to the required one.

Frequent patches could work, but I'm afraid they'd be more difficult to produce than simply updating the install SVN.

Patches take manual labor.

With an Install SVN you don't have to do ANYTHING;  just update the art or binaries as they come;  and let the players decide if they want the latest update or play at some older, stable version.  I suspect they'd do both.  Try the latest when playing alone;  play a recommended version online.

As far as marketing is concerned, everybody tends to assume that what everybody else does is the best.

You got Tesla selling cars online, and the customers love it, as they always hated the dealership experience.

But until Tesla did something different NOBODY questioned the dealership model;  and even now none of the big companies has actually switched away from the dealership model.  They will watch in disbelief all the way to their demise, it seems.  I'd be weary of copying what anybody else does in marketing, as the public hate marketing for a reason.

 

Edited February 16, 2021 by DanW58

[wraitii]

You're really not addressing my point. The technical side is trivial to an extent. The fact is that the gameplay would change, and that's not a good player experience. Changing the revision while joining a game, so you don't even know what you're playing? Absurd.

That being said, including an easy way to auto-update, even with semi-regular releases, is likely a good idea. Whether it should be a priority, I'm not so sure. We'll take patches, should they be sufficiently well written.

[DanW58]

4 minutes ago, wraitii said:

You're really not addressing my point. The technical side is trivial to an extent. The fact is that the gameplay would change, and that's not a good player experience. Changing the revision while joining a game, so you don't even know what you're playing? Absurd.

Well, having choices is not absurd.  If most players agree that some revision is "THE official revision for online play", they will be sure to be familiar with it.  But if someone is hosting a game with bleeding edge revision, if you are not familiar with it you just don't join.

 

Anyways, here's a first screenshot of the new shader.

Look at those big vasses inside the building.  You should all see it while turning the camera around...  It is a quite believable depiction of metallic specularities occluded by the mess of columns.  How does it know the specularities should be occluded?  From the AO,  and with some math tricks...

What I don't understand, off the bat, is why some of the people look good, and some look black.  Most curious of all is my cavalry, human looking ok, horse looking blacker than a primordial black hole.

[Stan`]

Could be they have normal maps / spec maps, or use player color, or something else.

Everything looks very green?

[DanW58]

50 minutes ago, Stan` said:

Could be they have normal maps / spec maps, or use player color, or something else.

Everything looks very green?

The reason everything looks so green is I made the ground color green, in the shader, and we are looking at a downward slant, so the reflections off things all hit a green ground.  Not that most things should have any specular color at all but black, but unfortunately everything has specularity.  For now I just have to change the color in the shader to gray and problem fixed.  I'll do it now and attach a new screenshot.  Eventually I will need to get the color of the ground into the shader from the C++ side of things.  I don't think meshes have vertex colors, so that will problably involve sampling the terrain texture around a building at the moment of creation.  I'll cross that bridge when I get there.

As for the rest, it could be; not sure how to check;  investigating...  Maybe it's an all female swat team...  Their heads look normal;  it's only from the neck down.

Another solution to the game-play changing is to NOT change the game-play.  Hold back from releasing game-play affecting features.  You can still update the graphics, models and whatnot on a daily basis.

Adding a shot WITHOUT metal detection at the bottom, for comparison.

Edited February 16, 2021 by DanW58

[badosu]

4 hours ago, DanW58 said:

P.S.  The graphics on that Bar screenshot look HORRIBLE.  Its players must be masochists.

Look again :-)

https://www.beyondallreason.info/

https://github.com/beyond-all-reason/Beyond-All-Reason

Edited February 16, 2021 by badosu

[badosu]

1 hour ago, wraitii said:

Changing the revision while joining a game, so you don't even know what you're playing? Absurd.

Engine changes are compartmentalized, the lobby is dissociated from the version going on but can detect if a different map/engine is being used. Also having dedicated servers handle that, current the clusters run on very inexpensive VPS instances.

That said I'm not sure that would be practical for 0ad, just having it weekly would be great already (e.g. updates on wednesdays or thursdays to identify issues with new version before weekend).

[DanW58]

Well, okay, BAR have the same problem we do:  lack of ambient shadowing.

Or we could do what I suggested in the last edit, namely hold back on releasing game-play affecting changes.

We can release art updates and graphics updates on a daily basis without touching anything that might affect balance.  Just save such updates for a major new release.

 

Edited February 16, 2021 by DanW58