Graphics rendering issue in A26

[hyiltiz]

Description: During a game play, entities such as trees, units and buildings seem to be colored/shaded with "anticolor" (kinda like negative film look).

What I tried to fix it:

• This happens in both singleplayer against Petra bot and multiplayer
• In multiplayer, while it is happening in my Debian, it doesn't happen to the other player connected through local network via macOS
• it almost never happens in the first 15min of the game, but once it happens, it never goes away, even after saving the game, restarting 0ad then loading that save
  • Some maps seem to lead to the issue more earlier
• it seems only changing "Shadow" settings would have an impact on it, and no other setting has any noticable effect:
  • Turning off "Shadow" sometimes fixes it; but keeping Shadow off can still trigger it in the same game later or in the next game
  • In "Shadow" submenu, sliding the "Cutoff distance" sometimes has an affect

Here are videos of the effect I am talking about: https://drive.google.com/drive/folders/13z3abqfR2-rc7dOMQPF-3kuWG3-xyzjG?usp=share_link

Various logs from 0ad: https://paste.tomsmeding.com/fxSf5Vb3

This *never* happened in A25 in the same computer and system...

@Stan` @vladislavbelov

Edited November 6, 2022 by hyiltiz
@ ppl

[vladislavbelov]

Hi! Thanks for the report! My first thought is an out of video memory (because of new art). But not sure. Could you enable debug checks via adding the following lines to your user.cfg and reproduce the bug?

renderer.backend.debugcontext = "true"
renderer.backend.debugmessages = "true"
renderer.backend.debuglabels = "true"
renderer.backend.debugscopedlabels = "true"

Also could try to decrease the textures quality in options to low?

[hyiltiz]

Your hunch about graphics card memory is probably right; now I recall that after that rendering issues goes on for a while:

• the game sometimes hangs and I have to hot-reboot the OS (by holding the power button, yikes...)
• if I do not wait until the game hangs or I got lucky that it doesn't, when I close the game, I find out that my Firefox rendering all gets messed up (the tab bar becomes a flat rectangle, and no icons show up in tab bar nor text etc.)

I can play Xonotic or an array of other games just fine. Is it possible that something with my config+hardware is leading to memory leaks?

 

Updated my user.cfg as suggested, reproduced the bug and uploaded two videos in the same drive link. Also including a .tar.xz file including all my config and logs right after this reproduced gameplay. I added two new videos:

• first video shows when the rendering issue just showed up, and you can see enabling Shadows somehow "works around" the rendering issue (no more negative/cyan colored buildings -- these buildings could even disappear later on)
• second video shows that after playing for a few minutes with "Shadows" on, now even "Shadows on" setting has a different rendering glitch; the terrains now has an ugly-looking grid-like pattern. Toggling "Shadows" only allows selecting one of the two rendering issues, but both issue persists.

@Stan` @vladislavbelov

I am in #0ad and #0ad-dev, if that is more convenient.

Edited November 7, 2022 by hyiltiz

[hyiltiz]

Just encountered a few different variants of this issue; the bad rendering now even flickers, or becomes purple/pink etc. I was also able to re-produce when the structures disappear. All captured in videos here: https://drive.google.com/drive/folders/13z3abqfR2-rc7dOMQPF-3kuWG3-xyzjG?usp=share_link

[Norse_Harold]

I have written some troubleshooting tips for graphics problems here. Summary: check factory overclocking and monitor refresh rate sync compatibility charts.

Also verify that you have an adequate and reliable power supply for your graphics card, and that your graphics card isn't overheating. What brand and model of each of these components do you have? (userreport_hwdetect.txt just says "AMD BONAIRE")

Another idea is to disable GLSL, since the shader precompiler may have memory corrupting bugs. You can do this by clicking Settings, Options, Graphics (advanced), Renderer backend: "OpenGL ARB".

[hyiltiz]

This is a laptop. I can confirm that it heats up a lot when that issue happens, and I am using the manufacturer's power supply adapter/chords etc. The monitor is laptop's internal monitor, no external display/monitors connected. I can try OpenGL ARB.

Neofetch says: AMD ATI Radeon R9 M280X
lspci says: VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon R9 M280X] (prog-if 00 [VGA controller])
lsmod says:

Spoiler

└─# lsmod | rg drm
drm_dp_helper         159744  2 amdgpu,radeon
cec                    61440  1 drm_dp_helper
drm_ttm_helper         16384  3 vmwgfx,amdgpu,radeon
ttm                    86016  4 vmwgfx,amdgpu,radeon,drm_ttm_helper
drm_kms_helper        196608  4 vmwgfx,drm_dp_helper,amdgpu,radeon
drm                   622592  20 vmwgfx,gpu_sched,drm_dp_helper,drm_kms_helper,amdgpu,radeon,drm_ttm_helper,ttm

└─# lsmod | rg amd
amdgpu               8130560  0
gpu_sched              53248  1 amdgpu
i2c_algo_bit           16384  2 amdgpu,radeon
drm_dp_helper         159744  2 amdgpu,radeon
drm_ttm_helper         16384  3 vmwgfx,amdgpu,radeon
ttm                    86016  4 vmwgfx,amdgpu,radeon,drm_ttm_helper
drm_kms_helper        196608  4 vmwgfx,drm_dp_helper,amdgpu,radeon
drm                   622592  20 vmwgfx,gpu_sched,drm_dp_helper,drm_kms_helper,amdgpu,radeon,drm_ttm_helper,ttm

└─# lsmod | rg radeon
radeon               1679360  21
i2c_algo_bit           16384  2 amdgpu,radeon
drm_dp_helper         159744  2 amdgpu,radeon
drm_ttm_helper         16384  3 vmwgfx,amdgpu,radeon
ttm                    86016  4 vmwgfx,amdgpu,radeon,drm_ttm_helper
drm_kms_helper        196608  4 vmwgfx,drm_dp_helper,amdgpu,radeon
drm                   622592  20 vmwgfx,gpu_sched,drm_dp_helper,drm_kms_helper,amdgpu,radeon,drm_ttm_helper,ttm

 

 

[hyiltiz]

I used OpenGL ARB, set both of the adaptive fps settings to 60, enabled VSYNC, disabled Shadows, set Number of Shaders to 0, but still got the same issue towards the mid-end of the game. It certainly seems like a performance issue: it tends to always coincide as my pop reaches beyond 100.

[hyiltiz]

Running the following two lines before running 0ad helped:
echo battery > /sys/class/drm/card0/device/power_dpm_state
powertop --auto-tune

It led to less heat-up, and GPU rendering worked throughout the game. I can now probably set it up so it persists over reboots.

 

[Norse_Harold]

1 hour ago, hyiltiz said:

It led to less heat-up, and GPU rendering worked throughout the game.

Okay, glad to hear that changing the power management settings for the GPU has reduced, maybe eliminated, the malfunction. Ideally you would check the temperatures.

You're using the open source AMDGPU driver for your GPU, right? You can check the temperature sensor readings with sysfs as explained here. What are the temperatures when the GPU is idle, when it's under light load (ie. running 0ad with low activity), when it's under heavy load (ie. running 0ad with high activity)< and when it's running Xonotic or another game?

I've read that 80 Celsius is a relatively safe maximum temperature for a GPU, and 90 Celsius is sometimes acceptable. Of course, the temperatures read by the sensors are not necessarily the peak temperatures of the IC.

If the temperatures are excessive then you can sometimes resolve the problem by cleaning the heatsink. Some laptops accumulate so much dust and hair on the inside of the main heatsink that the airflow is restricted, and the temperatures are excessive.

Cleaning the heatsink usually requires partial disassembly unless there's an access panel directly to the heatsink. Try to find a service manual for your model of laptop or a Youtube video demonstrating "disassembled", "disassembly", "teardown" or "replace the hard drive", or similar search terms including your laptop model or series.

Don't remove the heatsink without knowing what needs to be done for your specific model of laptop in order to reinstall the heatsink. Sometimes disposable thermal pads are necessary of the correct dimensions in order to properly cool various components other than the CPU and GPU.

Edited November 8, 2022 by Norse_Harold

[hyiltiz]

As reported from `lsmod` (attached above in Spoiler), I am using radeon. I tried amdgpu as well before, which can drive this GPU tho not quite sure which one would have better support for this. I know amdgpu has better and recent dev work. But either way, I can check temp via sysfs, for sure. In fact, I didn't even need to; earlier I could cook an egg on the surface, now it is only warm to the touch and doesn't even feel hot. I should probably clean the heat sink and the fans, and glue new coolant; I have done that before, and if needed, I can do that again. Thanks for the suggestion!

[vladislavbelov]

Have you tried the texture quality option to low?

[hyiltiz]

Yes; Texture Quality is set to low.

[vladislavbelov]

On 10/11/2022 at 6:31 AM, hyiltiz said:

Yes; Texture Quality is set to low.

Did it change the behaviour? Also you could open profiler (F11) to see amount of uploaded textures memory when the bug appears.

[hyiltiz]

The Texture Quality has been low throughout my testing; I turned it to High only once. It didn't seem to change the behavior noticably; it may have decreased time-to-issue-from-game-start, but I haven't been monitoring time too closely. I am booming almost the same way, and always attacking as I get close to 180 pop, so that is also about the time the issue appears, so Texture Quality didn't seem to have any noticable effect.