Tutorial - Ambient Occlusion and Normal map bake using Xnormal

[Lion.Kanzen]

What is XNormal.

xNormal is a program that is widely used in the industry for baking down your high res sculpts/ high geometry meshes out of programs like zBrush, Mudbox, etc. to you low res and game ready mesh, maintaining all the detail from your high sculpt/ geometry.

The program is free available and you can download it by visiting: http://www.xnormal.net/Downloads.aspx

 

How to use xNormal for normal map bakes and getting the best results

The challenge in using xNormal is to be able to get all the detail of your high res mesh onto your low res, game ready mesh and make your lower res game usable mesh appear like the high res mesh.

The defaults results will give you a pretty average result in giving you a lot of double projections and bad seams or simply that the rays won't hit your geometry at all during the bake.

By following this simple rules you'll be able to get all of your detail to your low res game ready mesh, while making it appear like the high res mesh.

Making sure that the low res geometry isn't too low and being able to catch all the rays from the bake

When you're creating your low res mesh, don't go too low on geometry. Always make sure that low res will catch all the curves and angles.

As a practical example, if you have a high res cylinder with 64 spans, you will get a bad bake if your low res cylinder is being only 4 spans. You low res cylinder in this example is so low that almost all the rays will miss it and giving you a very jaggy and bad bake.

Down below is an example of how low res looks like to get a nice bake.

Adjusting the ray distance and default mesh scale in xNormal

Since there are a multiple of 3D software packages in the industry and being used the scale from each to the next can be very different (for an example a grid unit in Maya is different that a grid unit in 3DS Max).

To get the best result in xNormal and making sure the rays hit a big enough mesh we should set the default mesh scale in xNormal to 16. This way we make sure the mesh is big enough to catch all the rays.

We should also set the ray distance to a higher value then the default one. The default value in xNormal is 0.5 which is too low and a lot of rays miss the target and double projections occur. A good value to start out with is to set our ray distance to 50 and tweak it from there.

Baking out separate pieces of your geometry

If you're still getting double projections and bad seams, one option to fix it is to bake out separate pieces of your mesh.

The issue that arises when you bake your whole mesh at once is that the edge padding will "bleed" over into other parts of your texture map and in some cases xNormal doesn't know the differences and it projects one piece over the other. This will especially happen if you got a tight packed UV map layout.

It is a bit more of a laborious task to break up your mesh into several pieces, bake them individually and then cut and composite those in Photoshop, but it will give you a nice clean bake with no double projections and edges "bleeding" into your other parts of the texture map.

Inverting the Y- value/ green channel of the bake for CRYENGINE

In the regular normal map bake settings for xNormal, the Y channel (or green channel in Photoshop) is set to Y+. CRYENGINE and Max use the inverse, meaning a value of Y-. There are two ways to fix that problem.

The first is to go to the settings in xNormal and set the Y+ value to Y- so we'll get the right bake out of it. Many artists want to read the normal map bake with it's normal settings to understand and see where eventually issues are. Reversing the Y- in xNormal makes it hard to understand.

So the second way is to bake it out normally from xNormal and then once everything is ok, to go to the green channel in Photoshop and reverse it manually. Both ways are viable and up to personal preference..

Using xNormal for baking out your Ambient Occlusion

With xNormal we are also able to bake out our Ambient Occlusion. To get a nice AO bake we have to tweak some settings. Preferably we want to set our Occlusion rays to 256 or higher. 256 is a good value for a 2048 x2048 image.

The higher the number on the occlusion rays the smoother and softer the AO will be, exponentially your render time will double. If you have a bigger size image like a 4096 x 4096 it is recommended that you set your occlusion rays potentially to 512, depending on how large your surfaces are.

If you have lots of smaller pieces and surfaces setting your occlusion rays to 512 won't give you much of a difference as compared to 256.

The next setting we have to adjust is the spread angle. The maximum value is 179.50. With a higher spread angle we make sure that the AO rays get wider spread hitting also the smallest gaps in our mesh and ensuring we get a nice smooth AO in those.

Setting the correct Bucket size and Antialising settings

The next setting we want to change is our bucket size. Normally we used to have the smallest bucket size (16) because the general impression was that it would be more detailed because it does 16 pixels per CPU core but after doing some research we found out that the highest (512) is faster. So if you use 512 and have a quad core with 8 threads, you will see 8 render blocks trying to render 512 pixels each in one go. What we don't want set higher is our AA setting. 1x AA is enough and the difference to a 4x AA is minimal.

The same image has been rendered once with an AA setting of 1x the other of 4x. The 4x AA took 43 minutes to render! The 1x AA took only roughly 3 minutes. The difference is minimal and doesn't justify the time. Plus also once our map gets reduced in size, like from a 2048 map to a 1024 Photoshop applies AA to image as it gets scaled down.

Curvature Map

The best use for having a good gloss map, render a curvature map  and that use as an absolute base for the gloss map.

It provides a mid gray image with every edge highlighted where a curve is.

To get it to render like this, you need to set up options like the image down below

The amount of rays you should set are the same as the AO and also their spread angle is the same.

To get it to render a grey image, change the "Tone Mapping" setting to monochrome.

ZBrush polypaint/Vertex Colour

Some people prefer to use ZBrush polypaint to texture assets. It is important to keep in mind when you do this that the mesh be as dense as possible considering it is vertex colour based. If the high poly mesh is too low, the texturing will turn out blurry and will look very low res.

To be able to render the polypaint/vertex color it is important to turn off the "ignore per vertex color" tick box on your high poly mesh. As the option suggest, if you don't, it will completely ignore it and show you a black render.

Now turn on the "Bake Highpoly's Vertex Colour" and asset is ready to render.

Texture Baking/UV transfer

You can use Xnormal as a "render to texture" tool with the "Bake Base Texture" option.

[Lion.Kanzen]

• xNormal is a free software that we used before the Substance Painter
• It is recommended to use Substance painter for all the baking possible
• But in some cases XNormal might be needed, for example
  • if the Substance Painter is buggy and slow and no other solution is available
  • with alpha maps, as it's not possible to get the alpha map easily out of Substance Painter
• When baking a normal map, use setting -Y when baking because the inverted way is used in our editors
• When baking a color id map you might want to use Antialiasing x1 to create crisp edges for masking in Substance Painter

Baking an alpha map in XNormal and adding it in Painter

• Start with XNormal
• Select the low poly mesh

 

WIP [Space reserved]

[Loki1950]

BTW it is only available for Win64 though an very old version for Win32 may still around somewhere there was talk several years ago of a Linux port never enough coders about so that fell by the way side.

Enjoy the Choice

[Stan`]

5 hours ago, Loki1950 said:

BTW it is only available for Win64 though an very old version for Win32 may still around somewhere there was talk several years ago of a Linux port never enough coders about so that fell by the way side.

Enjoy the Choice

I don't think the source is public ?

[Loki1950]

It is not public it's a one man project really used it about ten years ago and have it installed on my Win 10 machines.

Enjoy the Choice

[Stan`]

That's what I thought. A bummer though cause it's really good software ...

[vladislavbelov]

1 hour ago, Stan` said:

That's what I thought. A bummer though cause it's really good software ...

Does it make sense to use xNormal if Blender has a builtin baker?

[Lion.Kanzen]

7 minutes ago, vladislavbelov said:

Does it make sense to use xNormal if Blender has a builtin baker?

I also read about Substance Painter.

[Stan`]

1 hour ago, vladislavbelov said:

Does it make sense to use xNormal if Blender has a builtin baker?

It sometime does, as blender can have wonky results. For the general case it's alright.

1 hour ago, Lion.Kanzen said:

I also read about Substance Painter.

Substance painter is good! I have a license. But it doesn't fit our texture atlases workflow. Example

This texture could come from substance, the problem is it doesn't support reusing the same parts of the textures  (eg. for the arm) as the light bake will be incorrect

So it will only work for animals or objects that don't share a texture.

For buildings it's better to use Substance Designer. I made the dwarf struct with it.

https://github.com/0ADMods/dwarfs/blob/master/art/textures/skins/structural/dwarf_struct.png

 

 

[Trinketos]

6 hours ago, Stan` said:

It sometime does, as blender can have wonky results. For the general case it's alright.

Substance painter is good! I have a license. But it doesn't fit our texture atlases workflow. Example

This texture could come from substance, the problem is it doesn't support reusing the same parts of the textures  (eg. for the arm) as the light bake will be incorrect

So it will only work for animals or objects that don't share a texture.

For buildings it's better to use Substance Designer. I made the dwarf struct with it.

https://github.com/0ADMods/dwarfs/blob/master/art/textures/skins/structural/dwarf_struct.png

 

 

The two programs together cost 5976.70 MXN 

¡xD!

[Lion.Kanzen]

1 hour ago, Trinketos said:

The two programs together cost 5976.70 MXN 

¡xD!

Cuánto es eso en dólares.

[Trinketos]

299.99

[vladislavbelov]

10 minutes ago, Trinketos said:

299.99

Pretty cheap for a professional software ~3x of annual Netflix subscription, I remember Maya costs much more.

[Stan`]

Yeah maya is more like 3999€ per year.

[Lion.Kanzen]

I'm going to try to see how I make the textures work and make my rocks/stones low poly. You know use a  retopology technique.

Edited May 30, 2021 by Lion.Kanzen

[Loki1950]

2 hours ago, Stan` said:

Yeah maya is more like 3999€ per year.

And the student version's terms of use say that you do not own what you create with it

Enjoy the Choice

[Trinketos]

2 hours ago, Stan` said:

Yeah maya is more like 3999€ per year.

You can travel in time with maya?