glTF2, 0 A.D. and Godot

[Flamadeck]

Recently Godot posted an interesting post detailing the state of standard asset exchange formats for game engines.

Their post goes over previous and even previous version of glTF citing all kinds of problems, from ambiguity defects, crappy libraries to crappy coordinate conventions and even missing functionality for common assets in game engines.

The post conclusion comes with a very unambiguous message: 

Why we should all support glTF 2.0 as THE standard asset exchange format for game engines

https://godotengine.org/article/we-should-all-use-gltf-20-export-3d-assets-game-engines

Quote

glTF 2.0: What makes it great?

By just reading the spec, the advantages should become clear. Still, I will detail the key points below:

JSON based

GLTF is a JSON based format. It's very easy to parse and the data already comes typed. This significantly reduces the need to rely on complex third party libraries to read it. Many languages (e.g., Javascript, Python) even support this format natively.

Comparing the size (in lines of code) of both Godot's glTF 2.0 and Collada importers, the result is as follows:

• Collada: ~5000 loc
• glTF 2.0: ~2000 loc

Godot does not even use third party libraries to parse these files. This makes it more evident how simple the format is in comparison.

Theoretically, glTF2 should be less efficient to parse than Collada, as it requires parsing the whole JSON into memory. Collada can be streamed in by using a SAX XML parser but, in practice, glTF beats Collada hands down. This is because (besides many Collada exporters not obeying the specification, so they can't be parsed as SAX anyway) glTF has yet another killer feature:

Separate binary blob for arrays

glTF allows specifying an external file for the big data, in binary format. It even supports many common GPU data types, so in practice this file can be moved in chunks directly to the GPU memory. Yes, glTF is the first of these formats that can even be used efficiently as in-engine data.

Even if engines would prefer to import to their own format, like Godot does, glTF is extremely fast to export and import. This makes it ideal for making changes to a complex file in the 3D software and updating them to the 3D engine almost instantly.

No ambiguity

glTF's file structure is crystal clear. No extra or redundant data exists. There is only one way the scene definition can be understood, which gives no room for exporters to take shortcuts. It also makes life for importers simpler, warranting that all existing scenes will work on the first implementation attempt.

Collections of objects of the same type are stored in JSON arrays and referenced by indices. No more confusing IDs.

There is also no support for different coordinate systems or units, and this is great. Quoting one of the first paragraphs of the specification:

• glTF uses a right-handed coordinate system, that is, the cross product of X and Y yields Z. glTF defines the y axis as up.
• The units for all linear distances are meters.
• All angles are in radians.
• Positive rotation is counterclockwise.

Just one simple use case to care about. No extra conversions. All work is left to the exporter. As there are usually way more importers than exporters, this is an excellent design decision.

Modern features

glTF 2.0 fully supports skeletons and morph targets, which can be parsed easily and unambiguously.

It also supports PBR based materials using the Disney/Unreal format, which is what most engines and 3D modelling applications use nowadays, with albedo, metallic, roughness, normal, emission and ambient occlusion. It also handles two-sidedness and transparent materials, including alpha to coverage.

Extensions for handling shader material graphs are in the work.

Great animation support

Animation support is also well done. glTF 2.0 supports multiple animations per file, which is ideal for exporting character actions (though be careful with some official examples, they are old and incomplete). It also supports many key interpolation types, such as Catmull Rom and Cubic Spline.

Animations are also stored in the binary file, so they can be loaded quickly.

Clean documentation

The specification is very complete, I personally found it well worded and not lacking in any area. There are many sample files to test all functions.

Flexible extension support

glTF uses the same extension system as other Khronos specifications, which is proven to be centralized and well organized.

Strong backing

glTF is developed by individuals from many companies, such as Microsoft, Unity, Google, Adobe, NVidia, Oculus, etc. in collaboration with Khronos.

Best of all, it's completely open and everyone can contribute!

glTF is an open standard and the development process is also transparent.

Support glTF!

For glTF to succeed, it needs strong developer adoption. Ask your favorite 3D modelling software or engine developer to support it!

Currently, the Blender exporter is in the works and not complete, and there is no support at all for Autodesk products (export plug-ins need to be written).

This is the best chance we'll ever have in a long time for a proper, open standard. Let's work together to make it happen!

Would the glTF 2.0 format be interesting for the 0 A.D. developers to implement?

A standard asset exchange format for game engines with

a rich feature set and good documentation, tests, implementation and software ecosystem support would

be great for asset developers and modders.

What does everyone think about glTF 2.0?

Is it an interesting format to support or not? Reply and discuss in this thread.

Links:

https://www.khronos.org/gltf

https://en.wikipedia.org/wiki/GlTF

https://github.com/KhronosGroup/glTF

https://github.com/KhronosGroup/glTF/blob/master/README.md

https://github.com/KhronosGroup/glTF/blob/master/specification/2.0/README.md

https://github.com/KhronosGroup/glTF-Sample-Models

https://github.com/KhronosGroup/glTF-Blender-Exporter

 

 

Edited August 7, 2017 by Flamadeck
Fixed and improved Tags

[vladislavbelov]

It's like:

 

This format have pluses and minuses, i.e. Blender doesn't support this format. If we should change the format, why we should use the text one or mixed (like this this)? We could use something faster, most binary formats are faster and smaller. But yes, most of them have a license which we can't support.

Also converting is the really much cost operation, because we should convert not only files, but wiki, parsers (a lot of C++), tutorials, probably we should add some hacks for this format. Do we really need to change the format? Because we don't feel bounds of the Collada so much, because we use it pretty simple.

Another way we could use assimp, and we won't longer be depended on file formats.

Edited August 7, 2017 by vladislavbelov

[Flamadeck]

2 minutes ago, vladislavbelov said:

It's like:

 

This format have pluses and minuses, i.e. Blender doesn't support this format. If we should change the format, why we should use the text one? We could use something faster, most binary formats are faster and smaller.

Did you not read the post (considering the fast response that might be true you dirty meme spammer) before writing your reply?

GlTF 2 stands head and shoulders above the other formats in terms of specification, test cases, clearness, speed and features.

There is already a Blender exporter and glTF 2.0 supports specifying an external file for the big data, in binary format.

 

Quote

Separate binary blob for arrays

glTF allows specifying an external file for the big data, in binary format. It even supports many common GPU data types, so in practice this file can be moved in chunks directly to the GPU memory. Yes, glTF is the first of these formats that can even be used efficiently as in-engine data.

Even if engines would prefer to import to their own format, like Godot does, glTF is extremely fast to export and import. This makes it ideal for making changes to a complex file in the 3D software and updating them to the 3D engine almost instantly.

 

[vladislavbelov]

4 minutes ago, Flamadeck said:

Did you not read the post (considering the fast response that might be true you dirty meme spammer) before writing your reply?

GlTF 2 stands head and shoulders above the other formats in terms of specification, test cases, clearness, speed and features.

There is already a Blender exporter and glTF 2.0 supports specifying an external file for the big data, in binary format.

Yes, but did you read the original post? It's JSON based. Did you use blender? It doesn't support it by default, but with external addon-s we could add any other format.

Edited August 7, 2017 by vladislavbelov

[Flamadeck]

11 minutes ago, vladislavbelov said:

Yes, but did you read the original post? It's JSON based. Did you use blender? It doesn't support it by default, but with external addon-s we could any other format.

I know. Have you seen how other formats are structuring their data? It's much worse. Some things so much worse.

Also the Godot open source game engine supports glTF2 natively.

Do not fear Blender add-ons dude! Blender add-ons are necessary to go from noob to pro!

Many pros use them to enhance, accelerate and improve their workflow and artistic output.

Edited August 7, 2017 by Flamadeck
Fixed capitalization

[vladislavbelov]

Just now, Flamadeck said:

I know. Have you seen how other formats are structuring their data? It's much worse.

Also the Godot game engine supports glTF2 as natively?

It's much worse.

Do not fear blender add-ons dude! Blender add-ons are necessary to go from noob to pro!

Many pros use them to enhance, accelerate and improve their workflow and artistic capabilities.

Much worse? What does it mean? Currently I'm looking at the format and see that it has the fragmented input and many time for parsing.

It doesn't tell me something, that Godot supports it. Because it's not used for really big projects (AAA, AA), at least I don't know about it.

I agree that installing addons it's useful, but not necessary, because even with the best addons you won't be a pro.

Btw I don't tell that it's bad format, no. It could be really good. Just I need tests, benchmarks, results (numbers), but not words. So I wrote why changing the format is really complicated task.

[sphyrth]

I was only looking for 2 phrases that left me satisfied:
1. Easy Parsing
2. Open Standard

The sad thing is, I'm not an asset maker... meaning my opinion on this has little to no relevance.
 

[Lion.Kanzen]

@stanislas69 you have any idea?

Edited August 7, 2017 by Lion.Kanzen

[fabio]

KTX texture format was also proposed, but it went nowhere.

[Stan`]

Okay so I have to be that guy but I to put things in perspective.

Most if not all of our art assets use collada and XML format. Replacing all those would be a tremendous amount of work. I'm pretty sure that's 1000+ files If it was possible at all. We can't even import our assets exported from blender with blender. I imported the Gaul ram to make a Zapotec variant and I had to tweak every single bone and rotate them to get it in the same state. We currently have dae using z_up and y_up and respectively meters centimeters and inches as units. Some of the files can't be imported in blender without editing their content as they contain max materials.

So in order to have a smooth and smart transition to the Godot format we'd need to write a pretty strong parser.

Then comes the fun part. We have pmd and PSA files which are binary and therefore more optimised than any text format. We'd need to convert those too as the reading and converting code for those would have to be removed if we go the sane way.

Then all our texture maps are diffuse normal/parallax/spec/ao. We don't support PBR.

The opengl version is quite low so I can bet we don't have the optimizations needed to make it work properly. We also have a custom working engine we'd need to switch.

This was the bad part. Not here is the practical one.

Who will do this ? And who will support the good guy who will do this ? Is the license compatible, is the code compilable on every Linux distro we support as well as macOs ?

So @Flamadeck Unless you can provide us a patch been it a WIP that proves it can be done for our engine I don't think that kind of migration is possible.

I'm not closing this discussion and I'd like to be proven wrong but I think that reduces everything to something like this close to 0% of happening now

[leper]

All of this sounds a lot like what Collada was about a few years ago. If someone does all the work and thus allows us to drop that bundled FCollada (which we are "maintaining" in the sense that we try to not make it fall apart) that would most likely be appreciated, however the current state of support for that format seems a bit light right now.

[xan2622]

Substance Painter 2017.3 now supports gITF 2.0 :
https://support.allegorithmic.com/documentation/display/SPDOC/Version+2017.3