Test Case: Loading small binary glTF (glb) with big issues

1

Greetings.

In some other topic I promised to give you some test case in regards to slow shadow volume calculation. I have created a simple world with a dozen of instances of a “house”, all are Cashed=true. The model is neither too beautiful nor too complicated. It’s just for the test case and it has already proven to slowdown loading, and even crashing the game.

Before I proceed I’d like to say that all the issues listed below only happens when loading from a binary glb, and x3dv that have textures packed inside. The same models exported to json version of glTF load nearly instantly, also LOD doesn’t freeze the screen, the shadow volumes work without issues, and no crashes.

My tests were done with shadow volumes, and I divided them the into 2 main categories: without LOD, and with it. The LOD uses 2 lazy-made “middle” & “far” models but it’s good enough, the fps keeps being nice typically. However, my goal was to make them un-optimal.

The loading time is reasonable, having 12-20 instances of TScene. For single model it’s about 20 seconds, and for 3 LOD level it’s 2 times more as expected. When playing, I have to look around first - it makes the viewport freeze often for a short time. The models are supposed to be cashed but somehow need the reload of the the 2nd and 3rd level of LOD - after turning 360 degrees everything works quite fine.

However, memory usage is very questionable. Without LOD, having just 1 model takes some 40GB of my memory. Loading x3dv LOD took above 50GB (with peak at 59.4) ram and a whole 11GB vram. I don’t think it’s right.

For a comparison, json glTF with separate textures loaded in few seconds, and used 566MB plus few MBs of vram.

After loading it goes down to “just” 16GB sometimes, but not in every scenario. I understand that many players have this “just” 16GB for the whole system, so it’s still somehow above my expectations. GPU memory stays at full 11GB vram + 14.5GB from shared (part of the above 16).

When I loaded it with 20 models in Debug mode, I’ve got a crash with EOutOfMemory while the engine was trying to load some texture/picture. The “lod_far” model doesn’t use normal map or roughness as they’re not seen anyway. I don’t believe that’s the case, because memory doesn’t improve for the 2 other models.

Switching Shadows off on the directional light didn’t improve the memory consumption. Switching CastShadows on the scenes didn’t help either. So the issue isn’t because of shadow volumes.
Turning off textures in RenderOptions also doesn’t help, as the binary *.glb needs to be loaded anyway, but it’s supposed to be loaded once so it shouldn’t consume much memory?

In Release mode, 20 models consumed nearly all my memory but as debugger wasn’t loaded I could allocate up to some 61GB without crash.

Using more than 20 TScene?
One time I had an error dialog saying some stuff about objects’ parent being wrong but I couldn’t take the screenshot. The second run didn’t go any better - the message had funny white rectangles instead of letters, and I ended up rebooting the system - it’s fastest option to solve memory leaks IMHO.

The crash isn’t consistent, as it may happen that memory doesn’t peak above 61GB. Usually I can have 20 TScene and it runs at about 60 fps (my display limit).

I hope what I wrote makes sense, maybe it’s too chaotic, and that it’s helpful. I believe this test case would help to improve some bits and bobs somewhere in the future. It’s may also be of benefit to others, so they know that some file formats can cause serious issues.

The case files, zipped, are 150MB so I uploaded them to Proton Drive

shadow_volumes_out_of_memory
shadow_volumes_out_of_memory1322×509 205 KB

1 Like
2

The memory consumption actually makes sense if we consider that glb isn’t cached correctly. We’d have 20 copies of unpacked textures. The textures themselves take almost 1GB for each instance. Still the consumption is 3 times higher than physical size, but for every material you need to create metallicRoughness texture, etc.

How I understand it.

The issue happens not only for glb’s but also for x3dv’s that have embedded textures.

For json files that link to external texture, the cache list (simplified) would be Cache.List[‘textures/mytex.png’] so every new TCastleScene instance refers to exactly the same resource.

However, for packed files you need to assume that names can be not unique.
You require unique name so you add prefix/suffix.
Then, for every new Load(‘model.glb’) you process it as a completely unknown previously file, and create resource ‘mytex.png’, ‘mytex_01’ etc.

Now, when you ask Cache object if the resource is there, the Cache will say ‘No, I don’t have mytex_01.png’ and new cached object is created.

It really looks like the *.glb is processed every time for every new TCastleScene. If the Cache had an info that particular ‘castle-data:/model.glb’ was already loaded, then it would return link to already loaded resources without suffixes. And then, loading time wouldn’t increase, and also the memory consumption would stay at the same level as it does for json versions with physical files ‘textures/mytex.png’.

I know it’s simplyfied thinking, but it seems it explains the memory issue.

1 Like
3

I went through the code to see how far my simplification was from the truth. It seems the problem lies in DeepCopy and how files are loaded and translated into x3d nodes.

When I load a json model, with links to external resources (textures in this case) in a physical file system, like hard drive or network, the Cache loads the textures only once. DeepCopy of the model copies just the links - not the resources. Which is as expected.

However, when I load “virtual files system” like *.glb or *.x3dv with embedded resources, then the DeepCopy copies the whole resources, not just links to them. What I initially thought was that textures, sounds, etc are held in a separate tree. Then the copy would only copy the model pointing at these resources without copying the resources - exactly the same as would be with simple json/xml. In such case the Cache asked ‘Do you have this texture?’ would answer ‘Yes’ as it does with separate files and provide a link to the resource.

Currently, the only advantage of using Cached for *.glb is that the model doesn’t have to be loaded again from storage or network. In contrary to my previous comment where I wrongly considered the cache doesn’t know if this or that model was loaded, the cache in your engine does know. But it doesn’t know the ‘mytexture’ because the texture is just an x3d node with some data inside the model, and is copied to every instance as is.

So, although my simplified model was quite far from true, from my point of view it behaves kind of the same anyway. (My simplifications can sometimes sound unfair, but it’s not my intention to question the skills you guys put into this engine - I just hypothesise starting form most trivial to more in-depth)

Anyway. I don’t consider the scenario in this test case as a bug, it’s just an unexpected behaviour for me. And as I understand the remedy wouldn’t be very simple. Because the simple translation of a model into x3d objects (done by the file reader) would have to be made aware of the embedded resources and load them as a separate thing from inside the file. It’s somehow similar to unzipping ZIP files. You would have to unpack the resources into jpeg-image or png-image, or sound-wav, or whatever the embedded resource holds and keep them like you would keep physical files from a drive.

3 Likes
4

Thank you for the testcase! I just downloaded it, confirmed the problem and started to analyze it.

And then I saw you already did the analysis in following posts in this thread – huge thank you!

Yes, everything you concluded is right, except one thing:

Nah, I do consider it a bug and I think we can improve it :slight_smile: Although we recommend to use “unpacked” glTF (i.e. separate glTF, and binary data, and texture files) in our docs how to export from Blender, not “packed” GLB, but naturally people often will use GLB, and we still want to be reasonably efficient in this case.

And GLB files make sense when one wants to have self-contained (single file) models, e.g. to easily open with our mobile viewer. So we don’t really discourage from using GLB in all cases, we sometimes encourage it :), it makes sense.

As you concluded, indeed:

  • For each embedded texture in GLB we create TPixelTextureNode. This node doesn’t have any URL, it only keeps image contents, and it is just duplicated on each DeepCopy of the model. This results in extremely inefficient memory usage when the model is loaded multiple times, with or without TCastleScene.Cache (because TCastleScene.Cache only prevents downloading and parsing the file, but each model is still a DeepCopy of the graph, at least until we solve roadmap item “Allow animating selected node graph properties, without changing the graph, to be able to reuse graphs” ).

    Note: in case of textures in external files, we use TImageTextureNode, and we cache them based on URLs. This even happens regardless of TCastleScene.Cache, the “texture cache” is independent.

    So the current TPixelTextureNode behavior is always bad, regardless of TCastleScene.Cache. The TImageTextureNode is always efficient, also regardless of TCastleScene.Cache.

  • Note: We don’t use special URLs to access embedded textures inside the GLB. We just create TPixelTextureNode nodes as we read the glTF file. It was simpler to do than treating glTF like a zip with a virtual filesystem. Since we had no need so far to “extract” only a specific texture from GLB (without loading the whole 3D model), this is/was good enough so far.

Solution:

  • We should “invent” URLs for these embedded textures (like castle-data:/mymodel.glb#texture:textureid) and use these URLs to utilize cache inside TPixelTextureNode.
  • I see that our cache in TTexturesVideosCache is even prepared for this already, one can use it with invented URLs + loaded image contents.
  • Our TPixelTextureNode with TSFImage will need to be adjusted: TSFImage will have to be configurable whether it “owns” the image or not, and loading image from (potentially invented) URL.
  • I think I can also address “x3dv with embedded textures” in a similar way, I just need to always be capable of generating a unique texture id inside the model… which isn’t a problem, we can even invent id and put it in TPixelTextureNode metadata. Or just use texture number.

Stay tuned for fix to this, likely ~tomorrow.

2 Likes
5

Sorry for a delay with this, I have this on a list of TODOs for next week!