Adding TShapeNode to RootNode from additional thread

Hi.
I was trying to dynamically load multiple shapes (TShapeNode) into the already shown scene (into the active TX3DRootNode).
If I do it from the same thread, everything works fine (but slowly)
But if I do the same from additional thread, I get exceptions from the CGE sources.
Is there any way to fix this?

Hello, @kedzoo and welcome to the forum!

As a rule of a thumb you should never use any Castle Game Engine features from a thread other than the main thread (just as with other game engines).

You may try an unsafe hack - (re)construct the whole TX3DRootNode in a thread and load it into a TCastleScene in the main thread. However, note that loading something operates on cache, and that may mess with the main thread.

A much safer approach - preload all the assets you need in the main thread and then add after that (re)construct the whole TX3DRootNode in a thread. This should work.

If you need to add/remove multiple nodes to/from a scene, consider using A) Multiple scenes and set Scene.Enabled := true/false; and/or B) use TSwitchNode.WhichChoice and set it to -1 to hide element and 1 to set it to the first child node. This way you can operate huge amounts of elements safely and lighting-fast.

Hello, Eugene.
Thank you for advise. Idea was to load part of map when camera moves. And unload part that far from camera. It is required coz I want to implement support of extrahuge maps (up to 48kk hexagones with inner geometry). So I cannot just preload it. And I cannot decompose it to resonable amount of preloaded assets. Anyway, seems like I found some solution (thanks again for your advice). Seems like simple extension of scene class will works for 2 scenes case (at least it works now in my tests with single scene).

  TMX3DScene = class(TCastleScene)
  private
    FEnabled: Boolean;
    FSceneBusy: TCriticalSection;
    procedure SetEnabled(AValue: Boolean);
  protected
    procedure Render(const Params: TRenderParams); override; overload;
  public
    constructor Create(AOwner: TComponent); override;
    destructor Destroy; override;

    property Enabled: Boolean read FEnabled write SetEnabled;
  end;
  
procedure TMX3DScene.SetEnabled(AValue: Boolean);
begin
  if FEnabled = AValue then
    Exit;

  FSceneBusy.Acquire;
  try
    FEnabled := AValue;
    if FEnabled then
      Enable
    else
      Disable;
    Exists := FEnabled;
    Visible := FEnabled;
  finally
    FSceneBusy.Release;
  end;
end;

procedure TMX3DScene.Render(const Params: TRenderParams);
begin
  FSceneBusy.Acquire;
  try
    inherited Render(Params);
  finally
    FSceneBusy.Release;
  end;
end;

UPD: No. Each time when Scene was “enabled”, it allocates some memory. With no reason

Finally (actually, not sure about “finally”) implemented it like that: geometry is calculated and shape nodes are created in additional thread. And Root node is updated in main thread (wnd.OnBeforeRender).

But can easily see that screen is hanging while root update. And sometimes scene recalculating everithing (in task manager can see that a lot of memory released and allocated again. I guess its Scene.ChangedAll method). This also causes lags.


[dynamic loading of 5kk tiles map]

I returned to experiments with a full load in a separate thread. Yeah, I remember that rule “do not use another threads”. But what I can do? And seems like some engines support async load (Unreal, Unity, Panda3D and some another - that google said)

Back to my tests. This works in many ways. And works fast. But there is always one problem. Something in memory is not freed. But this is not a memory leak, these are some registered objects that freed correctly afterwards. Allocated memory grows for each iteration.

Wondering what it is and how to release it …

Reconstructing\recreating root node, Scene.FreeResources, Scene.ChangedAll, RootNode.UnregisterScene, RootNode.FdChildren.Changed and anything I found in sources - does not help

Be cautious of unexpected (they may be rare, or may be very rare, but I wouldn’t count on that) errors though. I’ve had a (very) painful experience of almost completely rewriting a large project because of that.

Also see https://castle-engine.io/wp/2020/07/19/asynchronous-non-blocking-downloading-using-tcastledownload-class-and-other-http-communication-features/ - however, as I didn’t try the feature yet, I’m not sure how it’ll work with actually loading models, but at least preloading them can be made from a thread.

Note, that this may be the feature of Windows/Linux memory management. The memory is freed correctly, but is still allocated for the program. This way they keep the program run faster by avoiding defragmenting the memory when not needed.

No, Eugene. If you right, then in single thread case it must allocate memory in same way. But its totally different.

Sure. Thats why I trying to solve it now, before to do anything else. But I think my final extension of TCastleScene works quit fine. I can manage any collision, and waiting when all Scene methods in main thread finish works after removing scene from Viewport. Yeah, maybe something is still there but there is no exceptions…

Thank you, I will check it.

Ah, ok. That makes sense.

Do you do FreeOnTerminate := true?

Sorry, I somehow missed your two previous messages… Looking at the image, it looks like not the best way to render that many tiles. Sorry, I’m busy right now and can’t go down into the details, but my first guess would be creating some sort of “LODs” (they can be autogenerated, but overall the idea is similar to GoogleEarth/GoogleMaps) and loading those instead of thousands of shapes which is heavy both for rendering and loading. Also splitting the whole world into multiple scenes can help greatly (to avoid recalculating one huge scene in ChangedAll), but I think you’ve done that already.

I use FreeOnTerminate := False and destroy Thread by myself on Loader object destructor. Coz thread implemented for many purposes, not just removing or adding shape nodes.

Previously I implemented it in 2D and it works fine, but amount of prepared sprites will be about 480k… I planned a lot of variations of landscape. Different indents depends to another tiles, vary river width and etc. So, I turned to 3D (orthogonal camera for now). This scale is just for testing, easy to see how it loads, I already planned what to do for so huge scales but it will be later.

not exactly, I glue hexagones in groups (can set group size, now using groups 10x10) and it works good enough, its even works faster then I expected and (normally) takes just about 800 MB of memory (video+ram, coz testing on notebook without discrete graphic card) for 5kk fully loaded map. What on screen - I think will be LOD 1