PositionToCamera

[Peardox]

Apologies if I’m asking a very stupid question that’s handled by some other unit but…

I want to design a nice looking UI, a sort of HUD, for my project and as I’m using 3D anyway all UI components can start off as being planes (for flat buttons) or rectangular boxes (3d controls).

Here’s my basic plan, I want to create a Scene to hold the UI elements then use BeforeRender to position the scene so it remains fixed relative to the camera (and by association the viewport). I can then detect user interaction with the Key / Mouse / Touch events and respond accordingly.

Two simple questions and then one problem…

1. Where do I get the extent of the ViewPort from? e.g. running full screen on my laptop I want it to return at least something like 1920, 1080. I’ve looked but this simple detail’s answer eludes me.

2. Make my UI always be drawn? I guess this is like giving it a draw distance of zero. One concern here I have is rounding errors - they could potentially glitch a part of an observed scene in front of the UI.

And the problem…

I tried out using the following code to test…

procedure TForm1.CastleControlBase1BeforeRender(Sender: TObject);
var
  ZPlane: Single;
  Projection: TVector3;
begin
  ZPlane := -1;

  if Viewport.PositionToWorldPlane(Vector2(128, 72), True, Zplane, Projection) then
    stateNotifications.Show( 'X : ' + FloatToStr(Projection.X) +
      ', Y : ' + FloatToStr(Projection.Y) + ', Z : ' + FloatToStr(Projection.Z))
  else
    stateNotifications.Show('Not possible');
end;

StateNotification is just a global TCastleNotifications so I can see what’s going on.

I sorta thought this would fail as it’s not taking the camera into account and it does indeed go wrong at certain rotations (I can see why now - everything’s world relative).

I’m obviously using the wrong function - this was just a quick first stab.

So I’m looking for something similar to PositionToWorldPlane that uses the Camera for the calculations.

[michalis]

Where do I get the extent of the ViewPort from?

You can get the current TCastleViewport size looking at

• RenderRect (this returns position and size in final, device pixels, after UI scaling is done)

• Or EffectiveWidth and friends EffectiveXxx. These return size before UI scaling is applied. (So in the same coordinate system as the Width property).

See the TCastleUserInterface docs, TCastleViewport just descends from TCastleUserInterface.

If you just want to know the current aspect ratio, which I suppose is what you want for 3D UI, then both options are fine. EffectiveWidth / EffectiveHeight will be equal to RenderRect.Width / RenderRect.Height.

Make my UI always be drawn? I guess this is like giving it a draw distance of zero.

It should be at distance > zero, if you use perspective view. In perspective view we have a “near clipping plane” and your UI has to be further than this to be visible. Query Viewport.Camera.EffectiveProjectionNear to know this value, and place your UI further than that.

To make the UI not collide with world geometry,

• One solution is to make sure you use collisions (Scene.Spatial := [ssDynamicCollisions, ssRendering]) and that your navigation radius (minimal distance from camera to collidable stuff) is larger than the possible distance to your UI. You can adjust this radius by Viewport.Navigation.Radius, if you use our standard navigation classes for walk/fly. See the " Set the camera and navigation" section on Viewport with scenes, camera, navigation | Manual | Castle Game Engine .

  This solution has the obvious drawback – you will effectively have to fight to “squeeze” your UI within some distance range: between camera “near clipping place” and “collision radius”. The other solutions overcome this limitation.

• You can place your UI in a separate TCastleScene descendant, and override LocalRender there to set RenderContext.DepthRange . This uses OpenGL trick to basically make sure that stuff rendered with RenderContext.DepthRange := drNear is always visible on top of stuff rendered with RenderContext.DepthRange := drFar. You can practically copy the TPlayer.LocalRender implementation: https://github.com/castle-engine/castle-engine/blob/master/src/game/castleplayer.pas#L1474 .

  GitHub - castle-engine/castle-game: First-person shooter style game in a dark fantasy setting, using Castle Game Engine is using this to display 3D weapons always on top of level.

• Yet another solution is to use multiple TCastleViewport, with the same position and size, as layers. This way the order of TCastleViewport (which are TCastleUserInterface descendants, they have order determined by InsertFront / InsertBack calls or dragging them by CGE editor) determines the drawing order. So you would place the UI stuff in the front viewport.

  Make sure to set the Transparent on all (except the last) viewports to true.

  And synchronize camera from one viewport to another (e.g. in BeforeRender, use Viewport.Camera.GetView / SetView to copy camera vectors).

  Our “Escape from the Universe” ( https://www.youtube.com/channel/UCL39xgCEpgymhIiuNikXiUg ) uses this approach to organize enemies / level elements into layers.

Note that if you want the GUI to be flat 2D, then you should just use the standard CGE UI ( User Interface | Manual | Castle Game Engine ). It’s way simpler than doing it yourself in 3D But then, if you want to place it in 3D and e.g. rotate with perspective, than indeed you have to make it a 3D object inside TCastleViewport. I plan to allow our “standard UI” work in 3D some day too (I have an idea how to do this), but it’s not yet done.

I sorta thought this would fail as it’s not taking the camera into account and it does indeed go wrong at certain rotations

PositionToWorldPlane does take current camera into account. And it should account for all possible perspective etc. settings. If it doesn’t work, please submit a bug with testcase – it really should work always

[Peardox]

Well, I’m not sure if this is a bug or just me being stupid…

The “Not Possible” is from Viewport.PositionToWorldPlane returning false

The project is a simple load object with auto controls + camera with F11 bound to Full Screen. i.e. load it and press the arrow keys to set it rotating

Project attached…

test03.zip (1.5 MB)

[michalis]

I made some tests.

The PositionToWorldPlane works as documented in my tests, but it may be not what you need. In 3D, the new PositionToCameraPlane may be closer to what you expect

I just pushed it to GitHub ( https://github.com/castle-engine/castle-engine/commit/a4810cc329aa203377645ccb2a3418e97420edac , https://github.com/castle-engine/castle-engine/commit/01bc479d74e7dd2b8c329fd09471b2e54ba319cb ) , the engine binary download on https://castle-engine.io/ will be automatically refreshed after a few hours by Jenkins (once it passes all the automatic tests).

Details:

I extended your test, showing the result of PositionToWorldPlane in 3D. Then I changed it to show PositionToCameraPlane (you can comment out two lines in CastleControlBase1BeforeRender to switch back and forth between these 2 routines). Attaching: position_to_plane_test.zip (1.5 MB)

It uses PositionToWorldPlane(CastleControlBase.MousePosition, ...). And visualizes the result by moving a trivial scene that shows a sphere. You can experiment how it works after you rotate the camera a bit.

Conclusions:

1. The PositionToWorldPlane is correct, although unintuitive It is literally making a collision between

   1. camera ray (that correctly takes into account a camera position and direction) and

   2. an infinite plane with equation “Z = constant” in world coordinates.

   In other words, the plane with which we collide is not parallel to screen, as you could expect. The plane has constant Z coordinate, in the world space. This works correctly, assuming that plane is not orthogonal (or almost orthogonal) to the camera, when the calculations go crazy.

   This has some use: in examples/2d_dragon_spine_game/ demo I needed exactly this. In general it’s useful if you use 3D view (any direction, with perspective or orthogonal camera) but your game is mostly around a Z = constant plane. So it’s like a 2D game, but with 3D camera

2. What you want in 3D game is probably something else. You probably want something like Unproject from OpenGL. Which means that you want to collide with a plane parallel to the camera (not necessarily with constant Z).

   This is implemented now as PositionToCameraPlane (more accurate but too long name would be PositionToPlaneInFrontOfCamera).

   Implemented, tested with above example.

[Peardox]

I also did some more tests and came to the conclusion that I hadn’t explained what I was trying to achieve sufficiently which was, of course, camera-plane relative positioning.

I discovered the word “Frustum” while experimenting and tried writing my findings at which point I discovered that my usually good technical vocabulary was sorely lacking when it came to 3d (highly frustrating).

Essentially I wanted to place a bounding box in front of the camera and then position models inside it.

It sounds like the new method provides the co-ordinates I need to supply to two vector3d values to create a box I can position things in relative to that box.

I wanted to take the 3d line so I could rasterize some SVGs and use them as textures I could place on a plane (model of a…). I went on to write the SVG rasterization stuff but then my test didn’t work because I didn’t have PositionToPlaneInFrontOfCamera (perfect naming IMHO)

As an added bonus camera-relative positioning, hopefully, allows for positioning planes with textures, applied with TImageTextureNode.SetUrl to a TX3DRootNode.DeepCopy of the plane (model of…). ThIs is a very useful technique I watched a video of recently on YouTube frequently used to make advertizing shorts in Blender for franchised movies (Marvel was the example in the video). After GIt’s updated I’ll throw a little test together, it’ll be interesting to see the results.

Addendum

Found that video… https://www.youtube.com/watch?v=-xLeFGHn26s

[Peardox]

Just changed the position_to_plane_test.zip to make the new object a cube

How do I back-out the camera’s rotation so that the SceneShowClicks object doesn’t rotate with the main scene, or more accurately can be rotated relative to the camera?

I need something like a Viewport.Camera.GetRotation I guess?

[michalis]

1. Yes exactly You can get Viewport.Camera.Direction, Viewport.Camera.Up. Or just use Viewport.Camera.GetView(Pos, Dir, Up) to get 3 vectors. (You can just ignore returned Pos).

   Then you can set TCastleTransform.Direction, Up properties. Or just use TCastleTransform.SetView(Dir, Up) (it has an overloaded version that doesn’t change position). IOW, do

   var
     Pos, Dir, Up: TVector3;
   begin
     Viewport.Camera.GetView(Pos, Dir, Up);
     SceneShowClicks.SetView(Dir, Up);
   end;
   

2. Alternative version is to use X3D Billboard node. We support it fully in X3D.

   The simplest example of using it is on https://github.com/castle-engine/demo-models/blob/master/navigation/billboard_simple.x3dv . See also the “Navigation” X3D component docs (I just improved them ) on https://castle-engine.io/x3d_implementation_navigation.php .

   In this case, you would need to wrap your model in an X3D node Billboard.

[Peardox]

Implementing (1) produced, as expected, the specific case of making the object orthogonal to the camera. I then decided to try rotating the object around the Y axis thinking that Y is up and everything is normalised so no problems (as if anything is ever that simple…)

I get to this adaption where I’m rotating at a constant speed per ms to fully rotate the object once in four seconds with direct manipulation of X + Z…

const
  // Number of seconds to spend rotating the object fully once
  SecsPerRot = 4;
var
  Pos, Dir, Up, Projection: TVector3;
  // Decalre some scratch variables
  rx, rz, theta: Single;
begin
  Viewport.Camera.GetView(Pos, Dir, Up);
  // Calculate the angle of the oject at the current time
  theta := ((CastleGetTickCount64 mod (SecsPerRot * 1000)) / (SecsPerRot * 1000)) * (Pi * 2);
  // 3D rotation in Y scratch variable for new X
  rx := (Dir.X * Cos(theta)) - (Dir.Z * Sin(theta));
  // 3D rotation in Y scratch variable for new Z
  rz := (Dir.X * Sin(theta)) + (Dir.Z * Cos(theta));
  // Update X with new value
  Dir.X := rx;
  // Update Z with new value
  Dir.Z := rz;

  if Viewport.PositionToCameraPlane(Vector2(256, 256),
    True, 200, Projection) then
  begin
    SceneShowClicks.SetView(Dir, Up);
    SceneShowClicks.Translation := Projection;
  end;
end;

Everything worked nicely and my cube was merrily rotating independent of the camera view (when rotZ = 0…).

Yep, you’re way ahead of me here… Soon as the Z rotation was non-zero my initial delight was somewhat tempered as the objects rotation became unpredictable - well, to myself it was unpredictable, to you it’ll be predictably ‘my concept of unpredictable’

This independently rotatable object concept may seem like it’s a pointless objective but it’s actually rather cool in specific circumstances.

The best examples I can think of easily are related with generating revenue.

I could have a cube that rotates showing four images (extensible to n-images by altering the texture and other associated data on the plane facing away from the camera once per pi/2 revolutions). These images could then, if clicked on, open a web browser to some defined URI.

One obvious use would be self-sold advertising / sponsorship slots. Unless your project has an identified list of potential advertisers / sponsors (hopefully before you start development) this one’s a hard sell. There is, however, the possibility of cross-promotion either for your own projects or those of other like minded developers.

Another, easier to realise use, would be acknowledgements to third parties. Taking view3dscene as an example (as I ain’t got a product yet) a list of Patreons of some donation level could be displayed with the links all going to a unified ‘Support us and get your name here page’. The Patreon usage example is common with youTubers, you support a channel for X bux PCM and you get a mention in the ending credits.

At least one spot on this virtual cube would, of course, be reserved for the ‘how to get added to the vcube’ (think I’ll make this the top face as it’s never normally seen)

(2) This is in some ways related to the above. I was going to ask you about object hierarchy at some point anyway but the Billboard stuff resulted in me finding the grouping section. As a result I’ve got some reading to do down the like at which point I’ll prob have some Qs about that too (not confused enough to ask intelligent questions on that subject ATM).

[Peardox]

Viewport.PositionToCameraPlane

(A)

Eek… I just ran the trivial test with a cube in FullScreen and noticed something I didn’t spot in Windowed mode. The cube is rotated, something barely visible to me initially, as it’s a near black on black.

While PositionToCameraPlane does exactly what it says I realised that a variant is required for this specific use - PositionToCameraSphere is, I imagine, the desired method. I think that using a Sphere for the projection should have the effect of an object would always be directly facing the camera with no rotation.

(B)

This also make me realise that BoundingBox may not describe the volume an object can be rotated to as to have an object always visible the BoundingBox describes as at a rotation of 45 degrees in X, Y and Z the object’s projected 2d representation will be greater than that when rotation is zero.

view3dscene always looks like there’s too much border although having just tried it out on the table model I observe that rotating in Z is accounted for (which is why it looked like there was too much border).

If I’m correct BoundingSphere looks like it’s the closest to the desired method. What’s the easiest way to get that for an object in a Viewport?

There’s no BoundingBoxContainingBoundingSpehereForBoundingBox method I note

[michalis]

I get to this adaption where I’m rotating at a constant speed per ms to fully rotate the object once in four seconds with direct manipulation of X + Z…

1. You can use RotatePointAroundAxisRad from CastleVectors unit to reliably rotate one 3D vector around any 3D axis. Like rotating Up vector around the Direction axis.

   This could be used to make your rotation work reliably in any 3D case?

2. You can also make a hierarchy of TCastleTransform (put one TCastleTransform as a child of another, which may be a child of another). This may allow you to rotate the object around a constant (local) axis using one (inner) TCastleTransform, and then adjust to camera using another (outer) TCastleTransform.

   ( And TCastleScene is also a TCastleTransform instance here, as it descends from TCastleTransform. )

PositionToCameraSphere

Indeed the point of PositionToWorldPlane is to generate positions on a plane parallel to the camera, so the exact distance between the generated position and camera position will differ slightly.

If you want to instead move the object around a sphere aroond the camera, you could use this:

function PositionToSphereAroundCamera(const Viewport: TCastleViewport;
  const Position: TVector2;
  const ScreenCoordinates: Boolean;
  const Radius: Single; out PlanePosition: TVector3): Boolean;
var
  R: TFloatRectangle;
  ScreenPosition: TVector2;
  RayOrigin, RayDirection: TVector3;
begin
  R := Viewport.RenderRect;

  if ScreenCoordinates then
    ScreenPosition := Position
  else
    ScreenPosition := Position * Viewport.UIScale + R.LeftBottom;

  Viewport.Camera.CustomRay(R, ScreenPosition, Viewport.Projection, RayOrigin, RayDirection);

  // we know that RayDirection is normalized now
  Result := RayOrigin + RayDirection * Radius;
end;

It’s a modified version of TCastleViewport.PositionToWorldPlane implementation, but instead of calculating TrySimplePlaneRayIntersection it just takes RayOrigin + RayDirection * Radius.

This also make me realise that BoundingBox may not describe the volume an object can be rotated to

There is TBox3D.BoundingSphere (to get bounding sphere of a box), and Box3DAroundPoint (to get bounding box from sphere).

Note that in CGE, TCastleTransform.BoundingBox should always account for the object transformation, including it’s rotation. So this box should always be valid. But it may be larger than necessary in some cases.

( And TCastleTransform is a child of another TCastleTransform, and you want world-space bounding box, you can use Scene.LocalBoundingBox.Transform(Scene.WorldTransform). )

[Peardox]

Slight error in PositionToSphereAroundCamera - remove the out parameter + change result type to TVector3 then assign it to the a TVector3 in the caller (as out is gone)

Anyway, RotatePointAroundAxisRad fixed my wobbly rotating cube but there’s still an issue as the returned TVector3 is still on a plane rather than rotated perfectly flat to the camera.

Trying to explain something visual in words is always hard so, a bit of code and an image make things far easier to explain and demonstrate what I’m trying (and failing) to do…

So, one project (somewhat modified trivial test)

sphere_to_plane.zip (890.3 KB)

And image

The cross in the lower left is displayed from slightly above while I want to display it so that the blue bar is angled slightly away from the viewpoint to that the red + green bard are orthogonal to the viewpoint.

I found RotationAngleRadBetweenVectors which appeared to be the solution. I’ve tried many combinations of position, direction and up vector from camera and the cross but can’t manage to get it to tilt away as I desire.

If you run the test project I’ve bound the X key to move the cross between lower, center and upper areas of the screen (it cycles, press X multiple times to see). The rotation I want to apply when in any vertical position is the same as displayed when it cross is in the center - essentially tilting the object away from Y.

I’ve also bound the shift and control keys so that…

Shift / Ctrl keys switch …
none -> Camera vectors
Shift -> Cross vectors
Ctrl -> Cube vectors
Ctrl + Shift -> Table vectors

Also R resets examine view, F11 toggle full screen

The cross is colour coded X, Y and Z axis are R, G and B (easy to remember) with the pointy tip being the positive axis (rightward, into, upward)

Oh, yeah. Is there an easy way to change the notification font to a system mono-spaced like courier new? Formatting is suspect in hard-coded Serif…