Get all geometry under raycast?

Hello, so it looks like I did not thoroughly tested, because I have found a strange issue. It does not work completely I figured now, but I do not understand why. Basically I have saved user interface file which I edit in the editor where I do have viewport as root, some background and then in the Items, I do have Camera, DirLight and all models as TCastleScene. One of them I do have generated as custom geometry through X3DNodes and is saved as X3D and loaded into scene (there I need to differentiate per shape with help of triangle, so I need extended collision result). The thing is, that when I move around and get mouse cursor over object like e.g. building there, and box behind it, it only registers the building and then proceeds to return nil as WorldRay result even if I previously make the building Pickable property false. But when there is that custom geometry behind building (but should be irelevant as it is loaded as X3D model), it does register both objects correctly. Where could possibly be a problem hidden here? Here is the code for method which I call to get all geometry in Viewport’s OnMotion procedure.

function CGEForm.AllItemsHitByRay(pos: TVector2; depth: integer): TList<TPair<TShape, TCastleTransform>>;
var
  RayHit: TRayCollision;
  RayOrigin, RayDirection: TVector3;
  Item: TPair<TShape, TCastleTransform>;
  Shape: TShape;
  rayColNode: TRayCollisionNode;
  rayColPos: Integer;
begin
  Result := nil;
  depth := Max(depth, 1);
  fCGEViewport.PositionToRay(pos, true, RayOrigin, RayDirection);
  RayHit := fCGEViewport.Items.WorldRay(RayOrigin, RayDirection);
  if RayHit <> nil then
  begin
    rayColPos := GetCorrectRayCollisionNodePos(rayHit);
    Result := TList<TPair<TShape, TCastleTransform>>.Create();
    while (RayHit <> nil) AND (rayColPos > -1) AND (depth > 0) do
    begin
      shape := nil;
      rayColNode := RayHit.Items[rayColPos];
      if (rayColNode.Item is TCastleScene) AND (rayColNode.Triangle <> nil) AND (rayColNode.Triangle.Shape <> nil) then
      begin
        if TCastleScene(rayColNode.Item).VerticesCount <> rayColNode.Triangle.Shape.VerticesCount then
          shape := rayColNode.Triangle.Shape;
      end;

      Result.Add(TPair<TShape, TCastleTransform>.Create(shape, rayColNode.Item));
      rayColNode.Item.Pickable := false;
      RayOrigin := rayColNode.Point + RayDirection.AdjustToLength(SingleEqualityEpsilon);
      FreeAndNil(RayHit);
      RayHit := fCGEViewport.Items.WorldRay(RayOrigin, RayDirection);
      rayColPos := GetCorrectRayCollisionNodePos(RayHit);
      Dec(depth);
    end;

    if RayHit <> nil then
      FreeAndNil(RayHit);

    for Item in Result do
    begin
      Item.Value.Pickable := true;
    end;
  end;
end;

Thanks for any further advice.