# How to judge the mouse on TDrawableImage

**URL:** <https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739>\
**Category:** 2D\
**Created:** [February 8, 2023, 7:57am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739 "2023-02-08T07:57:16Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [February 8, 2023, 7:57am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/1 "2023-02-08T07:57:16Z")

</div>

I use TDrawableImage to draw by myself, Everything is good at present

I met a problem, How to judge the mouse on TDrawableImage

1、examples/viewport\_and\_scenes/detect\_scene\_hit/  
This example not use TDrawableImage to draw

2、With TCastleTiledMap, you can use Map property and PositionToTile method to detect on which exactly tile we clicked.  
My TDrawableImage shape is irregular，PositionToTile ，It seems that I can’t achieve my goal

I previously used Zengl to judge the mouse on ImageTex by function glReadPixels (core function)

glReadnPixels return pixel data from the frame buffer，speed is fast

---

<div class="post-metadata">

**Author:** ![michalis](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/michalis/32/3_2.png) [@michalis](https://forum.castle-engine.io/u/michalis)\
**Post date:** [February 8, 2023, 5:12pm UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/2 "2023-02-08T17:12:21Z")

</div>

If you want a direct equivalent to glReadPixels, you can just use `Window.Container.SaveScreen` that will give you back TRGBImage. Use one of these overloads:

```delphi
function SaveScreen: TRGBImage; overload;
function SaveScreen(const SaveRect: TRectangle): TRGBImage; overload; virtual;
function SaveScreen(const SaveRect: TFloatRectangle): TRGBImage; overload;

```

Underneath they indeed use just `glReadPixels`, there’s nothing more fancy there:)

The TRGBImage is an image in regular memory, you can just read it’s pixel values, like `MyImage.Pixels[X, Y, 0]`, this returns TCastleColor.

Notes:

1. It’s not efficient to process many pixels one-by-one like this (iterating over array `TRGBImage.Pixels` is much more efficient) but I guess you just want to query a single pixel under mouse, in fact you probably just want to grab this single pixel value to 1x1 image.

2. `Window.Container.SaveScreen` will flush and redraw internally. This could be avoided by using `SaveScreen_NoFlush`, it’s a global routine:

3. You say `glReadPixels` works fast for you, I just have to make a disclaimer:

---

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [February 9, 2023, 5:25am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/3 "2023-02-09T05:25:31Z")

</div>

The example：Door shape is irregular， When mouse move on the door，can trigger event, so I need to determine whether the mouse is on the door picture

In tiledmap 2DGames， this is a very common design needs, such as the mouse moves to char、bookcase、Irregular entrance、switch on the wall etc.

If use Map property and PositionToTile method detect on which exactly tile（not door irregular shape, only standard size tile） we clicked. Can’t accurately determine whether the mouse is on the door .

If use examples/viewport\_and\_scenes/detect\_scene\_hit , It seems not suitable for tiledmap

Is there any other way?

 ![无标题](https://forum.castle-engine.io/uploads/default/original/2X/8/800464104c7be594f569512901392e5cd6eedd0a.jpeg)

---

<div class="post-metadata">

**Author:** ![michalis](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/michalis/32/3_2.png) [@michalis](https://forum.castle-engine.io/u/michalis)\
**Post date:** [February 9, 2023, 4:49pm UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/4 "2023-02-09T16:49:58Z")

</div>

> [@soastao](#):
>
> Is there any other way?

Admittedly not right now. But I can foresee how to implement a “proper” system for the future for TCastleTiledMap (that knows about shapes). We can then do a specialized raycast, looking at 2D geometry of the shapes. So it would detect whole rendered shapes (even when they are larger than map tiles). To avoid detecting shapes where pixels are transparent, It could resolve the point to a specific point in texture atlas, query the alpha of it, and look at lower layer if it is transparent.

Admittedly this is not ready yet and would be some work to add.

If you want to go with solution of grabbing the pixel under the mouse, and you know it performs well for you, go ahead and use it now. And I’m adding another TODO for TCastleTiledMap for future 🙂

---

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [February 14, 2023, 6:33am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/5 "2023-02-14T06:33:49Z")

</div>

I thought about it recently, CGE still can’t do it

I can describe the details as much as possible, please forgive me for poor English

Tiledmap map only needs to judge some objects， such as the mouse moves to char、door、bookcase、Irregular entrance、switch on the wall etc

So I traversed all object element，check if the mouse is above them.

First, check whether the mouse is in the area where the graphic rectangle is located,.This one is very fast, because only the coordinates need to be compared. Can exclude most of the elements.

For the situation of the mouse in the image rectangular area, get this image Texture Data( in ZenGL) , check whether the mouse points X,Y has pixel data

In fact, I only take a little data（only one image），and it is not obtained from the map，get this image texture. If it is transparent, explain that the mouse is not there.

SaveScreen get data from a map after rendering， not one image，This method can’t do it

If have TDrawableImage.SaveScreen ,Maybe can do , It seems impossible

The following is the code in ZenGL, judging whether the mouse is on the picture

 ![无标题](https://forum.castle-engine.io/uploads/default/original/2X/d/d3e3a85cde59496cf14a1d81444763a3dd8a65ee.png)  
 ![99](https://forum.castle-engine.io/uploads/default/original/2X/4/4d73e656229bbb9bfcf2a3654f03dcd94001c25d.png)

---

<div class="post-metadata">

**Author:** ![michalis](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/michalis/32/3_2.png) [@michalis](https://forum.castle-engine.io/u/michalis)\
**Post date:** [February 14, 2023, 8:21am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/6 "2023-02-14T08:21:59Z")

</div>

> [@soastao](#):
>
> The following is the code in ZenGL, judging whether the mouse is on the picture

So the ZenGL is doing something close to what I mentioned in _“But I can foresee how to implement a “proper” system for the future for TCastleTiledMap (that knows about shapes)…”_ in my earlier post. They analyze the original image, since that has the alpha channel.

I have added a GitHub issue [Tiled maps: detect the tile indicated by mouse, taking into account texture tiles with any shape, and with any alpha channel · Issue #448 · castle-engine/castle-engine · GitHub](https://github.com/castle-engine/castle-engine/issues/448) to deal with it in CGE.

If you want to implement it with your custom system, you want to look at original image’s data. `TDrawableImage` has a reference to `Image: TEncodedImage`, you can use it to obtain the original image data. If it’s not using GPU compression, you can just cast it to e.g. `TCastleImage` and query pixels there, like

```delphi
function AlphaOfPixel(const X, Y: Integer): Single;
begin
  Result := (MyDrawableImage.Image as TCastleImage).Colors[X, Y, 0].W;
end;

```

> [@](#):
>
> SaveScreen get data from a map after rendering， not one image，This method can’t do it

The rendered image indeed has everything “flattened”.

To be clear, I did point to `SaveScreen` adding a disclaimer that it’s not perfect, because you asked for analogy to `glReadPixels` at the beginning of this thread 🙂

Unless ZenGL is using `glReadPixels` for more things, maybe they use it even to get original image data? If that is so: For CGE, this is in `TDrawableImage.Image`, there’s no need for `glReadPixels`.

( I can also imagine a solution that would still use `SaveScreen` – you’d need to make a temporary render of everything, setting all textures to a solid color, specific to given tile, but still with original alpha. Though I would recommend to pursue a solution without this involved hack .)

---

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [February 15, 2023, 1:06am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/7 "2023-02-15T01:06:50Z")

</div>

Thanks michalis, AlphaOfPixel can do it,I have tested and passed(I will test the performance in the next step). My current problems seem to be solved, I will continue to explore CGE.

This also shows that a complete game example is the importance of the engine.

Simple examples cannot reveal the functions that need to be used in actual use.

---

<div class="post-metadata">

**Author:** ![michalis](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/michalis/32/3_2.png) [@michalis](https://forum.castle-engine.io/u/michalis)\
**Post date:** [February 15, 2023, 3:05am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/8 "2023-02-15T03:05:17Z")

</div>

I agree about the examples – we need to have a “simple Diablo game” example, this would explore features needed by games of this type. In the past, I tested Tiled support on “examples/tiled/strategy\_game\_demo/”, but that’s too simple use-case, it didn’t require various features that “Diablo” would.

---

<div class="post-metadata">

**Author:** ![valterb](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/valterb/32/26_2.png) [@valterb](https://forum.castle-engine.io/u/valterb)\
**Post date:** [February 17, 2023, 10:27pm UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/9 "2023-02-17T22:27:22Z")

</div>

I apologize for the banality of my observation which I am sure derives from my difficulty in understanding but I’m wondering: why not simply implement in the engine the possibility of creating trivial hotspots by drawing them around a door, a bed, a wall?  
“The example：Door shape is irregular， When mouse move on the door，can trigger event, so I need to determine whether the mouse is on the door picture”.  
Isn’t that the purpose of a hotspot in a game engine?  
I had a lot of fun with different engines such as Wintermute, AGS, Point & Click, Visionaire, Unity, and a common point in all of them was the presence of hotspots, fundamental in point and click games. Couldn’t one envisage including them in the CGE?

---

<div class="post-metadata">

**Author:** ![michalis](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/michalis/32/3_2.png) [@michalis](https://forum.castle-engine.io/u/michalis)\
**Post date:** [February 19, 2023, 2:18am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/10 "2023-02-19T02:18:10Z")

</div>

Indeed you could set up manually “hotspots” like this (this is already possible using CGE to some extent, see below) but it would be additional work, esp. for a big map, to keep such hotspots synchronized with the actual Tiled map.

So the point of related GitHub issue ( [Tiled maps: detect the tile indicated by mouse, taking into account texture tiles with any shape, and with any alpha channel · Issue #448 · castle-engine/castle-engine · GitHub](https://github.com/castle-engine/castle-engine/issues/448) ) is to detect the tile image under mouse – using just the information we already have about the Tiled map (including alpha channel of the tiles).

As for hotspots: You can already detect scene, shape, material etc. under the mouse cursor, see example “examples/viewport\_and\_scenes/detect\_scene\_hit/”. So you can use any TCastleTransform descendant as a “hotspot” this way, just set Visible=false on it. You could set up hotspots:

- using primitives like TCastleBox, TCastleSphere
- using polygons designed in Blender, loaded to TCastleScene
- to draw polygons right within the CGE editor you can use `TCastlePolygon2D` from: [Draw 2D geometry inside CGE editor using Castle Lines 2D components – Castle Game Engine](https://castle-engine.io/wp/2023/01/13/draw-2d-geometry-inside-cge-editor-using-castle-lines-2d-components/) .

---

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [December 5, 2023, 1:44am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/11 "2023-12-05T01:44:18Z")

</div>

Has the problem been resolved now？

Considering that this is a frequently used feature in 2D Tiledmap games, can increase the priority？

---

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [December 5, 2023, 1:46am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/12 "2023-12-05T01:46:06Z")

</div>

In other engines, I can use irregular images like doors as a special object for mouse collision detection. At the current 2D level of cge, I cannot find a reasonable detection approach (tiledmap of a large number of sub images)

If possible, it would be best to have a 2D example

---

<div class="post-metadata">

**Author:** ![eugeneloza](https://forum.castle-engine.io/user_avatar/forum.castle-engine.io/eugeneloza/32/12_2.png) [@eugeneloza](https://forum.castle-engine.io/u/eugeneloza)\
**Post date:** [December 5, 2023, 8:05am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/13 "2023-12-05T08:05:43Z")

</div>

Well, doing that for TDrawableImage is pretty trivial:

[pixel-detection.zip](https://forum.castle-engine.io/uploads/short-url/ddyBNDYx0ouFxUO0g2z2uZdc5Xp.zip) (19.4 KB)

The core of the detection is in `Motion` method:

```pascal
  X := (Event.Position.X - DestRect.Left) / DestRect.Width;
  Y := (Event.Position.Y - DestRect.Bottom) / DestRect.Height;
  if (X >= 0) and (Y >= 0) and (X < 1) and (Y < 1) then
  begin
    ImgX := Round(SrcRect.Width * X - SrcRect.Left);
    ImgY := Round(SrcRect.Width * Y - SrcRect.Left);
    if (ImgX >= 0) and (ImgY >= 0) and (ImgX < Image.Width) and (ImgY < Image.Height) then
    begin
      if (Image.Image as TRGBAlphaImage).PixelPtr(ImgX, ImgY)^.W > 10 then
        LabelFps.Caption := 'OVER PIXEL'
      else
        LabelFps.Caption := 'OUT OF PIXEL'
    end else
      LabelFps.Caption := 'Mouse is within the screen rectangle but outside of the image rectangle, the results are ambiguous. However, even here we can try to play with Image.RepeatS and Image.RepeatT';
  end else
    LabelFps.Caption := 'OUT OF IMAGE'; 

```

The question is how “scalable” this algorithm is 🙂 E.g. a `TDrawableImage` not necessarily contains a CPU `Image`. Also if used for something “bigger” like pixel-perfect collision detection (next natural step of this feature) it quickly results in poor performance as requires “random reading of memory”. I’ve seen an idea for solution through custom “mipmaps”-like downscales of the image to speedup collision detection, but in turn it eats up memory 🙂

So, I guess the question is “what exactly this feature should do in all possible usecases”.

---

<div class="post-metadata">

**Author:** ![soastao](https://forum.castle-engine.io/letter_avatar_proxy/v4/letter/s/67e7ee/32.png) [@soastao](https://forum.castle-engine.io/u/soastao)\
**Post date:** [December 5, 2023, 8:27am UTC](https://forum.castle-engine.io/t/how-to-judge-the-mouse-on-tdrawableimage/739/14 "2023-12-05T08:27:14Z")

</div>

![99_副本](https://forum.castle-engine.io/uploads/default/original/2X/6/661b90e87168fe0329806e11fb660e01699d6af1.jpeg)

For example, for non rectangular doors, when the mouse moves over the image of the door, the mouse becomes a small hand. Clicking on it can detect a collision with the door, triggering door related events

Similar to a treasure chest, I don’t want the mouse to become a small hand when there is still a distance from the treasure chest. The effect I want is that the mouse only changes when it is on the picture of the treasure chest. Similar to the switches on the wall
