Support of international chars in CGE

I am trying to do some experiments with handling Cyrillic (Russian) chars in CGE.
In the example:
\examples\2d_standard_ui\edit_test\edit_test.exe
If I enter Russian chars into TCastleEdit they are displayed as ?
While I may enter latin chars without any problems.
Now I added TCastleControlBase on TForm and OnPress I try to see that are returned by
procedure CastleControlBase1Press(Sender: TObject;
const Event: TInputPressRelease);

For example, when I press Russian ф (f) char which is located on the same button as latin a then I see in Debug - Local Variables:

EVENT={EVENTTYPE = ITKEY, KEY = KEYA, KEYSTRING = ‘ф’, MODIFIERSDOWN = [], KEYREPEATED = false, MOUSEBUTTON = MBLEFT, FINGERINDEX = 0, POSITION = {DATA = {0, 0}}, MOUSEWHEELSCROLL = 0, MOUSEWHEELVERTICAL = false}

But they I try to print KEYSTRING on put it into string:

procedure TForm1.CastleControlBase1Press(Sender: TObject;
		const Event: TInputPressRelease);
var
   st: string;
begin
   case Event.EventType of
   	 itKey: begin
   		 st:= 'key: '+IntToStr(integer(Event.Key))+', keychar: '+Event.KeyCharacter+', key str: '+Event.KeyString;

So I see that in st it’s recorded just as:
key: 65, keychar:

Could you please comment it? Why it’s not possible to enter Russian chars into TCastleEdit
and why I cannot catch Event.KeyString into st? (It works fine if I switch keyboard into EN layout and press Latin/English chars.

I am new to Lazarus while I have 20+ years of Delphi development exprience.
In Lazarus I created a simple application. I put TEdit on TForm.
Then I do:

procedure TForm1.Edit1KeyPress(Sender: TObject; var Key: char);
begin
  ShowMessage(Key);
end;  

Then I press ф button. I am on Windows.
It’s strange that in Local Variables i’ts displayed as ?
Then in ShowMessage it’s displayed as ?
But on TEdit it’s printed well as ф
Could someone comment on it? Maybe I should simply enable something in Lazarus? I know that Lazarus is using UTF-8 strings.

screenshot000208

This problem is connected to those characters unavailable in default font (which contains only ASCII characters). You need to manually specify the font to render the string or specify default font in configuration.

E.g. create a data/CastleSettings.xml file like this:

<?xml version="1.0" encoding="utf-8"?>
<castle_settings>
  <default_font>
    <regular
      url="DejaVuSans.ttf"
      size="25"
      sizes_at_load="20 40"
      anti_aliased="true"
      sample_text="йцукенгшщзхъфывапролджэячсмитьбюёЙЦУКЕНГШЩЗХЪФЫВАПРОЛДЖЭЯЧСМИТЬБЮЁ"
    />
  </default_font>
</castle_settings>

Add DejaVuSans.ttf font (or any other, adjusting the variable above as necessary) to the data folder too.

And then in the game load this settings file:

Window.Container.LoadSettings('castle-data:/CastleSettings.xml');

Note, that on Windows you’ll need to have corresponding 32 or 64 bit FreeType library available on PATH or in the game folder. Pick the necessary DLL from tools\build-tool\data\external_libraries\

I’m not sure about that. It should work fine. Note absence of key str: text in st output. It looks like KeyCharacter is some sort of a control character, which cuts the next content of the line. Also note that Windows console output works in Win-1251 only (for Russian Windows) and it often gets confused when getting UTF-8 characters.

This is a different problem. Note, that Key is passed as char - char is a 8-bit symbol, i.e. an ASCII symbol, which by default doesn’t support UTF-8 characters, such as “ф” - so, don’t expect anything reasonable in Key here. I don’t remember if there is a workaround - there most certainly should be, as I remember to have been successfully using Cyrillic characters in TEdit.

Some additional notes:

  • Both Castle Game Engine (whether using TCastleWindowBase or TCastleContolBase) and Lazarus (LCL) use UTF-8 in Strings. We should handle any (including multi-byte) characters in such strings.

  • Input of international characters should work both with TCastleWindowBase and TCastleContolBase, although it goes through different code paths.

    In case TCastleContolBase it should work because it works in LCL (I recall I fixed it in ~2018 to support French characters).

    In case of TCastleWindowBase we have our own code that communicates with WinAPI (on Windows, on Unix we use GTK etc.). Eugene did work on it to support international characters long time ago, and a month ago I improved it to handle (hopefully) even Chinese input, https://github.com/castle-engine/castle-engine/issues/159 .

  • The display of international characters is an independent matter. To display international characters in CGE (e.g. using TCastleEdit or TCastleLabel) you need to have a font loaded that supports the necessary glyphs. Eugene above already nicely shown how to achieve this, more info in manual https://castle-engine.io/manual_text.php and we also have an example “examples/fonts/test_local_characters” in CGE sources.

    Other ways of displaying international characters (using LCL or console) are outside of my knowledge, bu they should work :slight_smile:

Let us know whether it helps :slight_smile: As you see the situation is a bit complicated – “input” of international characters is one thing, “display” of international characters is another thing. And “input” may be done through various ways (depending on whether you use TCastleWindowBase or TCastleControlBase, and platform). It all should work – if it doesn’t please report :slight_smile:

Eugene and Michalis,

I tried to follow to Eugene’s instructions and edit_test works fine: it displays Russian fonts.
But then I installed French keyboard layout on my Windows.
BTW Deja vu in French is Déjà vu
Thus i switched to French keyboard layout and typed Déjà vu
it could be seen as: D?j? vu
I switched to Russian keyboard layout and I was able to successfully type and see Russian chars.
Then I closed the edit_test and in CastleSettings.xml I removed the line
sample_text=
I started edit_test. I see that I cannot type Russian chars now.

The idea is:
For example, I have a game and a player needs to type his name into TCasteEdit field. Thus he cannot do it.Because the language is hardcoded into CastleSettings.xml
But in Lazarus LCL if I put TEdit - I switch keyboard to Russian - I may type in Russian. Then I switch keyboard to French layout - I may type in French.
See the screenshot:

screenshot000213

So on the same TEdit I see Russian and French chars. This is because TEdit is using UTF-8.

Furthermore: if I have sample_text= with Russian chars and I may type Russian chars then I type Russian text in Notepad++ or Word and I try to copy/paste the text into edit_test. I see ??? instead of the right text. Thus copy/paste does not work.

I also found many articles about support of Russian in LCL and console (yes, it’s using different encoding). I will try to find out how to work with it.
For me it’s not the urgent question. I just wanted to know how international chars are supported in CGE and Lazarus. I will try to study it more in details in upcoming weeks.
Eugene and Michalis - thanks for your help!

From your description, it does seem like things actually work as expected – the input of international characters works, however their display is limited to the characters specified at sample_text= when loading.

Indeed this means that players cannot e.g. type their name using any language they like. You need to

  1. provide a font file that supports all languages you want, and
  2. you need to specify in sample_text= all the local characters you need.

This is indeed the limitation of the current font support. I will definitely remove the 2nd problem someday, and enable loading either all characters from the font, or/and enable loading them dynamically as needed. Thus we would be able to display all the glyphs that are present in TTF/OTF file, without the need for listing them in sample_text= or anything like that.

Furthermore: if I have sample_text= with Russian chars and I may type Russian chars then I type Russian text in Notepad++ or Word and I try to copy/paste the text into edit_test. I see ??? instead of the right text. Thus copy/paste does not work.

Oh, and this is just a separate bug. The copy-paste should work. If your sample_text= contained Russian characters, and you paste Russian characters, it should work. We are most likely missing proper Windows<->UTF-8 conversions in TWinApiClipboard in src/window/windows/castlewindow_winapi.inc . Recorded in https://trello.com/c/pgX67VRU/48-fix-twinapiclipboard-to-enable-pasting-copying-strings-with-international-characters-eg-convert-russian-characters-between-windo .