AnsiString vs Utf8String properly declared, Delphi: we no longer force AnsiString to UTF-8

News
1
Running Castle Game Engine application on Android using Delphi

( This announcement is a bit non-exciting small improvement around the engine codebase. For something more exciting, that I was working on through September, see the screenshot attached here :), and stay tuned for more news soon. )

Previously, our engine did SetMultiByteConversionCodePage(CP_UTF8) forcing all AnsiStrings (with both FPC and Delphi) to have UTF-8 encoding. This is also consistent with what Lazarus LCL was (and still is) doing.

We have extended our possibilities now, and changed what happens with Delphi by default (with FPC, nothing changes by default):

  1. Throughout the engine, we use now always Utf8String when we want to say “8-bit string with UTF-8 encoding”. Aside from the practical gains (see below), this naturally just looks better: we explicitly express our intention.

    We use AnsiString in code only when (very seldom) we really mean “8-bit string with whatever encoding happens to be the platform default”.

    We have tested that with Delphi (unfortunately, not so with FPC) the proper implicit conversions are always done when you assign between AnsiString (even if they have non-UTF-8 content) and Utf8String. See lots of new automatic tests in

    Note: 90% code of the engine still uses just String (not Utf8String or AnsiString). That is deliberate and simple: we want to just use default string type, which means UnicodeString (String=UnicodeString with Delphi, or FPC Unicode RTL) or AnsiString (FPC, without FPC Unicode RTL). Nothing changes in this regard.

  2. We added defines to control whether the engine does SetMultiByteConversionCodePage(CP_UTF8), forcing all AnsiString to contain UTF-8. And by default, it is no longer done when compiled with Delphi.

    • CASTLE_ANSISTRING_FORCE_UTF8: This is the previous behavior. And it is still default with FPC.
    • CASTLE_ANSISTRING_UNCHANGED: This is the new possible behavior. It is (since now) the default with Delphi.

    Note that CASTLE_ANSISTRING_UNCHANGED is not supported with FPC, due to FPC (at least 3.2.2) not doing AnsiString vs Utf8String conversions automatically (but you can force it by defining also I_UNDERSTAND_THAT_NON_ASCII_CHARACTERS_ARE_BROKEN if you really need it, and understand the consequences).

    We changed Delphi default to CASTLE_ANSISTRING_UNCHANGED, because it works flawlessly with Delphi in our tests, and it seems safer — we don’t mess with global settings if we don’t need to. This is nice if you have an existing codebase that assumes AnsiString don’t hold UTF-8 encoded data. This is also important when our code is run as part of Delphi IDE, when you install our Delphi packages — this way we don’t mess assumptions that may be made by Delphi IDE or other 3rd-party Delphi packages you have.

  3. Along the way, we have also fixed and simplified some details around how Unicode characters are handled.

    • Reading JSON files (with our FpJson) with non-ASCII text now works on Delphi reliably. Our FpJson unit (and friend) use String or Utf8String, and don’t mess with encoding.

    • FPHashCompatibility unit, complicated compatibility hack for Delphi, is gone.

    • A few cases when we used UTF-8 with Delphi are fixed. E.g. with Delphi, one cannot do S := S + C when S is Utf8String and C is AnsiChar and C may be only a part of multi-byte character: Delphi will pad C to be a valid Unicode character. See TTestCompiler.TestAddingIncompleteUtf8 test in tests/code/testcases/testcompiler.pas for details.

    • CastleReadLink on Delphi/Linux fixed.

    • GetTempFileNamePrefix on Delphi/Windows fixed.

    • We cleaned some unused Vampyre Imaging files (like dglOpenGL copy used by Vampyre demos) to not confuse developers what is used by our engine.

For more details about how CASTLE_ANSISTRING_FORCE_UTF8 and CASTLE_ANSISTRING_UNCHANGED work see

If you like our work, please support us on Patreon!