Zip instead data/

Hello.

As I understood documentation, a standard way of distributed application to access its media resources (models, textures, etc) is to place them all in a plain form under a directory tree “data”.

Can I have all my media resources be contained in a single (compressed) file at runtime instead, so the engine classes loads its data directly from a single zip file (by relative pathname)?

Thanks.

  1. A solution that works now (but is not efficient, see below): you can distribute your data as zip, and at the beginning of the 1st application execution unpack it (somewhere to the user-specific directory, i.e. ApplicationConfig) and then set ApplicationDataOverride global variable. This way you can use castle-data:/ protocol (or ApplicationData function, see https://castle-engine.io/manual_data_directory.php ) to refer to files inside your zip. To unpack the zip, FPC offers standard units.

    But it’s not a solution I would recommend – you introduce extra loading time for the 1st application run, and user has game data (which may be large) duplicated on disk. And your code gets more complicated – you need to unpack the zip, store somewhere your application version (to unpack the zip again, if application version changes)… So it seems like extra work with little gain :slight_smile:

  2. Another solution that works: You are not forced to use the castle-data:/ approach documented on https://castle-engine.io/manual_data_directory.php. You can write your own zip loader, and provide the data to all CGE loading routines as TStream instances.

  3. Future 1 : One way this could be possible in the future is that I want to add registration of custom URLs to CastleDownload unit. So you could implement your own zip loader, register it to load files from zip e.g. when you load my-zip-data:/folder_inside_zip/my_image.png, and then you would set ApplicationDataOverride := 'my-zip-data:/' .

  4. Future 2 (more distant): I want to allow you to just use URL to automatically refer to files inside zip. Once this is implemented, you could just load files like this: castle-data:/my-package.zip:/folder_inside_zip/my_image.png . The idea is that we will then automatically load file called folder_inside_zip/my-image.png inside castle-data:/my-package.zip. So all the internal loading would be performed using CGE routines.

For now (until at least one of “future” stuff in AD 3 or AD 4 is implemented), I would advise to go the “most standard route”, that is: do not put your data in a zip. This way the standard CGE things (including castle-data:/ URL) will work out-of-the-box, and you can use the build tool ( https://github.com/castle-engine/castle-engine/wiki/Build-Tool ) to easily package your games and it will work for all the platforms. When the games are packaged, they are compressed anyway (when packaging for Windows, we actually create a zip file with everything), so size is not an issue.

1 Like

I might add a tiny bit here…

  1. At the present moment Castle Game Engine can load some assets (at least XML-based files) from separate GZ archives. Which may be a better option (see below).

Using a single large ZIP archive for the whole folder has a lot of disadvantages:

  • Large archive requires either large RAM to store unzipped content (and game content can be gigabytes of data) OR it will increase loading times. As far as I remember ZIP can provide access to compressed files faster than RAR, but anyway, there’s a lag, not only caused by ZIP algorithms itself, OR will require unzipping the data into some temporary folder, which will increase HDD usage and loading times (remember those gigabytes of data again).
  • Zipping game data normally doesn’t provide for serious data reduction, while significantly increases loading times. I.e. normally game assets are already provided in compressed format, such as PNG and OGG. Zipping those has either very poor (just compressed a random PNG file and got 0.6% HDD space saved versus uncompressed file size), or even a negative effect (yes, zipped PNG or OGG/MP3 file may be even larger than the original). Only those spacy text-based formats like XML are affected.
  • Many, many games do today store data in an extensive folder structure - however, compressing single “meaningful blocks” of game data may be reasonable.

You might also be interested in trying Lazarus Resources, see http://wiki.freepascal.org/Lazarus_Resources (they work with “Bare” FPC too as far as I know :)). As far as I know those can also be compressed, not sure of the algorithm thou. This is a nice way for a “small program/game” that would keep everything small and in a single executable/binary file, which can in turn be compressed by binary shrinkers, such as UPX (also see notes at http://wiki.freepascal.org/Size_Matters#UPX ).
I think Resources should automatically work with Castle Game Engine (Castle Game Engine uses those :)), but @michalis will be more likely to answer correctly how far can resources go :slight_smile:

Thanks for an answer, and I’m impressed with your engine.
You’ve done a great job !

Definitely I’ll go with #2. The question was if I’m able to load data from custom streams.

Probably I need to make some clarification: an intent is not about a size, but rather some kind of protection (not very strong, but still) against end user to meddle with assets and possible cheats.
There are also additional benefits to have a single file instead of multiple ones.

Such approach exists in Irrlicht graphics engine:

// C++ pseudocode
IFileSystemInterface* fs = engine->getFileSystem();
fs->addDirectory( "./data" );
fs->addFileArchive( "./myassets.zip", optional_password );

// load assets either from directory or archive, whichever found first
myModel->load("relative/path/collada.dae");
myTexture->load("relative/path/file.png");
myData->load("relative/path/data.xml");

You already have some kind of file system abstraction (as data-URLs), so I thought it would be nice to have similar feature, even in form of LoadFromStream examples.

Castle Game Engine has a simple, but relatively strong blowfish (Blowfish (cipher) - Wikipedia ) protection for XML files (i.e. the ones where the game variables, such as health and score should be stored) for such purpose: https://castle-engine.io/apidoc-unstable/html/CastleXMLUtils.html#URLReadXML . Those can also be gzipped.

  • FPC resources are cross-platform, work also without LCL, and can be used to “embed” any file in the exe. They don’t provide compression out-of-the-box as far as I know, so indeed you would need to just attach a compressed file (like zip) to the EXE (or compress whole EXE with resources). And watch out that binary files may not compress much, as Eugene notes.

  • If you’re more interested in protecting the data, consider encrypting (or at least signing) some critical files. As Eugene notes, we have ready routines to load XML with BlowFish encryption, and you can hardcode the decryption key in the source code. I use this technique to protect some “game balance” XML files in Cat-astrophe Games distributed applications on Android/iOS (we don’t want users to hack them, as then global “leaderboards” would be easy to hack). The BlowFish is actually implemented as part of standard FPC units, there are other algorithms too, BlowFish was just simplest to use :slight_smile:

If you want to protect the whole data, then putting it in a zip and encrypting (or signing), with hardcoded key inside data, may be reasonable. But be careful about performance, as Eugene also notes – depending on the approach, you will use more memory and/or disk space and/or the loading time will suffer. I would advice making a test (e.g. with Timer from CastleTimeUtils) and compare the loading time with/without encryption, to be sure you’re not losing too much time. If it’s too much, then consider only encrypting/signing some critical files.

2 Likes

Thanks guys.

I’m in a process of consideration of new game engine (all in one) to rewrite and improve already working (in technical part) application written on Irrlicht/Newton/SDL/OpenAl (C++) and partly FPC (yes, a complicated one). My targets are desktops, and my game is not Fallout :slight_smile: so there shouldn’t be any RAM/load time issues, even if it will be Pascal instead of C++.

My main goal is to prevent (easy way for) user to change model geometry and textures, as it will give unfair advantages. Simple hiscore text encryption will not work.

So what I’m really interested is how/if it is possible to use LoadFromStream :slight_smile: now or in some near future.

OK, seems reasonable :slight_smile:

I went ahead and added the possibility to register custom URL handlers (since it seems a nice feature in general, for other purposes too), in this commit https://github.com/castle-engine/castle-engine/commit/f720cf28efa833bae00b5bcf347a79032e525b46 (and later improved in https://github.com/castle-engine/castle-engine/commit/d79573ee127407f5530b53afaf681e10f8f0b7b4 , simplified the callback).

I created an example application that does exactly what you asked for :), keeps data in a zip file, and engine seamlessly reads it now. See https://github.com/castle-engine/castle-engine/tree/master/examples/custom_url_handler . You can now read data from ZIP by accessing URLs like my-packed-data:/xxxx. Moreover example sets ApplicationDataOverride := 'my-packed-data:/' and in effect you can use regular castle-data:/xxx protocol (like in https://castle-engine.io/manual_data_directory.php ) to load files from ZIP too.

2 Likes

Thank you, will check.

Took a look. Thank you.

Although provided example uses temp directory to extract files to disk, it looks like I will be able to use RegisterUrlProtocol and my own ReadUrl which will return my own T(Memory)Stream to which I could feed any arbitrary data.

Am I right?

Yes, exactly.

The implementation on https://github.com/castle-engine/castle-engine/blob/master/examples/custom_url_handler/gameinitialize.pas#L53 is deliberately the simplest possible implementation, it seemed to me that it’s simplest to use FileUnzip from Unzip51g unit and unpack to a temporary directory. I did not find in FPC units a simple function that would return the file from ZIP as a stream, but I honestly did not search too long.

A proper implementation should avoid using a temporary file/directory, and should just return any TStream instance that returns the data.

Also a pull request that improves this example (but, preferably, keeps the code simple and only using standard FPC units) would be welcome too :slight_smile:

1 Like

I’m going to work on it, as soon as I’ll get familiar with another parts of engine, since I’m personally interested in this feature.

1 Like