Using TCastleClientServer for binary data?

1

I am trying to base my terrain server on the CGE client/server network example. I have the indy package installed. I want to the client to be able to make requests that the server then fills. So the request to the server could be a string, but I want binary results, like an array of 65536 singles or whatever size the results are. It looks like the client code assumes all messages from the server are strings? How could this be adapted to allow for binary messages?

I did try adding a writebuffer procedure to the CastleClientServer but is unclear how I would make the client receive this data. Maybe I should be using the writestream/readstream from IdIOHandler?

2

I guess you would update TCastleTCPClientThread to not just do readln over and over? Or maybe it can do that, but then a specific LMessage switches it to listen for a binary data block? Or maybe a header that says the type of message?

3

I figured out how to do what I want I believe. I can send a byte or whatever that identifies the kind of message. Then I can read a string, or read a block of binary data.

   while not Terminated do
    begin
      try
        MsgType := FClient.IOHandler.ReadByte;

        case msgtype of 1 :
        begin
          FMessageList.Add('ping');
          Queue(FOnMessageReceived);
        end;
        2 :
         begin
           LMessage := FClient.IOHandler.ReadLn; //Result is empty when timeout reached.
           FMessageList.Add(LMessage);
           Queue(FOnMessageReceived);
         end;

        end;
      except
        //Exception occurs when connection was disconnected (gracefully).
        Terminate;
        if Assigned(FOnDisconnected) then
          Synchronize(FOnDisconnected); //Must be synchronize, not queue, because queued messages get lost.
      end;

I had to make ReadByte not throw exception if timeout. Then it waits for the byte. So I think this will work.

1 Like
4

I see you found a solution. I was wondering if that works with bigger chunks of data, but you can send 4*64k bytes easily so I think itā€™s a nice thing.

However, I was considering sending it as a ZIP. Letā€™s say client requests ā€˜?TerrainDataā€™ and server responds with a [virtual] path to the file. Then, by using TCastleZip thingy, you could process the resource with regular Castle utils.

It would cache the resources and speed up client requests - especially when more than 1 client needs them, - but Iā€™m not sure if that would play well for highly dynamic resources. When new ZIP version is created (because of data change) then you would either return ā€œWait a momentā€ or use alternating files, because client may send requests any time.

Iā€™m just thinking out a loud. Iā€™m probably just over-complicating it :upside_down_face:

5

This isnā€™t necessarily going to be multiuser. The goal is to offload processing and memory to the terrain server. It will probably handle my water flow too. I donā€™t know yet if it will be fast enough with sockets on the same machine. If not, I will add some sort of memory sharing, but that might be platform specific. Each terrain tile is currently 40,000 bytes a max resolution. I donā€™t think you were around when I was posting a lotā€¦ this is my worldā€¦

live-world_screen_127
live-world_screen_1271920Ɨ1200 146 KB

1 Like
6

cs
cs1709Ɨ1237 354 KB

1 Like
7

I see you solved it by implementing the missing functionality using Indy. Indeed, that is what I would recommend. The IOHandler in Indy should provide all necessary features one may need.

Our CastleClientServer in a ā€œsimple wrapperā€ around Indy deliberately, allows only passing Strings (which we assume have typical encoding, so UTF-8 with FPC and UTF-16 with Delphi), just because that seemed enough for its needs back when it was implemented :slight_smile: A PR to extend CastleClientServer, to send/receive any binary buffer, would be welcome, itā€™s of course a sensible feature to add (and you donā€™t want to ā€œoveruseā€ strings for arbitrary binary data due to Unicode encoding assumptions and FPC/Delphi differences).

If someone wants to tackle this:

As API, I propose to add overloads with Buffer suffix to send binary blobs, and pass them around them as const ABuffer; const BufferSize: Int64 (so itā€™s just like TStream.WriteBuffer API). So TCastleTCPServer gets new methods:

// existing
procedure SendToClient(const AMessage: String; AClient: TClientConnection); virtual;
procedure SendToAll(const AMessage: String); virtual;
// new methods
procedure SendToClient(const ABuffer; const BufferSize: Int64; AClient: TClientConnection); virtual;
procedure SendToAll(const ABuffer; const BufferSize: Int64); virtual;

TCastleTCPClient gets new:

// existing
procedure Send(const AMessage: String); virtual;
// new
procedure SendBuffer(const ABuffer; const BufferSize: Int64); virtual;

And they are received using new callbacks:

// existing
TMessageReceivedEvent = procedure(const AMessage: String) of object;
TClientMessageReceivedEvent = procedure(const AMessage: String; AClientConnection: TClientConnection) of object;
// existing
TMessageBufferReceivedEvent = procedure(const ABuffer; const BufferSize: Int64) of object;
TClientMessageBufferReceivedEvent = procedure(const ABuffer; const BufferSize: Int64; AClientConnection: TClientConnection) of object;

( I wondered about a single callback to receive both types of message, maybe like record case MessageType: TMessageType of 0: Data: Pointer; DataSize: Int64; 1: Message: String ā€¦ ) but it seems safer to expose separate callbacks for string receiving and binary blobs receiving. Opinions to the contrary are welcome :slight_smile: )

The examples should be extended to report it to, you can test passing e.g. TVector3 around.

Warning that it may take some effort to do this ā€“ CastleClientServer has actually 2 implementations (Indy, or using Android service) and we want them both to be functional (and able to call each other).

2 Likes
8

That is basically what I have done, adding parallel methods for sending the buffer. I am unconcerned about android for my project. If I get around to caring about android, I can implement the android code. I use a msg_type byte that I send first to identify if the server will return string or binary.

1 Like
9

I liked the clean look of your untyped buffer. But indyā€™s SendBuffer wants a dynamic array TIdBytes. If i send it a raw buffer that is just a block of memory, it crashes because it is trying to get the length from the dynamic array. How would you implement your

procedure SendBuffer(const ABuffer; const BufferSize: Int64); virtual;
10

I grepped Indy code for const Buffer, and I was lucky ā€“ found example of doing the conversion you ask for:

function TIdBaseStream.Write(const Buffer; Count: Longint): Longint;
begin
  if Count > 0 then begin
    Result := IdWrite(RawToBytes(Buffer, Count), 0, Count);
  end else begin
    Result := 0;
  end;
end;

Indeed, thereā€™s utility

function RawToBytes(const AValue; const ASize: Integer): TIdBytes;

defined in IdGlobal unit exactly for this purpose. Itā€™s rather simple, just allocates TIdBytes (dynamic array) and copies the contents:

function RawToBytes(const AValue; const ASize: Integer): TIdBytes;
{$IFDEF USE_INLINE}inline;{$ENDIF}
begin
  SetLength(Result, ASize);
  if ASize > 0 then begin
    Move(AValue, Result[0], ASize);
  end;
end;
1 Like