Re: [Boost-users] [Serialization] Speedding up client-server communication

Subject: Re: [Boost-users] [Serialization] Speedding up client-server communication
From: Brian Wood (woodbrian77_at_[hidden])
Date: 2010-02-04 21:25:23

Ruediger Berlich writes:

> Hi there,
>
> I am in the process of speeding up communication between a server and its
> clients. Communication involves serialized class data. Messages can be as
> large as 100 kilobytes.

> I have done some measurements which have shown that, in a cluster with
> Gigabit networking, most overhead of the parallelisation seems to come
from
> the Broker infrastructure and the process of (de-)serialization. Network
> latency and/or bandwidth seems to play only a minor role in this
> environment.

> Hence, apart from optimizing my broker, I'm looking for ways to optimize
the
> serialization process, as used in my application. As messages are
discarded
> as soon as they reach the recipient, versions of serialized data do not
play
> an important role.

I don't find any mention of "message" in the 1.42 Boost
Serialization documentation.  Are you using MPI?

The obvious thing you've not mentioned is compression.  I use
bzip2 to compress and decompress data -

http://webEbenezer.net/misc/SendCompressedBuffer.hh
http://webEbenezer.net/misc/ReceiveCompressedBuffer.hh

Besides using bzip2, I use variable-length integers to encode
the size of the compressed data.  It works to use a fixed-size
integer as well, but frequently you can shave a couple of
bytes off the total by using a variable-length integer.

Brian Wood
http://webEbenezer.net
[phone]

"The kingdom of heaven is like a treasure hidden
in the field, which a man found and hid again;
and from joy over it he goes and sells all that
he has and buys that field."