Re: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string

Subject: Re: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string
From: Roland Bock (rbock_at_[hidden])
Date: 2009-03-01 05:19:33

Jens Weller wrote:
> Hm, maybe boost::spirit::karma could do you a favor here, but I think it will not gain much perfomance, if at all.
> 
> Maybe you could think about a functor which does it on your own.
> Do the numbers repeat often, then you could also build a table <int,string> to only convert once to string.
> 
> Also parellizing with boost Thread would be an option.

Hi Jens,

hmm. Documentation about boost::spirit::karma seems to be well hidden. 
Even a google search like

karma site:www.boost.org/doc/libs/1_38_0/libs/spirit

did not reveal any hits. Can you give me a link?

Writing an own functor might be an option, I guess. The numbers do not 
repeat often, though. Quite the opposite: It is for certain that every 
number appears just once. And it could be anything within a 64bit range.

Parallelization certainly could be helpful in some scenarios, but I a 
have a multi-threaded application already. So splitting up the 
serialization might increase speed for the serialization itself, while 
slowing down the total process due to increased context switches. I 
therefore want to have the single threaded operation as efficient as 
possible.

Regards,

Roland

> 
> mfg.
> 
> Jens Weller
> 
> -------- Original-Nachricht --------
>> Datum: Fri, 27 Feb 2009 18:46:07 +0100
>> Von: Roland Bock <rbock_at_[hidden]>
>> An: boost-users_at_[hidden]
>> Betreff: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string
> 
>> Hi,
>>
>> I have a program which produces a vector of integers (several million 
>> entries). I need to write that into a human-readable string of space 
>> separated numbers.
>>
>> I wonder, what would be the fastest way?
>>
>> My first attempt was
>>
>> stringstream resultStream;
>> copy(integers.begin(), integers.end(), 
>> ostream_iterator<int>(resultStream, " "));
>> string result = resultStream.str();
>>
>>
>> But that requires copying the string.
>>
>>
>> Thus I was wondering if I could use boost to write to the target string 
>> directly?
>>
>> boost::iostreams allow me to do the following:
>>
>>
>> string result;
>> boost::iostreams::filtering_ostream 
>> out(boost::iostreams::back_inserter(result));
>> copy(integers.begin(), integers.end(), ostream_iterator<uint64>(out, "
>> "));
>>
>> This is faster by almost a factor of 2.
>>
>> Any ideas how to increase speed even more?
>>
>>
>> Thanks in advance,
>>
>> Roland
>> _______________________________________________
>> Boost-users mailing list
>> Boost-users_at_[hidden]
>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>