Re: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string

Subject: Re: [Boost-users] Fastest way of serializing a huge vector of ints into a human readable string
From: Jens Weller (JensWeller_at_[hidden])
Date: 2009-03-01 08:02:26

Hi Roland,

I'm not sure if karma speeds up the process at all.
Also it runs afaik on an iterator interface, so you'd have the option to control what happens underneath.

> Jens Weller wrote:
> > Hm, maybe boost::spirit::karma could do you a favor here, but I think it
> will not gain much perfomance, if at all.
> > 
> > Maybe you could think about a functor which does it on your own.
> > Do the numbers repeat often, then you could also build a table
> <int,string> to only convert once to string.
> > 
> > Also parellizing with boost Thread would be an option.
> 
> Hi Jens,
> 
> hmm. Documentation about boost::spirit::karma seems to be well hidden. 
> Even a google search like
> 
> karma site:www.boost.org/doc/libs/1_38_0/libs/spirit
> 
> did not reveal any hits. Can you give me a link?

Karma seems not to be documented yet well enough in the docs.
But there are some examples which are shipped with boost:
boost/libs/spirit/examples/karma/
f.e.
http://www.boost.org/doc/libs/1_38_0/libs/spirit/example/karma/quick_start1.cpp

> Writing an own functor might be an option, I guess. The numbers do not 
> repeat often, though. Quite the opposite: It is for certain that every 
> number appears just once. And it could be anything within a 64bit range.

hm, I ask my self, how much time the conversiont number->string takes.
Once you remove all the copying of tempstrings etc. you probably already end up at a pretty fast way.

> Parallelization certainly could be helpful in some scenarios, but I a 
> have a multi-threaded application already. So splitting up the 
> serialization might increase speed for the serialization itself, while 
> slowing down the total process due to increased context switches. I 
> therefore want to have the single threaded operation as efficient as 
> possible.

Well, ofcourse you'd have to do the optimisation first. Just using more processors doesn't make it faster in the processing.

regards

Jens Weller

> > -------- Original-Nachricht --------
> >> Datum: Fri, 27 Feb 2009 18:46:07 +0100
> >> Von: Roland Bock <rbock_at_[hidden]>
> >> An: boost-users_at_[hidden]
> >> Betreff: [Boost-users] Fastest way of serializing a huge vector of ints
> into a human readable string
> > 
> >> Hi,
> >>
> >> I have a program which produces a vector of integers (several million 
> >> entries). I need to write that into a human-readable string of space 
> >> separated numbers.
> >>
> >> I wonder, what would be the fastest way?
> >>
> >> My first attempt was
> >>
> >> stringstream resultStream;
> >> copy(integers.begin(), integers.end(), 
> >> ostream_iterator<int>(resultStream, " "));
> >> string result = resultStream.str();
> >>
> >>
> >> But that requires copying the string.
> >>
> >>
> >> Thus I was wondering if I could use boost to write to the target string
> >> directly?
> >>
> >> boost::iostreams allow me to do the following:
> >>
> >>
> >> string result;
> >> boost::iostreams::filtering_ostream 
> >> out(boost::iostreams::back_inserter(result));
> >> copy(integers.begin(), integers.end(), ostream_iterator<uint64>(out, "
> >> "));
> >>
> >> This is faster by almost a factor of 2.
> >>
> >> Any ideas how to increase speed even more?
> >>
> >>
> >> Thanks in advance,
> >>
> >> Roland
> >> _______________________________________________
> >> Boost-users mailing list
> >> Boost-users_at_[hidden]
> >> http://lists.boost.org/mailman/listinfo.cgi/boost-users
> > 
> _______________________________________________
> Boost-users mailing list
> Boost-users_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-users

-- 
Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger01