boost::iostreams zlib compression
Ok I think this sucks. :-) Here is my code to implement zlib compression/decompression via boost. However since is uses streams I cannot simply pass a buffer. This code is intended for compress socket buffers, but is pathetically slow. HAs anyone done this? What can I improve? The copy seems to be the bottle neck. thx jc Here is my code: #include <boost/iostreams/copy.hpp> #include <boost/iostreams/filter/zlib.hpp> #include <boost/iostreams/device/file.hpp> #include <boost/iostreams/stream.hpp> #include <boost/iostreams/filtering_stream.hpp> #include <boost/iostreams/detail/config/zlib.hpp> using namespace boost::iostreams; std::string compress(std::string data) { std::stringstream compressed; std::stringstream decompressed; decompressed << data; boost::iostreams::filtering_streambuf<boost::iostreams::input> out; out.push(boost::iostreams::zlib_compressor()); out.push(decompressed); boost::iostreams::copy(out, compressed); return compressed.str(); } std::string decompress(std::string data) { std::stringstream compressed; std::stringstream decompressed; compressed << data; boost::iostreams::filtering_streambuf<boost::iostreams::input> out; out.push(boost::iostreams::zlib_decompressor()); out.push(compressed); boost::iostreams::copy(out, decompressed); return decompressed.str(); } int main() { std::string data = compress("boo!"); std::cout << data << std::endl; std::cout << decompress(data) << std::endl; return 0; }
JC, I am not an expert on Iostreams, but I was able to obtain a significant performance improvement by removing temporaries and passing by reference instead of by value. I have attached the code. Notice, that I am using back_inserter and make_iterator_range instead of stringstream operations. Hope This Helps, Justin On Monday 03 March 2008 15:13:32 j.c. wrote:
Ok I think this sucks. :-) Here is my code to implement zlib compression/decompression via boost. However since is uses streams I cannot simply pass a buffer. This code is intended for compress socket buffers, but is pathetically slow. HAs anyone done this? What can I improve? The copy seems to be the bottle neck.
thx jc
Oops, I had some bugs in this code ... the attached should work better. Justin On Monday 03 March 2008 17:57:10 KSpam wrote:
I am not an expert on Iostreams, but I was able to obtain a significant performance improvement by removing temporaries and passing by reference instead of by value. I have attached the code. Notice, that I am using back_inserter and make_iterator_range instead of stringstream operations.
Thanks Justin, Do you think the speed is OK for data transmission over udp sockets? jc On Mar 3, 2008, at 4:59 PM, KSpam wrote:
Oops, I had some bugs in this code ... the attached should work better.
Justin
On Monday 03 March 2008 17:57:10 KSpam wrote:
I am not an expert on Iostreams, but I was able to obtain a significant performance improvement by removing temporaries and passing by reference instead of by value. I have attached the code. Notice, that I am using back_inserter and make_iterator_range instead of stringstream operations.<main.cpp>
Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
JC,
Do you think the speed is OK for data transmission over udp sockets?
Unfortunately, this is a difficult question to answer, so I will resort to the standard engineering response ... it depends :-) There are many parameters that can play into the answer, and they all depend on the nature of the application. For instance: What frequency are messages being sent? What is the average message size? What is the speed and reliability of the network connection? There will obviously be some overhead required in compressing the messages. If the network connection is unreliable or slow, then the overhead may be worthwhile. If the network is reliable and fast (or if your sending or receiving processor is slow), the compression overhead may not be worthwhile. Of course, the beauty of Boost.Iostreams is that adding, removing, or changing the compression type is very easy to do. When you prototype your application, you can perform throughput analysis to determine the optimal setup. I modified the example I sent to add a bzip2 compression test (ALGO 4) and a no compression test (ALGO 3). Here are the results from running a debug Linux build on my machine: ALGO 0 : 10000 cycles took 1.261 seconds! ALGO 1 : 10000 cycles took 1.2 seconds! ALGO 1 : Improvement 4.83743 % ALGO 2 : 10000 cycles took 0.652 seconds! ALGO 2 : Improvement 48.295 % ALGO 3 : 10000 cycles took 0.218 seconds! ALGO 3 : Improvement 82.7121 % ALGO 4 : 10000 cycles took 0.928 seconds! ALGO 4 : Improvement 26.4076 % For grins, here are the release build results: ALGO 0 : 10000 cycles took 0.927 seconds! ALGO 1 : 10000 cycles took 0.88 seconds! ALGO 1 : Improvement 5.07012 % ALGO 2 : 10000 cycles took 0.345 seconds! ALGO 2 : Improvement 62.7832 % ALGO 3 : 10000 cycles took 0.028 seconds! ALGO 3 : Improvement 96.9795 % ALGO 4 : 10000 cycles took 0.597 seconds! ALGO 4 : Improvement 35.5987 % As you can see, the no compression case (ALGO 3) is significantly faster. In this case, zlib compression (ALGO 2) is faster than bzip2 compression (ALGO 4). Bzip2 compression generally compresses the data better than zlib compression though. There's always a tradeoff :-) Hope This Helps, Justin
It's p2p, so lost of connections, packets, etc.. Could I get an updated copy? That would be most helpful. Again, thanks so much. On Mar 3, 2008, at 9:43 PM, KSpam wrote:
JC,
Do you think the speed is OK for data transmission over udp sockets?
Unfortunately, this is a difficult question to answer, so I will resort to the standard engineering response ... it depends :-) There are many parameters that can play into the answer, and they all depend on the nature of the application. For instance:
What frequency are messages being sent? What is the average message size? What is the speed and reliability of the network connection?
There will obviously be some overhead required in compressing the messages. If the network connection is unreliable or slow, then the overhead may be worthwhile. If the network is reliable and fast (or if your sending or receiving processor is slow), the compression overhead may not be worthwhile. Of course, the beauty of Boost.Iostreams is that adding, removing, or changing the compression type is very easy to do. When you prototype your application, you can perform throughput analysis to determine the optimal setup.
I modified the example I sent to add a bzip2 compression test (ALGO 4) and a no compression test (ALGO 3). Here are the results from running a debug Linux build on my machine:
ALGO 0 : 10000 cycles took 1.261 seconds! ALGO 1 : 10000 cycles took 1.2 seconds! ALGO 1 : Improvement 4.83743 % ALGO 2 : 10000 cycles took 0.652 seconds! ALGO 2 : Improvement 48.295 % ALGO 3 : 10000 cycles took 0.218 seconds! ALGO 3 : Improvement 82.7121 % ALGO 4 : 10000 cycles took 0.928 seconds! ALGO 4 : Improvement 26.4076 %
For grins, here are the release build results:
ALGO 0 : 10000 cycles took 0.927 seconds! ALGO 1 : 10000 cycles took 0.88 seconds! ALGO 1 : Improvement 5.07012 % ALGO 2 : 10000 cycles took 0.345 seconds! ALGO 2 : Improvement 62.7832 % ALGO 3 : 10000 cycles took 0.028 seconds! ALGO 3 : Improvement 96.9795 % ALGO 4 : 10000 cycles took 0.597 seconds! ALGO 4 : Improvement 35.5987 %
As you can see, the no compression case (ALGO 3) is significantly faster. In this case, zlib compression (ALGO 2) is faster than bzip2 compression (ALGO 4). Bzip2 compression generally compresses the data better than zlib compression though. There's always a tradeoff :-)
Hope This Helps, Justin _______________________________________________ Boost-users mailing list Boost-users@lists.boost.org http://lists.boost.org/mailman/listinfo.cgi/boost-users
participants (3)
-
j.c. -
Julian Cain
-
KSpam