Re: [boost] GIL io_new review

Subject: Re: [boost] GIL io_new review
From: Mateusz Loskot (mateusz_at_[hidden])
Date: 2010-12-13 12:53:24

On 10/12/10 19:27, Domagoj Saric wrote:
> 
> "Mateusz Loskot" <mateusz_at_[hidden]> wrote in message 
> news:4D00FF4B.7070901_at_loskot.net...
> 
>>>> 4. Cutting TIFF to PNG involves compression. If we are
>>>> interested in raster access, RIO, I/O speed, perhaps we could
>>>> stick to TIFF as output format as well. What you think?
>>> 
>>> That depends on what exactly are we trying to test here, the C++ 
>>> wrappers (e.g. io_new vs io2) and/or the backends (e.g. LibTIFF
>>> vs WIC) and/or something third...
>> 
>> Actually, that was part of my initial question.
> 
> Hi, sorry for the delay...

Sorry for delay too

Here is the tiling benchmark for GDAL:

https://github.com/mloskot/workshop/tree/master/benchmarking/tiling/gdal

The gdal_image_tiles_test.cpp + Makefile for those who would like to run
it in their environments.
The results.txt file includes timing + tiles number + total size of
tiles for PNG and JPEG output.

Shortly, my results on Intel P8600 + 4GB RAM with Linux (amd64)

PNG: 11:30 - 11:50 min
JPG:  2:10 -  2:30 min

RAM usage observed for both is less than 5MB.

> This code http://codepad.org/eGMixUK1 (io2 using the WIC backend, 
> unfortunately large TIFF support was added only to the latest WIC 
> available on Windows 7 so the same code will not work on WinXP)
> opens the input TIFF and hacks it up into separate tile files. With
> an Intel i5_at_4.2 GHz with 4 GB RAM I got the following results:

Looks like run on ~1.5x faster machine, considering clock of single CPU.

> 200x200 PNG tiles ~ 65 seconds 512x512 PNG tiles ~ 69 seconds

What compression level did you use? See results.txt for my details.

> 512x512 TIFF tiles ~ 27 seconds

I assume it's no compression, right? I haven't tried TIFF.

> RAM usage was below 5 MB the whole time (after working around an 
> apparent leak in WIC that otherwise cause the RAM usage to crawl up
> to ~15 MB)

Similar to the GDAL run.

> and the binary is about 45kB...

The size of binaries in GDAL test is: 26K test program + 11M libgdal.so
Note, libgdal.so is built with support of more than 120 dataset formats.

> ... I guess these tasks are no longer so hard for modern hardware as
> my work desktop churns them out pretty fast...

Indeed, your results are very impressive.

I'm going to run your benchmark and submit my results here
https://github.com/mloskot/workshop/tree/master/benchmarking/tiling/gil
but I'm missing some details about how to configure and build it.
Can I find it anywhere?

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org