Re: [boost] GIL io_new review

Subject: Re: [boost] GIL io_new review
From: Phil Endecott (spam_from_boost_dev_at_[hidden])
Date: 2010-12-08 10:58:51

Domagoj Saric wrote:
> "Phil Endecott" <spam_from_boost_dev_at_[hidden]> wrote in message 
> news:1291751283459_at_dmwebmail.dmwebmail.chezphil.org...
>
>> Having said that, I suspect that even with lots of extensions GIL might 
>> not really be what I need.  Here's a typical problem that I will have to 
>> address in the next few weeks:  I have a 1e12 pixel image - a map - which 
>> is supplied as a few thousand 5000x5000 TIFF tiles; I have to join them 
>> together, tweak the colours, draw some lines on top, and re-chop it into a 
>> few million 256x256 PNG tiles.  The issue of course is that that image 
>> can't be held in RAM.  I will do this with my own wrappers around libtiff 
>> and libpng, in rows or chunks.

> Hi Phil,
> if you are willing to try (and can work/test with MSVC on Windows) I think 
> io2 should already be able to help you here (partial ROI based access to big 
> images) at least with the WIC backend...

I'm doing this on Linux.  Wikipedia tells me WIC is "Windows Imaging Component".

> Example code demonstrating a possible (skeleton) solution: 
> http://codepad.org/WD7CpIJ8 ...

I don't really follow what that code is doing, and it's not obvious to 
me what its memory footprint will be.  In contrast, I think I can write 
something that's not much longer and more obviously correct by using 
the sort of simple wrappers around the libraries that I have been 
proposing and explicitly managing the tiled input and output:

class TiledReadImage {
   typedef shared_ptr<ReadTiff> readtiff_ptr;
   readtiff_ptr images[1400];  // One row of input tiles
   int rownum;  // in pixels
   void open_next_row() {
     for (int c = 0; c < 1400; ++c) {
       images[c] = new ReadTiff(input_tile_filename(c,rownum/5000);
     }
   }
public:
   TiledReadImage(): rownum(0) {}
   read_row(pixel_t* data) {
     if (rownum%5000 == 0) {
       open_next_row();
     }
     for (int c = 0; c < 1400; ++c) {
       images[c]->read_row(data+5000*c);
     }
     ++rownum;
    }
};

// Something similar for TiledWriteImage

TiledReadImage i;
TiledWriteImage o;
pixel_t data[700000];
for (int row=0; row<1300000; ++row) {
   i.read_row(data);
   o.write_row(data);
}

Anyone can look at that and see that it keeps 1400 input files open 
(bad) and needs 700 kwords of buffer memory (good).  If you wanted to 
have fewer files open you could code explicitly something else that had 
one input file open (good) and buffered complete tiles (probably 2 x 
5000 x 700,000 = 7 GBytes) (bad).

> ps. unfortunately I do not have access to such a huge image to test whether 
> WIC can actually handle such monster images...

If you're interested in experimenting, I suggest making random input 
tiles or just replicating them.

>> On the grounds that this is better than what is currently there, and on 
>> the assumption that Domagoj Saric is not imminently going to post 
>> something else that would supersede this, I believe that it should be 
>> accepted.
>
> If we've already waited so long, why rush if some of us agree that there are 
> still things that need 'polishing'...

To get the ball moving on GIL extensions, and because this is better 
than what GIL currently has.  Past experience suggests that the 
existence of this io extension will not prevent other similar and 
incompatible things (i.e. yours) from being accepted in the future.

> You said you already did some LibXXX wrappers of your own...if this is 
> so...why not join the effort even if only temporary to make sure you get 
> what you want...Christian also seems open for cooperation...

My wrappers have all been written to implement some subset of the 
functionality that I needed at the time.  For example I have never 
implemented any of the error handling stuff, and in some cases I have 
only read or write but not both.

Regards,  Phil.