[boost] [IOStreams] windowing in mapped_file

From: Benjamin Collins (ben.collins_at_[hidden])
Date: 2007-09-03 16:36:55

I have, on occasion, had a need to read large files.  Sometimes,
boost::iostreams::mapped_file works just fine, as long as I have
enough memory space available to map the file in its entirety.
Sometimes I don't.

I have been thinking of making io::mapped_file support the notion of
memory-mapped files with sliding windows.  I have tried a little bit
of code, but without much success because I hadn't thought through the
design very well (and part of that I blame on my current position on
the learning curve of developing boost libraries).  In particular, I
tried adding a couple of parameters to the mapped_file_params struct,
and then coding support for it in the mapped_file_source class.  This
doesn't get very far in getting what I want for several reasons mostly
because it doesn't address how the upper layers of iostreams will use
the device.  The data(), begin(), and end() member functions return
simple char* (the iterator type is just typedef char *).

I'd like to take another approach, which I think would be better than
my initial attempt.  I'd like to do something approximating the
following:

template<class W = default_window_manager>
 class mapped_file_source {
 public:
     typedef W::iterator iterator_type;

     // ...
     iterator_type begin() const;
     iterator_type end() const;
     iterator_type data() const;

     // ...
 private:
     W _manager;
 };

Where 'class W' would have to model something like the following:

 class window_manager {
 public:
     struct iterator {
         operator++() {
             if(current_ptr+1 >= current_map_end)
                 // remap file to the next block, re-adjust pointers
         }

         operator--() {
             if(current_ptr-1 <= current_map_begin)
                 // remap file to the previous block, re-adjust pointers
         }

         // operators for +,- might also need to do something non-trivial
     };
 private:
     // member functions to perform re-mapping

     current_ptr;
     current_map_begin;
     current_map_end;
 };

Of course, the above does not deal with random access at all.
'default_window_manager' could just provide a windowed memory map with
a single window that is exactly the size of the file to make
mapped_file behave as it currently does.

It seems to me that this could be generally useful, but perhaps
reality is that I need to be saved from myself.  Comments?

I apologize in advance if this is has been hashed over before.  My
searches of the archives turned up nothing, but the internet is too
big a place to prove a negative.

-- 
Benjamin A. Collins <ben.collins_at_[hidden]>
http://bloggoergosum.us