Re: [Boost-users] [iostream] Problems opening large mapped_file

Subject: Re: [Boost-users] [iostream] Problems opening large mapped_file
From: Heiko Klein (Heiko.Klein_at_[hidden])
Date: 2012-08-07 06:50:05

I checked now the boost-sources. My version was 1.40 not 1.44 as I 
thought previously. 1.40 does not have build-in large-file support, 
restricting it to 2GB files, this has been corrected in later version, 
e.g. from a diff to 1.41:

     #ifdef _LARGEFILE64_SOURCE
         flags |= O_LARGEFILE;
     #endif

Heiko

On 2012-08-07 08:58, Heiko Klein wrote:
>
>
> On 2012-08-07 06:38, Vitaly Budovski wrote:
>> On 7 August 2012 02:43, Heiko Klein <Heiko.Klein_at_[hidden]> wrote:
>>> Hi,
>>>
>>> I wanted to test the mapped_file_source iostream device. My code is
>>> working
>>> well with a normal file_source device and files >= 6GB. When I change
>>> the
>>> constructor to a mapped_file_source, it works still well for smaller
>>> files
>>> (e.g. 150MB), but for files > 2GB I get (tested with 2.5GB):
>>>
>>> exception occured: failed getting file size: Value too large for defined
>>> data type
>>>
>>>
>>> And for files > 4GB (tested with 6GB) I get:
>>>
>>> exception occured: failed opening file: Bad file descriptor
>>>
>>>
>>>
>>> My constructor is as follows:
>>>
>>>          using namespace boost::iostreams;
>>>          typedef stream<mapped_file_source> mmStream;
>>>          mmStream* mmistream = new mmStream();
>>>          mmistream->open(mapped_file_source(fileName));
>>>
>>>
>>>
>>> I've tested against boost 1.44 and a 32bit linux (Ubuntu 10.04).
>>>
>>>
>>> Are these general mmap problems, or am I missing some flags for
>>> mapped_file_source?
>>>
>>> Best regards,
>>>
>>> Heiko
>>
>> Hello,
>>
>> 32bit systems have only approximately 3GB of user addressable memory.
>> You should upgrade to a 64bit system.
>>
>>
>> Regards,
>>
>> Vitaly
>
> Vitaly,
>
> thanks for your explanation. This makes sense for the problems with the
> 6GB file. But it does not explain the 'failed getting file size: Value
> too large for defined data type' with the 2.5GB file. Do you have an
> idea why this operation fails?
>
> Since I cannot upgrade to 64bit everywhere, I catch now the exception
> and open the stream again as file_source.
>
> Heiko

-- 
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY