Re: [Boost-users] iostream large file - performance

From: Asif Lodhi (asif.lodhi_at_[hidden])
Date: 2007-03-09 11:25:49

Hi Aljaz,

On 3/8/07, Aljaz <aljaz.fajmut_at_[hidden]> wrote:
>
> I've been testing fstream libary to read a huge text file (45 megabytes),
> which has around 2 million lines..
>
> When I do (using fstream):
>
>
> Code:
> int line = 0;
> string str;
> ifstream is("file.txt");
> while (!is.eof()) {
>         getline(is, str);
>         line++;
> }It takes about 9 seconds to read the whole file..
>
> If I open file it with any 'good' text editor (editplus, notepad++ for
> instance) it takes a few seconds to read/load the whole file..
>
> So I'm wondering.. What am I doing wrong?
> What should I do to achieve that speed - better preformance?
>

As others have said earlier, text editor read your files in a piecemeal
fashion - a little at a time appropriately when you need to move to
corresponding lines/pages. I am newbie to boost myself but I feel your
problem has more to do with the relationship between the file allocation
unit size of your OS and that of your buffer/array and how large a buffer
you allocate to read the data.  Also, it looks like you want to "count" the
lines instead of reading. Anyway, I think looking at the appropriate OS
system services, which are efficiently related to corresponding OS file
allocation unit sizes, and then looking at the source code of the library
(boost or whatever) you are using would be helpful.

--
Best,
Asif