RE: [boost] library for managing a large number of FILE objects

From: Michiel Salters (michiel.salters_at_[hidden])
Date: 2004-04-16 04:43:36

I'm wondering whether this can be generalized. Basically,  I imagine a
class "virtually" holding objects of some type T, with the restriction that
it may not hold more than N actual objects of type T, and holds
data<T> objects to create the other objects. You'd have a policy<T>
which would create a T from a data<T> and later destroy it.
The assumption is that a data<T> would be a lot cheaper than an
actual T.

Your case would be a specialization with T==FILE* and data<T>
being just the file name.  The policy<FILE*>::create(data<FILE*>)
method would call fopen, the policy<FILE*>::destroy() would fclose it.
The default policy<T>::create( data<T> data_T ) can probably call
new T( data_T ) and return that.

Regards,
Michiel Salters

> -----Original Message-----
> From: boost-bounces_at_[hidden]
> [mailto:boost-bounces_at_[hidden]]On Behalf Of Jody Hagins
> Sent: vrijdag 16 april 2004 1:57
> To: boost_at_[hidden]
> Subject: [boost] library for managing a large number of FILE objects
> 
> 
> 
> I know that C++ IOStreams are supposed to take over the world, and
> cstdio with FILE is considered taboo by some C++ 
> efficionados.  However,
> due to many reasons, lots of code still uses cstdio FILE I/O. 
>  One issue
> that comes up from time to time is the need to manage more files than
> the OS will allow a single process to have open simultaneously.  I
> process tons of data, and this is an issue for me.  I have developed a
> small library that provides on-demand caching of FILE 
> pointers, so that
> an application can "open" as many FILEs as necessary, and use them as
> normal.  A simple LRU eviction algorithm is used to reclaim FILEs when
> "all" have been used.
> 
> I was discussing this library with another developer, and he 
> said he has
> seen several questions recently about a similar issue, and he 
> advised me
> to ask if there was interest here.  This library, however, 
> does not seem
> "on the edge" enough though...
> 
> A very simple example of how you can use the library (of course there
> are better ways to do the following, but it is meant to be a 
> small easy
> to use example).
> 
> 
> // Open 10,000 FILEs
> wjh::file_cache file_cache;
> std::vector<wjh::cached_fptr> fp;
> for (int i = 0; i < 10000; ++i)
> {
>   std::stringstream strm;
>   strm << "FILE_" << i;
>   wjh::cached_fptr fptr = file_cache.open(strm.str().c_str(), "w");
>   if (!fptr)
>   {
>     std::cerr << strm.str() << ": " << strerror(errno) << std::endl;
>     break;
>   }
>   fp.push_back(fptr);
> }
> 
> // Randomly write to a particular file.
> for (int i = 0; i < 200000; ++i)
> {
>   int x = rand() % fp.size();
>   fprintf(fp[x], "file %d, iteration %d\n", x, i);
> }
> 
> 
> Is it something useful for more than just me (and the pitiful 
> souls who
> work for me and must use it) on this list?
> 
> Is it something worth posting here?
> 
> _______________________________________________
> Unsubscribe & other changes: 
> http://lists.boost.org/mailman/listinfo.cgi/boost
> 

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.