[boost] boost::tokenizer

From: Dirk Gregorius (dirk_at_[hidden])
Date: 2004-10-14 12:46:49

Hi,

I like to break a file into tokens for processing. The file contains 
comments which are introduced by "//", "#" and ";". Can I setup the 
tokenizer directly such that the comments are skipped? If no, what would you 
suggest to erase the comments from my string before processing?

Here is what I do right now:

// CODE
ifstream is( "file.txt" );

string file, line;
file.reserve( 2 * 1024 * 1024 );
while ( getline( is, line ) )
{
 TrimHead( line );
 if ( line[0] != '/' && line[1] != '/' )
  file.append( line + "\n" );   // Need to append "\n" again to get the 
right tokens - not very nice
}

typedef tokenizer<char_separator<char> > Tokenizer;
char_separator<char> sep(" \t\n");
Tokenizer tokens( file, sep );
// END CODE

Another idea was to the following:

// CODE
ifstream is( "file.txt" );

string line( ( istreambuf_iterator<char>( is ) ), 
istreambuf_iterator<char>() );
EraseComments( line );
// END CODE

Any help is appreciated.

-Dirk