[Boost-users] Tokenizer reset

From: Umut Tabak (umut.tabak_at_[hidden])
Date: 2007-05-06 06:03:49

Dear all,

I am new to boost and tried to use the Tokenizer lib. But I have a 
problem. When I initialize a tokenizer object in a loop and tokenize 
that, it is ok on the first iteration. But on the second and subsequent 
iterations, even if I try to initialize the object with some different 
string, the object always tokenizes the first initialization. So I see 
that I have to reset the tokenizer object at each iteration. But I could 
not understand how to do this from the tokenizer man pages.

Here is some code :
// define tokenizer and separator
typedef boost::tokenizer<boost::char_separator<char> > tokenizer;
boost::char_separator<char> sep(" = ");
..
.
.
//initialize and use in a loop
while(getline(inFile,readLine)){
      //skip empty lines for count operation
      if(!readLine.empty()){
        // each line is tokenized
        tokenizer tokens(readLine, sep);  
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!<======
        // tokens are pushed on a vector
        for (tokenizer::iterator tok_iter = tokens.begin();
          tok_iter != tokens.end(); ++tok_iter)
          lineTokens.push_back(*tok_iter);
        // lineTokens hold the tokens
        // check if the configuration file is valid by the count
        // function of the set
        if(configVars.count(lineTokens[0])==1){  // means existent so 
this is valid
        // assign the read value to the right class data member
          switch(line_count){
            // switch can be extended depending on the configuration file
            // decide on the line number of the configuration file
            case 0:// line 1
              vlInputFileName=lineTokens[1];
              break;
            case 1:// line 2
              vlProjectFileName=lineTokens[1];
              break;
            case 2:// line 3
              vlcatvbsFileName=lineTokens[1];
              break;
            case 3:// line 4
              vlVbModuleName=lineTokens[1];
              break;
            case 4:// line 5
              pathToVLProjectFile=lineTokens[1];
              break;
            // case lines can be added here
            // ...
            default:
              ;
              break;
          }
        }
        else
          throw std::runtime_error("The configuration file variable(s) 
are not valid!");
        // clear the line for the next operation
        readLine.clear();
        line_count+=1;
        // I should reset tokenizer object but I could not do  
<========================= Reset tokenizer after each iteration

Your help is very much appreciated

Best regards,

Umut