I am getting a result from the boost_tokenizer that I do not expect. I wonder if anyone could explain.

I have a line which only contains one occurence of the delimiter and it happens to be the first character. I would expect tokenization to yield two tokens, The first would be the empty string, the next would be the word that occurs after the delimiter. What actually happens is there is only one token, the word I am after. I do not get the expected empty string first. This seems wrong to me. Can anyone comment please? Here is the (simplified) code:

boost::char_separator<char> emSeparator("M");
boost_tokenizer emTokenizer(line, emSeparator);
boost_tokenizer::const_iterator it2 = emTokenizer.begin();
std::cout << "first token = [" << *it2 << "]" << std::endl;

I am using boost version 1.42.0 on Windows-XP SP3, compiling with VS 2005 (aka vc8).

Regards,

Andrew Marlow

___________________________________________________________
This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and delete this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is prohibited.

Please refer to http://www.bnpparibas.co.uk/en/information/legal_information.asp?Code=ECAS-845C5H  for additional disclosures.