[Boost-users] Tokenizer / offset_separator misbehaving - boost 1.47.0 / Windows 2003 Standard Server 32-bit / Visual C++ 2010 Express

Subject: [Boost-users] Tokenizer / offset_separator misbehaving - boost 1.47.0 / Windows 2003 Standard Server 32-bit / Visual C++ 2010 Express
From: asif saeed (asif.lse2_at_[hidden])
Date: 2012-02-10 10:16:57

Hi,

I wrote the following code to test offset_separator

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
int main (int argc, char **argv)
{
using namespace std;

typedef boost::offset_separator sep_type_t;
typedef boost::tokenizer<sep_type_t> tokenizer_t;
typedef tokenizer_t::const_iterator const_iterator;
typedef tokenizer_t::iterator iterator;

const string sentence = "Marry had a little lamb!";

int endMarkers[] = {5,3,1,6,4};

sep_type_t sep(endMarkers,endMarkers+3);

tokenizer_t t_(sentence,sep);

for (tokenizer_t::iterator i=t_.begin();i!=t_.end();++i)
cout << endl << *i;

return 0;
}
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

I get the following result:

Marry
 ha
d
 a li
ttl
e
 lamb
!

Any idea how I can fix it? I want each token on a separate line. I am
providing zero-based end indices to separator.

Thanks in advance,

-Asif