[Boost-users] Re: tokenizer and wstring with VC7.1

From: Bronek Kozicki (brok_at_[hidden])
Date: 2004-02-22 07:23:34

On Sat, 21 Feb 2004 09:10:43 -0000, Keith MacDonald wrote:
>     MyTokenizer token(string_t(_T("abacadaeafag")), sep);

if you take a look into tokenizer constructor

    template <typename Container>
    tokenizer(const Container& c,const TokenizerFunc& f)
      : first_(c.begin()), last_(c.end()), f_(f) { }
    
you will notice that it's not storing copy of its string argument;
instead it stores only its begin and end iterator. When string variable
is destroyed (and in your example it's temporary variable; thus its
destroyed at the end of expression) these iterators are no longer valid.
Problem you are experiencing here is unusual manifestation of undefined
behaviour - you are working in invalid interators. I think that program
crash would be better indicatation that you have serious problem, but
undefined behaviour may manifest in any other way - this time it's just
as if tokenizer is empty. Of course, slight change of program or
compilation options may result in crash (or anything else), until you
remove undefined behaviour:

>     string_t    s(_T("abacadaeafag"));
>     MyTokenizer token(s, sep);

B.