Proposed Changes to Tokenizer

From: jbandela_at_[hidden]
Date: 2001-09-09 21:28:42

I was thinking about the following changes to Tokenizer. I would like 
some feedback before I implement them.

1. char_delimiters_separator and offset_separator would take an extra 
template parameter to specify whether the tokens are assigned using

tok+=*next; OR
tok.assign(cur,next);

(Thanks to Gennadiy E. Rozental for this idea)

2. escaped_list_sepator would allow specifying multiple characters 
for escape, separator(c), and quote. This would solve the problem 
that Johan Nilsson brought up about empty tokens

3. escaped_list_separator would take another template parameter, to 
specify the functor to be used to process escapes. The default would 
be tokenizer_default_escape_processor which would have the same 
semantics as present. Also available would be a c_escape processor 
that would process the C escapes, and a url_escape_processor which 
would process url escapes ( ie + for space, and %hh where hh are hex 
digits). The last would ease processing of cgi parameters.

Regards,

John R. Bandela