Update to tokenizer

From: jbandela_at_[hidden]
Date: 2001-09-15 19:13:18

I have updated tokenizer to fix/enhance escaped_list_separator. It 
now correctly handles terminating empty tokens. In addition, you are 
now able to specify multiple characters as escape, quote, and 
separator characters.

This change was motivated mostly in response to the comments of Johan 
Nillson and Jens Maurer regarding empty fields and tokens.

Cheers,

John R. Bandela