Boost logo

Boost :

Subject: Re: [boost] RFC: interest in Unicode codecs?
From: Mathias Gaunard (mathias.gaunard_at_[hidden])
Date: 2009-07-18 02:14:30

Rogier van Dalen wrote:

> Freestanding transcoding functions and codecvt facets are not the only
> thing I believe a UTF library would need, though.

I've personally purposely chose not to use codecvt facets in my unicode
library at all, but maybe I should provide them anyway for compatibility
with the iostreams subsystem.
I don't really find those practical to use.

  I'd add to the list:
> - compile-time encoding (meta-programming);

Didn't think of that.

> Iterator
> adaptors, I found, are a pain to attach error policies to and write
> them correctly. For example, with a policy equivalent to your
> "ReplaceCheckFailures", you need to produce the same code point
> sequence whether you traverse an invalid encoded string forward or
> backward. I've got code for UTF-8 that passes my unit tests, but the
> error checking and the one-by-one decoding makes it much harder to
> optimise.

For now my iterator adaptors (and the codecs they're based on for that
matter) perform full checks, including checking that we don't go past
the end of the input range (one way or the other).
While I wanted both versions with checks and without initially, only
having one does make it easier to use.

An error policy isn't really enough though, because to do full checks
you need each iterator to know about the begin and the end of the range
it's working on which could be avoided altogether when trusting the input.

They're fairly simple implementations and were never benchmarked
(benchmarking my library isn't even scheduled at the moment), but
they're quite correct (proper unit tests are in the works).

> I believe that Mathias Gaunard is working on a library at
> <>. I don't know how
> complete it is, but from the documentation it looks well thought-out
> so far. I'm looking forward to seeing where that's going!

I'm in the writing of several tutorials to make it easier to understand
how it's designed. (plus I still need to actually implement some stuff
that is in that version of the docs)

Boost list run by bdawes at, gregod at, cpdaniel at, john at