[boost] codecvt facets for utf8/uft16/utf32 - any interest?

From: Alberto Barbati (abarbati_at_[hidden])
Date: 2002-11-04 19:14:24

Hi Boosters,

I read in the list archives that there was a proposal by Vladimir Prus 
about codecvt facets for uft8-ucs2 and ucs2-ucs2 conversion. The 
proposal dates back to last year. I wonder what happened of it.

However, I would like to up the ante and propose a much wider choice of 
codecvt facets that could be used to effectively process Unicode text 
files. This new proposal aims to be fully conformant to Unicode 3.2 
requirements. Therefore, I will refer to the utf-8, utf-16 and utf-32 
encodings of Unicode code points, disregarding the ucs-2 and ucs-4 
counterparts.

The proposal shall include facets to convert:

   external    internal
   utf-8    -> utf-16* (BMP only - no surrogates)
   utf-8    -> utf-16* (all planes - surrogates allowed)
   utf-16LE -> utf-16
   utf-16BE -> utf-16
   utf-16** -> utf-16
   utf-8    -> utf-32
   utf-16LE -> utf-32
   utf-16BE -> utf-32
   utf-16** -> utf-32
   utf-32   -> utf-32

Notes:
(*) There are two utf-8 -> utf-16 facets because a 4-bytes utf-8 code 
unit sequence is mapped to a utf-16 surrogate pair. If the application 
won't handle surrogates anyway, it can opt for a more optimized facet 
(such processing is probably not conformant, is this "optimization" 
really needed?)

(**) external utf-16 facets will detect an initial BOM (U+FEFF) to 
select the endian-ness of the external stream (what to do if there is no 
BOM? default to the endian-ness of the platform?)

All proposed facets will be implemented as class templates, in order to 
avoid any explict reference to wchar_t or any other fixed-size integral 
type. Simply, a compile-time assertion will be used to ensure that the 
supplied type is large enough to hold the internal characters. (For 
platforms where wchar_t has less than 32 bits an application that wants 
to use utf-32 facets will thus be responsible of choosing a suitable 
integral type, defining char_traits and specializing basic_*stream 
accordingly.)

(future directions) The facets could use template policies, for example 
  to customize error handling (for instance, if a non-character is 
encountered the conversion may either signal an error o ignore the 
non-character).

The library is explicitly directed to platforms were char is an 8-bits 
type. Support for other platforms can be included in subsequent 
revisions, according to interest.

Is there any interest in this proposal? Any feedback is appreciated.

Alberto Barbati