Boost logo

Boost :

Subject: Re: [boost] [unicode] Interest Check / Proof of Concept
From: Phil Endecott (spam_from_boost_dev_at_[hidden])
Date: 2008-11-19 14:43:08

James Porter wrote:
> Over the past few months, I've been tinkering with a Unicode string
> library. It's still *far* from finished, but it's far enough along that
> the overall structure is visible. I've seen a bunch of Unicode proposals
> for Boost come and go, so hopefully this one will address the most
> common needs people have.

Hi Jim,

Mine was probably one of those proposals that you looked at; for the
record the code is all available at

and nearby directories. I was reasonably happy with my implementations
of the most common character sets (i.e. unicode, ASCII, iso8859), but I
wanted to explore some of the more esoteric ones to understand the
implications that they would have on how a general-purpose framework
should work. For example, I wanted to explore how error handling
policies could be specified and what conditions they would need to
handle. The last work that I did with this code was a general-purpose
command-line conversion utility that could be used to benchmark the
conversions. Input and output character sets and error policies could
be set from the command-line, but the problem that I hit was that
making these things template parameters led to a code-size and
compilation-time explosion. That means that I'll need to rethink a few
things, but it has been low on my to-do list.

> The library is based on two (immutable) string types: ct_string and
> rt_string. ct_strings are _C_ompile _T_ime tagged with a particular
> encoding, and rt_strings are _R_un _T_ime tagged with an encoding.

Mutable vs. immutable strings is something that has been briefly
discussed before. My personal preference has been for mutable strings,
but without the O(1) random access guarantee of a std::string. I also
considered strings where the only mutation allowed is appending, i.e.
there's a back_insert_iterator. Why do you prefer immutable strings?

One argument for mutable strings is simply that std::string is mutable,
and that a proposal is more likely to prove popular if it changes less
w.r.t. existing practice.

I also have run-time and compile-time tagging. My feeling now is that
compile-time-tagging is the more important case. Data whose encoding
is known only at run-time can be handled using a more ad-hoc method if
necessary. I also struggled to find good names for these things; I
don't find ct_string and rt_string great. Do any readers have suggestions?

> This is to allow for faster conversion when the encoding is known at
> compile-time, but to allow for conversion at run-time (useful for
> reading XML!).
> General usage would look something like this:
> ct_string<ct::utf8> foo("Hello, world!");

typedef ct_string<ct::utf8> utf8string;

> ct_string<ct::utf16> bar;
> bar.encode(foo);

Well it's actually decoding the utf16 and encoding the utf8. Maybe
"transcode", and preferably as a free function:


equivalent to:


> rt_string baz;
> baz.encode(bar,rt::utf8);

So the encoding of the rt_string is not stored in the string?

> Note the use of ct::utf8 and rt::utf8. As you might expect from the
> syntax, ct::utf8 is a type, and rt::utf8 is an object. Broadly speaking,
> to create an encoding, you create a class with read and write methods,
> and then you create an instance of an rt_encoding<MyEncoding>. Most of
> this is laid out in the comments of my code, so I won't go into too much
> detail here.

I'll try to find time to have a look, but I do encourage you to post
more details to the list. That tends to generate more discussion than
"please look at the code" proposals do.

> There's still a lot missing from the code (most notably,
> dynamically-sized strings and string concatenation),

So what is your underlying implementation? Not std::string?

> but here's a
> rundown of what *is* present:
> * Compile-time and run-time tagged strings
> * Re-encoding of strings based on compile-/run-time tags
> * Uses simple memory copying when source and dest encodings are the same
> * Forward iterators to step through code points in strings
> If you'd like to take a look at the code, it's available here:
> . I've tested it in gcc
> 4.3.2 and MSVC8, but most modern compilers should be able to handle it.
> Comments and criticisms are, of course, welcome.

One of my priorities has been performance; it would be good to compare
e.g. utf8-to/from-utf16 conversion speed.

My feeling about the way forward is as follows:

- A complete character set library is a lot of work.

- A library that only understands Unicode is less work, but is it what
people need?

- Is there a consensus about mutable vs. immutable strings? Perhaps we
should start by defining a new string concept, removing the
character-set-unfriendly aspects of std::string like indexing using
integers, and see what people think of it. I have been trying to use
only std::algorithms and iterators with strings in new code, but it can
often be simpler to use indexes and the std::string members that use or
return them.

- It would be useful to factor out the actual Unicode bit-bashing
operations. I have implementations of them that I have carefully
tuned, and they are ready for wider use even though the rest of my code isn't.

Regards, Phil.

Boost list run by bdawes at, gregod at, cpdaniel at, john at