Re: [boost] [rfc] Unicode GSoC project

Subject: Re: [boost] [rfc] Unicode GSoC project
From: Mathias Gaunard (mathias.gaunard_at_[hidden])
Date: 2009-05-15 10:04:30

Graham wrote:

> A good reloadable character library is in the vault.

I'll be reviewing it in a while.
I'm not too sure about the memory layout it uses (__uni_char_data could 
really be compressed to use less memory for example), nor about the 
interface it exposes, but it does seem to work well.

About is_grapheme_break though, isn't the implementation for legacy 
grapheme cluster rather than extended ones though?

> I think that a grapheme is more of an iterator concept than a data type
> concept. By specialising it you will unnecessarily complicate any
> library. Don't forget that, for example, the current grapheme may start
> as one character, then suddenly 'grab' the surrounding characters as it
> makes a combined glyph.
> I have never found a use case in practise where specialising the
> grapheme as other than a validated series of code points was helpful.

A grapheme is nothing more than a subrange of code points, at least in 
my current design.

> The two cases where graphemes are important is in display [which
> requires intermediate glyph conversion anyway, and works just as well on
> runs of code points, so code points are fine] and in editing - and the
> grapheme-ness here alters during typing.

It's also useful for grapheme-level searching.

Searching for the substring "foo", in the string "foo\u20d7" shouldn't 
match anything, because the extremities of the match are not at grapheme 
boundaries.

>  if you
> can do graphemes then you can do words, paragraphs etc as they are all
> just attributes of the characters with simple rules. Graphemes come in
> to their own for text display and editing and you would need these as
> well to be able to support that.

Those are not as important in my opinion, and given the time I have is 
restricted focus won't be on these.

Adding them later makes perfect sense, however.