- Boost - lists.boost.org

Unicode string
by Vladimir Prus 07 Apr '04

07 Apr '04

Miro Jurisic wrote: > On the other hand, in order to manipulate a Unicode string without > violating constraints on well-formedness, you have to consider the string > as a sequence of abstract characters (unless, of course, you constrain > yourself to string transformations which operate on code point sequences > yet guarantee that strings remain well-formed; there are few such > transformations -- concatenation is one of them under certain constraints). [snip] > capital letter C; combining caron; lowercase letter e > > it contains two abstract characters, but three UCS4 code points; therefore, > removing the first character from that string means removing the first two > code points of three. Removing just the first code point would leave you > with a combining caron followed by a lowercase letter e, which is not a > well-formed Unicode string. Hi Miro, so the point is that when using string-as-code-point-container, even searching and removing a character/substring might get invalid string? E.g. even looking for string 'foo' you theoretically can find string 'foo' followed by composing character, and removing just 'foo' will be invalid? > basic_string is not the abstraction you are looking for, but it's also the > only one that is readily available in STL/boost today. It may serve as a > good starting point (questionable, IMNSHO), but it should most definitely > not be treated as the right thing to use for Unicode in the long term. I wonder what's the right abstraction then? Is it necessary to have a class to represent abstract character, with all composing characters? - Volodya

3 5

RE: [boost] Re: Numeric conversions lib comments
by Rozental, Gennadiy 07 Apr '04

07 Apr '04

> > In most part proposed design looks reasonable. I have only > one major > > IMO concern. > > > > Issue: (Major) Why traits are template parameter of > converter? I've > > expressed my opinion in the matter before: Traits - as name suggest > > could not be policy. IOW should not be template parameters. > Traits are > > type specific and an ability to supply them independently to the > > primary type in most cases leads to invalid/too complicated design. > > Traits are not only template parameters of public > converter class > > but many other classes. IMO all of them should use S and T > as template > > parameters instead. > > > I have to disagree. > Traits are used as policies in too many designs, including > the STL itself, and I don't see a problem with that. I agree I don't say I like it either. Major issue with basic_string design IMO is that char_traits is template parameter > that traits classes are type specific, but they are used as > policies precisely because of that: if the user of a > parametrized facility wants a different treatment for a given > type, she can't change *the* traits for the type, all she can > do is pass in a different traits as a policy. Here is how I see it. Trait is something type specific, IOW specific to type, IOW uniquely defined by type. If you could provide trait along with the type as a template parameter, that mean you could change it independently from the type, IOW you could define different value for trait still using the same type. For example if you have component C defined like: template<typename T, typename Trait> struct C { ... }; You could instantiate it like C<MyT,TraitValue1> or C<MyT,TraitValue2>. So as you could see trait value is not uniquely defined by the type. Proper design would be to have trait stand alone class, like this: template<typename T> struct Trait { value = ... }; template<typename T> struct C { here we are using Trait<T>::value, where we used to use Trait. }; Now if the user needs to define trait value (or change from using default value provided by most traits designs), she just need to define explicit/partial specialization for this type or types family. On the other hand policy is something that is orthogonal to the primary type (actually in policy based design there may not be any primary type, but just an orthogonal policies), IOW you could use different policies in conjunction with the same type. And natural place for the policy is along with type in component definition among the template parameters. > If this were > not a policy, users would be stuck with the definition of the > default traits class. I agree that this opens a window to all > sort of problems in case a user plugs in an invalid traits, > but I think that should not constitute a reason for fixing > the traits into the converter. As always, "trust the programmer" So again Trait is not a Policy and let's not mix them > > 4. converter<>::convert converts only rvalue. Why? > > IMO we need second operation that works with lvalues. > What if I want > > to convert int& into char&? > This requires the target lvalue as a parameter instead of a > return value. It could be done yes, but I don't see the > reason. Why would want to do that? For example for [de]serialization solutions. What is in your opinion is problem domain for numeric conversion? Gennadiy.

11 50

RE: [boost] regex lib: regex_grep w/ match_posix broken?
by Benzinger, Ralph 07 Apr '04

07 Apr '04

> > Is it just me or does regex_grep return overlapping matches > > when using the match_posix flag? More specifically, grepping regex > > "a+" over text "-aaa-" will find matches "aaa", "aa", and "a" at > > offsets 1, 2, and 3, respectively. > > Yep, there's a fix going into cvs now, the patch is: > > Index: perl_matcher_common.hpp > < search_base = position = (*m_presult)[0].second; > > search_base = position = m_result[0].second; John, Thanks for the quick response. This works almost :-), but now the regex_grep misses consecutive matches: grepping regex "aaa" over "-aaaaaa-" will find only one match "aaa" at offset 1, but not the one at offset 4. Grepping over "-aaa-aaa-" works, though. Ralph

1 0

Re: [boost] [BGL] Tarjan's offline LCA algorithm.
by Cromwell Enage 07 Apr '04

07 Apr '04

No, my brother reclaimed the old computer :~ Cromwell Enage __________________________________ Do you Yahoo!? Yahoo! Small Business $15K Web Design Giveaway http://promotions.yahoo.com/design_giveaway/

1 0

regex lib: regex_grep w/ match_posix broken?
by Benzinger, Ralph 07 Apr '04

07 Apr '04

Hello, Is it just me or does regex_grep return overlapping matches when using the match_posix flag? More specifically, grepping regex "a+" over text "-aaa-" will find matches "aaa", "aa", and "a" at offsets 1, 2, and 3, respectively. This does not happen using the match_perl flag. I've confirmed this on NT and Linux platforms. The problem seems to be somewhere near perl_matcher::match_all_states(), where the inner if(!(this->*proc)()) statement is entered multiple times and thus indirectly sets (*m_presult)[0].second to an incorrect value, leading to an incorrect restart point. I know that regex_grep is deprecated, but I still prefer it over the iterator approach. Any ideas or comments? Thanks, Ralph

2 1

regex doc issue
by David Abrahams 07 Apr '04

07 Apr '04

On the match_results doc page, there's a broken link to #m12. Should be #m13? -- Dave Abrahams Boost Consulting www.boost-consulting.com

2 1

RE: [boost] RE: [program_options] Unicode support
by Ferdinand Prantl 07 Apr '04

07 Apr '04

Hi Volodya, > From: Vladimir Prus [mailto:ghost@cs.msu.su] > > > I would make program_options support an imbue() function > that allows a > > locale to be specified (otherwise use the default locale) > > Why bother? As I've indicated, internal processing will work > just fine with UTF-8, and no interface of the library will > expose a UTF-8 encoded string. Yes - that's it. There can be a locale en_US.UTF-8, and then the application must accept it. If it does not, it has conversion support, imbuable or not. Application should not require the library to have a unified encoding of char * on all environments; that's not STL style. > imbue() is a good thing for specifying what encoding char** > input has, but it's orthogonal to the rest of the library -- > it's used only on interface boundary. Yes, that's also my opinion. > > This provides much more flexibility than just supporting UTF-8. > > UTF-8 is a really impractical encoding for almost any > locale where the > > majority of text is not ASCII like and the user may well prefer to > > encode text is Shift-JIS or other encodings. > > Again, in my case the user does not use UTF-8 string, so why > would he care how the strings are encoded internally? Yes again, now it seems to me we try to persuade each other about the same thing :-) Ferda > - Volodya

2 1

RE: [boost] RE: Re: [program_options] Re: Unicode support
by Ferdinand Prantl 07 Apr '04

07 Apr '04

Hi Volodya, > From: Vladimir Prus [mailto:ghost@cs.msu.su] > > > UCS-2 held all characters in Unicode 1.1, There was a need for more > > unique numbers and UCS-4 was introduced in Unicode 2.0. > Unfortunately > > there is no 4-byte character specialization for > basic_string<> in STL yet. > > Or, to be exact, there's no agreement if wchar_t should be > 32-bit or 16-bit. > Linux (or gcc specifically) uses 32 bits, and Windows 16, > which means wstring only suitable for UCS-2. Wow, I did not know that about gcc, what a smart compiler... ;-) MSVC does it usually because Win32 API id ANSI/UCS-2, so the usage is straightforward then. > Besides, UCS-2 > has a mechanism to represent characters outside of 16-bit > space with two elements, which, I suspect, won't work if > wchar_t is 16 bit. Actually, not UCS-2, but another variable-sized character encoding was introduced in Unicode 2.0 - UTF-16. It serves for the 16-bit characters similarly like UTF-8 for the 8-bit ones. Using numbers from the unused (and forbidden) space it allows to encode characters in more than one element and so break the constraint of the 16-bits. It works with wchar_t, if only UCS-2 characters are stored in it; because the special characters are forbidden in UCS-2 anyway. However, UTF-16 brings the inconvenience of variable-sized character and those, who used UCS-2 and "enjoyed" the fixed character size should switch to UCS-4 to carry on... Not all applications need it, indeed, UCS-2 is enough, usually. > > Generally speaking UTF encodings (e.g. UTF-8 or UTF-16) are not > > suitable for the fast in-memory usage because some operations, for > > example at(int) and length() have not O(1) but O(n) to give > a result. > > I believe UTF-8 is popular party because such operations are > believed to be rare. Really? string.length() ? You must have been joking... :-) UTF-8 is popular because the vast amount of texts contains much more ASCII characters than the accented ones. It saves space to store them in this encoding. It was with the UTF-7 encoding also the first one, thus the oldest. It can be also comfortable for the application not to perform any conversion and work with the text in its native encoding - here can help the stringized interface working with UTF-8. Not for all applications, however. For example, XML parsers tend to work in UTF-16 internally, because UCS-4 eats much memory and UTF-8 as well for East Asian encodings. I also feel, that the applications, which parse texts and extracts string parameters to be used in some structures or as user input are processed usually with fixed-size strings, because the string operations are easier. > > They (UTF, > > T=Transformation) are better for storing texts as they save > place by > > using variable number of bytes for a character. However UCS (e.g. > > UCS-2 or > > UCS-4) are fast for memory operations because they use > fixed character > > size. > > What about representing values with two 16-bit values (that's > what I've mentioned above)? You mean UTF-16. Yes, it is also a possiblity, but having wchar_t not fixed sized, you would have to provide basic_string<char16_t>... I see it pretty the same like your UTF-8 option, because both encodings use variable-sized character, being so incompatible with the shared implementation of basic_string<> for char and wchar_t. Moreover, if basic_string<char> from the API would produce UTF-8 string, that it would be only consistent, if basic_string<wchar_t> produced UTF-16 and not UCS-2. To say nothing of support of 4-byte wchar_t and UTF-32 :-) > (BTW, for reference to those interested, > http://www.cl.cam.ac.uk/~mgk25/unicode.html#ucsutf > talks about different encodings). > > > That is why I would not like to use basic_string<utf8char> > in memory, > > rather basic_string<ucs4char> instead, but I would not > generalize it > > for all possible applications. > > In case of program options, I suspect that everything will > work for UTF-8 strings. IOW, the library does not care that > any given 'char' might be part of some Unicode character. > That's why it's attractive for me. It depends on the way of usage of basic_string<> inside and outside of the program_options. The implemenation of basic_string<> is done for fixed-size characters. It does not do searching/iterating/length counting with regards to the character size, which can vary from 1 to 6 bytes in UTF-8. For example string.find() would produce unexpected results if in haystack was stored UTF-8 and a needle was in ANSI, where all 255 numbers are allowed. Not only the library must be aware of it, but also the users, that for example basic_string<char>.size() does not deliver the count of characters but the count of bytes in a string then. That is why I do not like usage of basic_string<> for variable sized characters; basic_string<char> would not have consistent behavior for different encodings; as long as it is used for different ANSI encodings, the only misunderstanding is when two different characters in two local alphabets share the same character above ASCII. For other methods is basic_string<char> consistent. > > You can expect initialization from (const char * argv []) on all > > platforms or (const wchar_t * argv []) on Windows in UCS-2. > With the > > basic_string<> you already have support for the parameters from the > > current locale (char) and for parameters in UCS-2. > > What do you mean by 'parameters from the current locale'? If you set LC_CTYPE to something like "cs_CZ.iso8859-2" on UN*Xes or choose "Czech" in Windows Control Panel, command line shell (well, not only it) will start to accept and deliver in (char * argv []) also characters from a local alphabet (here Czech, for example). Such string you would have convert into UTF-8, if you wanted to have an UTF-8 interface. It means, you would have to do more than basic_string<char>(argv[x])... > I > am not sure that ctype::widen is required to care about > user-selected character encoding. > Not do I think it's requires from default codecvt facet. AFAIK it must care; the method widen() needs locale to provide an extension from char to a templated char_type. STL in MSVC uses correctly mbcstowc() to perform the conversion from the local alphabet to the UCS-2. I hope that no-one simply casts to wchar_t, being so reliable only for 7-bit characters. Anyway, it is always possible to write an own converting facet and force its usage for widen by with imbue. > If my locale on Linux is ru_RU.KOI8-R I don't think standard > requires codecvt<wchar_t, char> facet in default instance of > 'locale' to do meaningfull conversion from KOI8-R into unicode. Hmm, I am not sure about the ISO/C++ definition. If the standard implementation were something like "return (wchar_t) x" then it would be useless for anything except 7-bit ASCII. MSVC works with locale and it should be according to the standard, hopefully. Yet delivers the shell the variable "char * argv []" in ru_RU.KOI8-R to you. > > If we take an option to read parameters into > basic_string<wchar_t> or > > basic_string<ucs4char_t>, where the character size or > encoding is not > > the same as the native encoding on the command line, there is an > > affinity to streams. Some shells allow usage of UTF-8 encoded > > parameters or, generally, usage of characters out of the current > > locale. It means, that a program can choose the way, how to > encode all > > characters from Unicode to char. UTF-7/8, etc. I would like > to have a solution similar to streams: > > imbue(). Having this, you could convert internally every > argv[x] using > > imbue(y) applied to a stringstream, where the facet y > provides the caller. > > That's right. There should be some mechanism to convert from > current 8-bit encoding and source code encoding into unicode. > At least at linux there's a mechanism to do the first thing, > but I'm not aware about Windows. There are such methods in stdlib.h, which produce a wide character or a string of wide characters: int mbtowc(wchar_t *pwc, const char *s, size_t n); size_t mbstowcs(wchar_t *pwcs, const char *s, size_t n); These methods work with the locale supported in the operating system and thus you can always get a wchar_t from a char. And these should be used in the default facet in STL. > > On the other hand, such a conversion can be performed also > by a user. > > The parameters sent to main() are char* or wchar_t* and thus > > program_options can give them back just as they are in > > basic_string<char> and basic_string<wchar_t>. The client > can use his > > facet to imbue a basic_stringstream<> initialized with the > parameter from program_options. > > Or a conversion library could be used to perform the conversion, > > something like lexical_cast<> does for type converions. It > is a matter > > of convenience only - separate converions library (no support for > > encoding in > > program_options) or imbue(), performing the conversion inside the > > program_options. However, the conversion should not be > implemented for > > program_options only, that is why I suggested an existing > interface - > > facets. > > I agree with basing the mechanism on facets. OTOH, it's > really can be made orthogonal to program options. So > initially program_options can support only ascii (strict > 7-bit) and unicode, for which conversion is trivial. Hmm, if it was orthogonal to program options, like facets are to streams, then there is no need to declare support of encodings. char * and basic_string<char> are simply in the local encoding and wchar_t * and basic_string<wchar_t> are wide according to the understanding of "wideness" on a platform (Unicode). It will also support the input, which you can expect for program_options from main(): char * in the current locale encoding. If someone wants something else, he can use facet and streams to convert it (which could be hidden in program_options with imbue, but not necessarily). I think the point is not to support UTF-8 or UCS-2 but to support encodings explicitely declared or not. I would not implicitely return UTF-8 or UTF-16 basic_string<>s, not only because STL handles their chars implicitely according to the current locale, but also because basic_string<char>(argv[x]) does not behave this way. Having the encoding independent on program_options using some external conversions, possibly imbuable into program_options, make the library thinner and more concentrated on the problem: parse the command line. Like the streams - read characters from input. Those, who say "we need full Unicode, UTF-8 or UCS-4" have the same option as if working with streams - imbue or convert. Otherwise they have currebnt locale char from istream or wchar_t from wistream. > > You can find come converting facets for UTF-8 raedy for > imbue() to a > > stream in the files section on yahoo. Unfortunately not finished or > > not reviewed... > > Yea, I can find some facets, including one written by myself > ;-( And yea, unfortunately they are not in Boost yet. Great! You can probly prepare it for review, if your time allows it... :-) Ferda > > - Volodya > > > > _______________________________________________ > Unsubscribe & other changes: > http://lists.boost.org/mailman/listinfo.cgi/boost >

2 1

Uploaded/checked in Wave V 1.1.4
by hartmutkaiser＠t-online.de 07 Apr '04

07 Apr '04

Hi all, I've just uploaded a new version of the Wave preprocesor library (V1.1.4) to the yahoo files section here: http://tinyurl.com/yul67. The same version is checked into the boost-sandbox CVS. There were a lot of changes recently: - Adjusted to use the new version of the program_options library. - Fixed a bunch of minor bugs in the expression evaluation engine. - Made a lot of other rather unrelated bug fixes (see in the ChangeLog file contained inside the distribution). Besides passing its own test suite Wave now passes 98% of the preprocessor validation suite published here: http://tinyurl.com/yul67. The remaining open issues are in the field of wide character and multibyte character handling, which are implementation depended anyway. But I'm still working hard on fixing the remaining issues. Regards Hartmut

2 2

Tokenizer doc problems/confusion (?)
by David Abrahams 07 Apr '04

07 Apr '04

1. http://www.boost.org/libs/tokenizer/tokenizer.htm says: template < class TokenizerFunc = char_delimiters_separator<char>, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ class Iterator = std::string::const_iterator, class Type = std::string > class tokenizer Yet char_delimiters_separator is officially deprecated. Is that really intentional? Wow, it appears to be using the deprecated class template for the default! Now, I wanted to tokenize an input stream, without putting it in a string first. It seems to be much harder than neccessary: #include <map> #include <string> #include <iostream> #include <boost/tokenizer.hpp> #include <boost/lambda/lambda.hpp> #include <iterator> int main() { typedef std::map<std::string, unsigned> fmap; // Seems awfully complicated boost::tokenizer< boost::char_delimiters_separator<char>, std::istreambuf_iterator<char> > t( (std::istreambuf_iterator<char>(std::cin)) , std::istreambuf_iterator<char>() ); fmap f; std::string s; using namespace boost::lambda; std::for_each(t.begin(), t.end(), ++var(f)[_1]); for (fmap::iterator p = f.begin(), e = f.end(); p != e; ++p) std::cout << p->second << ": " << p->first << "\n"; } I can think of lots of ways to simplify the interface, most of which center on eliminating redundant mentions of istreambuf_iterator<char>. When I throw the following text at it: ------ how much wood could a woodchuck chuck, if a woodchuck could chuck wood? ------ I get: 2: a 2: chuck 2: could 1: how 1: if 1: much 2: wood 2: woodchuck as desired. But if I replace char_delimiters_separator with char_separator, I get: 15: What's up with that?? Even if char_separator did what it was advertised to (and it's not clear that it does), it wouldn't give me the simple "find the words functionality" of char_delimiters_separator... so I'm baffled by the deprecation. 2. http://www.boost.org/libs/tokenizer/char_separator.htm says: explicit char_separator() The function std::isspace() is used to identify dropped delimiters and std::ispunct() is used to identify kept delimiters. In addition, empty tokens are dropped. which seems strange in light of the fact that there's no ctor taking _functions_ to be used to determine kept/dropped delimiters, and nowhere in the text do you indicate that functions are called internally. Help? Thanks, -- Dave Abrahams Boost Consulting www.boost-consulting.com

1 0