- Boost - lists.boost.org

Comment on string / unicode discussion
by Sean Parent 07 Jul '06

07 Jul '06

I don't have enough time to delve deeply into this thread but I thought I'd make a few passing comments. Adobe has a fairly major string class problem (we joke that every project must have it's own string class - which is nearly true). There isn't such thing as a single type of string - there are _many_ purposes and you need to be able to handle things like language and style runs and large, large blocks of text with efficient edits, UI substations (which are aware of things like split negation and masculine/feminine forms), language based ordering, different encodings... We need another string class like a hole in the head. What we do need - are good standard algorithms which can be applied to any string class. I believe this is doable with the current iterator interface. I believe it's possible (meaning I've done some quick experiments) to define an input iterator (actually as strong as a non-mutating forward iterator) and output iterator, which do conversions. This means that you can define operations in terms of unicode encoding (though some operations such as ordering may still require a locale). Consider - to_lower(first, last, output) to_upper(first, last, output) such transformations can work with any encoding (you can uppercase UTF-8 into UTF-32). They can't work in-situ (but I don't think to_upper or to_lower really can work in-situ - certainly not in UTF-8 and probably not in UTF-16, and I believe there are some multi- character forms that even break in UTF-32...). It is possible though to wrap them with a replace function for in-place operations. The current std::find() will work with such iterator adapters to find single UTF-32 character (in any encoded sequence). Currently with ASL we're taking such an approach for localization strings (replacing an existing string class for localized strings at Adobe with a small set of functions and _any_ string class (any sequence of code units), including std::string, std::vector (or deque or list). You might take a look here for some ideas: <http:// opensource.adobe.com/group__asl__xstring.html>. Sean

6 15

Adobe Source Libraries, GIL, and Boost
by Sean Parent 07 Jul '06

07 Jul '06

Lubomir pointed me at the GIL thread on the list - and I wanted to make a few comments about how I see the relationship between Boost and ASL. The charter for Boost is to vet libraries for possible inclusion in the standard. The charter for ASL is to explore the use of generic and declarative techniques to develop large applications. Our goal is to replace 85% of the imperative code being developed with small declarative constructs. Certainly there is going to be some overlap between the two project and ASL makes _heavy_ use of Boost and borrows ideas from Boost quite freely. For parts of ASL, inclusion into Boost makes perfect sense - and we try to actively defer to Boost when a Boost library can meet our needs. Boost certainly has a significantly larger user base than ASL and far more active developers (anyone interested in helping out ASL???). I _really_ wish that I had enough time/resources on my team to take some parts of ASL and contribute them to Boost. However, my teams focus currently needs to stay on the higher level libraries (which I don't think are ready for Boost yet) and we simply can't afford the time to go through the submission process. But I'm 100% supportive of Lubomir's contribution of GIL (Lubomir is not on my team, and I'm thrilled that he can take the time for this submission to Boost - just I'm I'm thrilled that we were able to release it as part of ASL). GIL is a very significant piece of work. I will also support anyone who would like to borrow portions of ASL to contribute them to Boost. Some libraries that I think would be interesting to get into Boost with some comments about what I see as still needing to be done. --- Forest <http://opensource.adobe.com/group__asl__tutorials__forest.html> I've started to map the concepts from forest into the terminology of BGL <http://opensource.adobe.com/wiki/index.php/ Edge_Interface_For_Forest_Iterators>. There is also some work here about generic algorithms for node based iterators (see the undocumented code for reverse_append here <http:// opensource.adobe.com/reverse_8hpp-source.html>. With a bit more polish I think this could be a _really_ nice collections of node based algorithms and concepts, which could be applied to a single linked list and other pre-traversed structures such as forest. There are also some forest specific algorithms I'd like to see such as the rotate necessary for balancing AVL trees (any forest is a binary tree, and vice versa - see Knuth). Range Based Algorithms (using the boost range library) <http:// opensource.adobe.com/group__standard__extensions.html> I've just started busting these out into separate headers (along the boost style - for example adobe/algorithm/reverse.hpp) - for inclusion in Boost I'd like to see this complete. This would be pretty simple to incorporate into Boost (and they are quite handy!). CMath C99 compatibility <http://opensource.adobe.com/group__cmath.html> There are also some rounding modes which were missing - This seems like a pretty straight forward "grab". ZUIDs <http://opensource.adobe.com/classadobe_1_1zuid__t.html> I'd like to see some review of the platform dependent code which gathers entropy to ensure we're getting enough entropy. A short tutorial would be good. ZUIDs would be a good addition to the serialization library. md5 and SHA code <http://opensource.adobe.com/classadobe_1_1md5__t.html> <http://opensource.adobe.com/group__adobe__sha.html> Review and a short tutorial ONCE <http://opensource.adobe.com/group__adobe__once.html> The STATIC_INSTANCE stuff might be able to be removed - this was done for the Mac where static initializer on DLLs were getting executed on a thread after load so other functions in the compilation unit could be executed concurrently. That may have been fixed (I haven't tracked the issue) - this library would be a good addition to Boost threads. ThreadID <http://opensource.adobe.com/group__asl__thread__id.html> Another good addition to Boost threads. Final <http://opensource.adobe.com/group__adobe__final.html> I don't use inheritance much but this would be a nice addition to Boost utilities. Counter <http://opensource.adobe.com/classadobe_1_1counter__t.html> If boost would surface the lightweight mutex used in shared_ptr - this would go away. (This is a heavyweight one - as I'm counting on Boost providing a lightweight one in the future :-) ). value_t <http://opensource.adobe.com/classadobe_1_1value__t.html> I think there are some nice ideas here which could be stolen for boost::any - value_t also captures equality comparison, makes use of a variant to avoid heap allocations for some common types (I'm not sure but I think boost any does two heap allocations, and value_t never does more than one). regular_object <http://opensource.adobe.com/ classadobe_1_1regular__object.html> This is a generalization of value_t (and boost any) - the idea is to be able to parameters an "any" object by concept. Mat Marcus has been doing a fair amount of work with this library. Needs better docs and tutorial. dictionary_t, array_t, name_t <http://opensource.adobe.com/ group__asl__tutorials__dictionary__t.html> ASL using these heavily to communicate with the runtime languages - could be used as part of the Python bindings (we're currently looking at doing Lua bindings with them. dictionary_t needs a closed hash implementation - there is one in the /future/ directory in ASL inspired by the Lua table structure, I haven't had a chance to compare it in depth to the Google hash implementations. Copy-On-Write <http://opensource.adobe.com/ classadobe_1_1copy__on__write.html> This is a fairly polished library - I've done lots of experimenting with COW (I prefer the term copy-on-modify - but COM confuses everyone...) over the last 10 years or so. This is a nice, simple, and efficient implementations (but it does need a better counter - see above). XML Parser <http://opensource.adobe.com/ classadobe_1_1xml__parser__t.html> This is a subset of an XML parser developed for xstrings. In our upcoming .18 release it has been expanded to support more uses by Eric Berdahl and Foster Brereton. It is designed to be a very efficient parser which pareses in place, and allows for preorder and postorder handling of elements (allowing for inplace substitutions). Needs a fair amount of work still to be a general XML parser - but I'd like to see that happen. --- Wow - that's not a complete list (much longer than I set out to write - I forget how large of a library we've created!). If you're not aware of ASL - it's all released under the MIT license (making it fairly straight forward to get through a code audit). I'd be happy to help out anyone who has the time to prepare some of these libraries (or any other library from ASL) for inclusion in Boost - ASL would also benefit from such an effort. Sean

6 9

[asio][thread] a new model for boost::asio (coroutine)
by 李慧霸 07 Jul '06

07 Jul '06

I have used the recent boost::asio library in my small project, I found it very difficult to correctly write the asynchronous code. So I tried another approach ---- a Half-Sync/Half-Async, I found it much better. The basic idea is very simple, utilize the coroutines(also called light weight threads, or user level threads) to handle multiple connections simultaneously and collaboratively. This approach has the advantange of simplicity in the Synchronous IO model, as well as the high performance in the Asynchronous IO model. Unfortunately, there isn't any coroutine support in boost::thread, and I had to make my experiment with the fiber support in ms windows. So I suggest coroutine be support. The benefit isn't only confined to boost::asio, in fact, it's an important way to describe algorithm flow, and many dynamic languages, both old and new, are supporting this feature. -------------- lihuiba 2006-07-05

3 3

updated cvalarray
by Michael Tegtmeyer 06 Jul '06

06 Jul '06

I have updated the cvalarray tarball in the vault in response to comments. Changes include documentation fixes, minor bug fix, and additional test cases. See the changelog for specifics. Also in response to comments, I will add another document describing this particular implementation. Until then, highlights include: - expression templates for math operations - explicit assignment loop unrolling for small arrays - fundamental type recognition for memcpy for larger arrays Thanks for the comments. As I mentioned to some folks privately, I have no experience with VC++ so if anyone would care to try and compile a test case, I'd be grateful to hear about the experience. Thanks Mike

1 0

Is there interest for iterator based conversion library?
by Alexander A Potocki 06 Jul '06

06 Jul '06

Hi boost community! I would like to propose my iterator based conversion library. It allows declaring and using transformations by following way: std::wstring utf16string(L”some sequence to convert”); typedef cvt::encoder<wchar_t, char> coder_t; coder_t coder = cvt::utf16() >> cvt::utf8(); That piece of code declares converter from wchar_t to char sequences and then instantiate such converter by utf16 to utf8 transformation. Then that converter instance can be used by following way: typedef cvt::convert_input_iterator<coder_t, std::wstring::const_iterator > it_t; - declaration of input iterator std::string result; std::copy( it_t(coder, utf16string.begin(), utf16string.end()), it_t(coder, utf16string.end(), utf16string.end()), std::back_inserter(result) ); Now we have utf8 encoded string in result container. That’s all. Library contains implementation for following transformations/algorithms: utf7, utf8, utf16(be/le), utf32(be, le), Unicode simple folding, base64, rc5(cipher/decipher). Library allows creating pipeline converters, e.g.: typedef cvt::encoder<char, char> coder_t; // char to char sequence transformation coder_t coder = cvt::utf7() >> cvt::utf16() >> cvt::utf8() / cvt::int8() >> cvt::rc5<32,12,8>(key) / cvt::int8() >> cvt::base64(); That expression defines pipeline transformation. Utf7 encoded sequence transforms to utf16 encoded sequence, then to utf8 encoded, then performs rc5 cipher with specified key, then performs base64 encoding. Library can be found at http://sourceforge.net/projects/trotter-cvt/ Alexander Potocki

1 0

Re: [boost] Comment on string / unicode discussion
by Reece Dunn 06 Jul '06

06 Jul '06

Sean Parent wrote: > What we do need - are good standard algorithms which can be applied > to any string class. Agreed. This is where Boost.StringAlgorithms come in. What I am interested in is *efficient* codepage -> codepage conversion. For example, I may want to read a file in that is stored as 8-bit ASCII as UTF8. Likewise, I may want to take UTF8 data and save it as UTF16, MacRoman or some other encoding. What you need is an encoding -> UTF32 converter and a UTF32 -> encoding converter. Ideally, I would like each of these to be as efficient as possible. They should also be able to accept partial data. That is, if I am reading in a UTF8 file in blocks, it is possible to hit the middle of a character. This can be solved in a random access stream by seeking to the previous character, but this is not always possible. Consider: std::basic_ostringstream< uchar32_t > utf; utf.set_locale( utf8_to_utf32_cvt()); // not sure on exact code here std::string utf8 = some_utf8_data(); std::copy( utf8.begin(), utf8.end(), stream_inserter( utf )); The problem with this is that the stringstream type is uchar32_t, but has an *input* character type of char. The conversion mappings are of the form: n source characters -> m destination characters where these may be encoded byte sequences (e.g. UTF8), surrogate pairs (e.g. UTF16) or combining characters (e.g. a + umlaut). I am not sure how good locales are for this kind of functionality and also how good C++ streams are for this. However, it would be nice to have a stream interface (i.e. << and >>). - Reece _________________________________________________________________ Be one of the first to try Windows Live Mail. http://ideas.live.com/programpage.aspx?versionId=5d21c51a-b161-4314-9b0e-49…

2 1

quan was [PQS]
by Andy Little 06 Jul '06

06 Jul '06

Hi, After all the interest in the PQS review, I have opted to rehouse the project on sourceforge and rename the project Quan. I am pleased to announce that the first release of Quan is now available. http://sourceforge.net/projects/quan/ There are many changes from pqs-3_1_x, based on the comments in the PQS review. See the quan-0_1_0 release notes on sourceforge for details: http://sourceforge.net/project/shownotes.php?release_id=429568&group_id=170… The project is also available via CVS: http://quan.cvs.sourceforge.net/quan/ Please also get in touch if you are interested in contributing to the project. regards Andy Little

3 5

Re: [boost] Comment on string / unicode discussion
by Martin Bonner 06 Jul '06

06 Jul '06

Just one comment: From: Sean Parent > Consider - > > to_lower(first, last, output) How do you handle that with just a forward iterator? You need to read a whole word before you can decide whether "SS" should be "ß" or "ss" in the output. (I suppose you could read the word into an internal buffer - but that seems extreme). > such transformations can work with any encoding (you can uppercase > UTF-8 into UTF-32). They can't work in-situ (but I don't think > to_upper or to_lower really can work in-situ - certainly not in UTF-8 > and probably not in UTF-16, and I believe there are some multi- > character forms that even break in UTF-32...). It is ß -> SS that break in-situ conversion! -- Martin Bonner Martin.Bonner(a)Pitechnology.com Pi Technology, Milton Hall, Ely Road, Milton, Cambridge, CB4 6WZ, ENGLAND Tel: +44 (0)1223 203894

1 0

(no subject)
by Sean Parent 06 Jul '06

06 Jul '06

> This is very close to what I have in mind. The main difference is that > the functions/algorithms in my mind take ranges instead of iterators. > Thus: > > to_lower(src, dest) > to_upper(src, dest) So long as you don't require ranges (or a pair of iterators makes a valid range and dest can still be an output iterator). That's fine - these should work on char* as well as container types. I don't know what kind of ranges you have for dest which allow dest to change size - seems a bit problematic. I want iterators that can handle the encoding transform. I want to be able to write items like the following: std::string s = get_some_utf_8_xml_data(); // Find the BOM character as a UTF-32 character utf_iterator_t i = std::find(utf_iterator_t(s.begin()), utf_iterator_t (s.end()), UL0x0000FEFF); assert(*i.base() == U0xEF); // base iterator points to start of UTF-8 character Sean > With these, I could make Fusion like wrappers that transform them into > something like: > > some_string s1 = to_lower(src); > some_string s2 = to_upper(src); > > where to_lower and to_upper return cheap views that are in and by > themselves valid strings/ranges. They are cheap because the actual > conversions/transformations are done on demand-- think lazy > evaluation. > So, like those done by expression template techniques, there are > no expensive temporaries when you perform seemingly expensive tasks > like: > > some_string s = f1(f2(f3(f4(src)))); > > And yes, because they are generic, those string algorithms can work > on any string type that satisfy some basic requirements. > > Regards, > > -- > Joel de Guzman > http://www.boost-consulting.com > http://spirit.sf.net

3 2

Re: [boost] Exception lib?
by Sohail Somani 05 Jul '06

05 Jul '06

> -----Original Message----- > From: boost-bounces(a)lists.boost.org > [mailto:boost-bounces@lists.boost.org] On Behalf Of Emil Dotchevski > > Well for some people the only way to fix a const correctness related > > error is to remove the const. I can imagine people doing > all sorts of > > (wrong)casts. IMHO, I think the api should constrain that freedom > > seeing as there *is* only one right way to do it! > > Please let's keep this to the point. There are no const_casts in the > proposed Boost Exception library. If you think that the > dynamic_cast can be > avoided, I'm very much interested to hear how. If you think > the need for > dynamic_cast is a result of poor library design, I'm very > much interested to > hear about any improvement ideas. You're missing my point. I didn't say that dynamic_cast in this way is bad, nor did I say someone would use const_cast. I was only pointing out that there are those who will just do what they can to get it to look like it works. Restrict those people. Sohail

1 0