- Boost - lists.boost.org

token_iterator comments, suggestions
by Gennadiy Rozental 07 Jun '04

07 Jun '04

Hi, For long time in my daytime projects I was using my class token_iterator designed based on old iterator_adaptor design (adopted for old compilers). Now when need arose for such functionality in Boost.Test. I looked in direction of boost::token_iterator. After some struggle I end up adopting my version to new design. Here some of comments and issues that made me opt so. 1. token_iterator follows in a sense a Policy-based design (PBD). Where TokenizerFunc is a Policy that implements most of the functionality. The problem here IMO (unfortunately you see it now quite frequently), that I do not see a reason to use such a design. PBD makes sense it your generic component do a lot of work, provide common interface, combine several independent policies. In this case most what token_iterator does is just a forwarding (it become especially clear if you switch to use input_iterator_adaptor). It wouldn't be that bad, but PBD in this case causes significant usability inconvenience. Instead of plain and simple: token_iterator it( "a:b", ":" ); I have to write something like: token_iterator<char_delimiters_separator<char> > it( char_delimiters_separator<char>( ":" ), "a:b" ). All of that just because we have separate notion of Tokenizer function. IMO it too big price to pay. My suggestion is to get rid of the TokenizerFunction notion. Just implement many different iterators with the same scope of interest: tokenization. We will need to find proper names for iterators and instead of TokenizerFunc model description we may have page explaining how to write custom token iterators (if necessary, which I doubt). 2. token_iterator deals with begin/end iterators. I understand that in some rare cases there is need to iterate over range of iterators (BTW, std::input_iterator gonna be bad example - did you ever tried to use token_iterator over it?). But in most case we want to tokenize a string. And in most cases it either std::string or [char|wchar_t] const*. The fact that Iterator is a template parameter makes token_iterator usage quite inconvenient. In most cases the only template parameter that is required is IMO character type. My suggestion: lets have 2 different types if token iterators: one to iterate over string and separate to iterate over range of iterators. 3. I found it confusing in many cases how delimiters gets specified in token_iterator: token_iterator( "...", ":", " " ) Which string represent dropped delimiters, which kept? And what if I just want empty tokens? I need to write token_iterator( "...", "", "", keep_empty_tokens ) What are these two empty strings doing here?My suggestion is to use named parameters scheme. Like this: token_iterator( "...", dropped_delimiters = ":", kept_delimiters = " " ); 4. 2 Tokenizer functions use Traits as *policy* parameter. What actually is required in Compare predicate, that may or may not be implemented using char_traits. While std::basic_string references inside tokenizer function implementation should refer to traits by name std::char_traits 5. What if I want to use isspace as kept delimiters policy? There is no way to specify that. IMO delimiter policy should be separated (as runtime policy). 6. You use iterator_category as a selector of assigner. It's better to use traversal instead 7. In many places you use #if !defined(BOOST_MSVC) || BOOST_MSVC > 1300 typename #endif use BOOST_DEDUCED_TYPENAME instead 8. What is the purpose of token_iterator_generator. I do not see a need for one with new iterator facade design. 9. Why would we need all the access methods? Iterator base()const{return begin_;} Iterator end()const{return end_;}; TokenizerFunc tokenizer_function()const{return f_;} Type current_token()const{return tok_;} bool at_end()const{return valid_;} I couldn't imagine why do we need them. 10 Token iterator asserts in attempts to dereference or iterate end iterator. I don't believe it's correct. In many cases (mostly in design that assumes some optional tokens in the end) it's much more convenient this does not happened. My suggestion is to return empty token on dereference and stay quite on iteration. 11. In simpler design there won't be need in template copy constructor for token_iterator 12. Documentation need to be updated IMO, cause even as things stands now it dies not represent the best practice moved from type generators usage. 13. Files under libs/tokenizer lacks any structure required by boost standards. 14. There is no real test suite only some small examples supplied. 15. Why escaped_list_separator was named this way. It seems strange to emphasize in a name the fact that it use escape symbols, while major separations is that it is CSV list iterator - and this is how I would name it I implemented string_token_iterator and range_token_iterator for use in Boost.Test. See here (in a couple hours) http://tinyurl.com/28db9 for implementation and here http://tinyurl.com/287uk for test suite. Regards, Gennadiy.

5 16

Serializer and when to stop
by Darren Cook 07 Jun '04

07 Jun '04

I'm using boost::serializer between two programs: the first creates the data then serializes it, the second loads that data in and analyzes it. When I had just 420 data samples it all worked fine: I create a vector<Sample> in memory, serialized it and loaded it in the other side. When I moved to 7000 data samples the first program, the creator, sucked all the memory from the machine and then some. So I moved to serializing each data sample as it was created. To read it back in I changed my code from this: std::ifstream input_file(fname,std::ios::binary); boost::archive::binary_iarchive archive(input_file); std::vector<Sample> samples; archive>>samples; input_file.close(); Into this: std::ifstream input_file(fname,std::ios::binary); boost::archive::binary_iarchive archive(input_file); std::vector<Sample> samples; samples.reserve(8000); while(!input_file.eof()){ Sample s; archive>>s; samples.push_back(s); } input_file.close(); But every time it fails with an assert [1]. This seems to be when it has reached end of file. I cannot know in advance how many data samples I'll write to disk, so it seems I have three options: A: Make a special "zero" version of Sample to mark end of file B: At end of program 1 write the number of samples to a special file, and read that in before starting the above loop, so I know when to stop. C: Alter the serializer class to throw exception instead of an assert; I can then catch it and carry on. What is the best way? Is adding a terminator byte to boost::serializer an option (i.e. so I don't have to make a terminator version of each data structure I want to serialize). Incidentally my above loop does a copy of each sample as it adds it to the vector. Is the above code going to be common enough to make it worth including into boost::serializer as a helper function, which could then be optimized with in-place construction or something clever like that? (And then it could handle the termination handling itself as well.) Darren [1]: /usr/local/src/boost_1_31_0/boost/archive/basic_binary_iprimitive.hpp:119: void boost::archive::basic_binary_iprimitive<Archive, IStream>::load_binary(void*, unsigned int) [with Archive = boost::archive::binary_iarchive, IStream = std::basic_istream<char, std::char_traits<char> >]: Assertion `count == static_cast<size_t>(is.gcount())' failed.

2 1

RE: [boost] Re: regression test tools ported to vacpp
by Rozental, Gennadiy 07 Jun '04

07 Jun '04

> Gennadiy Rozental wrote: > > > "Toon Knapen" <toon.knapen(a)fft.be> wrote in message > > news:40BF16BA.8080009@fft.be... > > > >>The regression test tools are finally ported to vacpp version 6 and > >>the > > > > I just spend half an hour to clean my code after this port. I do > > understand that you just wanted to make it work on your > compiler. But > > lets in a future first discuss it here on a list before you commit > > this kind of modifications. > > > > There was a discussion about all these issues for about a > week now with > clearly marked in the header that it was about the test > library. Since > you only responded to the last of the series of mails, I thought you > were OK with the other modifications. Yes. I was Ok with basic_cstring modification. Not the rest. I was actually hoped you (somebody familiar with PP) find a fix to IBM VA issues with BOOST_PP. As it stands now I simply disable access control for this compiler, what is wrong. > Generally I always contact the developer personally before > commiting a > change. However, I already have tried to contact you privately but I > never got a reply. It could be. I had to abandon three of my accounts lately. One because I moved. One because it's no longer free. One because it became unusable due to spam. Use gennadiy <dot> rozental <at> thomson <dot> com. > Anyway, in the future I will again try to contact you before > committing > soth. > > May I also ask to elaborate on what you found not to be clean > about the > port and how and why you changed these changes. This might be > interesting information for future ports. You couldn't just manually unfold macro definition all over the code. It makes code look like a mess IMO. See, what I did in cvs. Regards, Gennadiy.

1 0

Borland and BOOST_NO_FUNCTION_TEMPLATE_ORDERING
by Robert Ramey 07 Jun '04

07 Jun '04

Using Borland 5.5 I compile the following: #ifdef BOOST_NO_FUNCTION_TEMPLATE_ORDERING #error "NO function template ordering supported" #else #error "YES function template ordering supported" #endif And I get: Fatal F1003 c:\boost_1_31_0\libs\serialization\test\test_zmisc.cpp 4: Error directive: "YES function template ordering supported" Yet C:\boost_1_31_0\more\borland_cpp.html seems to suggest that I should get NO. What am I missing here? Robert Ramey

2 1

RE: [boost] Re: input_iterator_facade (was: Re: [iterator facade] Why isdereference const?)
by Rozental, Gennadiy 07 Jun '04

07 Jun '04

> I guess we're back to your old disagreements about > flexibility. Flexibility is > fine, but it adds complexity, both for implementors and the user, so > flexibility should be added only when absolutely necessary. > The eof_iterator > is intented to be as simple as possible. > > - Volodya Actually after some consideration I incline to agree with you on most points. Look in cvs on how my version looks now. Gennadiy.

2 1

[assign library post review again]
by Thorsten Ottosen 07 Jun '04

07 Jun '04

Dear all, a new zip file can be downloaded in the files section in the assignment folder: http://f3.grp.yahoofs.com/v1/4NHBQHrGLW_kAudslwozEsdMfU_iY8Ja3CZu5N40ZMlffd… or http://tinyurl.com/27ldw We still need to find a name. If nobody else besides Pavel and Tom has an opinion, I will go for Boost.Assignment with a majority vote of 2 to 1. br Thorsten

3 3

Re:[boost] Serializer and when to stop
by Robert Ramey 06 Jun '04

06 Jun '04

Darren Cook wrote: > I'm using boost::serializer between two programs: the first creates the > data then serializes it, the second loads that data in and analyzes it. > When I had just 420 data samples it all worked fine: I create a > vector<Sample> in memory, serialized it and loaded it in the other side. > When I moved to 7000 data samples the first program, the creator, sucked > all the memory from the machine and then some. 7000 samples? That doesn't seem very large in today's environment. I can't see why creating an archive should consume any significant memory at all. Nothing significant is constructed. Perhaps is a bug somewhere. In any case, you do raise in interesting issue which would crop up if serialization were used in something like data logging. > So I moved to serializing each data sample as it was created. > To read it back in I changed my code from this: > std::ifstream input_file(fname,std::ios::binary); > boost::archive::binary_iarchive archive(input_file); > std::vector<Sample> samples; > archive>>samples; > input_file.close(); > Into this: > std::ifstream input_file(fname,std::ios::binary); > boost::archive::binary_iarchive archive(input_file); > std::vector<Sample> samples; > samples.reserve(8000); > while(!input_file.eof()){ > Sample s; > archive>>s; > samples.push_back(s); > } > input_file.close(); > But every time it fails with an assert [1]. This seems to be when it has > reached end of file. I cannot know in advance how many data samples I'll > write to disk, so it seems I have three options: > A: Make a special "zero" version of Sample to mark end of file > B: At end of program 1 write the number of samples to a special file, > and read that in before starting the above loop, so I know when to stop. > C: Alter the serializer class to throw exception instead of an assert; > I can then catch it and carry on. I think I changed the assert/exception to check before loading rather than after for the next version. This would permit your code above to work as you expect. I'll have to double check. > What is the best way? Is adding a terminator byte to boost::serializer an > option (i.e. so I don't have to make a terminator version of each data > structure I want to serialize). > Incidentally my above loop does a copy of each sample as it adds it to > the vector. Is the above code going to be common enough to make it worth > including into boost::serializer as a helper function, which could then > be optimized with in-place construction or something clever like that? > (And then it could handle the termination handling itself as well.) The default implementation of STL collections create a new element on the stack and copy into the collection. Its as clever as I can make it without getting tripped up in things like exception safety, objects without default constructors, etc. Robert Ramey

2 1

Wave/flex_string and replace< InputIterator > bug
by Reece Dunn 05 Jun '04

05 Jun '04

Hi all, I am implementing test cases to validate the implementation of my fixed_string class and basic_string_impl (based on flex_string) and noticed a defect in the implementation of: template< typename InputIterator > inline string_type & replace( iterator i1, iterator i2, InputIterator j1, InputIterator j2 ); It handles the Integral case fine, but when the InputIterators are actually iterators, it asserts false, where the standard dictates it returns: replace( i1, i2, basic_string( j1, j2 )); or equivalent behaviour, e.g. in the implementation I'm using: replace( i1, i2, std::distance( j1, j2 ), char_type()); std::copy( j1, j2, i1 ); Is there any specific reason for this, or is it an implementation bug? Regards, Reece _________________________________________________________________ Want to block unwanted pop-ups? Download the free MSN Toolbar now! http://toolbar.msn.co.uk/

3 2

Re: [Boost-announce] Formal Review Tribool
by Haim Cohen 05 Jun '04

05 Jun '04

Hi, I am happy to contribute my review for the tribool library. I am a software engineer developing EDA (Electronic Design Automation) software, so I think I can contribute my 2 cents to the review of tribool, as I think EDA tools will benefit from this library. I did not compile the library. I have made a quick reading of this library, but an in-depth study of this problem domain before. First of all, my first comment is about the library's name, as well as the class name. In logic circuit simulation, there are 4 values : 0, 1, x ( which is tribool referred to as undetermine ), and z ( represents high impedance - or disconnection, and sometime referred to as "tri state" ). So, I think users will find it quite confusing when they see the 'tri' in the library's name. Since the library implements a logic value which has some value that is unknown, I would suggest 'unknown bool' or 'x bool'. Now to the interface. I think that a macro which enables renaming of the x value should not be supplied. This will make people's code less standard. Same as we all know what 'true' and 'false' mean, we should adopt a unique name for 'unknown yet quite determined logic value'. I think the default 'undetermine' name is not a good name. My suggestions : logic_x, x_value, x_logic, unknwon_bool . I also think the default tribool constructor should create a tribool object which is 'undetermine'. (Not false, as in the documentation) Besides the above, I think the interface of the class is good and succinct. Some point for thought : Users might want arrays of tribools, with all the support of bitwise operations. Is this should be a part of tribool as well ? I do not know. Now I will refer to the implementation. In real-life problems, programs might make a heavy usage of the tribool class for simulations of logic circuits. A logic circuit (at least the combinatoric part) is a DAG where nodes are logic gates, and directed archs are wires. This graph might be the result of a synthesis process, where a high level language for hardware description (e.g Verilog) is compiled into this DAG. Note : This is a very general and simple description of the process. Life are much more difficult. So, outputs will be calculated by performing lexicographic order of the DAG, and calculating from inputs to outputs. I am telling this, to convince that performance is *very* important here. With a quick look at the implementation code, I think that calculating the value using branches (if else) will not result with good performance. Calculations with true/false values only is fast - CPU vendors dedicate special hardware for this, so calculation can be made in one machine cycle. Even bitwise calculation (up to some word width) is done in one machine cycle. Compilers are making use of this, and (hopefully) compiles boolean and bitwise operators to the appropriate instruction. Unfortunately, I did not see any CPU which supports built in true/false/X arithmetic. And even if there are such CPUs available, we will not be able to use it, since boost libraries should be portable. So, it is very tempting to think - how can we implement efficient true/false/X arithmetic using the existing boolean arithmetic ? One tempting idea is to represent true/false/X with some integers, and use bitwise arithmetic to get the right results. It seems there are no such integer values. I think I even managed to proof this. Since there is already a library implementing 4 values logic (0,1,x,z) - which is SystemC - I took a quick look on how they do it with their 4-states logic data type called sc_logic. They use an approach which I think should be also used in tribool. It seems that they maintain lookup tables for each operation: AND, OR, NOT, XOR (!=) and XNOR (==). As the internal representation of tribool is enumerated type, we can use the integer value of this enumerated type to access the lookup tables (primitive 2D fixed-size and fixed-values arrays) and get the result of the calculation. This will be much faster since most CPUs has a special addressing mode for accessing arrays. So, I suggest to change implementation to use class-static fixed array to perform calculations. I know that the spirit of Boost is portability and not performance, but since what is suggested here is actually very similar to a primitive data type which might be heavily used in applications, I think it is important to consider performance as well. I hope this review would be found useful. Haim -- _____________________________________________________ Haim Cohen Software Engineer haim.cohen(a)analog.com / haimcohen2002(a)hotmail.com Analog Devices Israel (972) - 9 - 9715 406

4 4

Re:RE: [boost] serialization request
by Robert Ramey 05 Jun '04

05 Jun '04

Jeff Garland wrote: > > Robert Ramey wrote: > > > > >Russell Balest wrote > > > > > >>Does someone know where in the code I can implement this easily. > > >>I just want to ignore the comments. > > >> > > >easy to accommodate. Just create a use the following in your > > >serialize > > >functions: > > > > > >Template<class Archive> > > >Void serialize(Archive, my_class & t){ > > > std::string comment; > > > ar & BOOST_NVP("comment", comment); > > > ar & BOOST_NVP("myclass_data", t);; > > >} > > > > > >In which case your XML output will look like: > > > > > >... > > ><comment></comment> > > ><myclass_data>..contents of t</myclass_data> ... > > > > > > > > > > > Good idea. I'll probably do that. > Another approach would be to preprocess your input files and strip the > added comments using XSLT, perl, or a simple regex program. That way you > don't have to modify the data structures under test. Someone might complain that my original method has 2 inconvenience features a) it requires modification to the serializations b) it includes the (null) comment string place holder in all the archives including the binary one. So another alternative would be to derive from the included xml_?archive classes included in the package and redefine the override for named-value pair so that it inserts/extracts  before or after every tag. This method has the following features: a) doesn't require any kluging up of class serializations b) has the advantage of automatically includes the  for all object instances c) has the disadvantage of automatically includes the  for all object instances d) doesn't add any overhead in other archives. If c) above is intolerable you would have the option of making your own wrapper similar to NVP - (NVT) that you would use in your serializations rather than NVP whenever you wanted a comment place holder. That's really a variation on my first suggestion. Take your pick and good luck Robert Ramey

1 0