- Boost - lists.boost.org

[Endian] Performance
by Phil Endecott 13 Sep '11

13 Sep '11

Dear All, I've just done some quick benchmarks of Beman's proposed byte-swapping code using the following test program: #include <cstdint> #include <iostream> #include <arpa/inet.h> // for htonl() using std::uint32_t; static inline uint32_t byteswap(uint32_t src) { #if defined(USE_GCC_BUILTIN) return __builtin_bswap32(src); #elif defined(USE_HTONL) return htonl(src); #elif defined(USE_REV_INSTR) uint32_t target; __asm__ ( "rev %0, %1\n" : "=r" (target) : "r" (src) ); return target; #elif defined(USE_BEMAN) const char* s (reinterpret_cast<const char*>(&src)); uint32_t target; char * t (reinterpret_cast<char*>(&target) + sizeof(target) - 1); *t = *s; *--t = *++s; *--t = *++s; *--t = *++s; return target; #else #error "Define USE_*" #endif } int main() { uint32_t s = 0; for (uint32_t i = 0; i < 1000000000; ++i) { s += byteswap(i); } std::cout << s << "\n"; // Expect 506498560 } I've tested this on two systems: (A): Marvell ARMv5TE ("Feroceon") @ 1.2 GHz, g++ 4.4 (B): Freescale ARMv7 (i.MX53, Coretex A8) @ 1.0 GHz, g++ 4.6.1 Compiled with: --std=c++0x -O4 Execution times in seconds are: A B USE_GCC_BUILTIN 29.4 3.0 USE_HTONL 12.6 3.0 USE_REV_INSTR n/a 3.0 USE_BEMAN 17.6 8.0 ARM architecture version 6 (IIRC) introduced a "rev" byte-swapping instruction. On system B, which supports this, it is used by the gcc builtin and glibc's htonl() giving better performance than Beman's code. On system A which does not support this instruction, the gcc builtin makes a library call; I've not looked at what htonl() does. What do people see on other platforms? How important do we consider performance to be for this library? Regards, Phil.

11 37

[stopwatches] About reducing the scope of the library
by Vicente J. Botet Escriba 13 Sep '11

13 Sep '11

Hi, I'm not completly satisfied with the reporting interface/design I did for Boost.Stopwatches. I need to think more about the design so it can provide Locale facilities to the user. Do you think that it is worth proposing an addition to Boost.Chrono that provides only stopwatches (without reporting)? Best, Vicente

4 9

Re: [boost] [Submission] RawMemory (24-bit integers and runtime-dispatch)
by Adder 13 Sep '11

13 Sep '11

Hi there ! And thank you for your time and for your questions. Before I answer the first question regarding 24-bit integers, etc., please allow me to kindly ask you: Where are you going to store the 24-bit integer that you read from "raw memory" (= untyped memory = array of bytes) ? Or where is the 24-bit integer that you plan to write to "raw memory" stored ? (In what type of variable ?) Regarding the second question, regarding run-time dispatching of operations (as opposed to compile-time dispatching), I will start with a couple of examples that supports your question even more: It is not just the endian-ness that might not be known at compile-time. The alignment of data might also not be known before explicitly checking it at run-time. Let me quote from the ugly documentation that I have written (http://adder.iworks.ro/Boost/RawMemory/#UsersManual) "When we are dealing with an array of values (i.e. contiguous values) and performance is critical, we might want to check whether the data is aligned and specifically demand unchecked_memory_xfer if it is." Ultimately, the number of bytes that are used to represent the integer may be variable. Such is the case with the "mapping pair offsets" used by NTFS to locate the contiguous runs of clusters that make up the (non-MFT resident portion of the) file. In order to save precious MFT (Master File Table) entry space, signed offsets are stored in variable-sized (1 to 8-byte long) integers. My answer is probably obvious by now. I am thinking about letting the programmer use the ultimate weapon she has in her toolchest: The mighty "if" instruction. For the specific example you kindly gave, I am thinking that the following syntax might be useful for reading UCS-2 characters from a file saved with Notepad (in "Unicode" or "Unicode Big Endian" format): const uint16_t nByteOrderMask = raw_memory <uint16_t, little_endian>::peek (pFileBuffer); const bool bBigEndian = nByteOrderMask == 0xFEFF; ... // The loop: { const uint16_t cCurrent = bBigEndian ? raw_memory <uint16_t, big_endian>::peek (pCurrent) : raw_memory <uint16_t, little_endian>::peek (pCurrent); ... } I await your thoughts... Thank you again ! (-: -- Yours truly, Adder On 9/7/11, Joe Mucchiello <jmucchiello(a)yahoo.com> wrote: > For a library designed to deal with the weirdness of binary file formats, > I'm surprised it does not support 24-bit integers of arbitrary > alignment/sign. > >> uint32_t x = raw_memory <uint32_t, big_endian>::peek (&vbBuffer [5]); > > Do you also support a non-template "endianness" parameter? Sometimes you > don't know the endian-style of the file until you open it and read some > value out of it. > > Joe > > _______________________________________________ > Unsubscribe & other changes: > http://lists.boost.org/mailman/listinfo.cgi/boost >

2 5

[Endian] Review
by Tomas Puverle 12 Sep '11

12 Sep '11

Hi Beman, > - What is your evaluation of the design? The design of the integer types seems reasonable to me, even though I would prefer not to have the overloaded operators on them. I feel that having the operators has the potential to break the boundary between program input and the program's processing. What do you I mean by that? Typically, I try to structure my programs as follows: 1) Read input data 2) Verify input invariants, construct internal representation 3) Process internal representation 4) Output result The problem with having operators in the endian types is that they encourage the user to continue using them past the point 2). However, as I said, that's my personal preference and last year plenty of people wanted to see the operators included. Perhaps they could be a policy? endian<..., bool EnableOperators_ > ? As for the rest of the library, I have the following comments: - As many other people have mentioned, floating point support and user defined types are a must. - I am happy that the "conversion functions" have been added, Unfortunately, I find them lacking in several departments: - (Note: I am going to try very hard not to get drawn into another naming discussion. That's pretty much what ended my efforts last year - 50 people commenting, each proposing a different set of names :D ) - While the endian types support unaligned data, the endian functions don't appear to do so. I don't see how the endian types could be implemented using them. - No support for user defined types, arrays/ranges of user defined types etc. This is extremely useful. A frequent example I come across is the following (using our syntax): //load a file into memory //... swapInPlace<BigToNative>(data.begin(), data.end()); Depending on the platform, that last statement may get compiled out to a no-op. This is also the big reason why our version of "reorder()" is not unconditional but depends on the "swap type". - reorder(x) will work fine for integral types, however, has potential to cause issues for e.g. floating types. This has been discussed extensively during my proposal last year. Also, from my practical experience, having a floating point number as an input, I find the assembly typically includes (unncecessary) floating point load/store instructions. Something along the lines of e.g.: template<typename T_> T_ big_to_native(raw_memory<T_>) and template<typename T_> swapped_data<T_> native_to_big(T_) might be better and result in better performance. Of course, those two function signatures don't solve the "reorder" signature, but that could be changed correspondingly. - reorder() should use platform specific instructions (such as bswap) for doing endian swapping, otherwise falling back on one of the (many) variants which are currently being discussed. - I think the alternatives such as little_to_big() and big_to_little() are also needed. - I would like to see a few standard traits/typedefs so that I don't have to keep checking pre-processor macros, such as typedef boost::mpl::true_/false_ platform_is_big_endian; or something similar. - While I haven't needed this myself, last year someone mentioned the need for a "general reorder" functionality. Rather than simply dealing with little/big endianness, allow the user to specify ANY byte ordering for the specific data size. - I would like to see an endian_swap_iterator (and a raw_endian_iterator, which can iterate over unaligned data, too) > - What is your evaluation of the implementation? Other than the implementation of reorder() and its performance, which is already being discussed on the list, (and the inclusion of platform specific instructions, as per above) I would like to see endian<> to be implemented in terms of the "free" functions. It seems strange to have two implementations of the same task. > - What is your evaluation of the documentation? Fine. > - What is your evaluation of the potential usefulness of the > library? Very useful, long overdue in boost. > - Did you try to use the library? With what compiler? Did you > have any problems? Tried the integer types last year, no problems. Didn't try the "new" swapping functions. > - How much effort did you put into your evaluation? A glance? A > quick reading? In-depth study? I spent a fair bit of time looking through the docs, the code and reading the followup posts etc. > - Are you knowledgeable about the problem domain? Reasonably. > And finally, every review should answer this question: > > - Do you think the library should be accepted as a Boost library? I see the library as having two pieces, both useful to different people. I would say conditionally accept the endian<> class if floating point support is added. I would vote no for the inclusion of the free function interfaces in their current form. Thanks! Tom

6 14

[graph][iterators] csr graph and reverse iterator problem
by Takatoshi Kondo 11 Sep '11

11 Sep '11

Hi, All I'm using Boost.Graph and Boost.Iterators. When I use the compressed_sparse_row_graph with reverse_iterator, I ran into a problem. When I dereference the reverse_iterator of compressed_sparse_row_graph's iterator, invalid memory access occurs. On g++ version 4.6.0 with -Os option, The output is below. 0 1 1 0 0 1 134526756 3218710936 1 0 And I also checked where the invalid memory access occurred using valgrind. ==2341== Invalid read of size 4 ==2341== at 0x8048DE7: test_csr_graph() (rev_iter.cpp:45) ==2341== by 0x8048FB3: main (rev_iter.cpp:59) ==2341== Address 0xbe825f88 is just below the stack ptr. To suppress, use: --workaround-gcc296-bugs=yes ==2341== 1 ==2341== Invalid read of size 4 ==2341== at 0x8048E31: test_csr_graph() (rev_iter.cpp:46) ==2341== by 0x8048FB3: main (rev_iter.cpp:59) ==2341== Address 0xbe825f88 is just below the stack ptr. To suppress, use: --workaround-gcc296-bugs=yes ==2341== This result indicates the code below is suspicious. std::cout << *rib++ << std::endl; // NG Why? Line 45 std::cout << *rib++ << std::endl; // NG Why? Line 46 But... boost::tie(ib, ie) = boost::vertices(g); std::cout << *--ie << std::endl; // OK std::cout << *--ie << std::endl; // OK It is OK. Here is the source code that reproduces the problem. ------------ begin ------------- #include <boost/iterator/reverse_iterator.hpp> #include <boost/graph/adjacency_list.hpp> #include <boost/graph/compressed_sparse_row_graph.hpp> #include <iostream> #include <cassert> void test_graph() { typedef boost::adjacency_list<> graph_type; graph_type g(2); boost::graph_traits<graph_type>::vertex_iterator ib, ie; { boost::tie(ib, ie) = boost::vertices(g); std::cout << *ib++ << std::endl; // OK std::cout << *ib++ << std::endl; // OK assert(ib == ie); } { boost::tie(ib, ie) = boost::vertices(g); boost::reverse_iterator<boost::graph_traits<graph_type>::vertex_iterator> rib(ie); boost::reverse_iterator<boost::graph_traits<graph_type>::vertex_iterator> rie(ib); std::cout << *rib++ << std::endl; // OK std::cout << *rib++ << std::endl; // OK assert(rib == rie); } } void test_csr_graph() { typedef boost::compressed_sparse_row_graph<> graph_type; std::pair<int, int> e(0, 1); std::pair<int, int> es[] = { e }; graph_type g(boost::edges_are_unsorted, &es[0], &es[1], 2); boost::graph_traits<graph_type>::vertex_iterator ib, ie; { boost::tie(ib, ie) = boost::vertices(g); std::cout << *ib++ << std::endl; // OK std::cout << *ib++ << std::endl; // OK assert(ib == ie); } { boost::tie(ib, ie) = boost::vertices(g); boost::reverse_iterator<boost::graph_traits<graph_type>::vertex_iterator> rib(ie); boost::reverse_iterator<boost::graph_traits<graph_type>::vertex_iterator> rie(ib); std::cout << *rib++ << std::endl; // NG Why? Line 45 std::cout << *rib++ << std::endl; // NG Why? Line 46 assert(rib == rie); } { boost::tie(ib, ie) = boost::vertices(g); std::cout << *--ie << std::endl; // OK std::cout << *--ie << std::endl; // OK assert(ib == ie); } } int main() { test_graph(); test_csr_graph(); } ------------ end ------------- Why does the invalid memory access occur? Am I misusing the libraries? Thanks, Takatoshi

5 10

[type_erasure] ODR violation promotion
by Vicente J. Botet Escriba 10 Sep '11

10 Sep '11

Hi Steven, I'm interested in your TypeErasure library. Where can I find the on going version? The example for concept map seams not complete, as the specialization for less_than_comparable<std::type_info> is not used in any type erasure type. Could you complete it? Before the Boost.Conversion review someone signaled that the Boost.Conversion review was promoting ODR violations as two libraries could specialize a conversion for types S and T and make them incompatible for an end user as the ODR is violated. I think that your library suffer from the same issue, when two libraries need to specialize a concept provided by the library for a 3rd part concrete class using the proposed concept map mechanism. Do you agree? If not, how do the library avoids this kind of issue? Best, Vicente

3 10

[fusion] Algorithm/auxillary documentation
by Daniel James 10 Sep '11

10 Sep '11

Hi, The generated documentation for the algorithm/auxillary section hasn't been checked in. It should be at: http://svn.boost.org/svn/boost/trunk/libs/fusion/doc/html/fusion/algorithm/… http://svn.boost.org/svn/boost/trunk/libs/fusion/doc/html/fusion/algorithm/… Daniel

4 10

[Review] Boost.Endian i/o
by Vicente J. Botet Escriba 10 Sep '11

10 Sep '11

Hi, previous releases of Boost.Endian included a binary stream. Is there something else that can be used to get the same behavior? Best, Vicente

2 1

[Endian] Unaligned fields in packed structs
by Phil Endecott 10 Sep '11

10 Sep '11

Dear All, One of my all-time worst bugs was due to calling htonl() on an unaligned field in a packed struct. It worked in debug mode but not in production code, and there were no compiler warnings about the broken alignment requirement. So I thought I'd try this: inline void inplace_byteswap(uint32_t& val) { val = (val>>24) | ((val>>8) & 0x0000ff00) | ((val<<8) & 0x00ff0000) | (val<<24); } struct S { uint16_t a; uint32_t b; } __attribute__ ((packed)); int main() { S s; s.b = 0x12345678; byteswap(s.b); } This fails to compile: "error: cannot bind packed field ‘s.S::b’ to ‘uint32_t& {aka unsigned int&}’". So it's good that it doesn't just fail unpredictably at runtime like my old nightmare, but it's not ideal. If instead I write: inline uint32_t byteswap(uint32_t val) { return (val>>24) | ((val>>8) & 0x0000ff00) | ((val<<8) & 0x00ff0000) | (val<<24); } ... s.b = byteswap(s.b); ... then of course it does work; the unaligned accesses don't "escape" from where the compiler knows about them and can apply fixups. Anyway, before I tried this I was already wondering about the choice of function signatures: void endian::reorder(T&); // in-place byteswap void endian::reorder(T src, T& dest); // out-of-place byteswap What is the rationale (or precedent) for these and not T endian::reorder(T); ? If sin(theta) returns its result, why shouldn't reorder(val) do so too? Regards, Phil.

3 4

[boost/foreach] What is wrong with BOOST_FOREACH(const std::string &name, m_modules)?
by Demetrio Car 10 Sep '11

10 Sep '11

I am trying to compile this example http://www.boost.org/doc/libs/1_42_0/doc/html/boost_propertytree/tutorial.h… and I am getting the error "/usr/local/boost_1_47_0/boost/property_tree/detail/ptree_implementation.hpp:795:54: error: request for member ‘put_value’ in ‘tr’, which is of non-class type ‘bool’" in this line: BOOST_FOREACH(const std::string &name, m_modules){ pt.put("debug.modules.module", name, true); } My full error message is: make all Building file: ../src/Boost_property_tree.cpp Invoking: GCC C++ Compiler g++ -I/home/mori/Demetrio/WSs/Test/VanManagerParser/json_spirit -I/usr/local/boost_1_47_0/ -I/usr/local/boost_1_47_0/boost/spirit -I/usr/local/curl-7.21.7/ -O0 -g3 -Wall -c -fmessage-length=0 -MMD -MP -MF"src/Boost_property_tree.d" -MT"src/Boost_property_tree.d" -o "src/Boost_property_tree.o" "../src/Boost_property_tree.cpp" ../src/Boost_property_tree.cpp: In member function ‘void debug_settings::save(const std::string&)’: ../src/Boost_property_tree.cpp:80:5: error: invalid initialization of reference of type ‘std::string&’ from expression of type ‘const std::basic_string<char>’ In file included from /usr/local/boost_1_47_0/boost/property_tree/ptree.hpp:516:0, from ../src/Boost_property_tree.cpp:4: /usr/local/boost_1_47_0/boost/property_tree/detail/ptree_implementation.hpp: In member function ‘void boost::property_tree::basic_ptree<Key, Data, KeyCompare>::put_value(const Type&, Translator) [with Type = std::basic_string<char>, Translator = bool, Key = std::basic_string<char>, Data = std::basic_string<char>, KeyCompare = std::less<std::basic_string<char> >]’: /usr/local/boost_1_47_0/boost/property_tree/detail/ptree_implementation.hpp:817:13: instantiated from ‘boost::property_tree::basic_ptree<K, D, C>& boost::property_tree::basic_ptree<Key, Data, KeyCompare>::put(boost::property_tree::basic_ptree<Key, Data, KeyCompare>::path_type&, const Type&, Translator) [with Type = std::basic_string<char>, Translator = bool, Key = std::basic_string<char>, Data = std::basic_string<char>, KeyCompare = std::less<std::basic_string<char> >, boost::property_tree::basic_ptree<Key, Data, KeyCompare>::path_type = boost::property_tree::string_path<std::basic_string<char>, boost::property_tree::id_translator<std::basic_string<char> > >]’ ../src/Boost_property_tree.cpp:81:50: instantiated from here /usr/local/boost_1_47_0/boost/property_tree/detail/ptree_implementation.hpp:795:54: error: request for member ‘put_value’ in ‘tr’, which is of non-class type ‘bool’ make: *** [src/Boost_property_tree.o] Error 1 **** Build Finished ****

4 5