Re: [boost] IO Collection Framework (Output Formatters REDESIGN)

From: Reece Dunn (msclrhd_at_[hidden])
Date: 2004-11-03 15:47:25

Vladimir Prus wrote:

 > On Thursday 28 October 2004 12:26, Reece Dunn wrote:
 >
 >> Hi All,
 >>
 >> I have been working on the library and have got a local 
reimplementation of
 >> my Output Formatters library based on the review comments. There are 
still
 >> a few issues that need to be resolved and I would like to expand stream
 >> state support.
 >>
 >> I have created a preliminary evolution document available at:
 >>   http://uk.geocities.com/msclrhd/iocoll/evolution.html
 >> that outlines what the new functionality looks like. This redesign
 >> addresses several of the issues brought up during the review.
 >
 >
 > Hi Reece, here are my comments on the design.
 >
 >> Use () if you want to get the value of a decoration:
 >>
 >>   sep() == "|"; // instrinsinc value
 >
 >
 > Why would I want to do this?

In case you need to get the value directly and not lookup based on the 
stream.

 >>   sep( std::cout ) == "|"; // stream default if sep() == std::string()
 >
 >
 > So, this means: 'value in sep if any, otherwise value in stream'?

Yes. There is currently a bug in the implementation if you use "" as a 
decorator value (i.e. no decoration) where it will choose the default 
value. I am planning to use "\1 def" - a value that is not likely to be 
used.

 >> If you want to see if a decorator is on an input stream, you can use 
match:
 >>
 >>   io::match( std::cin, sep );
 >
 >
 > What are the use cases for this?

It is primarily used to implement the read function of a format object. 
e.g.:

    template< class TraitsT, typename F, typename S >
    std::basic_istream< CharT, TraitsT > &
    in( std::basic_istream< CharT, TraitsT > & is, F & f, S & s ) const
    {
       if( !(
          match( is, open )      && first.read(  is, f ) &&
          match( is, separator ) && second.read( is, s ) &&
          match( is, close )
       ))
          is.setstate( std::ios_base::failbit );
       return( is );
    }

 >> boost::io::wrapper_decorators< CharT, RetType = void >
 >
 >
 > While I see why it's called "wrapper", the name still sounds a bit 
strange for me. Maybe, "open_close_decorators"?

That is better. I'll update the name.

 >> Holds an (open, close) decorator set.
 >>
 >>   boost::io::wrapper_decorators< char > wrap(  "< ", " >" );
 >>   boost::io::wrapper_decorators< char > wrap2( wrap );
 >>
 >>   wrap.decorate(  "[[ ", " ]]" );
 >>   wrap2.decorate( wrap );
 >
 >
 > Why do I need the 'decorate' method as opposed to
 >
 >     wrap = wrapper_decorators<char>("<", ">")
 >
 > or
 >     wrap.assign("[[", "]]")
 >
 > The idea that I can decorate a decorator does not seems good for me. 
Why do I need to change decorators in this way?

At the moment, the default values on the stream applies to all types. 
That is, the default value is associated with the decorator id argument.

I can see why decorate would not be a good name, but assign is not good 
either. Consider:

    io::object( vec ).decorate( " / " ); // decorate io::object
    io::object( vec ).assign( " / " ); // assign what?

What I am trying to aim for is:

    io::object( vec ).separator( " | " )
       .openclose( "< ", " >" );

but I need to work on implementability (i.e. maintaining the current 
functionality). Maybe 'set' would be better for setting the decoration 
based on all 3 values or another sequence_decoration object.

 > To change them, I need to access decorators stored on the stream. How 
do I do that?

Use io::decorate< CharT > or the c/wdecorate typedefs like this:

    std::cout << io::cdecorate( "[[ ", " ]]" )
       << io::object( vec ) << '\n';

 >> RATIONALE: RetType -- a class may derive from this class and you may 
want to
 >> chain function calls. For example:
 >>
 >>   derived_class dc;
 >>   dc.decorate( "/ ", " /" ).derived_fn();
 >>
 >> RetType is needed to allow this type of behaviour.

 > Do you have the need for it at the moment? If not, I suggest you drop 
this template argument. I had something similar in program_options and 
was messy, did not work everywhere and added no value.

It is needed: the format objects can be passed as arguments to other 
format objects. Without the above, this code will not work:

    io::object( vec, fmt::container().decorate( " : " ));
-------------------------------------^
[error]: cannot find function 'write' in 'sequence_decorators'.

 >> [3]: sequence decorators
 >>
 >>   boost::io::sequence_decorators< CharT, RetType = void >
 >>
 >> Holds an (open, close, separator) decorator set. Derives from
 >> wrapper_decorations.
 >>
 >>   io::sequence_decorators< char > seq1;
 >>   io::sequence_decorators< char > seq2( " : " );
 >>   io::sequence_decorators< char > seq3( "( ", " )" );
 >>   io::sequence_decorators< char > seq4( "[ ", " ]", " | " );
 >>   io::sequence_decorators< char > seq5( wrap );
 >
 >
 > Do I ever need to use this constructor?

Do you mean seq5( wrap )? I don't know. It would depend on what you are 
using the library for. I have tried to keep the same arguments for the 
constructors and decorator() function.

 >> ===== TYPE TRAITS AND INTROSPECTION
 >>
 >> [4] type traits
 >>
 >> In order to tell the framework about a type, you need to register it
 >> using:
 >>
 >>   BOOST_IO_CLASSIFY_TYPE( nargs, tempate_type, type_class )
 >>
 >> where
 >>   nargs is the number of template arguments the class has
 >>   template_type is the name of the class (e.g. std::list)
 >>   template_class is the type to which the class belongs
 >>      (e.g. boost::io::seq_container_type)
 >>
 >>   BOOST_IO_CLASSIFY_TYPE( 1, std::complex,  io::nary2value_type );
 >
 >
 > Cool.

:)

 >> [5] boost::io::is_XXX< T >
 >>
 >> These MPL objects assert whether T is of type XXX, e.g.:
 >
 >
 > Cool.

:)

 >> [6] n-ary objects
 >>
 >> An n-ary object (currently, 2-ary, 4-ary and 8-ary objects are 
supported)
 >> is a statically-sized sequence of n elements. There is support here for
 >> describing how one of these types is constructed and for 
manipulating its
 >> elements.
 >>
 >> In order to get the nth element of an n-ary object ob, you can use 
getval:
 >>
 >>   std::cout << io::getval< n >( ob ) << '\n';
 >
 >
 > Isn't 'getval' an implementation detail of the library?

In that it is used to implement the fmt::pair_t and fmt::nary_t classes.

 >> BOOST_IO_NARY_PARAM2 is used to register a 2-ary type as containing two
 >> different types for elements 1 and 2, e.g. std::pair. 
BOOST_IO_NARY_PARAM1
 >> is used in all other cases.
 >>
 >>   BOOST_IO_NARY_PARAM2( std::pair, seperable_pair )
 >>   BOOST_IO_NARY_PARAM1( std::complex, inseperable_pair< T > )
 >
 >
 > And what about 4-ary objects? Still BOOST_IO_NARY_PARAM1? What if 
there are 4 different template parameters?

    BOOST_IO_NARY_PARAM1( boost::math::quaternion, nary4_type< T > )
    BOOST_IO_NARY_PARAM1( boost::math::octonion, nary8_type< T > )

The current implementation of n-ary types only supports different 
template parameters on 2-ary types. What would be better is something like:

    naryN_type< T1, ..., Tn = Tn-1 >

The fmt::nary_t format object would need to be extended to support 
FormatObject1..N format objects to render the individual elements.

 >> This is implemented like this:
 >>
 >>   template< int n, typename T1, typename T2 >
 >>   typename io::detail::selector< n, T1, T2 >::ref_type
 >>   io::refval( std::pair< T1, T2 > & p )
 >>   {
 >>      return( detail::selector< n, T1, T2 >::ref( p.first, p.second ));
 >>   }
 >>
 >> where the ref function of detail::selector takes 2, 4 or 8 arguments and
 >> returns a reference to the nth argument.
 >>
 >> An inseperable pair (io::inseperable_pair< T1, T2 = T1 >), 4-ary
 >> (io::nary4_type< T >) or 8-ary (io::nary8_type< T >) object cannot 
obtain
 >> a reference to individual elements. They need to be set at the same time
 >> using io::assignval:
 >>
 >>   io::assignval( ob2ary, a, b );
 >>   io::assignval( ob4ary, a, b, c, d );
 >>   io::assignval( ob8ary, a, b, c, d, e, f, g, h );
 >
 >
 > I think you're again discussing implementation, not interface. I 
don't think this function should be initially present in the library 
public interface.

I see why you would classify it as implementation. It really belongs in 
an "extenders guide" rather than a "users guide".

For example, if you want to add support for NSPoint (a 2D position 
structure from the MacOS Cocoa API), you need to know this information, 
e.g.:

    template< int n >
    float io::getval( const NSPoint & pt )
    {
       return detail::selector< n, float >( pt.x, pt.y );
    }

Likewise, if you are writing a format object that allows a different 
seperator for each division in the output, e.g.:

    open e1 s1 e2 s2 e3 s3 e4 close
    where eX are the elements in the n-ary type
      and sX are the seperators

you need to know how to use getval, refval and assignval in order to 
implement the format object in a generic way.

 >> [7] automatic type deduction
 >>
 >> The fmt::deduce function:
 >>
 >>   template< typename CharT, typename T >
 >>   [undefined] fmt::deduce( const T & ob );
 >>
 >> will return an instance to the nested format object construct that 
maps to
 >> the class T.
 >
 >
 > Ok, we've got to interesting part. I don't understand what's "nested 
format object", what's "construct", and what does it mean for 
"construct" to map to some class. And you haven't defined "format 
object" yet.

What I was trying to say with the above sentence is something like:

The type associated with ob may be a simple type or a more complex type 
like a container or pair type. If the type is a container or other 
sequence-like type, the elements in this type may be sequence-like types.

A format object is a class that is responsible for reading from and 
writing to a stream. There are format objects for different types (like 
containers, arrays and pairs). These take template arguments that 
specify the format object used to render the elements in the sequence 
associated with the format object.

The type of the object ob passed to fmt::deduce is mapped to the format 
object that can render ob. For example, if
    T = std::pair< std::list< char >, std::complex< float > >
then fmt::deduce will return a format object of the type
    fmt::pair( fmt::container(), fmt::pair())

 >> The deduction mechanism involves registering the format object with 
a type class.
 >
 >
 > Maybe, you should call that "type category" to avoid confusion.

Yes.

 >> In order to save writing code when several types are implemented by 
a format object, you can map these types into a formatter
 >> class type:
 >>
 >>   SEQ_TYPE_TO_FORMATTER( nary2value_type, pair_type );
 >>   SEQ_TYPE_TO_FORMATTER( nary2base_type,  pair_type );
 >>   SEQ_TYPE_TO_FORMATTER( nary2int_type,   pair_type );
 >
 >
 > Hmm... what's nary2int_type, nary2base_type and nary2value_type? I 
though there's only one type category for n-ary objects, now I see two more.

The nary2xxx_type categories belong to the type deduction system. They 
are used to distinguish between the different typedefs for the element 
types. Ideally, this would be better as a type trait:

element_type< T >::type
where
    element_type< std::complex >::type == std::complex::value_type
    element_type< boost::interval >::type == interval::base_type
    element_type< boost::rational >::type == rational::int_type
    etc.

This would make the deduction system simpler in that you would only need 
nary2_type. The problem is how this works with things like std::pair. 
One solution would be to return an mpl type list in this instance.

 > [snip]
 >
 >> As a user, you don't need this, but if you are adding a format 
object into
 >> the type deduction framework, you will need this to extract the nested
 >> format object.
 >
 >
 > Could you rephrase the above?

Yes. I shall update the evolution.html file with the revised wording.

 >> A format object is deduced by defining fmt::detail::deduce_type for the
 >> formatter class type
 >
 >
 > What's "formatter class type"? Is that "type category"?

More like a collapsed type category. For example, the fmt::container_t 
format object supports seq_container_type (sequential containers), 
assoc_container_type (associative containers) and set_container_type 
(set containers). Thus, the format class type for fmt::container_t is 
seq_container_type. SEQ_TYPE_TO_FORMATTER is used to perform this collapse.

 >> that the format object handles. It has the form:
 >
 >
 > Say I've used SEQ_TYPE_TO_FORMATTER( nary2value_type, pair_type ) 
already. Do I need to define this?

You only need to register it once. If you have another decleration like 
this it will most likely result in a compilation error.

 >>   template<>
 >>   struct fmt::detail::deduce_type< type >
 >>   {
 >>      template< typename CharT, typename T >
 >
 >
 > What's 'T' here? The type being output?

Yes

 >> [8] format object generators
 >>
 >> The format objects provided by the library supply generator functions to
 >> simplify the creation of format objects.
 >>
 >> A generator gen will have the form gen< CharT >() to specify the 
character
 >> type used by the format object to store the decoration values. It 
will have
 >> the form gen() that is equivalent to gen< char >().
 >
 >
 > Such overloads (non-template + template) used to cause problems for 
me. IIRC, on borland.

I am not sure on this because wrapper_decorators is producing compile 
errors on borland (5.6 and 5.5). I want to sort these out as I revise 
the code.

 >> In addition, the format object may provide additional forms to allow 
format object nesting.
 >
 >
 > I did not understood what's generator and when it should be used.

A generator is a function that will return a format object. This is 
similar to make_pair or make_tuple. For example, instead of doing:

    fmt::pair_t< char, fmt::container_t< char > >()

you can use the generator functions like this:

    fmt::pair( fmt::container())

 >> [9] format objects
 >> <2> pair format object
 >>
 >> fmt::pair_t< CharT, First, Second > will render 2-ary object types using
 >> First to render the first element and Second to render the second. The
 >> generator function is fmt::pair() and has the additional form:
 >>
 >>   [unspecified] fmt::pair( const First & f, const Second & s = basic());
 >
 >
 > "unspecified"? Shouldn't that be fmt::pair_t<charT, First, Second>?

It is actually

    fmt::pair_t< typename First::char_type, First, Second >

I use "unspecified" because the types can be long and complex, for 
example, io::object returns:

    io::object_t
    <
       CharT, T, typename fmt::detail::get_deducer
       <
          CharT, T
       >::type::format_object
    >

and I don't want the return type to distract from the functions 
specification.

 > And BTW, I'm strongly against the "_t" suffix. Does it add any 
information or you just _t because 'pair' is already used? What about 
calling the type "pair_formatter"?

I will change the name in the update I am performing.

 > And, why do I ever need to create the formatter explicitly? Can I 
call any methods of it?

You can save a format style, for example:

    fmt::pair_t< char > parens;
    parens.decorate( "( ", " )" );

    std::cout << io::object( p1, parens ) // ( a, b )
              << io::object( p1 ) // [ a, b ]
              << io::object( p2, parens ); // ( a, b )

You can also access properties and functions associated with the format 
object. For example:

    fmt::pair_t< char, fmt::pair_t< char > > po;
    po.first.decorate( " : " );
    po.decorate( "<: ", " :>" );

    std::cout << io::object( p, po ); // <: [ a : b ], c :>

 >> <7> state format object
 >>
 >> fmt::state_t< CharT, StateObject, bool pre, FO > will render the state
 >
 > object
 >
 >> defined by StateObject before the object being read or written if 
pre is
 >
 > true.
 >
 > I think the name "state format" is a bit unclear. I'd suggest some 
more explicit pre_format_hook/post_format_hook, or something like that. 
BTW, some thinks (and I'd agree) that using 'bool' as parameter to 
function/template is not good. If I look at
 >
 >    state<Foobar, true>()
 >
 > I have no idea what 'true' means.

I shall find a better name. I am considering using an enumeration to 
define pre, post and pre_post values. For example:

    pre_post_hook< char, pre >() // [hook] elem
    pre_post_hook< char, post >() // elem [hook]
    pre_post_hook< char, pre_post >() // [hook] elem [hook]

It may be better to drop this functionality from the initial release of 
the library.

 >> <8> object manipulator
 >>
 >> io::object_t< CharT, T, FO > is a special type of format object in 
that it
 >> provides I/O manipulator functionality
 >
 >
 > What's "I/O manipulator functionality"? Do you mean stream 
insertion/extraction operators?

Yes.

 > Finally: you mentioned you have revised version of the code. Looks 
like it's not in sandbox. Is it possible to take a look?

I want to work on it further before committing it to the sandbox. The 
current version is now available at:

    http://uk.geocities.com/msclrhd/iocoll/formatter.zip

NOTE: This is a work in progress.

Regards,
Reece