Re: [boost] IO Collection Framework (Output Formatters REDESIGN)

From: Vladimir Prus (ghost_at_[hidden])
Date: 2004-11-04 03:25:06

[I cc the mailing list] 

> >>Use () if you want to get the value of a decoration:
> >>
> >>   sep() == "|"; // instrinsinc value
> >
> > Why would I want to do this?
>
> In case you need to get the value directly and not lookup based on the
> stream.

Does the user ever need to do it? Or it's internal mechanism?

> >>   sep( std::cout ) == "|"; // stream default if sep() == std::string()
> >
> > So, this means: 'value in sep if any, otherwise value in stream'?
>
> Yes. There is currently a bug in the implementation if you use "" as a
> decorator value (i.e. no decoration) where it will choose the default
> value. I am planning to use "\1 def" - a value that is not likely to be
> used.

Why not fixing the bug? Using "\1 def" for empty decorator is not nice.

> >>If you want to see if a decorator is on an input stream, you can use
> >> match:
> >>
> >>   io::match( std::cin, sep );
> >
> > What are the use cases for this?
>
> It is primarily used to implement the read function of a format object.
> e.g.:
>
>     template< class TraitsT, typename F, typename S >
>     std::basic_istream< CharT, TraitsT > &
>     in( std::basic_istream< CharT, TraitsT > & is, F & f, S & s ) const
>     {
>        if( !(
>           match( is, open )      && first.read(  is, f ) &&

Ouch! So 'match' in fact reads 'sep' from the input stream. Your original 
comment made me thing that it somehow checks that 'sep' is equal to the 
decorator associated with the stream -- and that made me wonder why it's 
needed.

> >>   wrap.decorate(  "[[ ", " ]]" );
> >>   wrap2.decorate( wrap );
> >
> > Why do I need the 'decorate' method as opposed to
> >
> >     wrap = wrapper_decorators<char>("<", ">")
> >
> > or
> >
> >     wrap.assign("[[", "]]")
> >
> > The idea that I can decorate a decorator does not seems good for me. Why
> > do I need to change decorators in this way?
>
> At the moment, the default values on the stream applies to all types.
> That is, the default value is associated with the decorator id argument.
>
> I can see why decorate would not be a good name, but assign is not good
> either. Consider:
>
>     io::object( vec ).decorate( " / " ); // decorate io::object
>     io::object( vec ).assign( " / " ); // assign what?
>
> What I am trying to aim for is:
>
>     io::object( vec ).separator( " | " )
>        .openclose( "< ", " >" );
>
> but I need to work on implementability (i.e. maintaining the current
> functionality). Maybe 'set' would be better for setting the decoration
> based on all 3 values or another sequence_decoration object.

Probably. I agree that 'assign' in above example is not good. 

> >To change them, I need to access
> > decorators stored on the stream. How do I do that?
>
> Use io::decorate< CharT > or the c/wdecorate typedefs like this:
>
>     std::cout << io::cdecorate( "[[ ", " ]]" )
>        << io::object( vec ) << '\n';
>
> >>RATIONALE: RetType -- a class may derive from this class and you may want
> >> to chain function calls. For example:
> >>
> >>   derived_class dc;
> >>   dc.decorate( "/ ", " /" ).derived_fn();
> >>
> >>RetType is needed to allow this type of behaviour.
> >
> > Do you have the need for it at the moment? If not, I suggest you drop
> > this template argument. I had something similar in program_options and
> > was messy, did not work everywhere and added no value.
>
> It is needed: the format objects can be passed as arguments to other
> format objects. Without the above, this code will not work:
>
>     io::object( vec, fmt::container().decorate( " : " ));
> -------------------------------------^
> [error]: cannot find function 'write' in 'sequence_decorators'.

You mean that container_formatter derives from sequence_decorators and without 
RetType the value of fmt::container().decorate() will be sequence_decorators. 
Looks reasonable, but why not:

   fmt::container(open_close_delimiters("[", "]"))

> >>   io::sequence_decorators< char > seq5( wrap );
> >
> > Do I ever need to use this constructor?
>
> Do you mean seq5( wrap )? I don't know. It would depend on what you are
> using the library for. I have tried to keep the same arguments for the
> constructors and decorator() function.

I suspect that user will always specify formatting manually and has no need to 
convert formatting like this.

> >>[6] n-ary objects
> >>
> >>An n-ary object (currently, 2-ary, 4-ary and 8-ary objects are supported)
> >>is a statically-sized sequence of n elements. There is support here for
> >>describing how one of these types is constructed and for manipulating its
> >>elements.
> >>
> >>In order to get the nth element of an n-ary object ob, you can use
> >> getval:
> >>
> >>   std::cout << io::getval< n >( ob ) << '\n';
> >
> > Isn't 'getval' an implementation detail of the library?
>
> In that it is used to implement the fmt::pair_t and fmt::nary_t classes.

I'd suggest you clearly document that it's implementation detail to reduce the 
amount of information which must be read.

> >>BOOST_IO_NARY_PARAM2 is used to register a 2-ary type as containing two
> >>different types for elements 1 and 2, e.g. std::pair.
> >> BOOST_IO_NARY_PARAM1 is used in all other cases.
> >>
> >>   BOOST_IO_NARY_PARAM2( std::pair, seperable_pair )
> >>   BOOST_IO_NARY_PARAM1( std::complex, inseperable_pair< T > )
> >
> > And what about 4-ary objects? Still BOOST_IO_NARY_PARAM1? What if there
> > are 4 different template parameters?
>
>     BOOST_IO_NARY_PARAM1( boost::math::quaternion, nary4_type< T > )
>     BOOST_IO_NARY_PARAM1( boost::math::octonion, nary8_type< T > )
>
> The current implementation of n-ary types only supports different
> template parameters on 2-ary types. What would be better is something like:
>
>     naryN_type< T1, ..., Tn = Tn-1 >
>
> The fmt::nary_t format object would need to be extended to support
> FormatObject1..N format objects to render the individual elements.

Right. I wonder what are you going to do with general tuple<>

> >>   io::assignval( ob2ary, a, b );
> >>   io::assignval( ob4ary, a, b, c, d );
> >>   io::assignval( ob8ary, a, b, c, d, e, f, g, h );
> >
> > I think you're again discussing implementation, not interface. I don't
> > think this function should be initially present in the library public
> > interface.
>
> I see why you would classify it as implementation. It really belongs in
> an "extenders guide" rather than a "users guide".

Exactly!

> For example, if you want to add support for NSPoint (a 2D position
> structure from the MacOS Cocoa API), you need to know this information,
> e.g.:
>
>     template< int n >
>     float io::getval( const NSPoint & pt )
>     {
>        return detail::selector< n, float >( pt.x, pt.y );
>     }
>
> Likewise, if you are writing a format object that allows a different
> seperator for each division in the output, e.g.:
>
>     open e1 s1 e2 s2 e3 s3 e4 close
>     where eX are the elements in the n-ary type
>       and sX are the seperators
>
> you need to know how to use getval, refval and assignval in order to
> implement the format object in a generic way.

Understood. 

> >>[7] automatic type deduction
> >>
> >>The fmt::deduce function:
> >>
> >>   template< typename CharT, typename T >
> >>   [undefined] fmt::deduce( const T & ob );
> >>
> >>will return an instance to the nested format object construct that maps
> >> to the class T.
> >
> > Ok, we've got to interesting part. I don't understand what's "nested
> > format object", what's "construct", and what does it mean for "construct"
> > to map to some class. And you haven't defined "format object" yet.
>
> What I was trying to say with the above sentence is something like:
>
> The type associated with ob may be a simple type or a more complex type

"Type associated with ob"? Is it different from "type of ob"? 

> like a container or pair type. 
> If the type is a container or other 
> sequence-like type, the elements in this type may be sequence-like types.
>
> A format object is a class that is responsible for reading from and
> writing to a stream. There are format objects for different types (like
> containers, arrays and pairs). These take template arguments that
> specify the format object used to render the elements in the sequence
> associated with the format object.
>
> The type of the object ob passed to fmt::deduce is mapped to the format
> object that can render ob. For example, if
>     T = std::pair< std::list< char >, std::complex< float > >
> then fmt::deduce will return a format object of the type
>     fmt::pair( fmt::container(), fmt::pair())

Now, that is perfectly clear. One question though, if I use deduce(obj), what 
I can do with the resulting object, except for using it for output? Can I 
access the nested formatter? If yes, how? If not, then you don't need to 
deduce nested formatters, the above could return:

   fmt::pair(fmt::generic_formatter(), fmt::generic_formatter())

where generic_formatter() just calls deduce on the type it's passed and use 
the returned formatter. In other words: you would not need to have two 
template parameters for pair_t.

> >>In order to save writing code when several types are
> >>implemented by a format object, you can map these types into a formatter
> >>class type:
> >>
> >>   SEQ_TYPE_TO_FORMATTER( nary2value_type, pair_type );
> >>   SEQ_TYPE_TO_FORMATTER( nary2base_type,  pair_type );
> >>   SEQ_TYPE_TO_FORMATTER( nary2int_type,   pair_type );
> >
> > Hmm... what's nary2int_type, nary2base_type and nary2value_type? I though
> > there's only one type category for n-ary objects, now I see two more.
>
> The nary2xxx_type categories belong to the type deduction system. They
> are used to distinguish between the different typedefs for the element
> types. Ideally, this would be better as a type trait:
>
> element_type< T >::type
> where
>     element_type< std::complex >::type == std::complex::value_type
>     element_type< boost::interval >::type == interval::base_type
>     element_type< boost::rational >::type == rational::int_type
>     etc.

Understood.

> This would make the deduction system simpler in that you would only need
> nary2_type. The problem is how this works with things like std::pair.
> One solution would be to return an mpl type list in this instance.

How does it work now?

> >>As a user, you don't need this, but if you are adding a format object
> >> into the type deduction framework, you will need this to extract the
> >> nested format object.
> >
> > Could you rephrase the above?
>
> Yes. I shall update the evolution.html file with the revised wording.

Thanks.

> >>A format object is deduced by defining fmt::detail::deduce_type for the
> >>formatter class type
> >
> > What's "formatter class type"? Is that "type category"?
>
> More like a collapsed type category. For example, the fmt::container_t
> format object supports seq_container_type (sequential containers),
> assoc_container_type (associative containers) and set_container_type
> (set containers). Thus, the format class type for fmt::container_t is
> seq_container_type. SEQ_TYPE_TO_FORMATTER is used to perform this collapse.

Maybe using jsut "formatter class" would be clearer. And a overview describing 
the interaction between components is a must. Say, a diagram like:

   output:
         object   ---- (deduce) ---> formatter --- (operator<<) ---> stream

   deduce:
         type - (is_XXX) -> type_category -- (???) ---- > formatter class

where arrows join main concepts and paranthethis show the mechanism which 
converts between those concepts.

> >>   template<>
> >>   struct fmt::detail::deduce_type< type >
> >>   {
> >>      template< typename CharT, typename T >
> >
> > What's 'T' here? The type being output?
>
> Yes

I'm lost again. You have:

   fmt::detail::get_deducer< CharT, T >::type

and 

   template<>
   struct fmt::detail::deduce_type< type >

what is the relationship between the two?

> >>In addition, the
> >>format object may provide additional forms to allow format object
> >> nesting.
> >
> > I did not understood what's generator and when it should be used.
>
> A generator is a function that will return a format object. This is
> similar to make_pair or make_tuple. For example, instead of doing:
>
>     fmt::pair_t< char, fmt::container_t< char > >()
>
> you can use the generator functions like this:
>
>     fmt::pair( fmt::container())

You know what I'm  going to say: a comment is in order.

> >>[9] format objects
> >><2> pair format object
> >>
> >>fmt::pair_t< CharT, First, Second > will render 2-ary object types using
> >>First to render the first element and Second to render the second. The
> >>generator function is fmt::pair() and has the additional form:
> >>
> >>   [unspecified] fmt::pair( const First & f, const Second & s = basic());
> >
> > "unspecified"? Shouldn't that be fmt::pair_t<charT, First, Second>?
>
> It is actually
>
>     fmt::pair_t< typename First::char_type, First, Second >
>
> I use "unspecified" because the types can be long and complex, for
> example, io::object returns:
>
>     io::object_t
>     <
>        CharT, T, typename fmt::detail::get_deducer
>        <
>           CharT, T
>
>        >::type::format_object
>
> and I don't want the return type to distract from the functions
> specification.

OK.

> > And BTW,
> > I'm strongly against the "_t" suffix. Does it add any information or you
> > just _t because 'pair' is already used? What about calling the type
> > "pair_formatter"?
>
> I will change the name in the update I am performing.

Good.

> > And, why do I ever need to create the formatter explicitly? Can I call
> > any methods of it?
>
> You can save a format style, for example:
>
>     fmt::pair_t< char > parens;
>     parens.decorate( "( ", " )" );
>
>     std::cout << io::object( p1, parens ) // ( a, b )
>               << io::object( p1 ) // [ a, b ]
>               << io::object( p2, parens ); // ( a, b )
>
> You can also access properties and functions associated with the format
> object. For example:
>
>     fmt::pair_t< char, fmt::pair_t< char > > po;
>     po.first.decorate( " : " );
>     po.decorate( "<: ", " :>" );
>
>     std::cout << io::object( p, po ); // <: [ a : b ], c :>

Ok. I'm not sure how usefull it is, but we get this for free.

> > I think the name "state format" is a bit unclear. I'd suggest some more
> > explicit pre_format_hook/post_format_hook, or something like that. BTW,
> > some thinks (and I'd agree) that using 'bool' as parameter to
> > function/template is not good. If I look at
> >
> >    state<Foobar, true>()
> >
> > I have no idea what 'true' means.
>
> I shall find a better name. I am considering using an enumeration to
> define pre, post and pre_post values. For example:
>
>     pre_post_hook< char, pre >() // [hook] elem
>     pre_post_hook< char, post >() // elem [hook]
>     pre_post_hook< char, pre_post >() // [hook] elem [hook]
>
> It may be better to drop this functionality from the initial release of
> the library.

I think I'd like to be able to output vectors with prepended indices, but I 
don't know if such generic mechanism is needed. You can have 
"numbering_formatter", for example, which will output a number of then use 
nested formatter to output the object itself. If creating formatting is 
simple, this is better that pre_post_hook, since it's more general.

> > Finally: you mentioned you have revised version of the code. Looks like
> > it's not in sandbox. Is it possible to take a look?
>
> I want to work on it further before committing it to the sandbox. 

You can work in sandbox on a branch, if you don't want to break existing 
version. There's nothing wrong with adding unfinished version in sandbox -- 
nobody excepts it to be stable.

> I am 
> going to upload the files to my website shortly.

Would be good!

- Volodya