Re: [Boost-docs] Sphinx integration

Subject: Re: [Boost-docs] Sphinx integration
From: Mateusz Loskot (mateusz_at_[hidden])
Date: 2011-10-04 20:49:46

On 02/10/11 23:03, Dave Abrahams wrote:
> on Thu Sep 29 2011, Mateusz Loskot <mateusz-AT-loskot.net> wrote:
>> On 28/09/11 17:09, John Maddock wrote:
>>>
>>> And lets not forget that some of us really like quickbook ;-)
>>
>> I do like quickbook too.
>> Although may intention was to discuss the workflow of documenting Boost,
>> I think I have identified what is the biggest pain to me: Doxygen
>> integration in BoostBook.
>>
>> Doxygen is widely used C++ code documenting tool.
>> Fortunately or unfortunately, it is also used by many Boost authors.
>>
>> Doxygen does not support many of the terms/concepts/elements used across
>> Boost libraries. It is also becasue it is almost impossible to document
>> them well at source code levels.
>> However, things like "concept and model" exist at source code level,
>   do not?---------------------------------^
>
> That's true of concepts unless you're using Boost.ConceptCheck.  But
> even so, they will only be seen by Doxygen as regular C++ and not as the
> abstractions they represent.  And there are no model declarations. :-(

I understand your question, but I meant it as it reads.
I assumed concepts have representation in C++ code, either using
Boost.ConceptCheck or custom implementation of concepts.
So, I assumed they are documented and linked with corresponding models.
The documentation is either external (e.g. .xml, .qbk) or internal to
source code, it is Doxygen comments).

>> So, authors invent custom processors and tools to integrate. We have:
>> - bits of support in QuickBook (BoostBook) e.g. xinclude autodoc.xml
>> - Perl scripts (e.g. ref_tag_fix.pl in DateTime)
>> - complex (KLOCs) XSL Doxygen->BoostBook|QuickBook translator(s)
>> - custom Doxygen to BoostBook converters written in C++ using external
>> XML libraries (sic!)
> 
> Not to mention ReST in various places, and LiTRe, which was used to test
> Boost.Parameter docs (see trunk/libs/parameter/doc/index.rst and look
> for "@") and the C++ Template Metaprogramming book.

Interesting. I didn't know about it. It's a new thing to me.

>> My initial attempt to Boost.Geometry docs was based on XSL (idea I
>> copied from Boost.Asio). It worked well, but...we dropped it because of
>> its complexity that would make maintenance (and contributions from new
>> hackers) very difficult, unless we assume everyone's got XSLT-fu.
> 
> Yeah, it's nasty.

It is, but this particular implementation for Boost.Asio gives
best output I've seen so far, e.g.

http://www.boost.org/doc/libs/1_47_0/doc/html/boost_asio/reference/async_read.html

It is easy to navigate between type requirements/concepts and models
and other elements relying on those requirements. Linking template
parameters to type requirements in particularly nice feature, because
it quickly shows what parts of conceptual model take part in generating
which parts of final components. Shortly, it visualises what/how the
puzzles fit together.

>> I assume getting rid of Doxygen from Boost is not an option ;-)
>>
>> So, I started to think about Sphinx as a foundation of C++ documentation
>> integrator/generator, inspired by the basic Sphinx API for Doxygen:
>>
>> http://packages.python.org/sphinxcontrib-doxylink/index.html#confval-
>> doxylink
>> http://packages.python.org/sphinxcontrib-doxylink/index.html#confval-doxylink
> 
> Also https://github.com/michaeljones/breathe

Nice!

>> Long story short, I'm looking for Boost-wide solution for:
>> - defined workflow
> 
> Could you be more specific about what kind of workflow you have in mind?

Yes, the idea may seem vague indeed.
I mean overall set of well described conventions and tools to write and
process complete documentation of a Boost library. Complete means as
covering all aspects of Boost C++ library, including concepts.
Best practices and recommendations are also put in to the workflow
definition. The workflow definition in short:
- markup to use (Doxygen, BoostBook, QuickBook, ..)
- content processing depends on input (the markup used)
- BoostBook as common "raw" format later used as input for publishing
processors (HTML, PDF, .etc.)
- framework of content structure of Boost library (and libraries together)
- content integration into Boost documentation

>> - integrate source level documentation with documentation dedicated to
>>   end-user
> 
> Again, could you be more specific?  Is it deeper than simply allowing
> linking between the two?

Yes, I made it utterly unclear.
I meant what I have explained above now. It is to use source code
comments content as documentation content too, including interlinking
symbols (also concepts with models).

Simply, documentation of Boost.Asio is a good incarnation of the idea.

>> - squeeze all possible juices from Doxygen comments, add documentation
>>   from other sources (e.g. .qbk files) and transform it to complete
>>   content - make documentation looking well and consistent across Boost
> 
> Yeah, that would indeed be beautiful.

Yes. Shortly explained above already, it would mean accepting several
documentation sources/formats as input and producing BoostBook.

>>> Also some things like the Math docs have some complex quickbook
>>> templates for generating differing content for PDF and HTML output
>>> in order to support equations etc.
>>
>> Sphinx can produce PDF and many more:
>>
>> http://sphinx.pocoo.org/latest/builders.html
> 
> (note: you'd need to pass through LaTeX as an intermediate format;
> Unless something has changed since I last look the reportlab stuff is
> geared to forms and is hopelessly weak when it comes to anything that
> needs real page layout/line breaking.  That's what we did for C++
> Template Metaprogramming).

I see. I don't know if anything has changed. I have seen a few printouts
and they looked "not bad" though.

>>> the Boost-wide PDF build could incorporate the Restructured Text docs
>>> as well as the Docbook ones.
>>
>> I'm not trying to advocate reStructuredText in Boost.  I'm trying to
>> seed the idea of powerful and extensible tool like Sphinx, with no
>> relation to this particular format.
> 
> Mateusz, I commend you for thinking big!  

I'm already scared myself :-)

> Let me just add a goal: whatever is done, it should reduce the amount of
> code that is maintained specially for Boost's documentation.

I have to confess I have close to nothing about amount of code of Boost
docs impl. maintained currently, but I agree, certainly.

> The biggest problem we have in making a transition to a saner doc
> universe, as far as I can tell, is that we're dependent on bespoke
> tools: QuickBook, BoostBook, our own XSLT processors, etc...

Yes.

> What's the solution?  I think someone needs to survey the kinds of
> things people are doing with QuickBook and BoostBook, so that we can
> assess whether other systems can support our needs.

I'm willing to help with this.
Can we make a list of questions ("the kind of things") do we want to
answer with the survey?

> And while we're discussing alternate markup, I'd like to throw Markdown
> and Org-mode into the ring.

AFAIU, reStructuredText is extensible so it is possible to add custom
directives. Markdown is not, but it's light.

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
Charter Member of OSGeo, http://osgeo.org
Member of ACCU, http://accu.org