Re: [Boost-docs] Sphinx integration

Subject: Re: [Boost-docs] Sphinx integration
From: Dave Abrahams (dave_at_[hidden])
Date: 2011-10-02 22:03:28

on Thu Sep 29 2011, Mateusz Loskot <mateusz-AT-loskot.net> wrote:

> On 28/09/11 17:09, John Maddock wrote:
>>>> Perhaps quickbook to reStructuredText is a feasible solution.
>
>>>> Then all move to writing in reStructuredText, etc.
>>>
>>> But then there's the issue of lossiness.
>>
>> And lets not forget that some of us really like quickbook ;-)
>
> I do like quickbook too.
> Although may intention was to discuss the workflow of documenting Boost,
> I think I have identified what is the biggest pain to me: Doxygen
> integration in BoostBook.
>
> Doxygen is widely used C++ code documenting tool.
> Fortunately or unfortunately, it is also used by many Boost authors.
>
> Doxygen does not support many of the terms/concepts/elements used across
> Boost libraries. It is also becasue it is almost impossible to document
> them well at source code levels.
> However, things like "concept and model" exist at source code level,
  do not?---------------------------------^

That's true of concepts unless you're using Boost.ConceptCheck.  But
even so, they will only be seen by Doxygen as regular C++ and not as the
abstractions they represent.  And there are no model declarations. :-(

> so authors try to do their best with Doxygen.
> The resulting documentation is hopeless. It lacks of proper structure
> and does not make sensible use of the Hypertext (no links between
> properties of concept with properties of model). Examples:
>
> www.boost.org/doc/libs/1_47_0/libs/geometry/doc/html/geometry/reference/concepts.html
> http://www.boost.org/doc/libs/1_47_0/libs/pool/doc/index.html
>
> Conclusion: there is no need for a tool that can handle X number of
> programming languages and output formats, but a tool that can handle
> C++ programming language very very well, especially its "modern"
> aspects.

+1

> So, authors invent custom processors and tools to integrate. We have:
> - bits of support in QuickBook (BoostBook) e.g. xinclude autodoc.xml
> - Perl scripts (e.g. ref_tag_fix.pl in DateTime)
> - complex (KLOCs) XSL Doxygen->BoostBook|QuickBook translator(s)
> - custom Doxygen to BoostBook converters written in C++ using external
> XML libraries (sic!)

Not to mention ReST in various places, and LiTRe, which was used to test
Boost.Parameter docs (see trunk/libs/parameter/doc/index.rst and look
for "@") and the C++ Template Metaprogramming book.

> My initial attempt to Boost.Geometry docs was based on XSL (idea I
> copied from Boost.Asio). It worked well, but...we dropped it because of
> its complexity that would make maintenance (and contributions from new
> hackers) very difficult, unless we assume everyone's got XSLT-fu.

Yeah, it's nasty.

> I assume getting rid of Doxygen from Boost is not an option ;-)
>
> So, I started to think about Sphinx as a foundation of C++ documentation
> integrator/generator, inspired by the basic Sphinx API for Doxygen:
>
> http://packages.python.org/sphinxcontrib-doxylink/index.html#confval-
> doxylink
> http://packages.python.org/sphinxcontrib-doxylink/index.html#confval-doxylink

Also https://github.com/michaeljones/breathe

> Long story short, I'm looking for Boost-wide solution for:
> - defined workflow

Could you be more specific about what kind of workflow you have in mind?

> - integrate source level documentation with documentation dedicated to
>   end-user

Again, could you be more specific?  Is it deeper than simply allowing
linking between the two?

> - squeeze all possible juices from Doxygen comments, add documentation
>   from other sources (e.g. .qbk files) and transform it to complete
>   content - make documentation looking well and consistent across Boost

Yeah, that would indeed be beautiful.

> - (wishful thinking about ideal) make documentation interlinking
> content across Boost libraries - step-by-step guide on documenting
> Boost which fits on 1-2 sheets of A4 printout, otherwise the process
> is too complex and hardly anyone will bother.

I'd be happy to help with that.
>
>> Also some things like the Math docs have some complex quickbook
>> templates for generating differing content for PDF and HTML output
>> in order to support equations etc.
>
> Sphinx can produce PDF and many more:
>
> http://sphinx.pocoo.org/latest/builders.html

(note: you'd need to pass through LaTeX as an intermediate format;
Unless something has changed since I last look the reportlab stuff is
geared to forms and is hopelessly weak when it comes to anything that
needs real page layout/line breaking.  That's what we did for C++
Template Metaprogramming).

> (I imagine many would like to read Boost docs in Kindle :-) or browse
> it in QtHelp or Devhelp)
>
>> Which is not to say that Boost.Build couldn't support
>> RestructuredText to facilitate building such docs - 

Well, it does, to a degree.  There is a docutils toolset, which I wrote.
We used it for Boost.Iterators and several other libraries.

>> then if they can be built as PDF's (I assume there's a way?) 

Like I said, pass through LaTeX.

>> the Boost-wide PDF build could incorporate the Restructured Text docs
>> as well as the Docbook ones.
>
> I'm not trying to advocate reStructuredText in Boost.  I'm trying to
> seed the idea of powerful and extensible tool like Sphinx, with no
> relation to this particular format.

Mateusz, I commend you for thinking big!  

Let me just add a goal: whatever is done, it should reduce the amount of
code that is maintained specially for Boost's documentation.

The biggest problem we have in making a transition to a saner doc
universe, as far as I can tell, is that we're dependent on bespoke
tools: QuickBook, BoostBook, our own XSLT processors, etc...

What's the solution?  I think someone needs to survey the kinds of
things people are doing with QuickBook and BoostBook, so that we can
assess whether other systems can support our needs.

And while we're discussing alternate markup, I'd like to throw Markdown
and Org-mode into the ring.

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com