BoostBook and Boost.Build version 2

From: Douglas Paul Gregor (gregod_at_[hidden])
Date: 2003-03-12 17:03:15

As most of you probably know, I'm hoping to get support for building
BoostBook user documentation with BBv2, and I would like your
help.

This message is long. Really long. It describes my "vision" of
BoostBook documentation building with BBv2, along with implementation
ideas.

[Note: I'm CCing the Boost documentation list, but we should probably
keep the discussion just on the jamboost list]

=== XSLT ===
XSLT is a language that describes a transformation from an XML
document type into another document (or several documents). For our
purposes, an XSLT transformation can be described by:

1) An XML document for input
2) An XSLT stylesheet
3) A set of stylesheet parameters (name/value pairs)
4) An output file or an output directory

As I see it, we could have an "xslt" rule of the form:

rule xslt target : source : stylesheet : parameters *

If "target" is a normal file, then it's just a normal target in
Jam. If "target" is a directory, it would have to be a pseudo-target
because we can't generally guess at what the output files will be to
build a target set.

There is only a single source file, but like C++ source files it may
include (via XIncludes) other XML files, and should depend on those
files. We can probably restrict ourselves to XIncludes that look like
this:
<xi:include href="xml-file-we-depend-on"/>

XSLT stylesheets are also XML files, but with a slightly different
include mechanism. Stylesheet inclusion in XSLT takes one of these
forms:
<xsl:include href="xslt-file-we-depend-on"/>
<xsl:import href="xslt-file-we-depend-on"/>
Both XIncludes and XSL stylesheet inclusion the text in the href
attribute can be either a local file (relative to the file including
it; there is no search path) or a web address (e.g.,
http://www.etc.etc./). We should probably just ignore anything of the
form "http://*" and assume everything else is a file.

XSL stylesheet parameters can probably be implemented via BBv2
features, e.g.,

xslt array-proposal.docbook : array.xml : docbook.xsl
: <boost.generation.mode>standardese
;

However, unlike compiler features that are properties of the compiler,
stylesheet parameters are a property of the stylesheet so we can't
easily maintain a list of all of the parameters for each
stylesheet. We generally have to just "trust" the user. Perhaps an
"xsl:" prefix to an arbitrary string? For instance, something like:

xslt array-proposal.docbook : array.xml : docbook.xsl
: <xsl:boost.generation.mode>standardese
;

BBv2 would know nothing about "boost.generation.mode", but it would
see the "xsl:" prefix and create the property
(boost.generation.mode,standardese) for the target
array-proposal.docbook.

There are several XSLT processors available, so in its final form I
would expect we'll want XSLT toolsets just like we have compiler
toolsets. For now, however, I'm only recommending one XSLT processor
(xsltproc). Here is the command line information for using xsltproc:

xsltproc --xinclude <parameters> -o <target>? <stylesheet> <source>

<parameters> is composed of strings "--stringparam feature value" for
each property.

<target> is the target (file or directory; doesn't matter)

<stylesheet> is the stylesheet

<source> is the source file

So for the example above, the command line would be:
xsltproc --xinclude --stringparam boost.generation.mode standardese \
-o array-proposal.docbook docbook.xsl array.xml

(There's a little more information on the command line later, in the
section "XML Catalogs")

=== BoostBook, DocBook, HTML, and Man pages ===
Given a BoostBook document we apply a series of XSLT transformations
to get to end-user documentation (e.g., HTML or man pages). The
transformations are:
BoostBook file (.xml) --> DocBook file (.docbook) via stylesheet
tools/boostbook/xsl/docbook.xsl

DocBook (.docbook) --> single HTML file (.html) via stylesheet
tools/boostbook/xsl/html-single.xsl

DocBook (.docbook) --> HTML output directory via stylesheet
tools/boostbook/xsl/html.xsl

DocBook (.docbook) --> Man pages output directory via stylesheet
tools/boostbook/xsl/man.xsl

I expect that the Jamfile for these would look something like this:

type.register BOOSTBOOK : xml bbk ;
type.register DOCBOOK : docbook ;
type.register HTML : html ;
generators.register-standard boostbook.docbook : BOOSTBOOK : DOCBOOK ;
generators.register-standard boostbook.html-single : HTML : DOCBOOK ;

# ------- Most definitely broken attempt at a Jamfile --------
rule docbook ( target : source : parameters * )
{
xslt $(target)
: $(source)
: $(BOOST_ROOT)/tools/boostbook/xsl/docbook.xsl
: $(parameters)
;
}

rule html-single ( target : source : parameters * )
{
xslt $(target)
: $(source)
: $(BOOST_ROOT)/tools/boostbook/xsl/html-single.xsl
: $(parameters)
;
}

rule html ( target : source : parameters * )
{
xslt $(target)
: $(source)
: $(BOOST_ROOT)/tools/boostbook/xsl/html.xsl
: $(parameters)
;
}

rule man-pages ( target : source : parameters * )
{
xslt $(target)
: $(source)
: $(BOOST_ROOT)/tools/boostbook/xsl/man.xsl
: $(parameters)
;
}
# -----End most definitely broken attempt at a Jamfile ------

(The GNU make equivalent if some of these rules is at the end of this
e-mail)

The top-level target would reside in $(BOOST_ROOT)/doc/src and the
Jamfile would probably contain:

html-doc $(BOOST_ROOT)/doc/html : boost.xml ;

A sample invocation:

bjam html-doc xsl:boost.include.libraries="bind function ref"

This would execute:

xsltproc --xinclude --stringparam boost.include.libraries \
"bind function ref" -o boost.docbook \
$(BOOST_ROOT)/tools/boostbook/xsl/docbook.xsl boost.xml

xsltproc --xinclude --stringparam boost.include.libraries \
"bind function ref" -o $(BOOST_ROOT)/doc/html \
$(BOOST_ROOT)/tools/boostbook/xsl/html.xsl boost.docbook

=== XML Catalogs ===
I mentioned previously that XSL stylesheet inclusions may use web
addresses. When the XSLT processor needs the included stylesheets, it
will download them first and then continue processing. Big bottleneck
here (the DocBook HTML stylesheets alone are > 1MB) which can be
solved by XML catalogs. The same thing goes for DTDs.

XML catalogs remap remote web addresses (and, optionally, XML public
identifiers like "-//Boost//DTD BoostBook XML V1.0//EN") to local
files and directories. The XML catalog (just an XML file) is then
passed to the XSLT processor, which does the remappings
on-the-fly. XML catalogs can easily be generated when we know the
local directories for:
1) The DocBook XSL stylesheets
2) The DocBook DTD
3) The BoostBook DTD (in $(BOOST_ROOT)/tools/boostbook/dtd)

Ideally, the user could "configure" BBv2 with the locations of the
first two (so that the user need not specify the directories each
time), and BBv2 would build and use the XML catalog with whatever
directory information it has, omitting the rest of the information so
that the XSLT processor can download it as needed. For reference, the
XML catalog I use (generated by the configure script at the end of
this e-mail) is:

<?xml version="1.0"?>
<!DOCTYPE catalog
PUBLIC "-//OASIS/DTD Entity Resolution XML Catalog V1.0//EN"
"http://www.oasis-open.org/committees/entity/release/1.0/catalog.dtd">

<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
<rewriteURI
uriStartString="http://www.oasis-open.org/docbook/xml/4.2/"
rewritePrefix="/home/gregod/Projects/share/docbook-dtd-4.2//"/>
<rewriteURI
uriStartString="http://docbook.sourceforge.net/release/xsl/current/"
rewritePrefix="/home/gregod/Projects/share/docbook-xsl-1.60.1//"/>
<rewriteURI
uriStartString="http://www.boost.org/tools/boostbook/dtd/"

rewritePrefix="/home/gregod/Projects/Boost/boost/tools/boostbook/dtd/"/>
</catalog>

The uriStartString attribute values are hardcoded values representing
the DocBook XSL, DTD, and BoostBook DTD locations.The rewritePrefix
attributes are the local directories for each.

With xsltproc, the name of the XML catalog file is passed as an
environment vaiable XML_CATALOG_FILES, so the command line would look
like this when catalogs are used:
XML_CATALOG_FILES=catalog.xml xsltproc <etc, etc>

I have no idea how to accomplish this type of thing in BBv2 :(

=== Future issues: Doxygen, FO, PDF ===
At some point, there will be more potential steps in the documentation
generation pipeline. Doxygen, for instance, can output XML, which can
be collected into a single document (via an XSLT stylesheet) and then
transformed into BoostBook (via an XSLT stylesheet). No need to worry
about this now...

The DocBook XSL stylesheet can also output XSL Formatting Object files
(.fo), which are used for generating hardcopy versions of DocBook
documents. However, FO->PS and FO->PDF conversion isn't done with
XSLT, but with another external program (e.g., Apache's FOP). Again,
no need to worry about this now, and it should be quite easy, but it's
a hint that we may need yet another type of toolset in the future.

Thanks,
Doug

=== XML catalog configure script ===
#!/bin/sh

if [ -z $2 ]; then
echo "Syntax is ./configure <DocBook XSL path> <DocBook DTD path> [<FOP
Path>]"
echo "If you do not have the DocBook XSL Stylesheets or DTD, please
download"
echo "them from http://docbook.sourceforge.net/"
exit -1
fi

DOCBOOK_XSL_PATH=$1
if [ ! -r $DOCBOOK_XSL_PATH/html/chunk.xsl ]; then
echo "The DocBook XSL path given does not seem correct. Please check
the"
echo "path again and rerun configure. The DocBook XSL path given was: "
echo " $DOCBOOK_XSL_PATH"
exit -2
fi

DOCBOOK_DTD_PATH=$2
if [ ! -r $DOCBOOK_DTD_PATH/docbookx.dtd ]; then
echo "The DocBook DTD patch given does not seem correct. Please check
the"
echo "path again and rerun configure. The DocBook DTD path given was: "
echo " $DOCBOOK_DTD_PATH"
exit -3;
fi

BOOSTBOOK_XSL_PATH=`pwd`
BOOSTBOOK_XSL_PATH="${BOOSTBOOK_XSL_PATH%/build*}/xsl"
if [ ! -r "$BOOSTBOOK_XSL_PATH/docbook.xsl" ]; then
echo "Please execute configure from the BoostBook build directory"
exit -4;
fi

BOOSTBOOK_DTD_PATH=`pwd`
BOOSTBOOK_DTD_PATH="${BOOSTBOOK_DTD_PATH%/build*}/dtd"
if [ ! -r "$BOOSTBOOK_DTD_PATH/boostbook.dtd" ]; then
echo "Please execute configure from the BoostBook build directory"
exit -5;
fi

XSLTPROC=${XSLTPROC:-`which xsltproc`}
if [ ! -x $XSLTPROC ]; then
echo "Could not find the program xsltproc in the current PATH.
Stopping..."
exit -6
fi

XMLLINT=${XMLLINT:-`which xmllint`}
if [ ! -x $XMLLINT ]; then
echo "Could not find the program xmllint in the current PATH."
echo "This is not a problem per se, but you may not be able to validate"
echo "XML files against a DTD."
fi

if [ -z $JAVA_HOME ]; then
JAVA=${JAVA:-`which java`}
JAVA_BIN_PATH=${JAVA%/*}
JAVA_HOME=${JAVA_BIN_PATH%/bin}
fi

if [ ! -d $JAVA_HOME ]; then
echo "Could not find Java. PDF generation will not be available."
echo "If you have Java installed, set JAVA_HOME to its path and rerun "
echo "configure."
else
FOP=${FOP:-$3}
if [ ! -z $FOP ]; then
if [ -d $FOP ]; then
FOP=${FOP%/}
FOP="$FOP/fop.sh"
fi
fi

if [ -z $FOP ] || [ ! -x $FOP ]; then
echo "Could not find FOP. PDF generation will not be available."
echo "If you have FOP installed, pass the installation path as the
second"
echo "argument to configure, e.g.,"
echo " ./configure $DOCBOOK_XSL_PATH <FOP Path>"
echo ""
echo "If you do not have FOP and would like support for building PDF"
echo "documentations, please download FOP from:"
echo " http://xml.apache.org/fop/"
fi
fi

BOOSTBOOK_CATALOG="`pwd`/catalog.xml"

cat > Makefile.def << .
XSLTPROC=$XSLTPROC
XMLLINT=$XMLLINT
DOCBOOK_XSL_PATH=$DOCBOOK_XSL_PATH
BOOSTBOOK_XSL_PATH=$BOOSTBOOK_XSL_PATH
JAVA_HOME=$JAVA_HOME
FOP=$FOP
BOOSTBOOK_CATALOG=$BOOSTBOOK_CATALOG
.

cat > catalog.xml << .
<?xml version="1.0"?>
<!DOCTYPE catalog
PUBLIC "-//OASIS/DTD Entity Resolution XML Catalog V1.0//EN"
"http://www.oasis-open.org/committees/entity/release/1.0/catalog.dtd">

<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
<rewriteURI
uriStartString="http://www.oasis-open.org/docbook/xml/4.2/"
rewritePrefix="$DOCBOOK_DTD_PATH/"/>
<rewriteURI
uriStartString="http://docbook.sourceforge.net/release/xsl/current/"
rewritePrefix="$DOCBOOK_XSL_PATH/"/>
<rewriteURI
uriStartString="http://www.boost.org/tools/boostbook/dtd/"
rewritePrefix="$BOOSTBOOK_DTD_PATH/"/>
</catalog>
.

echo "BoostBook configuration is complete."

=== GNU make makefile ===
XSL=XML_CATALOG_FILES=$(BOOSTBOOK_CATALOG) $(XSLTPROC) --xinclude --nonet
CHECK_DTD=XML_CATALOG_FILES=$(BOOSTBOOK_CATALOG) $(XMLLINT) --xinclude
--noout --postvalid

%.docbook: %.xml
$(XSL) --stringparam boost.root $(BOOST_ROOT) -o $@
$(BOOSTBOOK_XSL_PATH)/docbook.xsl $<

%.fo: %.docbook
$(XSL) -o $@ $(BOOSTBOOK_XSL_PATH)/fo.xsl $<

%.pdf: %.fo
JAVA_HOME=$(JAVA_HOME) $(FOP) $< $@

%.html: %.docbook
$(XSL) -o $@ $(BOOSTBOOK_XSL_PATH)/html-single.xsl $<

[If you read this far, you deserve a cookie.]