This is slightly off topic.

I have a question how pdf's are build within the boost build system.
As far as I could figure it out the build system uses a proprietary
solution called RenderX XEP or so. I'm wondering if there are open
source solutions out there that could be used to generate boost's
documentation as pdf files?

Apache FOP is the only open source solution I know of: http://xmlgraphics.apache.org/fop/

Unfortunately it still hasn't got past a 1.0 release, and last I checked it's rendering was still far inferior to XEP - basically building with XEP is pretty much a "fire and forget" process, while building with FOP quite often either mangles the docs or uses rather unfortunate pagination choices :(

John.