Re: [boost] GDB-like debugger for template metaprograms

Subject: Re: [boost] GDB-like debugger for template metaprograms
From: Mikael Persson (mikael.s.persson_at_[hidden])
Date: 2014-12-02 18:25:16

Hi all,

I've been working on a re-vamped version of Templight (
https://github.com/mikael-s-persson/templight). So, I have a few things to
mention in relation to the points raised in this discussion that relate to
issues with the templight patch that metashell relies on.

First thing's first, my version of templight also includes a GDB-style
template meta-program debugger:

https://github.com/mikael-s-persson/templight#using-the-templight-debugger

The crucial difference that it runs on-line... i.e., breakpoints will pause
the compilation to provide you with a shell that allows you to query
information from the AST directly ("while it's hot"). It is not complete or
thoroughly tested yet, but it's pretty powerful (e.g., allows you to, for
example, evaluate the value of constants expressions in the context of the
template instantiation or to evaluate the actual type of typedefs and
similar dependent types). Technically, the possibilities are unlimited as
far as gaining information about whatever you want during the compilation
(suggestions are welcome!).

The point was raised that a problem with the original templight patch is
that it requires people to sync to a much older version of Clang, and
support for Windows is not great. My version of templight solves those
problems by having a much more up-to-date patch for Clang, that should work
on the latest svn trunk of it. Also, I generate separate applications for
Templight ("templight(.exe)", "templight++(.exe)" and "templight-cl.exe")
which means that you don't have to deal with a templight-enabled clang
executables that might interfere with your other workflows. And evidently,
with templight-cl.exe, you have a MSVC-compatible version of templight too,
based on the latest work on clang-cl.exe as a drop-in replacement for
cl.exe. The patch on Clang is very minimal (part of my re-vamping was to
pull most of the templight code out of the main clang code-base, and only
leaving the minimal hooks), that patch has been submitted to clang, but it
is still pending approval (I probably need more pressure from the community
to get this patch through).

Some of the finer points raised:

>From Mathias Gaunard:
> pretty printing complex types so that you can actually read them
> ...
> It can be hard to match '<' and '>' together for long symbol names
involving instantiations of templates where the parameters are themselves
template instantiations.

The basics of this has been implemented in my version of Templight. I
created a protobuf-based output format in which template names are
compressed by re-using the repeating sub-strings in the names. To achieve
this, I had to use the natural splits that exist in template names, based
on matching "<", ",", and ">" delimiters to extract template arguments from
the complex template instantiation names. I've tested this quite
extensively, on many of the example programs from Boost libraries like
Spirit and Proto, and it works quite well (I'm sure there are some
corner-cases where it breaks down, but I haven't encountered any so far).
When you output a protobuf-formatted template instantiation trace from
templight, you actually get a linked-tree type of structure which you could
easily use to make any kind of pretty-printing you like (or fancy
collapse-able stuff too).

Abel, I recommend that you check out that implementation, as it's really
simple yet effective (also, the compression ratio and processing speed
difference compared to the original YAML outputs is huge):

https://github.com/mikael-s-persson/templight/blob/master/templight_messages.proto
https://github.com/mikael-s-persson/templight/blob/master/lib/TemplightProtobufWriter.cpp
https://github.com/mikael-s-persson/templight/blob/master/lib/TemplightProtobufReader.cpp

>From Mathias Gaunard:
> full path and line information for each element in the backtrace is
needed. Ideally you want to output that in a format where most smart
editors can directly recognize it and jump there in a click.

Of course, this is necessary! That is what I have done on my templight
debugger. I don't know why Abel's MDB wouldn't have that. The whole point
of a GDB-style debugger is to run it behind a smart editor in the fashion
you describe.

>From Mathias Gaunard:
> I'd like to be able to use this tool to track how a specific
instantiation happens, and since I'm working on a large project I'd need to
have the reference of where the template being instantiated is and where
the instantiation occurs.

Currently, my templight debugger does not have a command to query the
location of where the template being instantiated is, only where the
instantiation occurs. But that is only for historical reason (this is what
templight traces record), but I could easily add this feature. I'll work on
it.

>From Edward Diener:
> Building Metashell on Windows seems pretty flawed:

Building anything on Windows is flawed ;)

For the record, my version of templight, that includes the interactive
debugger builds fine under Windows, and has the MSVC-compatible
templight-cl.exe program too.

>From Edward Diener:
> 2) Templight supposedly depends on an older version of Clang than what is
currently in the Clang svn source tree.

My version is up-to-date and much less intrusive on Clang, and therefore,
with fewer foreseeable problems in applying it to whatever version of the
Clang source tree you are sync'd with (as long as it is not too old).
However, my version of templight is not currently compatible with metashell
/ MDB, because that's not my responsibility (but it is my hope).

>From Edward Diener:
> Suppose I already have Clang regularly built from source using mingw/gcc.
Am I actually building another version of clang just for Metashell and
using that version instead as Clang when working with Metashell ? Ugh !!!
It would be much better if whatever is needed from Clang were part of the
latest clang source rather than some backdated version.

That is a point I have already raised with Abel some time ago. Clearly,
this is a very impractical situation to be in. This was the whole reason
for minimizing the patch necessary on Clang and then submitting it for
permanent inclusion in Clang. I have done this, just waiting for the Clang
gate-keepers to manifest any kind of interest.

I just wanted to point these things out. I'm very enthusiastic about the
future of this, and I believe it will have to go through my version of
Templight, to have any hope of success in the near future.

Cheers,
Mikael Persson.