Re: [boost] GDB-like debugger for template metaprograms

Subject: Re: [boost] GDB-like debugger for template metaprograms
From: Edward Diener (eldiener_at_[hidden])
Date: 2014-12-03 08:11:07

On 12/2/2014 6:25 PM, Mikael Persson wrote:
> Hi all,
>
> I've been working on a re-vamped version of Templight (
> https://github.com/mikael-s-persson/templight). So, I have a few things to
> mention in relation to the points raised in this discussion that relate to
> issues with the templight patch that metashell relies on.
>
> First thing's first, my version of templight also includes a GDB-style
> template meta-program debugger:
>
> https://github.com/mikael-s-persson/templight#using-the-templight-debugger
>
> The crucial difference that it runs on-line... i.e., breakpoints will pause
> the compilation to provide you with a shell that allows you to query
> information from the AST directly ("while it's hot"). It is not complete or
> thoroughly tested yet, but it's pretty powerful (e.g., allows you to, for
> example, evaluate the value of constants expressions in the context of the
> template instantiation or to evaluate the actual type of typedefs and
> similar dependent types). Technically, the possibilities are unlimited as
> far as gaining information about whatever you want during the compilation
> (suggestions are welcome!).
>
> The point was raised that a problem with the original templight patch is
> that it requires people to sync to a much older version of Clang, and
> support for Windows is not great. My version of templight solves those
> problems by having a much more up-to-date patch for Clang, that should work
> on the latest svn trunk of it. Also, I generate separate applications for
> Templight ("templight(.exe)", "templight++(.exe)" and "templight-cl.exe")
> which means that you don't have to deal with a templight-enabled clang
> executables that might interfere with your other workflows. And evidently,
> with templight-cl.exe, you have a MSVC-compatible version of templight too,
> based on the latest work on clang-cl.exe as a drop-in replacement for
> cl.exe. The patch on Clang is very minimal (part of my re-vamping was to
> pull most of the templight code out of the main clang code-base, and only
> leaving the minimal hooks), that patch has been submitted to clang, but it
> is still pending approval (I probably need more pressure from the community
> to get this patch through).
>
> Some of the finer points raised:
>
>  From Mathias Gaunard:
>> pretty printing complex types so that you can actually read them
>> ...
>> It can be hard to match '<' and '>' together for long symbol names
> involving instantiations of templates where the parameters are themselves
> template instantiations.
>
> The basics of this has been implemented in my version of Templight. I
> created a protobuf-based output format in which template names are
> compressed by re-using the repeating sub-strings in the names. To achieve
> this, I had to use the natural splits that exist in template names, based
> on matching "<", ",", and ">" delimiters to extract template arguments from
> the complex template instantiation names. I've tested this quite
> extensively, on many of the example programs from Boost libraries like
> Spirit and Proto, and it works quite well (I'm sure there are some
> corner-cases where it breaks down, but I haven't encountered any so far).
> When you output a protobuf-formatted template instantiation trace from
> templight, you actually get a linked-tree type of structure which you could
> easily use to make any kind of pretty-printing you like (or fancy
> collapse-able stuff too).
>
> Abel, I recommend that you check out that implementation, as it's really
> simple yet effective (also, the compression ratio and processing speed
> difference compared to the original YAML outputs is huge):
>
> https://github.com/mikael-s-persson/templight/blob/master/templight_messages.proto
> https://github.com/mikael-s-persson/templight/blob/master/lib/TemplightProtobufWriter.cpp
> https://github.com/mikael-s-persson/templight/blob/master/lib/TemplightProtobufReader.cpp
>
>  From Mathias Gaunard:
>> full path and line information for each element in the backtrace is
> needed. Ideally you want to output that in a format where most smart
> editors can directly recognize it and jump there in a click.
>
> Of course, this is necessary! That is what I have done on my templight
> debugger. I don't know why Abel's MDB wouldn't have that. The whole point
> of a GDB-style debugger is to run it behind a smart editor in the fashion
> you describe.
>
>  From Mathias Gaunard:
>> I'd like to be able to use this tool to track how a specific
> instantiation happens, and since I'm working on a large project I'd need to
> have the reference of where the template being instantiated is and where
> the instantiation occurs.
>
> Currently, my templight debugger does not have a command to query the
> location of where the template being instantiated is, only where the
> instantiation occurs. But that is only for historical reason (this is what
> templight traces record), but I could easily add this feature. I'll work on
> it.
>
>  From Edward Diener:
>> Building Metashell on Windows seems pretty flawed:
>
> Building anything on Windows is flawed ;)
>
> For the record, my version of templight, that includes the interactive
> debugger builds fine under Windows, and has the MSVC-compatible
> templight-cl.exe program too.
>
>  From Edward Diener:
>> 2) Templight supposedly depends on an older version of Clang than what is
> currently in the Clang svn source tree.
>
> My version is up-to-date and much less intrusive on Clang, and therefore,
> with fewer foreseeable problems in applying it to whatever version of the
> Clang source tree you are sync'd with (as long as it is not too old).
> However, my version of templight is not currently compatible with metashell
> / MDB, because that's not my responsibility (but it is my hope).
>
>  From Edward Diener:
>> Suppose I already have Clang regularly built from source using mingw/gcc.
> Am I actually building another version of clang just for Metashell and
> using that version instead as Clang when working with Metashell ? Ugh !!!
> It would be much better if whatever is needed from Clang were part of the
> latest clang source rather than some backdated version.
>
> That is a point I have already raised with Abel some time ago. Clearly,
> this is a very impractical situation to be in. This was the whole reason
> for minimizing the patch necessary on Clang and then submitting it for
> permanent inclusion in Clang. I have done this, just waiting for the Clang
> gate-keepers to manifest any kind of interest.
>
>
> I just wanted to point these things out. I'm very enthusiastic about the
> future of this, and I believe it will have to go through my version of
> Templight, to have any hope of success in the near future.

I think this is great. I surely hope that templight can become part of 
the clang build-from-source process, as well as having the clang 
compiler switches for templight documented. The ability to unroll 
template instantiations by the compiler, so that the template 
metaprogramming can see what is happening at each step when a template 
is instantiated, is a great boon for the template metaprogrammer.