[Boost-build] [boost-build] bug? Test target in boost-python tutorial example is not executed. (Was: What is run-pyd?)

Subject: [Boost-build] [boost-build] bug? Test target in boost-python tutorial example is not executed. (Was: What is run-pyd?)
From: Roland Puntaier (roland.puntaier_at_[hidden])
Date: 2012-11-05 19:30:53

I investigated into this problem and report here.
I did not solve the problem, but found a workaround for me.
There seems to be a bug in Jam I will describe in more detail further down.

The following debug options of b2 were useful::
-d13: shows you all debug messages including dependency graph and fate 
changes
-d1: show commands as they are done
-n: shows you all the commands without executing them
Then I used *ECHO* in jam files and also changed the Jam engine sources
(make1.c) to issue additional debug messages (A7).

At the end of the message there are attachments A1, A2, ... I will refer to.
"What is run-pyd?" I don't want to leave unansered and have attached it 
in A10.
I had to get an overview of the jam engine sources, which I share in A9.
A1-A4 is the tree and the files of the boost-python tutorial example.
I adapted it a little to my platform.

The problem:

When I run standard b2 I get the tree structure in A5. *hello.test* does 
not have any files.
--perserve-test-targets or -a (rebuild all files) options do not help.

Investigations:

Adding *ECHO* in testing.jam shows that capture-output is called with the
target "hello" (or more precisely: <pbin/hello.test/gcc-4.7.0/debug>hello),
but the capture-output actions are not executed. "b2 -d13" shows the target
dependencies (A8). The "hello" target is there twice on depth 2.

Here came first doubts about the soundness of this. Parallel 
dependencies must
not lead to duplicate targets in the tree. It's OK if they are references to
singleton targets. This role the files in the real file system can take 
over,
but what for internal targets. I found there is a *bindtarget()* function
which seems to bind the targets to unique hash table entries. Maybe this is
done in *make.c* before calling make1() in order to make *state_stack* for
*make1a(), make1b(), make1c()* reference the same targets. But with the
additional traces in make1.c (A7) I found that the two "hello" targets are
different, as can be seen by the different target field values (see
print_stack() in A7) ::

===== MAKE1B (2): <pbin/hello.test/gcc-4.7.0/debug>hello->UPDATE 
(<pbin/hello.test/gcc-4.7.0/debug>hello.test)
===== MAKE1A (0): <pbin/hello.test/gcc-4.7.0/debug>hello->STABLE 
(<pbin/hello.test/gcc-4.7.0/debug>hello.test)

The second "hello" target is marked internal. Actions are not executed 
for them.
For its NOCARE dependency "hello.output" the traces show ::

fate change <pbin/hello.test/gcc-4.7.0/debug>hello.output to STABLE from 
missing, no actions, no dependencies and do not care
made stable <pbin/hello.test/gcc-4.7.0/debug>hello.output

So both don't get made and therefore this branch is useless.

If the first "hello" target's actions had had been executed then 
"hello.output"
would have been there and the second branch would succeed. Why is this first
"hello" target not executed? It never reaches *make1c()* for execution 
because
it is popped from the *state_stack* here ::

if ( --t->asynccnt )
{
pop_state( &state_stack );
return;
}

If I replace ::

if ( --t->asynccnt )

with ::

if ( 0 )

Then b2 generates the "hello" test and I get the tree as in A6. But 
after "b2 --clean" and "b2" I see there are errors.
A second "b2" succeeds. So this asynccnt serves a purpose (as expected), 
but it seem to be buggy.
As far as I have understood, it expects the dependencies to add the same 
target again exactly as often
as they increased asynccnt, which is also done in *make1a()* ::

if ( pState->parent && t->progress <= T_MAKE_RUNNING )
{ ...
++parent_scc->asynccnt;
...
t->asynccnt = 1;
...
for ( c = t->depends; c && !intr; c = c->next )
push_state( &temp_stack, c->target, t, T_STATE_MAKE1A );
push_stack_on_stack( &state_stack, &temp_stack );

It seems to be a design deficiency with a buggy workarounds, but I can 
also be wrong, because I'm new to Jam and boost-build.
It would be great if somebody expert in boost-build could uncover a mere 
wrong usage
or else have found a clue to where the bug could be.

I'm not excited about Jam and boost-build and the fact that boost uses it.
What I don't like about boost-build is:
- Jam is a rather low-level language and a lot of Jam code is needed to 
build the scaffolding for proper usage.
- Jam has only this usage. Its syntax is very different from many other 
languages.
- Jam has no infrastructure like debugger and editor syntax coloring.
When I have time I'll have a look into WAF.
Python is a general purpose language with all the infrastructure around it.
So when an unexpected behavior comes up it's easier to investigate.
I could possibly have some lines of python code that do the whole build 
and test runs
in commented text to execute from inside the editor.

Attachments
===========

A1 ________________ tree

.
├── hello.cpp
├── hello.py
└── Jamroot

A2 ________________ hello.cpp

#include <boost/python/module.hpp>
#include <boost/python/def.hpp>

char const* greet()
{
return "hello, world";
}

BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}

A3 ________________ hello.py

import hello_ext
print(hello_ext.greet())

A4 ________________ Jamroot

import python ;

if ! [ python.configured ]
{
ECHO "notice: no Python configured in user-config.jam" ;
ECHO "notice: will use default configuration" ;
using python ;
}

project : requirements <library>/usr/lib/libboost_python3.so ;

python-extension hello_ext : hello.cpp ;

install convenient_copy
: hello_ext
: <install-dependencies>on <install-type>SHARED_LIB 
<install-type>PYTHON_EXTENSION
<location>.
;

local rule run-test ( test-name : sources + )
{
import testing ;
testing.make-test run-pyd : $(sources) : : $(test-name) ;
}

run-test hello : hello_ext hello.py ;

A5 ________________ tree with unpatched boost-build

.
├── bin
│ ├── config.log
│ ├── gcc-4.7.0
│ │ └── debug
│ │ ├── hello_ext.so
│ │ └── hello.o
│ └── hello.test
│ └── gcc-4.7.0
│ └── debug
├── hello.cpp
├── hello_ext.so
├── hello.py
├── Jamroot
└── libboost_python3.so

A6 ________________ tree with line "if ( --t->asynccnt )" in make1.c 
replaced by "if (0)"

.
├── bin
│ ├── config.log
│ ├── gcc-4.7.0
│ │ └── debug
│ │ ├── hello_ext.so
│ │ └── hello.o
│ └── hello.test
│ └── gcc-4.7.0
│ └── debug
│ ├── hello
│ ├── hello.output
│ ├── hello.py
│ └── hello.test
├── hello.cpp
├── hello_ext.so
├── hello.py
├── Jamroot
└── libboost_python3.so

A7 ________________ make1.c additional debug messages for -d13

git diff engine/make1.c
=======================
diff --git a/engine/make1.c b/engine/make1.c
index 2c9797f..6606b77 100644
--- a/engine/make1.c
+++ b/engine/make1.c
@@ -165,9 +165,10 @@ static state * push_state( stack * const pStack, 
TARGET * const t,
pState->parent = parent;
pState->prev = pStack->stack;
pState->curstate = curstate;
- return pStack->stack = pState;
-}
+ pStack->stack = pState;

+ return pStack->stack;
+}

/*
* Pushes a stack onto another stack, effectively reversing the order.
@@ -185,6 +186,31 @@ static void push_stack_on_stack( stack * const 
pDest, stack * const pSrc )
}

+static void print_asynccnt(TARGET *t, char const *after)
+{
+ printf( "asynccnt of %s (%s): %d\n", object_str( t->name ), after, 
t->asynccnt);
+}
+
+static char *state_str[3] = {"MAKE1A", "MAKE1B", "MAKE1C"};
+static char *fate_str[13] =
+ {"INIT", "MAKING", "STABLE", "NEWER", "SPOIL", "BUILD", "REBUILD", 
"MISSING", "NEEDTMP", "OUTDATED", "UPDATE", "BROKEN", "CANTMAKE"};
+static void print_stack(stack *pStack, char const *after)
+{
+ state *cs = pStack->stack;
+ printf( "after %s ======\n", after);
+ while ( cs )
+ {
+ printf( "===== %s (%d): %s->%s (%s)\n",
+ state_str[cs->curstate],
+ cs->t->asynccnt,
+ object_str( cs->t->name ),
+ fate_str[cs->t->fate],
+ cs->parent?object_str( cs->parent->name ):"" );
+ cs = cs->prev;
+ }
+}
+
+
/*
* make1() - execute commands to update a TARGET and all of its dependencies
*/
@@ -199,14 +225,26 @@ int make1( TARGET * const t )

/* Recursively make the target and its dependencies. */
push_state( &state_stack, t, NULL, T_STATE_MAKE1A );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"push in make1");

while ( 1 )
{
while ( ( pState = current_state( &state_stack ) ) )
{
if ( intr )
+ {
pop_state( &state_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"pop if intr in make1");
+ }

+ if ( DEBUG_FATE )
+ printf( " in %s (%d): %s->%s (%s)\n",
+ state_str[pState->curstate],
+ pState->t->asynccnt,
+ object_str( pState->t->name ),
+ fate_str[pState->t->fate],
+ pState->parent?object_str( pState->parent->name ):"" );
+
switch ( pState->curstate )
{
case T_STATE_MAKE1A: make1a( pState ); break;
@@ -266,6 +304,7 @@ static void make1a( state * const pState )
{
t->parents = targetentry( t->parents, parent_scc );
++parent_scc->asynccnt;
+ if ( DEBUG_FATE ) print_asynccnt(parent_scc,"+1 in make1a");
}
}

@@ -282,6 +321,7 @@ static void make1a( state * const pState )
if ( t->progress != T_MAKE_INIT )
{
pop_state( &state_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"pop if not T_MAKE_INIT in 
make1a");
return;
}

@@ -297,6 +337,7 @@ static void make1a( state * const pState )
* prematurely.
*/
t->asynccnt = 1;
+ if ( DEBUG_FATE ) print_asynccnt(t,"=1 in make1a");

/* Push dependency build requests (to be executed in the natural order). */
{
@@ -305,6 +346,7 @@ static void make1a( state * const pState )
for ( c = t->depends; c && !intr; c = c->next )
push_state( &temp_stack, c->target, t, T_STATE_MAKE1A );
push_stack_on_stack( &state_stack, &temp_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack, "push depends in make1a");
}

t->progress = T_MAKE_ACTIVE;
@@ -338,9 +380,12 @@ static void make1b( state * const pState )
/* If any dependencies are still outstanding, wait until they signal their
* completion by pushing this same state for their parent targets.
*/
- if ( --t->asynccnt )
+ --t->asynccnt;
+ if ( DEBUG_FATE ) print_asynccnt(t,"-1 in make1b");
+ if ( t->asynccnt )
{
pop_state( &state_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"pop if asynccnt>0 in make1b");
return;
}

@@ -353,11 +398,13 @@ static void make1b( state * const pState )
/* Append 't' to the list of targets waiting on semaphore. */
t->semaphore->parents = targetentry( t->semaphore->parents, t );
t->asynccnt++;
+ if ( DEBUG_FATE ) print_asynccnt(t,"+1 in make1b semaphore");

if ( DEBUG_EXECCMD )
printf( "SEM: %s is busy, delaying launch of %s\n",
object_str( t->semaphore->name ), object_str( t->name ) );
pop_state( &state_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack, "pop in semaphore in make1b");
return;
}
#endif
@@ -460,6 +507,7 @@ static void make1b( state * const pState )
if ( t->semaphore )
{
++t->semaphore->asynccnt;
+ if ( DEBUG_FATE ) print_asynccnt(t->semaphore,"+1 in make1b semaphore");
if ( DEBUG_EXECCMD )
printf( "SEM: %s now used by %s\n", object_str( t->semaphore->name
), object_str( t->name ) );
@@ -505,6 +553,7 @@ static void make1c( state const * const pState )
* been reused internally for some newly pushed state.
*/
pop_state( &state_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"pop in EXEC_CMD_OK in 
make1c");

/* Increment the jobs running counter. */
++cmdsrunning;
@@ -646,6 +695,7 @@ static void make1c( state const * const pState )
{
assert( t->semaphore->asynccnt == 1 );
--t->semaphore->asynccnt;
+ if ( DEBUG_FATE ) print_asynccnt(t->semaphore,"-1 in make1c semaphore");

if ( DEBUG_EXECCMD )
printf( "SEM: %s is now free\n", object_str(
@@ -677,6 +727,7 @@ static void make1c( state const * const pState )

/* Using stacks reverses the order of execution. Reverse it back. */
push_stack_on_stack( &state_stack, &temp_stack );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"pop and push in else 
branch in make1c");
}
}
}
@@ -908,6 +959,7 @@ static void make1c_closure
t->cmds = (char *)cmd_next( cmd );
cmd_free( cmd );
push_state( &state_stack, t, NULL, T_STATE_MAKE1C );
+ if ( DEBUG_FATE ) print_stack(&state_stack,"push in make1c_closure");
}

A8 ________________ dependency from "b2 -d13"

-> 0 Name: all
: Updating it
: NOTFILE
: Depends on <pbin/gcc-4.7.0/debug>hello_ext.so (stable) (max time)
: Depends on <p.>hello_ext.so (stable) (max time)
: Depends on <p.>libboost_python3.so (stable) (max time)
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello.test (update) (max time)
1 Name: <pbin/gcc-4.7.0/debug>hello_ext.so
Loc: bin/gcc-4.7.0/debug/hello_ext.so
: Stable
: Depends on bin/gcc-4.7.0/debug (stable)
: Depends on <pbin/gcc-4.7.0/debug>hello.o (stable) (max time)
: Depends on <p.>/usr/lib/libboost_python3.so (stable)
2 Name: bin/gcc-4.7.0/debug
: Stable
: NOUPDATE
: Depends on bin/gcc-4.7.0 (stable) (max time)
3 Name: bin/gcc-4.7.0
: Stable
: NOUPDATE
: Depends on bin (stable) (max time)
4 Name: bin
: Stable
: NOUPDATE
2 Name: <pbin/gcc-4.7.0/debug>hello.o
Loc: bin/gcc-4.7.0/debug/hello.o
: Stable
: Depends on bin/gcc-4.7.0/debug (stable)
: Depends on <p.-object(c-scanner)@180>hello.cpp (stable)
: Depends on <p.-object(c-scanner)@180>hello.cpp (internal node) (stable)
3 Name: <p.-object(c-scanner)@180>hello.cpp
Loc: hello.cpp
: Stable
3 Name: <p.-object(c-scanner)@180>hello.cpp (internal node)
: Stable
: NOTFILE
: Depends on <object(c-scanner)@180>boost/python/module.hpp (stable) 
(max time)
: Depends on <object(c-scanner)@180>boost/python/def.hpp (stable) (max time)
4 Name: <object(c-scanner)@180>boost/python/module.hpp
Loc: boost/python/module.hpp
: Stable
: NOCARE
4 Name: <object(c-scanner)@180>boost/python/def.hpp
Loc: boost/python/def.hpp
: Stable
: NOCARE
2 Name: <p.>/usr/lib/libboost_python3.so
Loc: /usr/lib/libboost_python3.so
: Stable
1 Name: <p.>hello_ext.so
Loc: hello_ext.so
: Stable
: Depends on . (stable)
: Depends on <pbin/gcc-4.7.0/debug>hello_ext.so (stable) (max time)
2 Name: .
: Stable
: NOUPDATE
1 Name: <p.>libboost_python3.so
Loc: libboost_python3.so
: Stable
: Depends on . (stable)
: Depends on <p.>/usr/lib/libboost_python3.so (stable)
-> 1 Name: <pbin/hello.test/gcc-4.7.0/debug>hello.test
Loc: bin/hello.test/gcc-4.7.0/debug/hello.test
: Updating it
:
: Depends on bin/hello.test/gcc-4.7.0/debug (stable)
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello (update) (max time)
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello (internal node) (stable)
2 Name: bin/hello.test/gcc-4.7.0/debug
: Stable
: NOUPDATE
: Depends on bin/hello.test/gcc-4.7.0 (stable) (max time)
3 Name: bin/hello.test/gcc-4.7.0
: Stable
: NOUPDATE
: Depends on bin/hello.test (stable) (max time)
4 Name: bin/hello.test
: Stable
: NOUPDATE
: Depends on bin (stable) (max time)
-> 2 Name: <pbin/hello.test/gcc-4.7.0/debug>hello
Loc: bin/hello.test/gcc-4.7.0/debug/hello
: Updating it
: Depends on bin/hello.test/gcc-4.7.0/debug (stable)
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello.py (missing)
: Depends on <pbin/gcc-4.7.0/debug>hello_ext.so (stable) (max time)
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello.py (internal node) 
(stable)
-> 3 Name: <pbin/hello.test/gcc-4.7.0/debug>hello.py
Loc: bin/hello.test/gcc-4.7.0/debug/hello.py
: Missing, creating it
: Depends on bin/hello.test/gcc-4.7.0/debug (stable)
: Depends on <p.>hello.py (newer) (max time)
: Depends on <p.>hello.py (internal node) (stable)
4 Name: <p.>hello.py
Loc: hello.py
: Newer
4 Name: <p.>hello.py (internal node)
: Stable
: NOTFILE
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello.py (missing)
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello.py (internal node) 
(stable)
5 Name: <pbin/hello.test/gcc-4.7.0/debug>hello.py (internal node)
: Stable
: NOTFILE
: Depends on <p.>hello.py (newer) (max time)
: Depends on <p.>hello.py (internal node) (stable)
2 Name: <pbin/hello.test/gcc-4.7.0/debug>hello (internal node)
: Stable
: NOTFILE
: Depends on <pbin/hello.test/gcc-4.7.0/debug>hello.output (stable) (max 
time)
3 Name: <pbin/hello.test/gcc-4.7.0/debug>hello.output
Loc: bin/hello.test/gcc-4.7.0/debug/hello.output
: Stable
: NOCARE

A9 ________________ overview of jam engine sources

I found some time to dig deeper into boost-build to answer my question and I
want to share it here.

In the engine folder there is the source for b2. It starts in *main()* 
in*jam.c*.
*parse_file* reads files provided by the -f option or else takes *Jambase*.
Then *make* generates the targets.

The file is parsed according to the *jamgram.y*, the core of which is 
the *rule* rule ::

rule : _LBRACE_t block _RBRACE_t
{ $$.parse = $2.parse; }
| INCLUDE_t list _SEMIC_t
{ $$.parse = pincl( $2.parse ); }
| ARG lol _SEMIC_t
{ $$.parse = prule( $1.string, $2.parse ); }
| arg assign list _SEMIC_t
{ $$.parse = pset( $1.parse, $3.parse, $2.number ); }
| arg ON_t list assign list _SEMIC_t
{ $$.parse = pset1( $1.parse, $3.parse, $5.parse, $4.number ); }
| RETURN_t list _SEMIC_t
{ $$.parse = $2.parse; }
| FOR_t local_opt ARG IN_t list _LBRACE_t block _RBRACE_t
{ $$.parse = pfor( $3.string, $5.parse, $7.parse, $2.number ); }
| SWITCH_t list _LBRACE_t cases _RBRACE_t
{ $$.parse = pswitch( $2.parse, $4.parse ); }
| IF_t expr _LBRACE_t block _RBRACE_t
{ $$.parse = pif( $2.parse, $4.parse, pnull() ); }
| MODULE_t list _LBRACE_t block _RBRACE_t
{ $$.parse = pmodule( $2.parse, $4.parse ); }
| CLASS_t lol _LBRACE_t block _RBRACE_t
{ $$.parse = pclass( $2.parse, $4.parse ); }
| WHILE_t expr _LBRACE_t block _RBRACE_t
{ $$.parse = pwhile( $2.parse, $4.parse ); }
| IF_t expr _LBRACE_t block _RBRACE_t ELSE_t rule
{ $$.parse = pif( $2.parse, $4.parse, $7.parse ); }
| local_opt RULE_t ARG arglist_opt rule
{ $$.parse = psetc( $3.string, $5.parse, $4.parse, $1.number ); }
| ON_t arg rule
{ $$.parse = pon( $2.parse, $3.parse ); }
| ACTIONS_t eflags ARG bindlist _LBRACE_t
{ yymode( SCAN_STRING ); }
STRING
{ yymode( SCAN_NORMAL ); }
_RBRACE_t
{ $$.parse = psete( $3.string,$4.parse,$7.string,$2.number ); }
;

There we have *RULE_t* (='rule' in *jamgramtab.h*), which is a jam rule. 
(Note that all three *rule* words mean something different.)
And then there is *ARG lol _SEMIC_t*, which calls a jam rule. *lol* is 
list of lists (:), *arg* is ARG (identifier) or inline function call 
(e.g. [ARG lol]).

The functions like *psetc* call *parse_make* with an according constant 
(*psetc --> PARSE_SETCOMP*) to
build a parse tree, which is compiled with *function_compile* (rule is 
synonymous to function) and then run by *function_run*.
Here is the according excerpt from the *parse_file* function ::

if ( yyparse() || !( p = yypsave ) )
break;
/* Run the parse tree. */
func = function_compile( p );
parse_free( p );
list_free( function_run( func, frame, stack_global() ) );
function_free( func );

*function_compile* makes a byte code representation of the jam file or 
rules therein (file itself is a rule).
*function_run* interprets it.
For instance *PARSE_SETCOMP* leads to *INSTR_RULE* (with arguments) in 
the code array (=*compiler*, via *compile_emit*).
*function_run* interprets this by calling *function_set_rule, 
new_rule_body*.
This basically builds a tree of rules in rules as given by the grammar.

When *INSTR_CALL_RULE* (corresponding to prule, PARSE_RULE) is 
encountered interpretation goes via *function_call_rule*,
*evaluate_rule*, which on its turn recurses to *function_run*.
A rule's scope is called *frame*. Actual arguments are handed over via a 
stack.

A rule can have an associated actions block (*ACTIION_t='actions'* with 
the rule name).
In *psete* (*PARSE_SETEXEC*) the block is only registered as string and 
later, during compilation,
parsed via *compile_emit_actions, function_compile_actions, 
parse_var_string, ...* and added to the compiler array with id 
*INSTR_ACTIONS*,
*parse_var_string* makes an array of constant strings and variables.
During interpretation the actions are associated to the rule.
In *evaluate_rule* before *function_run* actions get their sources and 
targets (both of type *TARGETS* as defined in *rules.h*).

*targets* are associated to a rule (*bindtarget*) by calling the builtin 
*DEPENDS $(<) : $(>) ;* and *INCLUDES $(<) : $(s) ;* or by the *on* 
(*ON_t*) syntax.
With actions and targets (and builtins) the jam language get its link to 
the system.
After parsing, compiling and interpreting all jam modules loaded 
starting from Jambase, *make()* deals with the execution of the actions.

In *jam.c* we have::

LIST * const targets = targets_to_update();
if ( !list_empty( targets ) )
status |= make( targets, anyhow );

The targets to update are provided by the user and are added to the list 
via the builtin *UPDATE* (*builtin_update*) in build-sytem.jam.
*make()* calls *make0()* for preparations and then *make1()* to execute 
the actions in three steps:
- *make1a()* to schedule builds
- *make1b()* for *make1cmds()*
- *make1c()* for *exec_cmd()* or schedule for parents

Jam grammar primitives are ::

! , != , & , && , ( , ) , += , : , ; , < , <= , = , > , >= , ?= , [ , ] 
, { , | , || , } ,
actions , bind , case , class , default , else , existing , for , if , 
ignore , in , include ,
local , module , on , piecemeal , quietly , return , rule , switch , 
together , updated , while

but also '$' or '@' get special treatment in the engine.

*class* with "class.jam" allows classes and instances.

The builtins are ::

ALWAYS, DEPENDS, ECHO, EXIT, GLOB, GLOB-RECURSIVELY, INCLUDES, REBUILDS, 
LEAVES, MATCH, SPLIT_BY_CHARACTERS,
NOCARE, NOTFILE, NOUPDATE, TEMPORARY, ISFILE, HDRMACRO, FAIL_EXPECTED, 
RMOLD, UPDATE, UPDATE_NOW, SUBST,
RULENAMES, VARNAMES, DELETE_MODULE, IMPORT, EXPORT, CALLER_MODULE, 
BACKTRACE, PWD, IMPORT_MODULE, IMPORTED_MODULES,
INSTANCE, SORT, NORMALIZE_PATH, CALC, NATIVE_RULE, HAS_NATIVE_RULE, 
USER_MODULE, NEAREST_USER_LOCATION,
CHECK_IF_FILE, PYTHON_IMPORT_RULE, W32_GETREG, W32_GETREGNAMES, COMMAND, 
MD5, FILE_OPEN, PAD, PRECIOUS, SELF_PATH, MAKEDIR,

A10 ________________ What is run-pyd?

Original question:

| python.jam refers to run-pyd and so does the jamroot test file.
| Grep for run-pyd in boost-build and googling did not give me any 
further insight.
| b2 --perserve-test-targets does not leave anything in the test folder 
either.
| I'm on Archlinux. When starting hello.py manually it works.
| Any hint would be appreciated.

In boost-python tutorial example's Jamroot we have::

local rule run-test ( test-name : sources + )
{
import testing ;
testing.make-test run-pyd : $(sources) : : $(test-name) ;
}

In python.jam there is ::

type.register RUN_PYD_OUTPUT ;
type.register RUN_PYD : : TEST ;

*type.register* refers to the rule *register* in the file *type.jam*.
*RUN_PYD* is *type* with base *TEST*, similarly for *RUN_PYD_OUTPUT*.

Then we have ::

generators.register
[ new python-test-generator python.capture-output : : RUN_PYD_OUTPUT ] ;

generators.register-standard testing.expect-success
: RUN_PYD_OUTPUT : RUN_PYD ;

Both register a generator from a source-type (none or *RUN_PYD_OUTPUT*) 
to a target-type (*RUN_PYD_OUTPUT* or *RUN_PYD*)
The rules *python.capture-output* and *testing.expect-success* prepare 
variables and the according *actions* generate the target.

Now what is *run-pyd*? The answer is in *type.register* where the 
following rule transforms "RUN_PYD" to "run-pyd" ::

rule type-to-rule-name ( type )
{
# Lowercase everything. Convert underscores to dashes.
import regex ;
local n = [ regex.split $(type:L) "_" ] ;
return $(n:J=-) ;
}

In *testing.make-test* the type is found by the rule name "run-pyd".