Re: [Boost-docs] [quickbook] Direct code import...

Subject: Re: [Boost-docs] [quickbook] Direct code import...
From: Daniel James (dnljms_at_[hidden])
Date: 2011-10-30 10:11:37

On 30 October 2011 04:14, Rene Rivera <grafikrobot_at_[hidden]> wrote:
> First, I need to say I'm not forcing this on anyone.

As far as I can tell, no one agrees with me on this. So it'll probably go in.

> Well, there's at least one problem with using "include".. It would mean
> duplicating the file parsing that "import" already does. And if we
> restructure so that the grammar can be used, it would then complicate the
> "import" also since it has to now also ignore the escape comment parts that
> "include" is using. Which I guess means that both the file inclusion and
> file import should be combined into a comprehensive file reading and parsing
> mechanic. And then based on the template name doing the parsing we either do
> import like things, or we do include like things.

It was easier to just port your implementation than to answer this:

https://svn.boost.org/trac/boost/changeset/75175/

I'm not entirely happy with that as it means that different blocks of
escaped quickbook are processed in different scopes, so a template
defined in one won't be available in the next.

>> I'd rather use explicit syntax to distinguish between paths and
>> patterns. This change breaks the windows path support, which I think
>
> Technically it doesn't break it. As long as you don't use a regex for the
> file it behaves as before.

I should have explained that your change thinks a windows path is a
regex, as '\' is used for file separators.

> OK. But would a more general concept of typed value quoting/escaping/parsing
> be better?

I wouldn't want to complicate the normal inclusion syntax for this
purpose, which almost always works fine for file names (using things
like square brackets isn't really portable).

>> I'd also use a version switch as well. I think things like this should
>> be approached with some caution.
>
> How would a version switch actually work here. I can only see it working
> with linear development. What I have is a branch.

The idea is that when integrated into trunk, the changes won't effect
files that were written for an older version of quickbook. It also
means language changes can be checked into trunk, without committing
to them. So I implement quickbook in a single branch, but can have
features that can be changed in the future.

>> I haven't tested it, but I don't think your code will work on windows.
>> For example, the line:
>>
>>     std::string f = i->path().filename().native();
>>
>> should probably be something like:
>>
>>     std::string f = detail::path_to_generic(i->path().filename());
>
> That would be true if 'f' was a full path. But it's only the file name part
> which can't have the characters that path_to_generic manipulates.
> Additionally this code is never executed in a backward compatibility context
> since it's only when one specifies a regex as the argument to the import.

This doesn't work because on windows files are encoded in UTF-16, so
'native' returns a wide string. 'native_string' would return the
correct string type, but wouldn't convert it correctly to UTF-8.
Filesystem can convert paths correctly using locales, but I wasn't
able to get that right. Boost.Locale might help here but I implemented
this before it was accepted, and I'd rather wait until it's known to
be stable before adapting it.

Also, I'm thinking of using urls for paths in the future, and I think
url syntax is slightly different to filesystem's generic path syntax,
so path_to_generic might do something extra. It'd be useful if paths
were always converted with it, even if it seems unnecessary, so that
when making changes I can be sure that's the case.

>> If there are matches at different points in the search path, they're
>> ordered by absolute path which may vary on different computers. I
>> think my preference would be to keep them in the order they appear in
>> the search path, with 'filename_relative' as a secondary sort field in
>> case of collisions. Or perhaps 'filename_relative' could be the
>> primary sort field, and the include path order the secondary sort
>> field.
>
> Good point. Although the ordering within different search paths is likely to
> never come up. As it's more likely that if one is concerned about some
> specific ordering one would put the imports/inclusions in a specific order..
> As I did in my use case.

I really want quickbook to always produce exactly the same
documentation for the same source, or as close as realistically
possible. The helps reduce changeset when people are checking
generated documentation into subversion, and makes it easier to
compare documentation generated at different times.

> But your idea is contradictory to your previous point about using "include"
> instead of "import". The above would be, according to your definition of the
> semantics, a combined import and include. Which is what I implemented. So
> perhaps we want to reconsider the semantics (and names) of "include" and
> "import".

No, the code snippets are only in scope within the included file. This
is the same as including a quickbook file (for 1.6). Although being
able to use a snippet from later in the file is different to using
templates.

I'll try to explain the motivation behind this. Currently when you
include a file, macros are scoped but templates aren't (there was a
'fixme' comment in the source saying that templates should be scoped).
To fix that there needs to be a mechanism for importing templates from
a file, my original thought was to adapt 'include' but then I realised
that the semantics matched the existing use of 'import' quite closely.
Hopefully it'll also be more intuitive, as 'include' is similar to a
C++ include, 'import' is similar to a python import.