Re: [OpenMethod] A pre-release feedback

From: Jean-Louis Leroy (jl_at_[hidden])
Date: 2025-04-14 20:43:18

Hi Andrzej,

Thank you for taking the time to take a look at my library. I agree with some
of your points, and I will try to explain some of the choices I made.

Before going over your remarks, a bit of context: OpenMethod is derived from
YOMM2 (https://github.com/jll63/yomm2, started in 2017), which is a rewrite of
YOMM11 (2013). I presented it at several conferences, the most prestigious being
CppCon, in 2018. My most recent talk was at Core C++ in Israel, last year. The
latest video of a talk is from using std::cpp in Madrid, here:
https://youtu.be/cOYE6OiuuMo?si=wMsSADt3_sEC-qz1.

I know that YOMM2 has been used in production in several places (like the Tokyo
Institute of Technology), by people who cared enough to send me PRs, bug reports
and suggestions. Also, vanity Google searches turned up conversations with
positive remarks on YOMM2, between people I never heard of.

> [1]. Animals as examples

> There are already a number of things in the library that suggest it might not
> be suitable for production use (I describe them below), but using class Animal
> as explanatory examples amplifies the impression that this is going to be a
> toy library.

> The arithmetic expression example is way better.

I agree that Animals doesn't look "serious". If you watch my talks, you will see
that I never use it. My favorite would be Matrix: it uses concepts everybody
knows already, AND it can be used to demonstrate both uni- and multi-methods. It
deals with some interesting cases, like matrix transposition, where you probably
want to return the same matrix if it is symmetrical. But it may give the
impression that it's all about multiple dispatch. Nay, "multi" is just the
cherry on the cake.

My second most favorite is the AST. But it does not demonstrate multiple
dispatch, nor `next`.

Animals, on the other hand, has all three: uni, multi, next. The Dog/Bulldog
example is the best way I've found to illustrate `next` in a snap. That is why I
use Animals in the tutorial, but I am open to switching to another example.

> [2]. The motivation

> In C++ we already have a dynamic-dispatch tool for "closed set of methods with
> open set of types": virtual functions. We also have a dynamic-dispatch tool
> for "open set of methods with closed set of types": the visitation of
> std::variant.

Why can't we have both at the same time?

The very fact that we have those two tools proves that extensibility is
desirable along both axes: behaviors and types. Currently there is no easy way
to write a library that is extensible in both directions. I believe that tying
polymorphism to membership was the main cause of the failure of the
Simula/Smalltalk/C++ way of OOP. Lisp was right.

> You need to provide a motivating example (more convincing than the zoo) that
> needs exactly this: "open set of methods with the open set of types". The
> arithmetic expression example doesn't do the job: there is a fixed and well
> known list of concrete types.

The papers authored by Stroustrup and his PhD students Pirkelbauer and Solodkyy
contain discussions of several examples. The most "recent" is N2216
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2216.pdf).

I disagree about AST. The narrative is: I have that AST library. One of my
apps needs to represent the AST in JSON. If I own the library, I can add a
virtual function, but then all the other apps get it, need it or not. And also
the function's dependencies, e.g. the iostream library if `to_json` writes to a
`std::ostream`.

I should explain in the tutorial how open-methods can help with decoupling. It's
not just about the Expression Problem. It's also about the banana-gorilla-jungle
problem.

If AST is a 3rd party library, it might provide a Visitor, if I'm "lucky". But
Visitor is so clumsy. See here
https://github.com/jll63/Boost.OpenMethod/blob/master/examples/slides.cpp for
implementations of `to_json` using type switches, function pointers, Visitor and
open-methods. This talk is worth watching too:
https://www.youtube.com/watch?v=QhJguzpZOrk .

When a program starts running, the set of operations and types is usually fixed
(barring dynamic loading). Many such programs still use virtual functions, for
many good reasons. They don't exist in isolation; they are part of a collection
of programs that use internal and 3rd party libraries. It's about architecture.

Ultimately, open-methods are a better version of virtual functions. More
flexible, not toxic.

> Also, the fact that Bjarne Stroustrup wanted to add open methods long ago is
> insufficient a motivation.

It is insufficient, but _not_ _insignificant_. I have a megaton of respect for
the man and his intelligence. D&E is one of my favorite books.

> [3]. Are open methods compatible with static type checking?

> It looks that when one of the two things -- number of types, number of methods
> -- is known statically, the compiler can verify if the programmer didn't
> forget to take some type or method into account. [...] The same seems not to
> be the case for open methods.

Correct. Full static validation can be done only with a complete view of the
entire program.

> The ISO C++ proposals quoted in the docs mentioned solving this by a
> mandated or encouraged linker error. Is it possible for this library to
> diagnose the incomplete specification of an open method in the
> `initialize()` function?

Yes. `initialize()` returns an object of an unspecified type that contains
various metrics and diagnostics, like the number of methods missing overrides
for some combinations of virtual arguments. By the time the review starts, I
will document some of this. Also, the trace contains that info.

I also have plans for a global analysis tool, based on clang, that would create
the dispatch tables as a generated source file; it would also generate hints for
static offsets in the v-tables, thus beating the speed of Stroustrup's
prototype. That tool would also report possible errors, without the need to run
the program.

> [4] In a similar vein, the decision to select an arbitrary specialization of
> an open method upon a tie, is another runtime surprise: not even a reported
> error. For multi methods it is not even clear how a static check for
> exhaustive unambiguous specialization should work.

I don't like it either. YOMM2 reports an "error" in that case. Stroustrup & co
argue that all best overriders are "correct", so why crash the program? Also, I
am lukewarm on using return types as tie-breakers, in some circumstances. As I
understand it, their reasoning is guided by considerations on dynamic loading.

The number of ambiguities is available in `initialize()`'s return value - to be
documented.

I am open to adding a customization point to make ambiguous calls runtime
errors.

> [5] It is concerning that so many things are defined in the global namespace.

Please bear in mind that OpenMethod (like YOMM2) attempts to emulate a
_language_ _feature_. The names it publishes are similar to language keywords.
`virtual_ptr` is like `virtual`, and `virtual` is global ;-)

> The docs say, "First we need to include the library’s main header. It defines
> a few macros, and injects a name - virtual_ptr - in the global namespace." But
> then the reference part says that virtual_ptr is in the namespace
> boost::openmethod. Who is right?

Both. <boost/openmethod.hpp> contains `using boost::openmethod::virtual_ptr`. OK
I can change "injects" to "aliases".

You can use the library with zero macro (well, just include guards) and zero
"injections" if you like - that's <boost/openmethod/core.hpp>. And if you want
the macros but not the alias, you can include <boost/openmethod/core.hpp> and
<boost/openmethod/macros.hpp>. I will improve the doc on this...

I am also not completely opposed to removing the injection of `virtual_ptr`.

>  The fact that as common a name as `next` is reserved inside open methods is
> also uncomfortable. Why not use a namespace-scoped name?

There are two `next`s. One is scoped in boost::openmethod::method (core API).
The other is inside the body of a BOOST_OPENMETHOD_OVERRIDE (keyword API). It
doesn't exist outside of it.

I can rename it to something else. In CLOS, it is called call-next-method. In
CLOS parlance, a method is a "generic" (defgeneric), and an overrider is a
"method" (defmethod). `next` can become call_next_overrider, no biggie...

> Next, the docs use the using-directive: `using namespace boost::openmethod;`.
> This is a bad idea: both for real programs and for the documentation: it is
> not clear if the library was designed with namespaces in mind.

Oh it was. And I thought a lot about `using namespace` in the doc.

Reading documentation is not the same thing as writing a real-life large
program. Long scoped names can get in the way while learning. The goal is
different.

I looked at the documentation of many Boost libraries. Some assume that their
namespace is in scope. Others use the `namespace bom = boost::openmethod; ...
bom::virtual_ptr` approach - which is just as "bad", right?

But I am flexible on this...

> Also, all the examples define open methods in the global namespace. I instead
> would like the docs to show how to declare open methods in my namespace, as
> this is a far more probable use case.

I should add an entry on namespaces to the tutorial.

[6] Friendship

> It puzzles me how the notion of friendship can be combined with this idea of
> flexibility in open methods. It is my understanding that open methods are
> added on top of the existing classes. That is: classes do not have to know
> that some open method will be added in the future. In other words, open
> methods are non-intrusive, right? So how would a class know who to grant the
> friendship to?

I used to agree with this. Targeted friendship seems so much in contradiction
with openness. On the other hand, all features of C++ should play ball with one
another. Friendship is part of the language.

Also, I am not a nanny.

I was often asked about friendship during the Q&A of my talks. I replied: You
are probably doing something wrong if you need it, but I already have a design
for it, and I will implement it when someone comes up with a good example where
it makes sense. And then someone did. Discussion is here:
https://github.com/jll63/yomm2/issues/7

Ponder this: polymorphism and encapsulation are orthogonal concepts, right? But
traditional virtual functions have access to the private and protected parts.
You just wanted polymorphism, but look!, you get access to private parts as
well, like it or not. Well, with open-methods, you can have _just_ polymorphism.
Better virtual functions, not toxic :)

> Also, the examples use the declaration `friend struct`, even though we are
> talking about methods. I think docs should mention the existence of these
> structs more prominently.

Maybe. It is mentioned in two places
(https://jll63.github.io/Boost.OpenMethod/#tutorials_friendship,
https://jll63.github.io/Boost.OpenMethod/#BOOST_OPENMETHOD_OVERRIDE).

I can see two reasons to need this info. You just want to grant friendship, in
which case you can just follow the recipe; or you are doing more evolved things,
like mixing core and keyword API and perhaps templates. In that case, you don't
shy away from reading and decanting the reference, and I think all the info is
there. But let's see what others say...

> [7] Function signatures
>
> Some macros in the library use function types in declarations:
> `BOOST_OPENMETHOD_OVERRIDERS(poke)<void(std::ostream&, virtual_ptr<Cat>)>`
>
> Others specify the type separately:
> `BOOST_OPENMETHOD_OVERRIDE(value, (virtual_ptr<Plus> node), int)`
>
> Is there a reason for this? Can the clever use of macros not offer the same
> function signature syntax?

These macros came out the way they are after trying many alternatives. CPP
macros are surprisingly versatile, but they're not Lisp macros.

I would like the syntax to be:

  BOOST_OPENMETHOD(poke(virtual_ptr<Animal>, std::ostream&), void);
  BOOST_OPENMETHOD_OVERRIDE(poke(virtual_ptr<Dog> dog, std::ostream& os), void)
  BOOST_OPENMETHOD_OVERRIDERS(poke(std::ostream&, virtual_ptr<Cat>, void)

But I don't think it's feasible.

However, this is probably possible:

  BOOST_OPENMETHOD_OVERRIDERS(poke, (std::ostream&, virtual_ptr<Cat>), void)

Which mirrors nicely:

  BOOST_OPENMETHOD_OVERRIDE(poke, (std::ostream&, virtual_ptr<Cat>, void)

Let me try this, it looks like a good idea.

> [8] Passing virtual pointers to const objects

> If `virtual_ptr<Cat>` is an analogue of this pointer, could we see an
> example in the docs that shows a virtual pointer to const objects? It would
> be `virtual_ptr<const Cat>`, I guess?

Yep! There, it looks like it's quite predictable, innit? :)

The tutorial is ready and willing to evolve. I wanted to start light because I
feel that, over the years, YOMM2's doc has become bloated, disorganized...

> [9] Is this library intended for production use?

Intended? Yes, definitely. Ready is a better question.

YOMM2 has existed for eight years. It has preserved backward compatibility
throughout. It has evolved a lot. I got bug reports. I fixed them. I know that
it has been used in production in more than one place.

Now OpenMethod is not only YOMM2, renamed. It also contains "new research", so I
cannot honestly say it is as field-tested as YOMM2; yet the new ideas address
real problems. For example, moving the method's return type to the end of the
parameter lists of BOOST_OPENMETHOD and BOOST_OPENMETHOD_OVERRIDE. It is not
(only) to look "more modern". It actually solves a real annoyance with YOMM2: it
does not cope well with return types that contain commas, so you have to use
workarounds.

> The library comes with a lot of clever macros, a lot of pointer type
> templates, a lot of new syntax and runtime surprises (mentioned earlier). I
> am concerned about how the debugging would look if something goes wrong
> with the usage of this library.

The inherent reduction in static checking has always been on my mind. I say
"inherent" because in presence of dynamic loading (which YOMM11, YOMM2 and
OpenMethod all support), what looks like a bug when you look at the main program
vanishes when you (dynamically) load a shared object or a DLL. You mentioned
plugins, you see what I mean.

In debug builds, the library performs all sorts of runtime checks. If you forget
to register a class, you will get an error message.

If you turn on trace, you will get a very detailed explanation of what classes,
methods, and overriders the library knows about, and how they relate. It will
explain how it selects the best and next best overriders for each virtual
tuple. It will tell you which slots in the v-tables are assigned to which
methods.

If you trace through a method call with a debugger - something I have done a lot
-, with a little bit of practice, you will understand what is going on. Even if
you make use of customization points. The code is organized to be step-in
friendly.

Try it :)

> Would you recommend it for use in production systems? Or is it for
> experimentation?

In 2013 I started my work on open-(multi-)methods for C++ because:
- I used multi-methods in previous projects in other languages.
- I worked on a project that would have really benefited from
  open-methods (open-uni-methods in this instance).
- I fantasized that I could help Stroustrup promote the idea by giving people a
  way to experiment with it. And then I would retire YOMM.

However, YOMM11, and its successors, were never "toys". For example, they build
redundancy free dispatch tables for multi-methods. That is not necessary for
proof-of-concept, but it is for a production capable library.

Recent releases of YOMM2 added customization points to address real-life
problems. For example, in the game industry, they disable RTTI, and use custom
systems instead. Because games are an important use-case, I make it possible to
interface with custom RTTI.

I came to recognize that we will have a colony on Mars before open-methods make
it into the standard. So I posit OpenMethod as the next best option.

> [10] Minor things

> Section Custom RTTI has sentence:
> > Now we can include the "core" header
>
> and then header `boost/openmethod/core.hpp` is not included. Is it a bug?

`boost/openmethod.hpp` includes `boost/openmethod/core.hpp` but you are right,
I'll fix the doc, thanks :)