- Boost - lists.boost.org

Re: [boost] Amazon SES
by Andrey Semashev 04 May '23

04 May '23

On 5/4/23 18:54, Sam Darwin via Boost wrote: > The comments about Message-ID: , References: , and In-Reply-To: > headers are very interesting. "The Amazon Simple Email Service (AWS > SES) rewrites the Message-ID header of outgoing emails." > Consider the very first email of a thread. It arrives at the mailman > server with a Message-ID that mentions "gmail", for example, and is > added to the Archives. The message is sent out to all list members, > who see a new Message-id of > ...010f0187dca41ec8(a)us-east-2.amazonses.com Later replies can set > the In-Reply-To header to refer the "@us-east-2.amazonses.com" > Message-ID. But the first archived message didn't yet have that ID, > since it arrived directly at the server, not through SES. The mailman > archives on lists.boost.org organize that first message in a separate > thread, presumably based on the original Message-ID. Can we disable Message-ID rewriting? Do we know how mailgun works wrt. Message-ID and other headers?

1 0

Re: [boost] Amazon SES
by Sam Darwin 04 May '23

04 May '23

The comments about Message-ID: , References: , and In-Reply-To: headers are very interesting. "The Amazon Simple Email Service (AWS SES) rewrites the Message-ID header of outgoing emails." Consider the very first email of a thread. It arrives at the mailman server with a Message-ID that mentions "gmail", for example, and is added to the Archives. The message is sent out to all list members, who see a new Message-id of ...010f0187dca41ec8(a)us-east-2.amazonses.com Later replies can set the In-Reply-To header to refer the "@us-east-2.amazonses.com" Message-ID. But the first archived message didn't yet have that ID, since it arrived directly at the server, not through SES. The mailman archives on lists.boost.org organize that first message in a separate thread, presumably based on the original Message-ID.

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Vinnie Falco 04 May '23

04 May '23

On Wed, May 3, 2023 at 11:44 PM Julien Blanc via Boost <boost(a)lists.boost.org> wrote: > the "replacing the ML with a forum" That's not happening, no one is replacing the ML, and our efforts to renovate the Boost website have exactly one overarching principle: "Do not fuck with the mailing lists" The mailing lists must not be touched, they need to work exactly the way they did before (with working threading). Thanks

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Julien Blanc 04 May '23

04 May '23

Le 2023-05-03 15:02, Sam Darwin via Boost a écrit : > Mailgun has been in production for many years. It costs $500 per month > to send smtp messages. It's a reasonable idea to try out another > service that's only 5% of the cost. Thanks for the detailed answer. From what is seen, it does indeed looks like amazon ses is sending emails with different message ids to each participant in the mailing list. Mailgun, on the other hand, was keeping the original message id in the email. > But at the same time, it's not impossible to switch back to mailgun, > if threading is partially disrupted, and we can't find a solution. Now that the source of the problem has been identified, i hope a solution can be found. Like Phil noted, mailgun is indeed very expensive. Best regards, Julien

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Andrey Semashev 04 May '23

04 May '23

On 5/4/23 10:20, Gavin Lambert via Boost wrote: > On 4/05/2023 18:34, Julien Blanc wrote: >> You're luckier than me. The current thread is semi-broken, but the >> "replacing the ML with a forum" one is completely broken for me. > > Sorry, that was sloppy reporting on my part. There was a large tree of > posts that were all together so I foolishly assumed that this was the > whole thread. > > There were *also* a bunch of disconnected replies earlier on, but since > these were completely disconnected they didn't "follow" the parent > higher in the list since I'm sorting by date. This meant they were out > of sight at the time I was thinking about this and writing my message. > >> I'm starting to wonder if we all receive the same emails (in the sense >> that they may not contain the same set of headers). For me, the >> initial message of Vinnie in the "replacing the ML with a forum" >> thread contains the following: >> >> Message-ID: >> <010f0187bde5599a-e76c27f2-225d-4d7e-b2c8-887e3a07ecb8-000000(a)us-east-2.amazonses.com> >> >> But Mashshall's answer, on the other side, contains the following >> headers : >> >> References: >> <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> >> In-Reply-To: >> <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> >> >> The "In-Reply-To" should match the Id of the initial message, for >> threading to work correctly. This is too obvious to be a client bug. >> Can someone check the Message-ID he has for the initial message of >> Vinnie in the "replacing the ML with a forum" thread? If it's >> different from mine, then that's probably the source of the issues. > > For me (via gmane), Vinnie's original message has: > > Message-ID: > <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> > > And Marshall's reply (did not thread properly) has: > > Message-ID: > <010f0187bde8bc2e-bebb94d1-2dfb-4165-ad8d-6af8e9b41381-000000(a)us-east-2.amazonses.com> > References: > <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> > In-Reply-To: > <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> > > Robert's reply (did thread properly) has: > > Message-ID: > <010f0187bf4b951d-1f6d7af8-3f2e-458b-9ffd-ca91ecfe04f9-000000(a)us-east-2.amazonses.com> > References: > <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> > In-Reply-To: > <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> > > These do indeed seem to be different from yours, which is unfortunate. > > It would be good if someone else not using gmane could check both > results as well (although that shouldn't be a factor, as I have other > groups that don't appear to be getting flattening). I'm reading the ML in Thunderbird. Vinnie's initial message has: Message-ID: <010f0187bde531b6-2cd00e40-57e1-476f-b30d-896ff8eab806-000000(a)us-east-2.amazonses.com> Marshall's reply: References: <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> Message-ID: <010f0187bde8ccef-31042da0-3d20-4f4a-95e8-74cd3770cc89-000000(a)us-east-2.amazonses.com> Robert's reply: Message-ID: <010f0187bf4ba591-77f55894-1b8b-40d4-b8ac-0d6af2b9ba6d-000000(a)us-east-2.amazonses.com> References: <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> For me, both Marshall's and Robert's replies, which are direct replies to the initial Vinnie's post, start their own threads with their subtrees of replies. The only descendant of the Vinnie's initial post is my own reply to it, which has this: Message-ID: <187be1c7608.280f.fb49792bc380c7e55e30f872b414f11c(a)gmail.com> In-Reply-To: <010f0187bde531b6-2cd00e40-57e1-476f-b30d-896ff8eab806-000000(a)us-east-2.amazonses.com> References: <010f0187bde531b6-2cd00e40-57e1-476f-b30d-896ff8eab806-000000(a)us-east-2.amazonses.com> So basically, every direct reply to the original post, except my own one, is a separate subtree (or a single out-of-tree message if there were no replies to it).

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Gavin Lambert 04 May '23

04 May '23

On 4/05/2023 18:34, Julien Blanc wrote: > You're luckier than me. The current thread is semi-broken, but the > "replacing the ML with a forum" one is completely broken for me. Sorry, that was sloppy reporting on my part. There was a large tree of posts that were all together so I foolishly assumed that this was the whole thread. There were *also* a bunch of disconnected replies earlier on, but since these were completely disconnected they didn't "follow" the parent higher in the list since I'm sorting by date. This meant they were out of sight at the time I was thinking about this and writing my message. > I'm starting to wonder if we all receive the same emails (in the sense > that they may not contain the same set of headers). For me, the initial > message of Vinnie in the "replacing the ML with a forum" thread contains > the following: > > Message-ID: > <010f0187bde5599a-e76c27f2-225d-4d7e-b2c8-887e3a07ecb8-000000(a)us-east-2.amazonses.com> > > But Mashshall's answer, on the other side, contains the following headers : > > References: > <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> > In-Reply-To: > <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> > > The "In-Reply-To" should match the Id of the initial message, for > threading to work correctly. This is too obvious to be a client bug. Can > someone check the Message-ID he has for the initial message of Vinnie in > the "replacing the ML with a forum" thread? If it's different from mine, > then that's probably the source of the issues. For me (via gmane), Vinnie's original message has: Message-ID: <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> And Marshall's reply (did not thread properly) has: Message-ID: <010f0187bde8bc2e-bebb94d1-2dfb-4165-ad8d-6af8e9b41381-000000(a)us-east-2.amazonses.com> References: <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> Robert's reply (did thread properly) has: Message-ID: <010f0187bf4b951d-1f6d7af8-3f2e-458b-9ffd-ca91ecfe04f9-000000(a)us-east-2.amazonses.com> References: <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187bde523cc-f60c3688-3843-44d4-b848-f37b2fd6a598-000000(a)us-east-2.amazonses.com> These do indeed seem to be different from yours, which is unfortunate. It would be good if someone else not using gmane could check both results as well (although that shouldn't be a factor, as I have other groups that don't appear to be getting flattening). The threads that are semi-flat appear to be because it did match one of the References but not all of them. For example, in this thread, your reply was: Message-ID: <010f0187e5785f8e-23fd3577-ced2-4f4a-a235-6b7a24d0022c-000000(a)us-east-2.amazonses.com> References: <010f0187d80eb0dd-93bebdd9-cabf-4ad6-a7f4-b5515bd22d31-000000(a)us-east-2.amazonses.com> <010f0187dca41ec8-478dfff5-d27d-4879-ab72-072be2f76176-000000(a)us-east-2.amazonses.com> <010f0187e09fef87-f959fd75-f882-47dc-b20c-f103c943383d-000000(a)us-east-2.amazonses.com> <010f0187e141f8b1-75bb05d0-8b61-4190-af34-2efb57ce70df-000000(a)us-east-2.amazonses.com> <CANEqPjFF80D2dp5pAZfo1aF7YHt73_cs5wFgENNZ40_piTZnvA(a)mail.gmail.com> <010f0187e2536549-44ef0b67-13ba-420a-8c95-50e5212ccbf5-000000(a)us-east-2.amazonses.com> <010f0187e400f951-96bb316d-68ca-4d38-a08a-6f9cfb56ec78-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187e400f951-96bb316d-68ca-4d38-a08a-6f9cfb56ec78-000000(a)us-east-2.amazonses.com> Replying to my message: Message-ID: <010f0187e400b04d-bdb05955-7e9a-4aae-b4f5-cde46033a981-000000(a)us-east-2.amazonses.com> References: <010f0187d80eb0dd-93bebdd9-cabf-4ad6-a7f4-b5515bd22d31-000000(a)us-east-2.amazonses.com> <010f0187dca41ec8-478dfff5-d27d-4879-ab72-072be2f76176-000000(a)us-east-2.amazonses.com> <010f0187e09fef87-f959fd75-f882-47dc-b20c-f103c943383d-000000(a)us-east-2.amazonses.com> <010f0187e141f8b1-75bb05d0-8b61-4190-af34-2efb57ce70df-000000(a)us-east-2.amazonses.com> <CANEqPjFF80D2dp5pAZfo1aF7YHt73_cs5wFgENNZ40_piTZnvA(a)mail.gmail.com> <010f0187e2536549-44ef0b67-13ba-420a-8c95-50e5212ccbf5-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187e2536549-44ef0b67-13ba-420a-8c95-50e5212ccbf5-000000(a)us-east-2.amazonses.com> Which was a reply to Andrey's: Message-ID: <010f0187e2536549-44ef0b67-13ba-420a-8c95-50e5212ccbf5-000000(a)us-east-2.amazonses.com> References: <010f0187d80eb0dd-93bebdd9-cabf-4ad6-a7f4-b5515bd22d31-000000(a)us-east-2.amazonses.com> <010f0187dca41ec8-478dfff5-d27d-4879-ab72-072be2f76176-000000(a)us-east-2.amazonses.com> <010f0187e09fef87-f959fd75-f882-47dc-b20c-f103c943383d-000000(a)us-east-2.amazonses.com> <010f0187e141f8b1-75bb05d0-8b61-4190-af34-2efb57ce70df-000000(a)us-east-2.amazonses.com> <CANEqPjFF80D2dp5pAZfo1aF7YHt73_cs5wFgENNZ40_piTZnvA(a)mail.gmail.com> In-Reply-To: <CANEqPjFF80D2dp5pAZfo1aF7YHt73_cs5wFgENNZ40_piTZnvA(a)mail.gmail.com> And yet I'm seeing both my message and your reply to it as siblings under Andrey's, because you appear to have a different id for my message than I do.

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Julien Blanc 04 May '23

04 May '23

Le 2023-05-04 01:44, Gavin Lambert via Boost a écrit : > This is what I'm seeing -- note that the current thread appears > semi-flat (even though most of these sibling messages were actually a > reply chain), but the "replacing the ML with a forum" thread appears > mostly normal. > https://imgur.com/a/GfTS6ig You're luckier than me. The current thread is semi-broken, but the "replacing the ML with a forum" one is completely broken for me. I'm starting to wonder if we all receive the same emails (in the sense that they may not contain the same set of headers). For me, the initial message of Vinnie in the "replacing the ML with a forum" thread contains the following: Message-ID: <010f0187bde5599a-e76c27f2-225d-4d7e-b2c8-887e3a07ecb8-000000(a)us-east-2.amazonses.com> But Mashshall's answer, on the other side, contains the following headers : References: <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> In-Reply-To: <010f0187bde530fb-ff3c1911-df7d-4399-b5d2-82f8062bf18c-000000(a)us-east-2.amazonses.com> The "In-Reply-To" should match the Id of the initial message, for threading to work correctly. This is too obvious to be a client bug. Can someone check the Message-ID he has for the initial message of Vinnie in the "replacing the ML with a forum" thread? If it's different from mine, then that's probably the source of the issues. Regards, Julien

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Gavin Lambert 03 May '23

03 May '23

On 4/05/2023 03:55, Andrey Semashev wrote: > The fact that the initial message is not part of the thread now is a > relatively recent issue. This is the mailman archive from a year ago: > > https://lists.boost.org/Archives/boost/2022/04/index.php > > Threading is fine there. In fact, threading was working correctly as > recently as in February 2023: > > https://lists.boost.org/Archives/boost/2023/02/index.php > > and has been broken in March: > > https://lists.boost.org/Archives/boost/2023/03/index.php > > If the system works (and worked) the way you describe, mailman was able > to associate the initial message with the thread before and now it > isn't. I suspect, this may have something to do with the headers that > are added by SES to the messages it sends to the subscribers. Threading only sometimes works in the gmane view as well. Usually it's not the mailing list itself at fault, but due to the mail client used by the person posting -- or sometimes that they're replying to a digest rather than the original mail, so the context headers are lost (even if they edit it to look like it was a reply to a single post). I have noticed an apparent increase recently in the number of posts that appear "flat" or semi-flat (still showing as a reply, but to a grandparent post rather than the real parent post) -- but other recent threads still do appear normally nested, so I don't think it's a systemic issue. This is what I'm seeing -- note that the current thread appears semi-flat (even though most of these sibling messages were actually a reply chain), but the "replacing the ML with a forum" thread appears mostly normal. https://imgur.com/a/GfTS6ig

1 0

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by Sam Darwin 03 May '23

03 May '23

Amazon SES is a basic SMTP gateway service. It has fewer features and options than mailgun. Incoming and outgoing mail are handled differently. When emails are sent out from the mailman server, they are proxied through SES. Only verified identities may send mail. Notice the sender says "boost(a)lists.boost.org", since lists.boost.org is a verified domain that is working, otherwise it would be blocked. Incoming emails go directly to mailman, not through any intermediary. SES is not involved with incoming emails to the mail server. This is the same situation as before. - Are emails being sent out? Almost certainly all the emails are being sent as shown by the mail logs on the mail server, and on SES it says "0.01% bounce rate", which is very good. - Is threading working? checking lists.boost.org, messages are organized into threads. However, it may be that threading is only partially correct. If that is what you are reporting. There are only a few settings exposed in the gateway to be able to adjust. One of them which hasn't been added yet is "Custom MAIL FROM domain", that could be modified from the default of amazonses.com to "a subdomain of lists.boost.org". As an experiment, I will set that today, and observe if it helps. Mailgun has been in production for many years. It costs $500 per month to send smtp messages. It's a reasonable idea to try out another service that's only 5% of the cost. But at the same time, it's not impossible to switch back to mailgun, if threading is partially disrupted, and we can't find a solution. On Wed, May 3, 2023 at 4:56 AM Andrey Semashev via Boost <boost(a)lists.boost.org> wrote: > > On 5/3/23 10:59, Julien Blanc via Boost wrote: > > Le 2023-05-02 15:25, David Sankel via Boost a écrit : > >> I wanted to call out Sam Darwin's work on this: > >> > >>> Transitioned boost mailing list gateway from mailgun to Amazon SES > >> > >> This transition has resulted in a $575 per month cost reduction in Boost > >> Foundation hosting expenses. Sam did an excellent job here. > > > > Did that happen around end of february / start of march? I've noticed > > that threaded view was broken around that date (and still is). At first, > > i blamed my client (evolution) but i see the same issue with another > > (roundcube). > > > > A quick look at mail headers seems to show wrong values in the > > References: header (but that is far beyond my expertise). > > I can see the threaded view is indeed broken in the mailman archive, for > example see here: > > https://lists.boost.org/Archives/boost/2023/04/index.php > > You can see the initial post is not grouped with the following thread. > > I should also mention that I was not totally correct when I said I don't > use threaded view when I read Boost ML. On my mobile email app that I > forgot about, I do use threaded view, and I did notice that some > messages were not grouped with the rest of the discussion. It didn't > occur to me until now whether those messages were the initial ones in > the discussion, but it may have been so. > > > _______________________________________________ > Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

2 1

Re: [boost] Q1 2023 Update: C++ Alliance's Efforts (PDF Attached)
by David Sankel 03 May '23

03 May '23

I wanted to call out Sam Darwin's work on this: > Transitioned boost mailing list gateway from mailgun to Amazon SES This transition has resulted in a $575 per month cost reduction in Boost Foundation hosting expenses. Sam did an excellent job here. Thanks Sam and C++ Alliance for this valuable work! David Boost Foundation

2 1