Changing virtual memory amount in Appveyor and Travis CI tests
Is it possible through .yml file for Appveyor and Travis CI to change the amount of virtual memory when running a CI test. I am running CI tests for Boost PP which heavily use the preprocessor, and some of the extended tests I am running fail on both Appveyor and Travis CI with out of memory conditions while on my own local computer, with about 32 GB of ram, the tests succeed. I think in the real world most users now have at least 32 GB of ram or more, while these virtual environments in Appveyor and/or Travis CI probably run their tests with much less overall virtual memory than an average user would have.
On 21/09/2020 14:14, Edward Diener wrote:
Is it possible through .yml file for Appveyor and Travis CI to change the amount of virtual memory when running a CI test. I am running CI tests for Boost PP which heavily use the preprocessor, and some of the extended tests I am running fail on both Appveyor and Travis CI with out of memory conditions while on my own local computer, with about 32 GB of ram, the tests succeed. I think in the real world most users now have at least 32 GB of ram or more, while these virtual environments in Appveyor and/or Travis CI probably run their tests with much less overall virtual memory than an average user would have.
According to their websites, Appveyor VMs have a limit of around 6 GB and Travis VMs have a limit around 8 GB for Windows and 4 GB for Linux/MacOS. Neither appear to be configurable in any way (but I am not an expert), which is understandable. I realise you're probably deliberately testing edge cases with pathologic memory usage, but normally there'd be something seriously wrong if a build comes remotely close to using that much. While yes, these days usually developers do have 32 GB RAM or more, the "popular" RAM size is still 16 GB or even less, and a lot of this space would be intended for parallel compiles or VMs and any single compile should assume it must use a lot less.
On 9/20/2020 11:59 PM, Gavin Lambert via Boost wrote:
On 21/09/2020 14:14, Edward Diener wrote:
Is it possible through .yml file for Appveyor and Travis CI to change the amount of virtual memory when running a CI test. I am running CI tests for Boost PP which heavily use the preprocessor, and some of the extended tests I am running fail on both Appveyor and Travis CI with out of memory conditions while on my own local computer, with about 32 GB of ram, the tests succeed. I think in the real world most users now have at least 32 GB of ram or more, while these virtual environments in Appveyor and/or Travis CI probably run their tests with much less overall virtual memory than an average user would have.
According to their websites, Appveyor VMs have a limit of around 6 GB and Travis VMs have a limit around 8 GB for Windows and 4 GB for Linux/MacOS. Neither appear to be configurable in any way (but I am not an expert), which is understandable.
I realise you're probably deliberately testing edge cases with pathologic memory usage, but normally there'd be something seriously wrong if a build comes remotely close to using that much.
I agree, which makes me think that the compilers/preprocessors involved limit the amount of memory why they allow themselves to use. But these "out of memory" type of conditions are occurring in gcc, clang, and vc++, so all three are limiting preprocessor memory in some way. Oh well, I do warn the end-user of Boost PP about pushing some of these limitations, but it is disappointing that the preprocessors can only reflect dubious "out of memory" type conditions rather than error out on what situations are really causing the preprocessors to fail.
While yes, these days usually developers do have 32 GB RAM or more, the "popular" RAM size is still 16 GB or even less, and a lot of this space would be intended for parallel compiles or VMs and any single compile should assume it must use a lot less.
Even if 16 GB, these CIs limiting memory to 4 GB or 6 GB seem a disservice to me.
On 21/09/2020 16:20, Edward Diener wrote:
Even if 16 GB, these CIs limiting memory to 4 GB or 6 GB seem a disservice to me.
While virtual memory is non-linear and weird (and virtual machine memory even more so), it still approximately boils down to however much physical RAM they can (affordably) fit onto the host, then divided by how many VMs they want to run per host. Giving each VM a larger memory allocation means you can fit fewer concurrent VMs on the same hardware, meaning that you either need more hardware or less concurrent users. And more hardware is expensive, especially at VM-host-server scale. And these CIs have a lot of users who don't pay for anything. So they have to draw a line in the sand somewhere. They have apparently decided that those particular limits work for most of their users (or at least the ones that they care sufficiently about). It's possible that they have some dedicated servers for paying customers with looser limits. But it's also possible that they don't, if it hasn't been an issue in the past.
On 9/21/2020 1:29 AM, Gavin Lambert via Boost wrote:
On 21/09/2020 16:20, Edward Diener wrote:
Even if 16 GB, these CIs limiting memory to 4 GB or 6 GB seem a disservice to me.
While virtual memory is non-linear and weird (and virtual machine memory even more so), it still approximately boils down to however much physical RAM they can (affordably) fit onto the host, then divided by how many VMs they want to run per host.
Giving each VM a larger memory allocation means you can fit fewer concurrent VMs on the same hardware, meaning that you either need more hardware or less concurrent users. And more hardware is expensive, especially at VM-host-server scale.
And these CIs have a lot of users who don't pay for anything.
So they have to draw a line in the sand somewhere. They have apparently decided that those particular limits work for most of their users (or at least the ones that they care sufficiently about).
It's possible that they have some dedicated servers for paying customers with looser limits. But it's also possible that they don't, if it hasn't been an issue in the past.
I understand that it is a free resource for Boost testing and that they may not offer much in the way of virtual memory, as well as having certain time limits when something is running. But nearly all my tests for intensive functionality in Boost PP and VMD run fine to completion on my local computer, using gcc, clang, and VC++ on Windows or Linux, whereas in the CI testing with Appveyor and Travis CI there are numerous "out of memory" and "process killed on timeouts" errors which prevents successful CI tests results. So for my purposes of CI testing these CI facilities are almost useless on the whole to tell me whether some change is valid or not. That's fine. I just will pay little attention to the overall result of the CI tests, since they do not reflect real world results because of the CI limitations.
On 25/09/2020 03:49, Edward Diener wrote:
I understand that it is a free resource for Boost testing and that they may not offer much in the way of virtual memory, as well as having certain time limits when something is running. But nearly all my tests for intensive functionality in Boost PP and VMD run fine to completion on my local computer, using gcc, clang, and VC++ on Windows or Linux, whereas in the CI testing with Appveyor and Travis CI there are numerous "out of memory" and "process killed on timeouts" errors which prevents successful CI tests results. So for my purposes of CI testing these CI facilities are almost useless on the whole to tell me whether some change is valid or not. That's fine. I just will pay little attention to the overall result of the CI tests, since they do not reflect real world results because of the CI limitations.
Perhaps you should disable those specific tests for CI builds, while keeping them for the normal test matrix? That would keep both useful.
On 9/24/2020 7:31 PM, Gavin Lambert via Boost wrote:
On 25/09/2020 03:49, Edward Diener wrote:
I understand that it is a free resource for Boost testing and that they may not offer much in the way of virtual memory, as well as having certain time limits when something is running. But nearly all my tests for intensive functionality in Boost PP and VMD run fine to completion on my local computer, using gcc, clang, and VC++ on Windows or Linux, whereas in the CI testing with Appveyor and Travis CI there are numerous "out of memory" and "process killed on timeouts" errors which prevents successful CI tests results. So for my purposes of CI testing these CI facilities are almost useless on the whole to tell me whether some change is valid or not. That's fine. I just will pay little attention to the overall result of the CI tests, since they do not reflect real world results because of the CI limitations.
Perhaps you should disable those specific tests for CI builds, while keeping them for the normal test matrix? That would keep both useful.
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
On 25/09/2020 15:39, Edward Diener wrote:
Perhaps you should disable those specific tests for CI builds, while keeping them for the normal test matrix? That would keep both useful.
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
I'm not familiar with Jamfiles (or CI, for that matter), so perhaps someone else should chime in at this point... ... but it does look like both Travis and Appveyor set the environment variable CI when building (they both set it to "true", but not with identical case; checking it against empty value is probably sufficient though), so you can possibly check that. (They also set TRAVIS or APPVEYOR respectively, if you need to distinguish them.)
Mere moments ago, quoth I:
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
I'm not familiar with Jamfiles (or CI, for that matter), so perhaps someone else should chime in at this point...
... but it does look like both Travis and Appveyor set the environment variable CI when building (they both set it to "true", but not with identical case; checking it against empty value is probably sufficient though), so you can possibly check that.
(They also set TRAVIS or APPVEYOR respectively, if you need to distinguish them.)
FWIW, Boost.Gil [1] and Boost.Phoenix [2] appears to have an example of changing compiler settings based on CI or not, which you could crib from. Most of the other libraries appear to do it differently -- they group the tests into different suites and their travis/appveyor files specify particular suites to run, or they have one test "outside" suites, with comments implying that the test matrix runs all of them but CI only runs the ones not in suites by default (I don't know if that's an accurate statement). [1] https://github.com/boostorg/gil/blob/master/test/Jamfile [2] https://github.com/boostorg/phoenix/blob/master/test/Jamfile
On Fri, 25 Sep 2020 at 07:19, Gavin Lambert via Boost <boost@lists.boost.org> wrote:
Mere moments ago, quoth I:
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
I'm not familiar with Jamfiles (or CI, for that matter), so perhaps someone else should chime in at this point...
... but it does look like both Travis and Appveyor set the environment variable CI when building (they both set it to "true", but not with identical case; checking it against empty value is probably sufficient though), so you can possibly check that.
(They also set TRAVIS or APPVEYOR respectively, if you need to distinguish them.)
FWIW, Boost.Gil [1] and Boost.Phoenix [2] appears to have an example of changing compiler settings based on CI or not, which you could crib from.
For GIL, I was piggybacking on Andrey Semashev's solution that Andrey just suggested in his response. It's simple and works well. Best regards, -- Mateusz Loskot, http://mateusz.loskot.net
On 9/25/2020 1:17 AM, Gavin Lambert via Boost wrote:
Mere moments ago, quoth I:
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
I'm not familiar with Jamfiles (or CI, for that matter), so perhaps someone else should chime in at this point...
... but it does look like both Travis and Appveyor set the environment variable CI when building (they both set it to "true", but not with identical case; checking it against empty value is probably sufficient though), so you can possibly check that.
(They also set TRAVIS or APPVEYOR respectively, if you need to distinguish them.)
FWIW, Boost.Gil [1] and Boost.Phoenix [2] appears to have an example of changing compiler settings based on CI or not, which you could crib from.
Most of the other libraries appear to do it differently -- they group the tests into different suites and their travis/appveyor files specify particular suites to run, or they have one test "outside" suites, with comments implying that the test matrix runs all of them but CI only runs the ones not in suites by default (I don't know if that's an accurate statement).
[1] https://github.com/boostorg/gil/blob/master/test/Jamfile [2] https://github.com/boostorg/phoenix/blob/master/test/Jamfile
Thanks ! I also have different aliases ( suites ) in my jamfiles, so I can use that also.
On 9/25/20 6:39 AM, Edward Diener via Boost wrote:
On 9/24/2020 7:31 PM, Gavin Lambert via Boost wrote:
On 25/09/2020 03:49, Edward Diener wrote:
I understand that it is a free resource for Boost testing and that they may not offer much in the way of virtual memory, as well as having certain time limits when something is running. But nearly all my tests for intensive functionality in Boost PP and VMD run fine to completion on my local computer, using gcc, clang, and VC++ on Windows or Linux, whereas in the CI testing with Appveyor and Travis CI there are numerous "out of memory" and "process killed on timeouts" errors which prevents successful CI tests results. So for my purposes of CI testing these CI facilities are almost useless on the whole to tell me whether some change is valid or not. That's fine. I just will pay little attention to the overall result of the CI tests, since they do not reflect real world results because of the CI limitations.
Perhaps you should disable those specific tests for CI builds, while keeping them for the normal test matrix? That would keep both useful.
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
You can set special environment variables in the CI config file, which you then test in the Jamfile. For example: https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f... https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f... https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f... https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f...
On 9/25/2020 4:15 AM, Andrey Semashev via Boost wrote:
On 9/25/20 6:39 AM, Edward Diener via Boost wrote:
On 9/24/2020 7:31 PM, Gavin Lambert via Boost wrote:
On 25/09/2020 03:49, Edward Diener wrote:
I understand that it is a free resource for Boost testing and that they may not offer much in the way of virtual memory, as well as having certain time limits when something is running. But nearly all my tests for intensive functionality in Boost PP and VMD run fine to completion on my local computer, using gcc, clang, and VC++ on Windows or Linux, whereas in the CI testing with Appveyor and Travis CI there are numerous "out of memory" and "process killed on timeouts" errors which prevents successful CI tests results. So for my purposes of CI testing these CI facilities are almost useless on the whole to tell me whether some change is valid or not. That's fine. I just will pay little attention to the overall result of the CI tests, since they do not reflect real world results because of the CI limitations.
Perhaps you should disable those specific tests for CI builds, while keeping them for the normal test matrix? That would keep both useful.
How do I tell in my jamfile when CI testing is occurring in order to disable the tests for CI builds.
You can set special environment variables in the CI config file, which you then test in the Jamfile. For example:
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f...
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f...
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f...
https://github.com/boostorg/log/blob/d98067a0edfe38e533c583864d58360b05ac17f...
Thanks ! I forgot about the os.environ functionality in Boost Build, although I have probably used it myself numerous times <g>.
participants (4)
-
Andrey Semashev -
Edward Diener -
Gavin Lambert -
Mateusz Loskot