Mutli index container - performance varation when erasing large amount of rows in different computers
Hi All, I am trying to use boost Multi Index Container in my company product to replace an old container that gave the same functionallity but with bad performance. The mutli index container contains 2 non unique ordered indices. The mutli index container will be used to insert large amount of rows sequentially and an erase using one of the ordered indices that will evantually erase around 10,000 rows in one erase operation since the container will contain many duplicate values of the delete key. Testing the insertion of 10,000 rows takes around 60-80ms on a standard PC 3.2GHZ. On a HP G4 server (2 xeon 3.2 cpu's, 4Giga of memroy) the performance is a little better. In both cases the insertion is consistent. However, while erasing elements using one of the indices which incur the erase of 10,000 elements takes around 30ms on the standard PC. However, on the HP G4 the deletion is inconsistent and far worse. It takes between 200ms to 1.4 seconds on different deletion cycles. I am using the same products exactly on both machines. The HP G4 does not have a development envoirment installed on it, the standard pc have visual studio 2005 installed on it. Does anyone have any idea what may cause such exterme difference? Thanks in advance, Avi Nice Systems ltd. -- View this message in context: http://www.nabble.com/Mutli-index-container---performance-varation-when-eras... Sent from the Boost - Dev mailing list archive at Nabble.com.
Note that i created a stand alone application to do only this performance tests compiled in visual studio 2005 without CLI support. When excuting this test program on other HP G4 servers they run as fast (faster) then the standard PC. I have only 1 HP G4 machine which the performance is as described. I tried all varation of compiler optimization but still. On this machine alone the performance is extermaly worse then all other computers when removing elements from the container. Thanks again, Avi Avi Revivo wrote:
Hi All,
I am trying to use boost Multi Index Container in my company product to replace an old container that gave the same functionallity but with bad performance.
The mutli index container contains 2 non unique ordered indices.
The mutli index container will be used to insert large amount of rows sequentially and an erase using one of the ordered indices that will evantually erase around 10,000 rows in one erase operation since the container will contain many duplicate values of the delete key.
Testing the insertion of 10,000 rows takes around 60-80ms on a standard PC 3.2GHZ. On a HP G4 server (2 xeon 3.2 cpu's, 4Giga of memroy) the performance is a little better. In both cases the insertion is consistent.
However, while erasing elements using one of the indices which incur the erase of 10,000 elements takes around 30ms on the standard PC. However, on the HP G4 the deletion is inconsistent and far worse. It takes between 200ms to 1.4 seconds on different deletion cycles.
I am using the same products exactly on both machines. The HP G4 does not have a development envoirment installed on it, the standard pc have visual studio 2005 installed on it.
Does anyone have any idea what may cause such exterme difference?
Thanks in advance, Avi Nice Systems ltd.
-- View this message in context: http://www.nabble.com/Mutli-index-container---performance-varation-when-eras... Sent from the Boost - Dev mailing list archive at Nabble.com.
Avi Revivo ha escrito:
Note that i created a stand alone application to do only this performance tests compiled in visual studio 2005 without CLI support.
When excuting this test program on other HP G4 servers they run as fast (faster) then the standard PC. I have only 1 HP G4 machine which the performance is as described. I tried all varation of compiler optimization but still. On this machine alone the performance is extermaly worse then all other computers when removing elements from the container.
Hello Avi, I don't see any reason why the erase part should perform so erraticaly on the HP server, specially when on other servers the program works fine. My hunch is that this could be related to the L1 and L2 cache configuration of the machine. Could you compare this particular setting with those of the servers where performance is adequate? Other than that I don't know what more to suggest. If you could share the test code I'd be happy to have a look at it. Also, maybe using a pool allocator like http://boost.org/libs/pool/doc/implementation/pool_alloc.html might have some beneficial effect and fix cache anomalies. Please keep me informed of your progress, if any, with this. Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Hi Joaquín, Thanks for the fast reply. After dissecting i found out that the only problem is the the actual invocation of delete(). Code executed: ********************************************************** int *arrayOfPointers[100000]; for(int i = 0; i < 100000; i++) { arrayOfPointers[i] = new int; } for(int i = 0; i < 100000; i++) { delete arrayOfPointers[i]; } ********************************************************** Excuting the for loop that deletes takes 25 times slower then any other computer tested. The average time for a single delete operation is longer in by factor of 25. Therefore, the container got nothing to do with it. I am still searching for the reason for this behavior, i am looking for possible hardware problems. Excuting this code in windows safe mode did not reflect any change. If anyone can recommend tools to diagnose such a problem it will be greatly appreciated. Many thanks, Avi Joaquín Mª López Muñoz wrote:
Avi Revivo ha escrito:
Note that i created a stand alone application to do only this performance tests compiled in visual studio 2005 without CLI support.
When excuting this test program on other HP G4 servers they run as fast (faster) then the standard PC. I have only 1 HP G4 machine which the performance is as described. I tried all varation of compiler optimization but still. On this machine alone the performance is extermaly worse then all other computers when removing elements from the container.
Hello Avi,
I don't see any reason why the erase part should perform so erraticaly on the HP server, specially when on other servers the program works fine. My hunch is that this could be related to the L1 and L2 cache configuration of the machine. Could you compare this particular setting with those of the servers where performance is adequate?
Other than that I don't know what more to suggest. If you could share the test code I'd be happy to have a look at it. Also, maybe using a pool allocator like http://boost.org/libs/pool/doc/implementation/pool_alloc.html might have some beneficial effect and fix cache anomalies.
Please keep me informed of your progress, if any, with this.
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- View this message in context: http://www.nabble.com/Mutli-index-container---performance-varation-when-eras... Sent from the Boost - Dev mailing list archive at Nabble.com.
Avi Revivo <revivoavi <at> hotmail.com> writes: [...]
After dissecting i found out that the only problem is the the actual invocation of delete().
Code executed: **********************************************************
int *arrayOfPointers[100000];
for(int i = 0; i < 100000; i++) { arrayOfPointers[i] = new int; }
for(int i = 0; i < 100000; i++) { delete arrayOfPointers[i]; }
**********************************************************
Excuting the for loop that deletes takes 25 times slower then any other computer tested. The average time for a single delete operation is longer in by factor of 25.
Umm... Still it looks like a cache issue. Can you try doing the deleteion in reverse order, from 99999 down to 0? Any difference, Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
Apperantly the cause of the problem was the usage of a utility called Global Flags that made tweaks in the registery and modified the operating system behavior when handling memory. It was not changed back by the person who performed the tweak and failed to let me know about. Thanks alot for your help, Avi Joaquín Mª López Muñoz wrote:
Avi Revivo <revivoavi <at> hotmail.com> writes:
[...]
After dissecting i found out that the only problem is the the actual invocation of delete().
Code executed: **********************************************************
int *arrayOfPointers[100000];
for(int i = 0; i < 100000; i++) { arrayOfPointers[i] = new int; }
for(int i = 0; i < 100000; i++) { delete arrayOfPointers[i]; }
**********************************************************
Excuting the for loop that deletes takes 25 times slower then any other computer tested. The average time for a single delete operation is longer in by factor of 25.
Umm... Still it looks like a cache issue. Can you try doing the deleteion in reverse order, from 99999 down to 0? Any difference,
Joaquín M López Muñoz Telefónica, Investigación y Desarrollo
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
-- View this message in context: http://www.nabble.com/Mutli-index-container---performance-varation-when-eras... Sent from the Boost - Dev mailing list archive at Nabble.com.
Avi Revivo wrote:
Apperantly the cause of the problem was the usage of a utility called Global Flags that made tweaks in the registery and modified the operating system behavior when handling memory.
It was not changed back by the person who performed the tweak and failed to let me know about.
Thanks alot for your help,
Whoa... now that's an obscure performance bottleneck. I'm glad you figured it out. - Michael Marcin
participants (4)
-
Avi Revivo -
Joaquin M Lopez Munoz -
Joaquín Mª López Muñoz
-
Michael Marcin