Re: [Boost-users] [multi_index] Core dump in multi-index library !!

From: Patel Priyank-PPATEL1 (priyankpatel_at_[hidden])
Date: 2006-05-10 16:49:21

Hi Joaquin,

As I said, I think the problem is during addition and not delete. I have tried to build my application with debug mode and have some more printouts for you. Please let me know if this helps or not. Everything goes smooth for a while but suddenly it stops working. I even
Removed delete calls but the problem still exists. One other thing, addition and deletion
Is done by separate threads in my application. I am trying now to lock/unlock pool when
I do addition/deletion and see whether that works or not. Let me know if you need any
More information relating this. Thanks a lot for your help. Thread id (8) adds procedures in procedure pool while thread id (4) removes
Procedure from procedure pool.

// SOME TRACE WHILE EXECUTING

LD_TRACE:           (8) In Method: Procedure_Executor::handle_timeout File: `src/apps/executor/Procedure_Executor.cpp' Line: 67
LD_TRACE:                     (8) In Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) Out Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) In Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) Out Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) In Method: Sample_Procedure::Sample_Procedure File: `src/apps/procedure/Sample_Procedure.cpp' Line: 9
LD_TRACE:                     (8) Out Method: Sample_Procedure::Sample_Procedure File: `src/apps/procedure/Sample_Procedure.cpp' Line: 9
LD_TRACE:                     (8) In Method: Access_Point::subscribe File: `src/apps/network_element/Access_Point.cpp' Line: 181
LD_TRACE:                               (8) In Method: AP_TCP_Connection::subscribe File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 81
LD_TRACE:                                         (8) In Method: Procedure_Pool::add_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 5
FIND CALL FROM ADD
LD_TRACE:                                                   (8) In Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
src/core/procedure/Procedure_Pool.cpp:47 Not able to found procedure for id: 1646 
LD_TRACE:                                                   (8) Out Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
Added procedure : 1646 
LD_TRACE:                                         (8) Out Method: Procedure_Pool::add_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 5
src/apps/connector/AP_TCP_Connection.cpp:88 Subscribed sample type procedure
LD_TRACE:                               (8) Out Method: AP_TCP_Connection::subscribe File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 81
LD_TRACE:                     (8) Out Method: Access_Point::subscribe File: `src/apps/network_element/Access_Point.cpp' Line: 181
LD_TRACE:                     (8) In Method: Sample_Procedure::start File: `src/apps/procedure/Sample_Procedure.cpp' Line: 15
LD_TRACE:                               (8) In Method: Procedure_Timer_Handler::schedule File: `src/core/procedure/Procedure_Timer_Handler.cpp' Line: 5
Timer id 24 scheduled.
LD_TRACE:                               (8) Out Method: Procedure_Timer_Handler::schedule File: `src/core/procedure/Procedure_Timer_Handler.cpp' Line: 5
LD_TRACE:                               (8) In Method: Tcp_Connection::send File: `src/core/connector/Tcp_Connection.cpp' Line: 78
src/core/connector/Tcp_Connection.cpp:92 Sent 4 bytes  on: Remote IP 173.6.64.7 , Remote Port 30501 Local Ip 173.6.64.7 Local Port 30000
LD_TRACE:                               (8) Out Method: Tcp_Connection::send File: `src/core/connector/Tcp_Connection.cpp' Line: 78
SENT=> P_ID: 1646 HEXDUMP 4 bytes
00 00 06 6e                                        ...n            
LD_TRACE:                     (8) Out Method: Sample_Procedure::start File: `src/apps/procedure/Sample_Procedure.cpp' Line: 15
LD_TRACE:           (8) Out Method: Procedure_Executor::handle_timeout File: `src/apps/executor/Procedure_Executor.cpp' Line: 67
LD_TRACE: (4) In Method: Tcp_Handler::handle_input File: `src/core/connector/Tcp_Handler.cpp' Line: 38
src/core/connector/Tcp_Handler.cpp:43 Received 4 bytes LD_TRACE: (4) Out Method: Tcp_Handler::handle_input File: `src/core/connector/Tcp_Handler.cpp' Line: 38
LD_TRACE: (4) In Method: Tcp_Handler::handle_output File: `src/core/connector/Tcp_Handler.cpp' Line: 73
src/core/connector/Tcp_Handler.cpp:78 Successfully dequeued messageLD_TRACE:           (4) In Method: AP_TCP_Connection::handle File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 119
LD_TRACE:                     (4) In Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
src/core/procedure/Procedure_Pool.cpp:44 Found procedure for id: 1646 
LD_TRACE:                     (4) Out Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
FOUND PROCEDURE FOR ID: 1646 
LD_TRACE:                     (4) In Method: Sample_Procedure::handle File: `src/apps/procedure/Sample_Procedure.cpp' Line: 53
RECV<= P_ID: 1646 HEXDUMP 4 bytes
00 00 06 6e                                        ...n            
LD_TRACE:                               (4) In Method: Access_Point::unsubscribe File: `src/apps/network_element/Access_Point.cpp' Line: 196
src/apps/network_element/Access_Point.cpp:199 UNSUBCRIBING: 1646
LD_TRACE:                                         (4) In Method: AP_TCP_Connection::unsubscribe File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 99
src/apps/connector/AP_TCP_Connection.cpp:103 UNSUBCRIBING: 1646 
LD_TRACE:                                                   (4) In Method: Procedure_Pool::remove_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 25
REMOVING: 1646 
LD_TRACE:                                                             (4) In Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
src/core/procedure/Procedure_Pool.cpp:44 Found procedure for id: 1646 
LD_TRACE:                                                             (4) Out Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
Removed procedure : 1646 
LD_TRACE:                                                   (4) Out Method: Procedure_Pool::remove_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 25
src/apps/connector/AP_TCP_Connection.cpp:108 UnSubscribed sample type procedureLD_TRACE:                                         (4) Out Method: AP_TCP_Connection::unsubscribe File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 99
LD_TRACE:                               (4) Out Method: Access_Point::unsubscribe File: `src/apps/network_element/Access_Point.cpp' Line: 196
LD_TRACE:                               (4) In Method: Procedure_Timer_Handler::cancel File: `src/core/procedure/Procedure_Timer_Handler.cpp' Line: 31
Timer id 24 cancelled.
LD_TRACE:                               (4) Out Method: Procedure_Timer_Handler::cancel File: `src/core/procedure/Procedure_Timer_Handler.cpp' Line: 31
DELETING PROCEDURE: 1646
LD_TRACE:                     (4) Out Method: Sample_Procedure::handle File: `src/apps/procedure/Sample_Procedure.cpp' Line: 53
LD_TRACE:           (4) Out Method: AP_TCP_Connection::handle File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 119
src/core/connector/Tcp_Handler.cpp:91 Successfully send message block to connectorLD_TRACE: (4) Out Method: Tcp_Handler::handle_output File: `src/core/connector/Tcp_Handler.cpp' Line: 73
LD_TRACE: (4) In Method: Procedure_Timer_Handler::handle_timeout File: `src/core/include/Procedure_Timer_Handler.h' Line: 37
LD_TRACE:           (4) In Method: Sample_Procedure::handle_timeout File: `src/apps/procedure/Sample_Procedure.cpp' Line: 41
LD_TRACE:           (4) Out Method: Sample_Procedure::handle_timeout File: `src/apps/procedure/Sample_Procedure.cpp' Line: 41
LD_TRACE: (4) Out Method: Procedure_Timer_Handler::handle_timeout File: `src/core/include/Procedure_Timer_Handler.h' Line: 37
LD_TRACE:           (8) In Method: Procedure_Executor::handle_timeout File: `src/apps/executor/Procedure_Executor.cpp' Line: 67
LD_TRACE:                     (8) In Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) Out Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) In Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) Out Method: Access_Points::next File: `src/apps/network_element/Access_Points.cpp' Line: 48
LD_TRACE:                     (8) In Method: Sample_Procedure::Sample_Procedure File: `src/apps/procedure/Sample_Procedure.cpp' Line: 9
LD_TRACE:                     (8) Out Method: Sample_Procedure::Sample_Procedure File: `src/apps/procedure/Sample_Procedure.cpp' Line: 9
LD_TRACE:                     (8) In Method: Access_Point::subscribe File: `src/apps/network_element/Access_Point.cpp' Line: 181
LD_TRACE:                               (8) In Method: AP_TCP_Connection::subscribe File: `src/apps/connector/AP_TCP_Connection.cpp' Line: 81
LD_TRACE:                                         (8) In Method: Procedure_Pool::add_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 5
FIND CALL FROM ADD
LD_TRACE:                                                   (8) In Method: Procedure_Pool::find_by_id File: `src/core/procedure/Procedure_Pool.cpp' Line: 41
Segmentation fault (core dumped)

// STACK TRACE

(gdb) where
#0  0x00000000 in ?? ()
#1  0x000ce15c in _ZNK5boost11multi_index13const_mem_funI9ProcedureiXadsrS2_NKS2_2idEvEEclERKS2_ (this=0x187548, x=@0x16d2a0)
    at /vob/wibb_test/tools/cots/boost/boost/multi_index/mem_fun.hpp:60
#2  0x000ce0f8 in _ZNK5boost11multi_index13const_mem_funI9ProcedureiXadsrS2_NKS2_2idEvEEclIPS2_EEiRKT_ (this=0x187548, x=@0x1345b0)
    at /vob/wibb_test/tools/cots/boost/boost/multi_index/mem_fun.hpp:55
#3  0x000ce368 in _ZN5boost11multi_index6detail18ordered_index_findINS1_18ordered_index_nodeINS1_15index_node_baseIP9ProcedureEEEENS0_13const_mem_funIS5_iXadsrS5_NKS5_2idEvEEEiSt4lessIiEEEPT_SE_RKT0_RKT1_RKT2_ (header=0x134598, key=@0x187548, x=@0xfe407618, comp=@0x187549)
    at /vob/wibb_test/tools/cots/boost/boost/multi_index/detail/ord_index_ops.hpp:65
#4  0x000cd6b0 in _ZNK5boost11multi_index6detail13ordered_indexINS0_13const_mem_funI9ProcedureiXadsrS4_NKS4_2idEvEEESt4lessIiENS1_9nth_layerILi1EPS4_NS0_10indexed_byINS0_14ordered_uniqueINS0_3tagI16procedure_id_tagN4mpl_2naESF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_EES5_SF_EESF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_SF_EESaIS9_EEENS_3mpl6v_itemISD_NSL_7vector0ISF_EELi0EEENS1_18ordered_unique_tagEE4findIiEENS1_14index_iteratorINS1_18ordered_index_nodeINS1_15index_node_baseIS9_EEEEEERKT_ (this=0x187548, x=@0xfe407618) at /vob/wibb_test/tools/cots/boost/boost/multi_index/ordered_index.hpp:376
#5  0x00064af0 in Procedure_Pool::find_by_id (this=0x187540, _id=1647) at src/core/procedure/Procedure_Pool.cpp:42
#6  0x0006462c in Procedure_Pool::add_by_id (this=0x187540, _procedure=0x173cd0) at src/core/procedure/Procedure_Pool.cpp:8
#7  0x00078268 in AP_TCP_Connection::subscribe (this=0x19c3a0, _procedure=0x173cd0) at src/apps/connector/AP_TCP_Connection.cpp:84
#8  0x00069b78 in Access_Point::subscribe (this=0x166728, _type=@0x12fc30, _procedure=0x173cd0) at src/apps/network_element/Access_Point.cpp:184
#9  0x000762e4 in Procedure_Executor::handle_timeout (this=0x139ee0, _time=@0xfe407930, _procedure=0x0) at src/apps/executor/Procedure_Executor.cpp:79
#10 0xff2c1940 in ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>::timeout () from /vob/wibb_test/tools/cots/ACE_5.5/ace/libACE.so.5.5.0
#11 0xff26f2e4 in ACE_TP_Reactor::handle_timer_events () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#12 0xff26f160 in ACE_TP_Reactor::dispatch_i () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#13 0xff26eff8 in ACE_TP_Reactor::handle_events () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#14 0xff254e0c in ACE_Reactor::run_reactor_event_loop () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#15 0x00074b30 in Executor_Reactor::svc (this=0x139f58) at src/apps/executor/Executor_Reactor.cpp:22
#16 0xff2668dc in ACE_Task_Base::svc_run () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#17 0xff266cb0 in ACE_Thread_Adapter::invoke_i () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#18 0xff266c1c in ACE_Thread_Adapter::invoke () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#19 0xff216d50 in ace_thread_adapter () at /mot/proj/ivtools/ppatel1/ACE_wrappers/ace/String_Base.inl:32
#20 0xfe8db730 in _thread_start () from /usr/lib/libthread.so.1
#21 0xfe8db730 in _thread_start () from /usr/lib/libthread.so.1
Previous frame identical to this frame (corrupt stack?)
(gdb) 

Priyank

 

-----Original Message-----
From: boost-users-bounces_at_[hidden] [mailto:boost-users-bounces_at_[hidden]] On Behalf Of JOAQUIN LOPEZ MU?Z
Sent: Wednesday, May 10, 2006 12:05 PM
To: boost-users_at_[hidden]
Subject: Re: [Boost-users] [multi_index] Core dump in multi-index library !!

Patel Priyank-PPATEL1 ha escrito:
> Hi Joaquin,
>  
> Seems the problem was happening when I call add and my add method 
> tries to first find out whether that id is already present in 
> container or not. It is not during remove method call.

(My hunch is that the problem will randomly show at various locations, please keep on reading.)

> I was concentrating on wrong portion of the code before.  Anyways, I 
> have changed and put some print out in code also at the end I have 
> included output that comes.  I have changed container to ordered index 
> as you have said, but seems to be the same problem.

This is a very valuable piece of information, and increases my suspicions that the problem lies in your code rather than in B.MI: hashed and ordered indices share almost no code at all, so if the problem persists after switching the index type it's probably because the error lies outside the library. Of course, I might be wrong.

> It seems that it is core dumping on following line:
> 
> bool Procedure_Pool::find_by_id(Procedure * _procedure) {  
> LD_TRACE("Procedure_Pool::find_by_id");
>  ACE_ASSERT(_procedure != 0);
>  ACE_DEBUG((LM_DEBUG, "PROCEDURE TO FIND: %d\n", _procedure->id()));  
> Procedure_By_Id::iterator it = procedure_by_id_.find(_procedure->id
());
>  ACE_DEBUG((LM_DEBUG, "AFTER FIND CALL\n"));  return (it != 
> procedure_by_id_.end()); }
>  
> Please let me know if I am doing something wrong here. Also, let me 
> know if you need any more information. Please see debug output at the 
> end of the email.

I was consiering two possible causes for the crash in my previous post (leaving out the possibility of a bug in
B.MI):

1. Procedure id inadvertently changed.
2. Procedure object deleted before having been removed
   from the Procedure_Pool.

The fact that the crash is now happening in find seems to rule out 1, because inconsistent id's might lead to wrong results in a query, but in no case to a core dump.

So, I'd say you are somehow deleting Procedure objects without removing them before from the Procedure_Pool.
Maybe you can try to check this as follows:

If you can locate all the places in your code where a Procedure object is deleted, i.e expressions of the form:

  delete p; // p is a Procedure *

try instrumenting *all of them* with the following check:

  if(p != 0){
    // Get the pool. I guess you've got some API to do this.
    Procedure_Hashed_Pool* pool=...;
    // Since we're deleting p, it musn't still be in pool!
    assert(pool->find_by_id(p->id()) == 0);
  }
  delete p;

Do you get an assertion? If so, here's your bug.

Please report back. If this doesn't clear up the problem we can explore some other paths.

Joaquín M López Muñoz
Telefónica, Investigación y Desarrollo
_______________________________________________
Boost-users mailing list
Boost-users_at_[hidden]
http://lists.boost.org/mailman/listinfo.cgi/boost-users