Re: [Boost-users] multi_array with large dimensions crashes

Subject: Re: [Boost-users] multi_array with large dimensions crashes
From: Pierre-Andre Noel (noel.pierre.andre_at_[hidden])
Date: 2011-08-25 09:19:04

> The code work now, thanks very much. But the programme crashes with a
run-time error when the dimensions of the array are huge 500x500x500.

In your examples, you were using "float" variable, which requires 4 bytes to
store. If you want a vector of 500*500*500=125000000 floats, that requires
500000000 bytes, i.e., 0.5 GB (or 0.466 GiB), of contiguous space in memory.

> This is probably due to the limitation of the 32 bit addrress space.

For 32 bits, the total amount of addressable space is 4 GB. So no, that is
not directly the problem.

> Is there any other work around. I heard of creating arrays with strides
and allocating them in different blocks of memory rather than in contiguos
memory?

I fail to see how strides may help. However, requesting memory in a
non-contiguous way may certainly help. The easiest way to do so would be to
use a vector at the first level.

typedef std::vector< boost::multi_array< data_type, 2 > >
vector_of_2D_multi_array;

When using this kind of object, remember that this is not a 3 dimensional
multi_array but instead a vector of two dimensional ones.

If you cannot reach 500x500x500 with this setting, try to see if your OS
imposes limitations to the amount of memory that can be allocated to a
single process.

Finally, if you aim for even larger sizes, the amount of system memory will
eventually become a problem. In such cases, you may want to consider
distributed computing...

Hope it helps,

Pierre-André

On Thu, Aug 25, 2011 at 5:04 AM, Jothy <jothybasu_at_[hidden]> wrote:

> Hi Pierre,
>
> Sorry for the late reply.
>
> The code work now, thanks very much. But the programme crashes with a
> run-time error when the dimensions of the array are huge 500x500x500. This
> is probably due to the limitation of the 32 bit addrress space. Is there any
> other work around. I heard of creating arrays with strides and allocating
> them in different blocks of memory rather than in contiguos memory?
>
> Thanks again
>
> Jothy
>
>
> On Wed, Aug 24, 2011 at 6:25 PM, Pierre-Andre Noel <
> noel.pierre.andre_at_[hidden]> wrote:
>
>> Yes, 500, 300 and 400 were the example dimensions you used in your earlier
>> message.
>>
>> Did you got it to work?
>>
>> Pierre-André
>>
>> PS: This email was not sent to the whole list.
>>
>>
>>
>> On Wed, Aug 24, 2011 at 11:55 AM, Jothy <jothybasu_at_[hidden]> wrote:
>>
>>> You mean 500,300,400 are the required dimesnions and xDim,yDim,zDim are
>>> site_type parameters.
>>>
>>> So the code should be like below I guess for an array of [x][y][z]
>>>
>>>
>>> array_ref_type::size_type xDim(x), yDim(y), zDim(z);
>>>
>>>   array_storage_type Storage(xDim*yDim*zDim);
>>>   array_ref_type
>>> Array(&Storage.front(),boost::extents[xDim][yDim][zDim]);
>>>
>>> Thanks
>>>
>>> Jothy
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 24, 2011 at 3:51 PM, Pierre-Andre Noel <
>>> noel.pierre.andre_at_[hidden]> wrote:
>>>
>>>> >> I am not familiar with multi_array, but it seems that you are
>>>> creating a >1GB object on the stack. This will almost always crash. Try
>>>> creating it on the heap instead. Be prepared for std::bad_alloc.
>>>>
>>>> > But how to do this? I am new to c++.
>>>>
>>>> I suggest you to either 1. read on the operators "new" and "delete" or
>>>> 2. use a std::vector to allocate the memory then use a
>>>> boost::multi_array_ref to refer to it. Here's an example:
>>>>
>>>> #include <vector>
>>>> #include <boost/multi_array.hpp>
>>>>
>>>> int main()
>>>> {
>>>>   typedef float data_type;
>>>>   typedef std::vector<data_type> array_storage_type;
>>>>   typedef boost::multi_array_ref<data_type,3> array_ref_type;
>>>>
>>>>   array_ref_type::size_type xDim(500), yDim(300), zDim(400);
>>>>   array_storage_type Storage(xDim*yDim*zDim);
>>>>   array_ref_type
>>>> Array(&Storage.front(),boost::extents[xDim][yDim][zDim]);
>>>>   // You may now use Array as usual.
>>>> }
>>>>
>>>> Hope it helps,
>>>>
>>>> Pierre-André Noël
>>>>
>>>>
>>>> On Wed, Aug 24, 2011 at 5:20 AM, Jothy <jothybasu_at_[hidden]> wrote:
>>>>
>>>>> Thanks for pointing this out.
>>>>>
>>>>> But how to do this? I am new to c++.
>>>>>
>>>>> Jothy
>>>>>
>>>>>
>>>>> On Tue, Aug 23, 2011 at 10:42 PM, Juraj Ivančić <
>>>>> juraj.ivancic_at_[hidden]> wrote:
>>>>>
>>>>>> On 23.8.2011. 12:18, Jothy wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have just started to use boost libs. I am trying to create a large
>>>>>>> float array (mandatory for my purpose). it works fine till certain
>>>>>>> limit, but crashes above that. I think it about allocating memory in
>>>>>>> the
>>>>>>> stack.
>>>>>>>
>>>>>>> My code is like this
>>>>>>>
>>>>>>> typedefboost::multi_array<**float,3>array_type;
>>>>>>> typedefarray_type::indexindex;
>>>>>>> array_typeArray(boost::**extents[xDim][yDim][zDim]);//**500,300,400
>>>>>>> for instance
>>>>>>>
>>>>>>>
>>>>>>> Can some one help me to resolve this?
>>>>>>>
>>>>>>
>>>>>> I am not familiar with multi_array, but it seems that you are creating
>>>>>> a >1GB object on the stack. This will almost always crash. Try creating it
>>>>>> on the heap instead. Be prepared for std::bad_alloc.
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>> ______________________________**_________________
>>>>>> Boost-users mailing list
>>>>>> Boost-users_at_[hidden]
>>>>>> http://lists.boost.org/**mailman/listinfo.cgi/boost-**users
>

>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Boost-users mailing list
>>>>> Boost-users_at_[hidden]
>>>>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Boost-users mailing list
>>>> Boost-users_at_[hidden]
>>>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Boost-users mailing list
>>> Boost-users_at_[hidden]
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-users
>>>
>>
>>
>