Re: [Boost-users] boost C++ load data from file

Subject: Re: [Boost-users] boost C++ load data from file
From: Jack Bryan (dtustudy68_at_[hidden])
Date: 2010-07-13 00:54:47

Thanks for your code, 
I am working on a multi-computer Open MPI cluster system. 
If I put these data files in /home/mypath/folder, is it possible that all non-head nodes can access the files in the folder ? 
I need to load some data to some nodes, if all nodes can access the data, I do not need to load them to each node one by one. 
If multiple nodes access the same file to get data, is there conflict ? 
For example, 
fopen(myFile) by node 1, at the same time fopen(myFile) by node 2. 
Is it allowed to do that on MPI cluster without conflict ? 
Are there some boost MPI routines to support this ? 
I cannot find them in boost.org.
Any help is appreciated. 
Jack
July 12  2010
From: binglongx_at_[hidden]
To: boost-users_at_[hidden]
Date: Tue, 13 Jul 2010 00:01:52 -0400
Subject: Re: [Boost-users] boost C++ load data from file

It's a wide shot without an example of your specific data 
organization, but this is something that might give your desired indexing 
(being efficient or not is disregarded here):
 
using namespace std; // excuse: just for 
brevity
typedef ... POWC; // What's the underlying type of this 
thing?
typedef map<POWC, float> PowcFloat;
typedef map<int, 
PF> YearPowcFloat;
typedef map<string, YearPowcFloat> 
Jumbo;
 
void load(const vector<string>& states, const 
vector<int>& years, const vector<POWC>& powcs, Jumbo& 
big)
{
 big.clear();
 BOOST_FOREACH(const string& state, 
states)
 {
  YearPowcFloat 
ypf;
  BOOST_FOREACH(int year, 
years)
  {
   PowcFloat 
pf;
   ifstream((state+boost::lexical_cast<string>(year)).c_str());
   for(size_t 
i=0; i<40; 
++i)
   {
    pair<POWC,float> 
ppf = ...; // populate ppf with each float from 
file
    pf.insert(ppf);
   }
   ypf.insert(pair<int,PF>(year,pf));
  }
  big.insert(pair<string,YearPowcFloat>(state,ypf));
 }
}
 
void test_load()
{
 vector<string> 
states;
 states.push_back("New 
York");
 states.push_back("Pennsylvania");
 vector<int> 
years;
 years.push_back(2009);
 years.push_back(2010);
 vector<POWC> 
powcs(40); // ...
 
 Jumbo big;
 load(states, years, powcs, 
big);
 cout<<"New York 2010 powcs[0] = "<< 
big["NewYork"][2010][powcs[0]] << endl;
}
 
 
 
 

From: Jack Bryan 
Sent: Monday, July 12, 2010 3:04 PM
To: boost-users_at_[hidden] 
Subject: [Boost-users] boost C++ load data from 
file

Dear All,

I need to load data (24 X 28 X 40) from 24 X 28 files into a datastructure 
so that my program can use
them to do operations. 

Each file has 40 data items, which are floating numbers.

I hope that the data structure can be indexed by item name not only by 
integer.

For example, for 24 states in USA, each state's power consumption is POWC 
in year Y. 

I hope that the data structure indexing is like this  

datastructure[newYork][year][powerconsumption]

My question is :

(1) how to load the data from 24 x 28 files by boost C++? 

It is boring to type file's name one by one in program. 

Is there an alternative to load the data by indexing the file names 
? 

for example, 

for (filename = name1; filename < filenum ; filename++)
{
load data from 
filename;
store loaded 
data in the datastructure.
}

(2) how to index the item of the datastructure by boost C++ ? 

Any help is appreciated. 

Jack 

July 11 2010

Hotmail is redefining busy with tools for the New Busy. Get more from your 
inbox. See how. 

_______________________________________________
Boost-users mailing 
list
Boost-users_at_[hidden]
http://lists.boost.org/mailman/listinfo.cgi/boost-users 		 	   		  
_________________________________________________________________
The New Busy is not the old busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_3