Boost logo

Boost-Commit :

Subject: [Boost-commit] svn:boost r49216 - in branches/release: . boost/detail libs/detail libs/tokenizer libs/tokenizer/test status
From: marshall_at_[hidden]
Date: 2008-10-09 14:05:36


Author: marshall
Date: 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
New Revision: 49216
URL: http://svn.boost.org/trac/boost/changeset/49216

Log:
Merged revisions 43283,43992,44388 via svnmerge from
https://svn.boost.org/svn/boost/trunk

........
  r43283 | bemandawes | 2008-02-16 18:01:32 -0800 (Sat, 16 Feb 2008) | 1 line
  
  Show output of example program
........
  r43992 | marshall | 2008-04-01 18:42:32 -0700 (Tue, 01 Apr 2008) | 1 line
  
  Replaced all occurrences of non-ASCII copyright symbol with '(c)' for people using non-ASCII code pages
........
  r44388 | grafik | 2008-04-13 16:55:58 -0700 (Sun, 13 Apr 2008) | 1 line
  
  Move tokenizer tests into canonical test subdir structure.
........

Added:
   branches/release/libs/tokenizer/test/
      - copied from r44388, /trunk/libs/tokenizer/test/
   branches/release/libs/tokenizer/test/Jamfile.v2
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/Jamfile.v2
   branches/release/libs/tokenizer/test/examples.cpp
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/examples.cpp
   branches/release/libs/tokenizer/test/simple_example_1.cpp
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/simple_example_1.cpp
   branches/release/libs/tokenizer/test/simple_example_2.cpp
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/simple_example_2.cpp
   branches/release/libs/tokenizer/test/simple_example_3.cpp
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/simple_example_3.cpp
   branches/release/libs/tokenizer/test/simple_example_4.cpp
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/simple_example_4.cpp
   branches/release/libs/tokenizer/test/simple_example_5.cpp
      - copied unchanged from r44388, /trunk/libs/tokenizer/test/simple_example_5.cpp
Removed:
   branches/release/libs/tokenizer/examples.cpp
   branches/release/libs/tokenizer/simple_example_1.cpp
   branches/release/libs/tokenizer/simple_example_2.cpp
   branches/release/libs/tokenizer/simple_example_3.cpp
   branches/release/libs/tokenizer/simple_example_4.cpp
   branches/release/libs/tokenizer/simple_example_5.cpp
Properties modified:
   branches/release/ (props changed)
Text files modified:
   branches/release/boost/detail/utf8_codecvt_facet.hpp | 2 +-
   branches/release/libs/detail/utf8_codecvt_facet.cpp | 2 +-
   branches/release/libs/tokenizer/char_sep_example_1.cpp | 2 +-
   branches/release/libs/tokenizer/tokenizer.htm | 26 ++++++++++++++++----------
   branches/release/status/Jamfile.v2 | 12 +-----------
   5 files changed, 20 insertions(+), 24 deletions(-)

Modified: branches/release/boost/detail/utf8_codecvt_facet.hpp
==============================================================================
--- branches/release/boost/detail/utf8_codecvt_facet.hpp (original)
+++ branches/release/boost/detail/utf8_codecvt_facet.hpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
@@ -1,4 +1,4 @@
-// Copyright © 2001 Ronald Garcia, Indiana University (garcia_at_[hidden])
+// Copyright (c) 2001 Ronald Garcia, Indiana University (garcia_at_[hidden])
 // Andrew Lumsdaine, Indiana University (lums_at_[hidden]).
 // Distributed under the Boost Software License, Version 1.0. (See accompany-
 // ing file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

Modified: branches/release/libs/detail/utf8_codecvt_facet.cpp
==============================================================================
--- branches/release/libs/detail/utf8_codecvt_facet.cpp (original)
+++ branches/release/libs/detail/utf8_codecvt_facet.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
@@ -1,7 +1,7 @@
 /////////1/////////2/////////3/////////4/////////5/////////6/////////7/////////8
 // utf8_codecvt_facet.cpp
 
-// Copyright © 2001 Ronald Garcia, Indiana University (garcia_at_[hidden])
+// Copyright (c) 2001 Ronald Garcia, Indiana University (garcia_at_[hidden])
 // Andrew Lumsdaine, Indiana University (lums_at_[hidden]).
 // Use, modification and distribution is subject to the Boost Software
 // License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at

Modified: branches/release/libs/tokenizer/char_sep_example_1.cpp
==============================================================================
--- branches/release/libs/tokenizer/char_sep_example_1.cpp (original)
+++ branches/release/libs/tokenizer/char_sep_example_1.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
@@ -1,4 +1,4 @@
-// © Copyright Jeremy Siek 2002.
+// (c) Copyright Jeremy Siek 2002.
 
 // Distributed under the Boost Software License, Version 1.0. (See
 // accompanying file LICENSE_1_0.txt or copy at

Deleted: branches/release/libs/tokenizer/examples.cpp
==============================================================================
--- branches/release/libs/tokenizer/examples.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
+++ (empty file)
@@ -1,156 +0,0 @@
-// Boost tokenizer examples -------------------------------------------------//
-
-// (c) Copyright John R. Bandela 2001.
-
-// Distributed under the Boost Software License, Version 1.0. (See
-// accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-
-// See http://www.boost.org for updates, documentation, and revision history.
-
-#include <iostream>
-#include <iterator>
-#include <string>
-#include <boost/tokenizer.hpp>
-#include <boost/array.hpp>
-
-#include <boost/test/minimal.hpp>
-
-int test_main( int /*argc*/, char* /*argv*/[] )
-{
- using namespace std;
- using namespace boost;
-
- // Use tokenizer
- {
- const string test_string = ";;Hello|world||-foo--bar;yow;baz|";
- string answer[] = { "Hello", "world", "foo", "bar", "yow", "baz" };
- typedef tokenizer<char_separator<char> > Tok;
- char_separator<char> sep("-;|");
- Tok t(test_string, sep);
- BOOST_REQUIRE(equal(t.begin(),t.end(),answer));
- }
- {
- const string test_string = ";;Hello|world||-foo--bar;yow;baz|";
- string answer[] = { "", "", "Hello", "|", "world", "|", "", "|", "",
- "foo", "", "bar", "yow", "baz", "|", "" };
- typedef tokenizer<char_separator<char> > Tok;
- char_separator<char> sep("-;", "|", boost::keep_empty_tokens);
- Tok t(test_string, sep);
- BOOST_REQUIRE(equal(t.begin(), t.end(), answer));
- }
- {
- const string test_string = "This,,is, a.test..";
- string answer[] = {"This","is","a","test"};
- typedef tokenizer<> Tok;
- Tok t(test_string);
- BOOST_REQUIRE(equal(t.begin(),t.end(),answer));
- }
-
- {
- const string test_string = "Field 1,\"embedded,comma\",quote \\\", escape \\\\";
- string answer[] = {"Field 1","embedded,comma","quote \""," escape \\"};
- typedef tokenizer<escaped_list_separator<char> > Tok;
- Tok t(test_string);
- BOOST_REQUIRE(equal(t.begin(),t.end(),answer));
-
- }
-
- {
- const string test_string = ",1,;2\\\";3\\;,4,5^\\,\'6,7\';";
- string answer[] = {"","1","","2\"","3;","4","5\\","6,7",""};
- typedef tokenizer<escaped_list_separator<char> > Tok;
- escaped_list_separator<char> sep("\\^",",;","\"\'");
- Tok t(test_string,sep);
- BOOST_REQUIRE(equal(t.begin(),t.end(),answer));
-
- }
-
- {
- const string test_string = "12252001";
- string answer[] = {"12","25","2001"};
- typedef tokenizer<offset_separator > Tok;
- boost::array<int,3> offsets = {{2,2,4}};
- offset_separator func(offsets.begin(),offsets.end());
- Tok t(test_string,func);
- BOOST_REQUIRE(equal(t.begin(),t.end(),answer));
-
- }
-
- // Use token_iterator_generator
- {
-
- const string test_string = "This,,is, a.test..";
- string answer[] = {"This","is","a","test"};
- typedef token_iterator_generator<char_delimiters_separator<char> >::type Iter;
- Iter begin = make_token_iterator<string>(test_string.begin(),
- test_string.end(),char_delimiters_separator<char>());
- Iter end;
- BOOST_REQUIRE(equal(begin,end,answer));
- }
-
- {
- const string test_string = "Field 1,\"embedded,comma\",quote \\\", escape \\\\";
- string answer[] = {"Field 1","embedded,comma","quote \""," escape \\"};
- typedef token_iterator_generator<escaped_list_separator<char> >::type Iter;
- Iter begin = make_token_iterator<string>(test_string.begin(),
- test_string.end(),escaped_list_separator<char>());
- Iter begin_c(begin);
- Iter end;
- BOOST_REQUIRE(equal(begin,end,answer));
-
- while(begin_c != end)
- {
- BOOST_REQUIRE(begin_c.at_end() == 0);
- ++begin_c;
- }
- BOOST_REQUIRE(begin_c.at_end());
-
- }
-
- {
- const string test_string = "12252001";
- string answer[] = {"12","25","2001"};
- typedef token_iterator_generator<offset_separator>::type Iter;
- boost::array<int,3> offsets = {{2,2,4}};
- offset_separator func(offsets.begin(),offsets.end());
- Iter begin = make_token_iterator<string>(test_string.begin(),
- test_string.end(),func);
- Iter end= make_token_iterator<string>(test_string.end(),
- test_string.end(),func);
- BOOST_REQUIRE(equal(begin,end,answer));
-
- }
-
- // Test copying
- {
- const string test_string = "abcdef";
- token_iterator_generator<offset_separator>::type beg, end, other;
- boost::array<int,3> ar = {{1,2,3}};
- offset_separator f(ar.begin(),ar.end());
- beg = make_token_iterator<string>(test_string.begin(),test_string.end(),f);
-
- ++beg;
- other = beg;
- ++other;
-
- BOOST_REQUIRE(*beg=="bc");
- BOOST_REQUIRE(*other=="def");
-
- other = make_token_iterator<string>(test_string.begin(),
- test_string.end(),f);
-
- BOOST_REQUIRE(*other=="a");
- }
-
- // Test non-default constructed char_delimiters_separator
- {
- const string test_string = "how,are you, doing";
- string answer[] = {"how",",","are you",","," doing"};
- tokenizer<> t(test_string,char_delimiters_separator<char>(true,",",""));
- BOOST_REQUIRE(equal(t.begin(),t.end(),answer));
- }
-
- return 0;
-}
-

Deleted: branches/release/libs/tokenizer/simple_example_1.cpp
==============================================================================
--- branches/release/libs/tokenizer/simple_example_1.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
+++ (empty file)
@@ -1,25 +0,0 @@
-// © Copyright John R. Bandela 2001.
-
-// Distributed under the Boost Software License, Version 1.0. (See
-// accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-
-// See http://www.boost.org/libs/tokenizer for documenation
-
-
-// simple_example_1.cpp
-#include<iostream>
-#include<boost/tokenizer.hpp>
-#include<string>
-
-int main(){
- using namespace std;
- using namespace boost;
- string s = "This is, a test";
- tokenizer<> tok(s);
- for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
- cout << *beg << "\n";
- }
- return 0;
-}
-

Deleted: branches/release/libs/tokenizer/simple_example_2.cpp
==============================================================================
--- branches/release/libs/tokenizer/simple_example_2.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
+++ (empty file)
@@ -1,24 +0,0 @@
-// © Copyright John R. Bandela 2001.
-
-// Distributed under the Boost Software License, Version 1.0. (See
-// accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-
-// See http://www.boost.org/libs/tokenizer for documenation
-
-// simple_example_2.cpp
-#include<iostream>
-#include<boost/tokenizer.hpp>
-#include<string>
-
-int main(){
- using namespace std;
- using namespace boost;
- string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3";
- tokenizer<escaped_list_separator<char> > tok(s);
- for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){
- cout << *beg << "\n";
- }
- return 0;
-}
-

Deleted: branches/release/libs/tokenizer/simple_example_3.cpp
==============================================================================
--- branches/release/libs/tokenizer/simple_example_3.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
+++ (empty file)
@@ -1,25 +0,0 @@
-// © Copyright John R. Bandela 2001.
-
-// Distributed under the Boost Software License, Version 1.0. (See
-// accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-
-// See http://www.boost.org/libs/tokenizer for documenation
-
-// simple_example_3.cpp
-#include<iostream>
-#include<boost/tokenizer.hpp>
-#include<string>
-
-int main(){
- using namespace std;
- using namespace boost;
- string s = "12252001";
- int offsets[] = {2,2,4};
- offset_separator f(offsets, offsets+3);
- tokenizer<offset_separator> tok(s,f);
- for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){
- cout << *beg << "\n";
- }
- return 0;
-}

Deleted: branches/release/libs/tokenizer/simple_example_4.cpp
==============================================================================
--- branches/release/libs/tokenizer/simple_example_4.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
+++ (empty file)
@@ -1,24 +0,0 @@
-// © Copyright John R. Bandela 2001.
-
-// Distributed under the Boost Software License, Version 1.0. (See
-// accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-
-// See http://www.boost.org/libs/tokenizer for documenation
-
-// simple_example_4.cpp
-#include<iostream>
-#include<boost/tokenizer.hpp>
-#include<string>
-
-int main(){
- using namespace std;
- using namespace boost;
- string s = "This is, a test";
- tokenizer<char_delimiters_separator<char> > tok(s);
- for(tokenizer<char_delimiters_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){
- cout << *beg << "\n";
- }
- return 0;
-}
-

Deleted: branches/release/libs/tokenizer/simple_example_5.cpp
==============================================================================
--- branches/release/libs/tokenizer/simple_example_5.cpp 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
+++ (empty file)
@@ -1,34 +0,0 @@
-// © Copyright John R. Bandela 2001.
-
-// Distributed under the Boost Software License, Version 1.0. (See
-// accompanying file LICENSE_1_0.txt or copy at
-// http://www.boost.org/LICENSE_1_0.txt)
-
-// See http://www.boost.org/libs/tokenizer for documenation
-
-/// simple_example_5.cpp
-#include<iostream>
-#include<boost/token_iterator.hpp>
-#include<string>
-
-#ifdef __BORLANDC__
-// compiler bug fix:
-template class boost::token_iterator_generator<boost::offset_separator>::type;
-#endif
-
-int main(){
- using namespace std;
- using namespace boost;
- string s = "12252001";
- int offsets[] = {2,2,4};
- offset_separator f(offsets, offsets+3);
- typedef token_iterator_generator<offset_separator>::type Iter;
- Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
- Iter end = make_token_iterator<string>(s.end(),s.end(),f);
- // The above statement could also have been what is below
- // Iter end;
- for(;beg!=end;++beg){
- cout << *beg << "\n";
- }
- return 0;
-}

Modified: branches/release/libs/tokenizer/tokenizer.htm
==============================================================================
--- branches/release/libs/tokenizer/tokenizer.htm (original)
+++ branches/release/libs/tokenizer/tokenizer.htm 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
@@ -4,7 +4,7 @@
 <head>
   <meta http-equiv="Content-Language" content="en-us">
   <meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
- <meta name="GENERATOR" content="Microsoft FrontPage 6.0">
+ <meta name="GENERATOR" content="Microsoft FrontPage 5.0">
   <meta name="ProgId" content="FrontPage.Editor.Document">
 
   <title>Boost Tokenizer Class</title>
@@ -16,8 +16,7 @@
   "86"><br></p>
 
   <h1 align="center">Tokenizer Class</h1>
- <pre>
- template &lt;
+ <pre> template &lt;
         class TokenizerFunc = char_delimiters_separator&lt;char&gt;,
         class Iterator = std::string::const_iterator,
         class Type = std::string
@@ -33,8 +32,7 @@
   iterator provided by begin.</p>
 
   <h2>Example</h2>
- <pre>
-// simple_example_1.cpp
+ <pre>// simple_example_1.cpp
 #include&lt;iostream&gt;
 #include&lt;boost/tokenizer.hpp&gt;
 #include&lt;string&gt;
@@ -50,7 +48,16 @@
 }
 </pre>
 
- <p>&nbsp;</p>
+ <p>The output from simple_example_1 is:</p>
+
+ <blockquote>
+
+ <p><code>This<br>
+ is<br>
+ a<br>
+ test</code></p>
+
+ </blockquote>
 
   <h3>Template Parameters</h3>
 
@@ -156,8 +163,7 @@
   <p>&nbsp;</p>
 
   <h2>Construction and Member Functions</h2>
- <pre>
-tokenizer(Iterator first, Iterator last,const TokenizerFunc&amp; f = TokenizerFunc())
+ <pre>tokenizer(Iterator first, Iterator last,const TokenizerFunc&amp; f = TokenizerFunc())
 
 template&lt;class Container&gt;
 tokenizer(const Container&amp; c,const TokenizerFunc&amp; f = TokenizerFunc())
@@ -226,7 +232,7 @@
   height="31" width="88"></a></p>
 
   <p>Revised
- <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->25 December, 2006<!--webbot bot="Timestamp" endspan i-checksum="38518" --></p>
+ <!--webbot bot="Timestamp" s-type="EDITED" s-format="%d %B, %Y" startspan -->16 February, 2008<!--webbot bot="Timestamp" endspan i-checksum="40414" --></p>
 
   <p><i>Copyright &copy; 2001 John R. Bandela</i></p>
 
@@ -235,4 +241,4 @@
   copy at <a href=
   "http://www.boost.org/LICENSE_1_0.txt">http://www.boost.org/LICENSE_1_0.txt>)</i></p>
 </body>
-</html>
+</html>
\ No newline at end of file

Modified: branches/release/status/Jamfile.v2
==============================================================================
--- branches/release/status/Jamfile.v2 (original)
+++ branches/release/status/Jamfile.v2 2008-10-09 14:05:35 EDT (Thu, 09 Oct 2008)
@@ -86,6 +86,7 @@
 build-project ../libs/system/test ; # test-suite system
 build-project ../libs/test/test ; # test-suite test
 build-project ../libs/thread/test ; # test-suite thread
+build-project ../libs/tokenizer/test ; # test-suite tokenizer
 build-project ../libs/tr1/test ; # test-suite tr1
 build-project ../libs/tuple/test ; # test-suite tuple
 build-project ../libs/type_traits/test ; # test-suite type_traits
@@ -180,16 +181,5 @@
 
     compile libs/timer/timer_test.cpp ;
 
- test-suite tokenizer
- : [ run libs/tokenizer/examples.cpp test_exec_monitor ]
- [ run libs/tokenizer/simple_example_1.cpp ]
- [ run libs/tokenizer/simple_example_2.cpp ]
- [ run libs/tokenizer/simple_example_3.cpp ]
- [ run libs/tokenizer/simple_example_4.cpp ]
- [ run libs/tokenizer/simple_example_5.cpp ]
- ;
-
-
-
 
 


Boost-Commit list run by bdawes at acm.org, david.abrahams at rcn.com, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk