[Boost-bugs] [Boost C++ Libraries] #5033: Property Tree JSON Parser cannot handle utf-8 string correctly.

Subject: [Boost-bugs] [Boost C++ Libraries] #5033: Property Tree JSON Parser cannot handle utf-8 string correctly.
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2011-01-02 12:57:38

#5033: Property Tree JSON Parser cannot handle utf-8 string correctly.
---------------------------------------------+------------------------------
 Reporter:  Lorin Liu <liu.lorin@…>          |       Owner:  cornedbee    
     Type:  Bugs                             |      Status:  new          
Milestone:  To Be Determined                 |   Component:  property_tree
  Version:  Boost 1.45.0                     |    Severity:  Problem      
 Keywords:                                   |  
---------------------------------------------+------------------------------
 Please refer to the following code fragment.

 {{{
     // This is a json utf-8 string {"value": "天津"}
     const char json[] = {0x7B, 0x22, 0x76, 0x61, 0x6C, 0x75, 0x65, 0x22,
 0x3A, 0x20,
                          0x22, 0xE5, 0xA4, 0xA9, 0xE6, 0xB4, 0xA5, 0x22,
 0x7D, 0x00};

     boost::property_tree::ptree pt;
     boost::format fmter("%1% : %2% \n");
     std::stringstream strm;
     std::string value;

     strm << json;

     read_json(strm, pt);

     value = pt.get<std::string>("value");

     // Print the individual char one by one.
     // However the wrong result appears. All chars printed to console are
 0x7F.
     // And the expected result should be the chars of 0xE5, 0xA4, 0xA9,
 0xE6, 0xB4 and 0xA5.
     BOOST_FOREACH(char c, value)
         std::cout << (fmter % (int)(unsigned char) c % c) << std::endl;
 }}}

 After my investigation, this might be a bug in
 boost/property_tree/detail/json_parser_read.hpp.

 My patch for this issue is as follows.

 {{{
 --- json_parser_read.hpp.orig   2010-12-24 15:49:06.000000000 +0800
 +++ json_parser_read.hpp        2011-01-02 10:26:37.000000000 +0800
 @@ -145,7 +145,7 @@
              a_unicode(context &c): c(c) { }
              void operator()(unsigned long u) const
              {
 -                u = (std::min)(u, static_cast<unsigned
 long>((std::numeric_limits<Ch>::max)()));
 +                // u = (std::min)(u, static_cast<unsigned
 long>((std::numeric_limits<Ch>::max)()));
                  c.string += Ch(u);
              }
          };
 }}}

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/5033>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.