[Boost-bugs] [Boost C++ Libraries] #10718: program_options: Multitoken option is limited to 32000 token

Subject: [Boost-bugs] [Boost C++ Libraries] #10718: program_options: Multitoken option is limited to 32000 token
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2014-10-30 10:08:19

#10718: program_options: Multitoken option is limited to 32000 token
------------------------------+-----------------------------
 Reporter:  hans.hohenfeld@…  |      Owner:  vladimir_prus
     Type:  Bugs              |     Status:  new
Milestone:  To Be Determined  |  Component:  program_options
  Version:  Boost 1.55.0      |   Severity:  Problem
 Keywords:                    |
------------------------------+-----------------------------
 Dear all,

 I've created an application using boost::program_options to handle command
 line parameters. Beyond several other parameters, it uses a multitoken
 parameter to handle input file names passed. The application is basically
 executed like:

 ./prog -i *.nii

 where *.nii resolves to a substantial number of files.

 The relevant part of my application's code is

 {{{
 typedef std::vector<std::string> FileContainer;
 FileContainer files;
 namespace po = boost::program_options;

 //...

 po::options_description opts("Test");
 opts.add_options() ("input,i",
 po::value<FileContainer>(&files)->multitoken(),
                     "The input file(s)");

 po::variables_map vm;
 po::store(po::parse_command_line(ac, av, opts), vm);
 po::notify(vm);
 }}}

 So far it works very well. A problem occurs, as soon as the number of
 input files and therefore the number of tokens exceeds 32000 (which is a
 regular scenario, as the application is running on a HPC cluster and is
 working with massive amounts of data). In such cases, the first 32000 file
 names are parsed, the remaining ones are skipped silently.

 I searched a little in the program_options code and found the reason for
 this behavior in

 boost/program_options/value_semantic.hpp

 from line 315

 {{{
        unsigned max_tokens() const {
             if (m_multitoken) {
                 return 32000;
             } else if (m_zero_tokens) {
                 return 0;
             } else {
                 return 1;
             }
         }
 }}}

 This is the case for all recent boost versions, including the latest from
 git.

 I replaced the 32000 with UINT_MAX from <climits> and it seems to work.
 Now my questions:

 1. Is 32000 intentional? UINT_MAX would also create an upper limit, but a
 much larger one.
 2. Is there a reason to "fail" silently, when the limit is reached? If
 not, I'd suggest to add an error/warning or something.

 I assume this is a rare use case and therefor won't cause any problems for
 the majority of applications and furthermore there are plenty of
 workaround (including simply not using boost::program_options), but
 nevertheless...

 If you need any further details, just let me know.

 Thank you.

 best regards,

 Hans

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/10718>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.