argparse add_mutually_exclusive_group should accept existing arguments to register conflicts

BPO	10984
Files	args_in_multiple_mutually_exclusive_groups.patch: Allow arguments to be passed into add_mutually_exclusive_group multigroup_1.patch multigroup_4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-01-22.23:01:55.200>
labels = ['type-feature', 'library']
title = 'argparse add_mutually_exclusive_group should accept existing arguments to register conflicts'
updated_at = <Date 2014-06-03.05:44:01.029>
user = 'https://bugs.python.org/gotgenes'

bugs.python.org fields:

activity = <Date 2014-06-03.05:44:01.029>
actor = 'paul.j3'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2011-01-22.23:01:55.200>
creator = 'gotgenes'
dependencies = []
files = ['30818', '30921', '30940']
hgrepos = []
issue_num = 10984
keywords = ['patch']
message_count = 13.0
messages = ['126862', '126879', '192444', '192765', '192844', '192954', '193071', '193184', '196771', '197960', '212458', '212482', '219660']
nosy_count = 5.0
nosy_names = ['bethard', 'gotgenes', 'micktwomey', 'paul.j3', 'jamadagni']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue10984'
versions = ['Python 3.3']

argparse supports registering conflicting arguments, however, it does so in a way that an argument may belong to at most one group of conflicting arguments. The inspiration for this bug is Stack Overflow question #4770576.
http://stackoverflow.com/questions/4770576/does-argparse-python-support-mutually-exclusive-groups-of-arguments

The most straightforward use case argparse can not accommodate is the following: the user has three flags, '-a', '-b', and '-c'. The flag '-b' is incompatible with both '-a' and with '-c', however, '-a' and '-c' are compatible with each other.

Current practice is to register a conflict by first defining a conflict group with parser.add_mutually_exclusive_group(), and then create new arguments within that group using group.add_argument(). Because the programmer is not allowed to create the argument prior to creating the group, an argument cannot be registered in two exclusive groups.

I feel it would be much more useful to be given the option to create exclusive groups after the programmer has defined and created the options, as is the design for ConflictsOptionParser
http://pypi.python.org/pypi/ConflictsOptionParser/

I'm definitely open to providing such functionality. I assume you're imagining something like:

parser = argparse.ArgumentParser()
a_action = parser.add_argument('-a')
b_action = parser.add_argument('-b')
c_action = parser.add_argument('-c')
d_action = parser.add_argument('-d')
parser.add_mutually_exclusive_group(a_action, c_action)
parser.add_mutually_exclusive_group(a_action, d_action)
...

If you can supply a patch, I'll take a look at it.

I've picked up on this as part of the EP 2013 sprints.

I've attached a patch which implements the behaviour described in the comments. It doesn't break any existing functionality but the help generated by argparse is definitely not quite right.

This approach of simply adding the existing actions to the group's _group_actions works fine, at least when it comes catching the error.

It may be difficult to get a useful usage line. In usage, arguments appear in the order in which they were created, optionals first, positionals after. Group notation is added if the subset of the arguments appear in its list in the same order.

In the patch .rst,

usage: PROG [-h] (--foo | --bar) [--baz]

the foo,bar group is marked correctly; the foo,baz group is not contiguous and is omited.

In bethard's example neither group will be marked
(class TestMutuallyExclusiveGroupWithExistingArguments)

But the problem isn't just with adding existing arguments.

class TestMutuallyExclusiveOptionalsMixed illustrates this with a case where group and parser arguments overlap.

In class TestMutuallyExclusiveOptionalsAndPositionalsMixed, the mix of optionals and positionals makes group marking impossible.

If the groups are in order, but overlap, usage can be a confusing mix

Groups ab, bc, cd, produce:
[-a A | [-b B | [-c C | -d D]

But if created in a different order, the usage can be:
[-a A | [-b B | -c C] -d D]

So there are 2 issues

if groups are not continuous or overlap, what is a meaningful usage?
any choice is likely to require a major reworking of the formatting logic.

Since confusing group markings are worse than none, a first step might be to flag a group added via this patch as 'do not mark'. Also add a note to the documentation that user may need to write their own grouping instructions (in usage, description or epilog).

One usage option is to make a subclass of HelpFormatter (the accepted way of customizing a formatter), and write a function that formats each group independently. For the example case, the resulting format might be:

usage: PROG [-h] [-b] [-a | -c] [-a | -d]

-h and -b are not part of any group. These are followed by the two groups. -a is repeated because it appears in both groups.

While playing with some examples, I found that exclusive group formatting has another failure case. If the usage line is long enough to wrap, optionals and positionals are formatted separately, with positionals appearing on a separate line(s). That means that if a group includes a positional, it will not be marked.

So (shortening lines for convenience sake), instead of:

usage: [-h] ... (-a | -b | x)

we get

usage: [-h] ... [-a] [-b]
       x

This is true even if arguments are added to the group in the normal way.

This patch adds a MultiGroupHelpFormatter that formats groups even if they share actions or the actions are not in the same order as in the parse._actions list. It sorts the groups so positional actions, if any appear in the correct order.

A long test case generates this help:

usage: PROG [-h] [-a A | -c C] [-a A | -d D] [-a A | -b B] [-b B | -d D]
            [-d D | x] foo [-b B | y]

positional arguments:
  x           x help
  foo         foo help        
  y           y help

optional arguments:
  -h, --help  show this help message and exit
  -a A        a help
  -b B        b help
  -c C        c help
  -d D        d help

In the 2nd usage line, the 2 groups, and action foo, are shown in the order in which x, foo, y were defined (and hence will be parsed), even though the groups were not defined in that order.

The default formatter could not format these groups, generating '[-h] [-a A] [-b B] ... x foo y' instead.

I have included the latest patch from http://bugs.python.org/issue11874. This splits the usage line generated by _format_actions_usage into parts that are groups or independent actions. The goal there is to correctly split long usage lines into multiple lines. Here it makes it easier to format groups and actions in new order.

If existing actions are added to new group as in the original patch for this issue, that group gets a no_usage = True attribute. The default formatter then will not attempt to format this group. The MultiGroupHelpFormatter ignores this attribute.

This patch needs better documentation. Test cases also need refinement, improving the names, and eliminating redundancies. Some of the new tests are copies of existing ones, but using the new formatter.

This patch produces the same usage as before, but I have rewritten _format_actions_usage() for both HelpFormatter and MultiGroupFormater.

The original HelpFormatter._format_actions_usage() formats the actions, splices in group markings, cleans up the text, if needed, tries to break it down into parts. But this is fragile, as shown here and in issues 11874, 18349).

Now _format_group_usage() and _format_just_actions_usage() format groups and actions directly, without the splice and divide steps. _format_actions_usage() for both classes call these to build a list of usage parts.

This change also solves http://bugs.python.org/issue11874 and http://bugs.python.org/issue18349, since it does not have to break up a formatted text line (and in the process get confused by [] and ()).

A possible further tweak is, in take_action(), test for conflicts before adding the action to 'seen_non_default_actions'

            if argument_values is not action.default:
                #seen_non_default_actions.add(action)
                for conflict_action in action_conflicts.get(action, []):
                    if conflict_action in seen_non_default_actions:
                       ...
                seen_non_default_actions.add(action)

This does not cause problems with any existing tests, but makes it possible to add an action twice to a group. Why do that? To prevent an action from occurring more than once. For some actions like 'count' and 'append' repeated use is expected, but for others it isn't expected, and may sometimes be a nuisance (the last occurrence is the one that sticks).

An example use would be:

    parser = argparse.ArgumentParser(prog="PROG",
        formatter_class=argparse.MultiGroupHelpFormatter)
    action = parser.add_argument('--arg', help='use this argument only once')
    group1 = parser.add_mutually_exclusive_group(action, action)
    args  = parser.parse_args()

calling this with:

python3 test_once.py --arg test --arg next

would produce this error message:

usage: PROG [-h] [--arg ARG | --arg ARG]
PROG: error: argument --arg: not allowed with argument --arg

The usage and error message aren't as clear as they might be if this feature was added 'from scratch'. But for a minor change like this, that may be an acceptable price.

Another situation in which this MultiGroupHelpFormatter helps is when one or more of the groups includes an optional positional.

The regular formatter moves all the positionals to the end, after the optionals. This move could easily break up a mutually exclusive group, and make formatting it impossible. But the MultiGroupHelpFormatter gives the group order priority.

Thus for example:

    p=argparse.ArgumentParser()
    # (formatter_class=argparse.MultiGroupHelpFormatter)
    g=p.add_mutually_exclusive_group()
    g.add_argument('-f')
    g.add_argument('foo',nargs='?')
    g=p.add_mutually_exclusive_group()
    g.add_argument('-b')
    g.add_argument('-c')
    g.add_argument('bar',nargs='*',default='X')
    print(p.format_usage())

produces (positionals at end, no group markings)

usage: PROG [-h] [-f F] [-b B] [-c C] [foo] [bar [bar ...]]

But the MultiGroupHelpFormatter produces:

usage: PROG [-h] [-f F | foo] [-b B | -c C | bar [bar ...]]

In this last case, the positionals are listed with their respective groups, and the groups are ordered by the relative ordering of the positionals.

I also wish to see argparse allowing me to define a group of arguments that conflict with another argument or another group of arguments and FWIW I feel the help output should be like:

prog [ --conflicter | [ --opt1 ] [ --opt2 ] ]

where --conflicter conflicts with --opt1 and --opt2 but those two don't conflict with each other and all are optional.

In http://bugs.python.org/issue11588 (Add "necessarily inclusive" groups to argparse) I propose a generalization to these testing groups that would solve your 'conflicter' case as follows:

    usage = 'prog [ --conflicter | [ --opt1 ] [ --opt2 ] ]'
    parser = argparse.ArgumentParser(usage=usage)
    conflicter = parser.add_argument("--conflicter", action='store_true')
    opt1 = parser.add_argument("--opt1", action='store_true')
    opt2 = parser.add_argument("--opt2", action='store_true')

    @parser.crosstest
    def test(parser, seen_actions, *args):
        if conflicter in seen_actions:
            if 0<len(seen_actions.intersection([opt1, opt2])):
                parser.error('--conflicter cannot be used with --opt1 or --opt2')

Groups, as currently defined, cannot handle nesting, and as a consequence cannot handle complex logic. My proposal is to replace groups with user defined conflict tests that would be run near the end of 'parse_args'.

This example shows, I think, that the proposal is powerful enough. I'm not sure about ease of use and logical transparency.

Formatting the usage line is a different issue, though the MultiGroupHelpFormatter that I propose here is a step in the right direction. For now a user written 'usage' is the simplest solution.

Another way to add an existing Action to a group is to modify the 'add_argument' method for the Group subclass. For example we could add this to the _MutuallyExclusiveGroup class:

    def add_argument(self, *args, **kwargs):
        # allow adding a prexisting Action
        if len(args) and isinstance(args[0], Action):
            action =  args[0]
            return self._group_actions.append(action)
        else:
            return super(_MutuallyExclusiveGroup, self).add_argument(*args, **kwargs)

With this the 1st example might be written as:

    group1 = parser.add_mutually_exclusive_group()
    a_action = parser.add_argument('-a')
    c_action = parser.add_argument('-c')
    group2 = parser.add_mutually_exclusive_group()
    group2.add_argument(a_action)
    d_action = parser.add_argument('-d')

This might be more intuitive to users.

I also wish to see argparse allowing me to define a group of arguments that conflict with another argument or another group of arguments and FWIW I feel the help output should be like:

prog [ --conflicter | [ --opt1 ] [ --opt2 ] ]

where --conflicter conflicts with --opt1 and --opt2 but those two don't conflict with each other and all are optional.

I know this is a very old issue, but this, to me, is an important feature, and I think it would be very nice if this issue could be re-visited.

The simplest use case could be wanting to break down your arguments into distinct groups visually, which is very useful when one has a large set of options as it aids readability, and still have mutually exclusive options. This doesn't seem to be possible without modifying argparse classes, which adds some overheard to users.

I haven't read the code, but it appears that "group" and "mutually exclusive group" are themselves... mutually exclusive. When logically, there isn't a good reason for them to be? It's natural to want to group options together. That doesn't mean those options can't have mutually exclusive relationships within themselves.

Currently, the way to have both groups and mutually exclusive groups without modifying the base class is to individually check and test each arg individually. If you have a dozen mutually exclusive args, that can easily add another 50 lines of code. It's not very practical :(

@rhettinger

@rhettinger tagging you as you seem to be the maintainer for this project. It appears @micktwomey wrote a patch which fixes this back in 2013. I imagine the code has changed a lot since this was first requested, but is there any chance of this being implemented?

feat(cli): break --help options into categories

I have just encountered this problem -- while I can write a workaround to functionally accomplish the desired behavior, I would prefer to do this via exclusive groups.

Navigation Menu

Repository navigation

argparse add_mutually_exclusive_group should accept existing arguments to register conflicts #55193

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

argparse add_mutually_exclusive_group should accept existing arguments to register conflicts #55193

Description

Activity

gotgenes commented on Jan 22, 2011

bethard commented on Jan 23, 2011

micktwomey commented on Jul 6, 2013

paulj3 commented on Jul 9, 2013

paulj3 commented on Jul 11, 2013

paulj3 commented on Jul 12, 2013

paulj3 commented on Jul 15, 2013

paulj3 commented on Jul 16, 2013

paulj3 commented on Sep 2, 2013

paulj3 commented on Sep 17, 2013

jamadagni commented on Feb 28, 2014

paulj3 commented on Feb 28, 2014

paulj3 commented on Jun 3, 2014

avalonv commented on Dec 5, 2022

avalonv commented on Dec 5, 2022

avalonv commented on Dec 5, 2022

LeggoMahEggo commented on Mar 16, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions