Skip to content

Parameters ignore ExampleGroup/Describe/'Context nesting (inconsistent depth-first parameter substitution) #359

@mtalexan

Description

@mtalexan

When Parameters are specified as part of a Describe block, the ExampleGroup/Describe/Context blocks are all but ignored. The expansion of the Parameters block is deferred to individual tests rather than following the syntax consistently and duplicating everything within the containing block.

This has a few different side effects, though I'm sure I'm missing some of the other negative consequences:

  1. The grouping of the tests doesn't match what's specified
  2. Parameter substitution isn't performed in titles of Describe/Context/ExampleGroup
  3. Before/After on ExampleGroups isn't processed in the correct order
  4. Before/After can't include parameter values

A simple example:

my_setup(){
    echo "entry" >> log.txt
}
my_teardown() {
    echo "exit" >> log.txt
}


Describe "for options that take strings"
    Parameters:block
        # CLI option, string value to pass it

        "label" "foo:bar"
        # can be set to a blank string
        "label" ""
        "annotation" "foo:bar"
        # can be set to a blank string
        "annotation" ""
        "include-glob" "some*pattern"
        "exclude-glob" "some*pattern"
    End

    Before "my_setup"
    After "my_teardown"

    ExampleGroup "$1 set to '$2'"
        It "'--$1 $2' w/ space"
            When run source "$SCRIPT_UNDER_TEST" "--$1" "$2" "--help"
            # minimal verification it prints help text and not an error about the options
            The word 1 of the line 1 of the stderr should not equal "[FATAL]"
            The word 1 of the line 1 of the stdout should equal "USAGE"
            The status should be success
        End
        It "'--$1=$2' w/ equals"
            When run source "$SCRIPT_UNDER_TEST" "--$1=$2" "--help"
            # minimal verification it prints help text and not an error about the options
            The word 1 of the line 1 of the stderr should not equal "[FATAL]"
            The word 1 of the line 1 of the stdout should equal "USAGE"
            The status should be success
        End
    End
End

The output of this looks like:

for options that take strings
   set to ''
    '--label foo:bar'w/ space
    '--label 'w/ space
    '--annotation foo:bar'w/ space
    '--annotation 'w/ space
    '--include-glob some*pattern'w/ space
    '--exclude-glob some*pattern'w/ space
    '--label=foo:bar' w/ equals
    '--label=' w/ equals
    '--annotation=foo:bar' w/ equals
    '--annotation=' w/ equals
    '--include-glob=some*pattern' w/ equals
    '--exclude-glob=some*pattern' w/ equals

You can see the tests are not grouped as specified. Instead of following what the syntax says and creating groups of 2 tests each, it creates a single group containing all the tests. This is inconsistent with all other DSL syntax.

Parameter expansion doesn't happen in the ExampleGroup, as can be seen in the output set to ''.

From this I can see that the Before and After function on the ExampleGroup are getting called for only 1 entry into the ExampleGroup rather than what would be consistent with all other syntax interpretation: 1x per parameter.

Because the ExampleGroup is only entered once rather than 1x per parameter, the Before/After would not be able to do parameter substitution while remaining naive about whether the next subshell it applies to is an ExampleGroup or Example.


I suspect this is caused by an implementation hurdle.
The use of $1, $2, etc primarily only work if passed as arguments to the subshell, and don't propagate to nested subshells. Since tests are what we want to run multiple of, I'm guessing the solution to the implementation problem was to do a roll-up of parameters as part of each subshell until an Example subshell was encountered, then the test was called as an iteration of subshell calls over the sets of parameters that were collected.

While this does at least give consistent behavior when ExampleGroups are defined as part of a parameterized block, and gives a way for a Parameters set to be extended within nested blocks, it doesn't match with the rest of the DSL syntax (at least not from how the Parameters block is currently described). Most of the documentation goes out of its way to make it clear that Example and ExampleGroup are both subshells, and are similar in a lot of ways. The Before/After is a good example of this that the same keyword is used to define something for the next ExampleGroup or Example. There seems to be no other case where a keyword applies only to a subsequent Example and not also to subsequent ExampleGroups.

The documentation of the Parameters currently reads more like a template as a result, and is somewhat naturally presumed to be defining a for loop over everything it shares an ExampleGroup block with.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions