When Parameters are specified as part of a Describe block, the ExampleGroup/Describe/Context blocks are all but ignored. The expansion of the Parameters block is deferred to individual tests rather than following the syntax consistently and duplicating everything within the containing block.
This has a few different side effects, though I'm sure I'm missing some of the other negative consequences:
- The grouping of the tests doesn't match what's specified
- Parameter substitution isn't performed in titles of
Describe/Context/ExampleGroup
Before/After on ExampleGroups isn't processed in the correct order
Before/After can't include parameter values
A simple example:
my_setup(){
echo "entry" >> log.txt
}
my_teardown() {
echo "exit" >> log.txt
}
Describe "for options that take strings"
Parameters:block
# CLI option, string value to pass it
"label" "foo:bar"
# can be set to a blank string
"label" ""
"annotation" "foo:bar"
# can be set to a blank string
"annotation" ""
"include-glob" "some*pattern"
"exclude-glob" "some*pattern"
End
Before "my_setup"
After "my_teardown"
ExampleGroup "$1 set to '$2'"
It "'--$1 $2' w/ space"
When run source "$SCRIPT_UNDER_TEST" "--$1" "$2" "--help"
# minimal verification it prints help text and not an error about the options
The word 1 of the line 1 of the stderr should not equal "[FATAL]"
The word 1 of the line 1 of the stdout should equal "USAGE"
The status should be success
End
It "'--$1=$2' w/ equals"
When run source "$SCRIPT_UNDER_TEST" "--$1=$2" "--help"
# minimal verification it prints help text and not an error about the options
The word 1 of the line 1 of the stderr should not equal "[FATAL]"
The word 1 of the line 1 of the stdout should equal "USAGE"
The status should be success
End
End
End
The output of this looks like:
for options that take strings
set to ''
'--label foo:bar'w/ space
'--label 'w/ space
'--annotation foo:bar'w/ space
'--annotation 'w/ space
'--include-glob some*pattern'w/ space
'--exclude-glob some*pattern'w/ space
'--label=foo:bar' w/ equals
'--label=' w/ equals
'--annotation=foo:bar' w/ equals
'--annotation=' w/ equals
'--include-glob=some*pattern' w/ equals
'--exclude-glob=some*pattern' w/ equals
You can see the tests are not grouped as specified. Instead of following what the syntax says and creating groups of 2 tests each, it creates a single group containing all the tests. This is inconsistent with all other DSL syntax.
Parameter expansion doesn't happen in the ExampleGroup, as can be seen in the output set to ''.
From this I can see that the Before and After function on the ExampleGroup are getting called for only 1 entry into the ExampleGroup rather than what would be consistent with all other syntax interpretation: 1x per parameter.
Because the ExampleGroup is only entered once rather than 1x per parameter, the Before/After would not be able to do parameter substitution while remaining naive about whether the next subshell it applies to is an ExampleGroup or Example.
I suspect this is caused by an implementation hurdle.
The use of $1, $2, etc primarily only work if passed as arguments to the subshell, and don't propagate to nested subshells. Since tests are what we want to run multiple of, I'm guessing the solution to the implementation problem was to do a roll-up of parameters as part of each subshell until an Example subshell was encountered, then the test was called as an iteration of subshell calls over the sets of parameters that were collected.
While this does at least give consistent behavior when ExampleGroups are defined as part of a parameterized block, and gives a way for a Parameters set to be extended within nested blocks, it doesn't match with the rest of the DSL syntax (at least not from how the Parameters block is currently described). Most of the documentation goes out of its way to make it clear that Example and ExampleGroup are both subshells, and are similar in a lot of ways. The Before/After is a good example of this that the same keyword is used to define something for the next ExampleGroup or Example. There seems to be no other case where a keyword applies only to a subsequent Example and not also to subsequent ExampleGroups.
The documentation of the Parameters currently reads more like a template as a result, and is somewhat naturally presumed to be defining a for loop over everything it shares an ExampleGroup block with.
When
Parametersare specified as part of aDescribeblock, theExampleGroup/Describe/Contextblocks are all but ignored. The expansion of the Parameters block is deferred to individual tests rather than following the syntax consistently and duplicating everything within the containing block.This has a few different side effects, though I'm sure I'm missing some of the other negative consequences:
Describe/Context/ExampleGroupBefore/AfteronExampleGroups isn't processed in the correct orderBefore/Aftercan't include parameter valuesA simple example:
The output of this looks like:
You can see the tests are not grouped as specified. Instead of following what the syntax says and creating groups of 2 tests each, it creates a single group containing all the tests. This is inconsistent with all other DSL syntax.
Parameter expansion doesn't happen in the
ExampleGroup, as can be seen in the outputset to ''.From this I can see that the
BeforeandAfterfunction on theExampleGroupare getting called for only 1 entry into theExampleGrouprather than what would be consistent with all other syntax interpretation: 1x per parameter.Because the
ExampleGroupis only entered once rather than 1x per parameter, theBefore/Afterwould not be able to do parameter substitution while remaining naive about whether the next subshell it applies to is anExampleGrouporExample.I suspect this is caused by an implementation hurdle.
The use of
$1,$2, etc primarily only work if passed as arguments to the subshell, and don't propagate to nested subshells. Since tests are what we want to run multiple of, I'm guessing the solution to the implementation problem was to do a roll-up of parameters as part of each subshell until anExamplesubshell was encountered, then the test was called as an iteration of subshell calls over the sets of parameters that were collected.While this does at least give consistent behavior when
ExampleGroups are defined as part of a parameterized block, and gives a way for aParametersset to be extended within nested blocks, it doesn't match with the rest of the DSL syntax (at least not from how theParametersblock is currently described). Most of the documentation goes out of its way to make it clear thatExampleandExampleGroupare both subshells, and are similar in a lot of ways. TheBefore/Afteris a good example of this that the same keyword is used to define something for the nextExampleGrouporExample. There seems to be no other case where a keyword applies only to a subsequentExampleand not also to subsequentExampleGroups.The documentation of the
Parameterscurrently reads more like a template as a result, and is somewhat naturally presumed to be defining a for loop over everything it shares anExampleGroupblock with.