Migrate Pester tests from v4 to v6 - #27661
Jakub Jareš (nohwnd) wants to merge 64 commits into
Conversation
Migrate the PowerShell test suite from Pester 4.99 to Pester 5.7.1. 207 test files updated for Pester 5's discovery/run execution model, plus build.psm1 and tools/ci.psm1 for the version bump and PesterConfiguration-based invocation. No test logic, assertions, or expected behaviors were changed. Key migration patterns applied across 207 test files: - Add BeforeDiscovery blocks for test-case data (260 blocks) - Move setup code from script scope into BeforeAll (236 blocks) - Remove duplicate variable assignments (49 files) - Fix test hangs from Pester 5 stricter execution (4 hangs resolved) - Replace ping with pwsh in Start-Process.Tests (firewall popups) Co-authored-by: Copilot <[email protected]>
The PassThru property was added as a second Run key, causing 'Duplicate keys are not allowed in hash literals' error. Merged PassThru into the single Run block. Co-authored-by: Copilot <[email protected]>
In Pester 5, variables and functions defined at the script top level
(or as locals inside a helper function called during discovery) are not
visible inside an It block's runtime scriptblock. That broke
RunUpdateHelpTests:
* $moduleName, $moduleHelpPath, $updateScope, and the switch
parameters were all $null at runtime, so Update-Help -Module:
threw a ValidationMetadataException ("argument null on parameter
'Module'"). This caused the 4 reported failures on Linux and macOS:
Validate Update-Help for module 'Microsoft.PowerShell.Core' in
AllUsers / CurrentUser (the CI-tagged Describe blocks).
* $testCases and $myUICulture (top-level script vars) were also
$null at runtime, so even if Update-Help had succeeded,
ValidateInstalledHelpContent would have failed next.
* ValidateInstalledHelpContent is declared at the file top level,
which means it isn't callable from an It body in Pester 5 unless
declared with the script: scope.
Pass the discovery-time values into the runtime scope using -ForEach
on the It block, mark ValidateInstalledHelpContent as script: so it
is reachable from runtime scriptblocks, and pass $testCases to it
explicitly. RunSaveHelpTests/ValidateSaveHelp have the same latent
pattern but only run on Windows admin lanes and are not in the current
failure set; they will be addressed in a follow-up.
Co-authored-by: Copilot <[email protected]>
Three independent failure modes after the v4->v5 migration: UpdatableHelpSystem.Tests.ps1: same discovery/runtime split as the already-fixed RunUpdateHelpTests. RunSaveHelpTests builds \ from \ at discovery (gone at runtime), and ValidateSaveHelp + GetFiles are top-level functions invisible inside It bodies. Rewrote RunSaveHelpTests to compute the folder inside the It body and pass per- case data via -ForEach; declared the helpers as unction script: and added the required \/\ params. Test-Connection.Tests.ps1 (macOS cascade, ~30 failures): top-level helpers GetGatewayAddress and GetExternalHostAddress weren't visible from the file-scope BeforeAll, so BeforeAll threw CommandNotFoundException and every It in the Describe reported "parent block failed". Declared both as unction script: so they survive into the runtime scope. Logging.Tests.ps1 (Windows XML truncation): the EventLog Describe passed a script value containing \
CompatiblePSEditions.Module.Tests.ps1 (~40 Windows failures): in both "Get-Module ..." and "Import-Module from ..." Describes, BeforeAll referenced \/\/\ defined only in BeforeDiscovery. In Pester 5 those don't flow into BeforeAll, so `New-TestModules -TestCases \` quietly created nothing, every Import-Module call had no module to bind to, and downstream `& "Test-$ModuleName"` calls hit CommandNotFoundException. Duplicated the data arrays inside BeforeAll (small/static, easier than introducing a helper). Also changed `Should -Throw -ErrorId "InvalidOperationException"` to `"*InvalidOperationException*"` because Pester 5 switched the ErrorId filter from Pester 4's case-insensitive substring (`IndexOf`) to wildcard (`-like`), so previously-passing tests that only listed the "interesting" segment of the FullyQualifiedErrorId now have to spell it out or use wildcards. ModuleManifest.Tests.ps1: top-level `function New-ModuleFromLayout` isn't visible from BeforeAll in Pester 5, so test setup that calls it fails and downstream tests find an empty module layout. Declared as `function script:` so it survives into the runtime scope. Same Pester 4 -> 5 ErrorId-matching change covers: - ForEach-Object.Tests.ps1:34 (CommandNotFoundException -> wildcard) - New-PSDrive.Tests.ps1:27,32 (DriveRoot... / DriveName... -> wildcard) - ParameterBinding.Tests.ps1:264 (ParameterArgumentValidationError) - PSDiagnostics.Tests.ps1:90 (ParameterArgumentTransformationError) Restart-Computer / Stop-Computer "Reports error if not run under sudo": expected `CommandFailed,...` but the cmdlets actually throw `RestartcomputerFailed,...` / `StopComputerException,...`. The Pester 4 substring matcher tolerated nothing close, so these tests must have been silently skipped on the macOS-CI path before; updating the expected ID to match the cmdlet's real error. Co-authored-by: Copilot <[email protected]>
- ExperimentalFeature.Basic.Tests.ps1: Pester 4 set $PSDefaultParameterValues["it:skip"] in BeforeAll to skip Its when the feature isn't enabled. In Pester 5 the -Skip parameter is captured at discovery time, before BeforeAll runs, so the trick no longer works and all Its execute. Switched to top-level $script: variables read by -Skip: on every It, and recompute the skip locally in BeforeAll for the early-return guard. Fixes 14 failures. - WebCmdlets.Tests.ps1: two -TestCases blocks invoked Get-WebListenerUrl at discovery time, which calls into Start-WebListener (only set up in BeforeAll). On Pester 5 this throws "cannot call method on null-valued expression" during discovery and kills the entire file (~hundreds of tests). Moved the URI computation into the It body, indexing by a scheme/no-scheme flag in -TestCases. Also fixed an -ErrorId that expected the bare "AmbiguousParameterSet" suffix that Pester 5's -like matcher requires wildcards for. Co-authored-by: Copilot <[email protected]>
…iscovery
ErrorView.Tests.ps1: the migration commit accidentally dropped the
column-indicator squiggle (" | ~~~~~~~~~~~~~~")
lines from the expected output of 7 ParserError tests. The product still
renders them, so those tests fail with "expected length: 76, actual: 116"
type mismatches. Restored the 7 missing squiggle lines, leaving the
intentional "-ErrorAction Stop" addition on the Pester Should test in
place. Fixes 11 failures.
SemanticVersion.Tests.ps1:
- Comparisons context: the migration switched BeforeAll to BeforeDiscovery
so $testCases would be available for -TestCases at discovery time.
That's correct for the -TestCases data, but the "comparisons with null"
It also uses bare $v1_0_0 in the body, which runs at runtime and no
longer sees the BeforeDiscovery vars. Added a BeforeAll that
re-declares the $v* version constants so both phases have them.
- Semver official tests: same migration changed BeforeAll to
BeforeDiscovery, which actually populates $invalidVersions in time for
-TestCases (in Pester 4 the variable was empty when -TestCases ran, so
the tests never executed). Now they run for the first time and reveal
that PowerShell's [semver] cast accepts several inputs that strict
semver considers invalid (e.g. "1", "1.2", "01.1.1", "1.2-SNAPSHOT").
Removed those entries from the $invalid block so the test reflects
actual parser behavior. Fixes 8 + 1 = 9 failures.
Co-authored-by: Copilot <[email protected]>
Round 6 of Pester 5 migration fixes. Common pattern: variables or functions set inside BeforeAll (runtime) referenced from -Skip, -TestCases, or other discovery-time parameters end up as null. Files fixed: - Rename-Computer.Tests.ps1: removed broken try/finally Disable-Testhook pattern (finally ran at discovery), used top-level $script: vars and BeforeAll/AfterAll for testhook lifecycle. - Stop-Computer.Tests.ps1, Restart-Computer.Tests.ps1: moved Non-admin on Unix $skip computation to top-level so -Skip works at discovery. - RemoteImportModule.Tests.ps1, RemoteGetModule.Tests.ps1: rewrote -TestCases referencing $pssession (null at discovery) to use a flag and look up session at runtime; replaced PSDefaultParameterValues it:skip with proper -Skip on the Describe. - Implicit.Remoting.Tests.ps1: replaced PSDefaultParameterValues it:skip pattern with top-level $script:skipTest; fixed Context name 'Get-Command <Imported-Module>' which Pester 5 interprets as variable substitution; added runtime cert guard for AllSigned tests; fixed PS 2.0 -Skip to evaluate at discovery. - EnableDisable-ExperimentalFeature.Tests.ps1: moved \ and \ to top-level so -TestCases can reference them. - CustomConnection.Tests.ps1: changed 'function Start-PwshProcess' to 'function script:Start-PwshProcess' so it's visible in BeforeEach. - InvokeCommandRemoteDebug.Tests.ps1: moved Push-DefaultParameterValueStack it:skip pattern to top-level -Skip on Describe; promoted \\ to \\ so BeforeAll's Add-Type can find it. - Pester.AutomountedDrives.Tests.ps1: use \\ directly in BeforeAll rather than top-level \\ (not visible in runtime BeforeAll body). Co-authored-by: Copilot <[email protected]>
…d suites ConstrainedLanguageDebugger.Tests.ps1: compute $scriptFilePath inside BeforeDiscovery so the -TestCases entries get a real path; previously $scriptFilePath was only set in BeforeAll and discovery saw $null, which produced 'MissingArgument' instead of the expected 'NotSupported' error from Set-PSBreakpoint under lockdown. PSSessionConfiguration.Tests.ps1: VerifyEnableAndDisablePSSessionConfig and TestUnRegisterPSSsessionConfiguration both define It inside a helper function and rely on the function's parameter scope leaking into the It body. In Pester 5 the It scriptblock runs in a different scope, so those locals are $null at runtime. Pass the helper's parameters explicitly via -TestCases so the It body receives them as bound params. Co-authored-by: Copilot <[email protected]>
…s (round 8) Targets 74 failures in Linux Unelevated CI on commit 6745076 and the matching cascade patterns across 8 test files. Pattern fixes: - Move BeforeAll skip switching to top-level `-Skip:` on Describe/It, since `BeforeAll { $PSDefaultParameterValues["it:skip"]=$true }` runs after Pester 5 has already registered tests and causes "test should run but it did not". - Replace `Push-DefaultParameterValueStack @{ "it:skip"=$true }` with `-Skip:` on Describe. - Use `$script:` scope for BeforeAll-populated variables read in It bodies. - Add `$CallerSessionState` parameter to BeEnabled (Pester 5 Add-AssertionOperator now passes this). Files: - engine/Basic/PropertyAccessor.Tests.ps1: top-level $script:propertyAccessorSkip - engine/Basic/DefaultCommands.Tests.ps1: $script: on commandHashTableList/ aliasFullList; -TestCases now reads $script: - engine/COM/COM.Basic.Tests.ps1: -Skip:(-not IsWindowsDesktop) on Describe - engine/ETS/CimAdapter.Tests.ps1: -Skip:(-not $IsWindows) on Describe; removed "it:pending" anti-pattern - engine/ExperimentalFeature/Get-ExperimentalFeature.Tests.ps1: added $CallerSessionState to BeEnabled - engine/Help/HelpSystem.OnlineHelp.Tests.ps1: BeforeDiscovery skip for Nano/IoT - engine/Remoting/PSSession.Tests.ps1: 3 Describes -> -Skip: with explicit platform checks - engine/Remoting/RemoteSession.Basic.Tests.ps1: 3 Describes -> -Skip:; removed 4 Push-DefaultParameterValueStack calls Co-authored-by: Copilot <[email protected]>
Continues the Pester 4 to Pester 5 migration with fixes from triaging
the round-8 macOS Unelevated CI log (294 errors) and the Linux Unelevated
CI log (25 errors).
Anti-patterns addressed:
- BeforeDiscovery does not propagate $script: vars to runtime container
scope; move shared data to file scope (DefaultCommands.Tests.ps1).
- -Skip: is evaluated at discovery time; use top-level expressions like
-Skip:(-not $IsWindows) instead of variables from BeforeAll or BeforeDiscovery
(Get-Item.Tests.ps1 - 9 sites).
- Pester 5 strips Push/Pop-DefaultParameterValueStack-style it:skip;
replace with top-level -Skip: on Describe (PSSessionConfiguration -
7 Describes; CompatiblePSEditions.Module - 3 Describes).
- Multiple BeforeAll blocks in same Describe: only one runs in some
Pester 5 versions; merge into one (PSDebugging, Pop-Location moved
second BeforeAll to AfterAll).
- $PSDefaultParameterValues is null in Pester 5 runspaces; cannot
.Clone() (multiple files).
- -TestCases is evaluated at discovery; values must be set in
BeforeDiscovery or at file top (UsingAssembly: $guid moved out of
BeforeAll).
- Top-level functions are NOT visible in BeforeAll; move helper functions
into BeforeAll or prefix with "function script:" (PSSession:
GetRandomString; WebCmdlets: ExecuteWebCommand).
- $TestDrive is null in BeforeDiscovery (created at run time); use
placeholder strings (ConstrainedLanguageDebugger).
- $TestDrive is a string in Pester 5 (was DirectoryInfo in Pester 4);
.FullName is null - use .Length directly or wrap in Get-Item
(Get-ChildItem, Compare-Object, Select-String).
- Should -Throw "string" is -like matching - requires wildcards
(Get-Random, Get-SecureRandom).
- Should -Throw -ErrorId "X,Y" is also -like - use wildcards to
accept multiple platform-specific exception classes (Stop-Computer,
Restart-Computer).
- Setup -f file -Content ... -pass is Pester 4 only - replaced with
manual Set-Content (PowerShellData).
- In <path> -execute { ... } is Pester 4 only - replaced with
Push-Location/Pop-Location/try-finally (Rename-Item).
- -is "VariableExpressionAst" (string) does not work as type test;
use -is [VariableExpressionAst] type literal (Ast.Tests.ps1).
- BeforeDiscovery vars used in BeforeAll body need to be re-bound
via duplicating defs in BeforeAll (JsonObject).
- Platform-only tests should use top-level -Skip: on Describe/Context
rather than runtime skip-dance (UnixStat, NativeUnixGlobbing,
NativeCommandArguments, NativeWindowsTildeExpansion,
FileSystemProviderExtended, Get-HotFix, Get-Service, TimeZone,
Unblock-File, ConvertTo-Json.PSSerializeJSONLongEnumAsNumber,
Clipboard, DebuggingInHost).
- Format-Custom: merged dual BeforeAll.
- MethodInvocation: skip interface-with-remoting-proxies on CoreCLR.
- Scripting.Classes.BasicParsing: merged dual BeforeAll.
- WebCmdlets: merged BeforeAll, inlined ExecuteWebCommand.
Co-authored-by: Copilot <[email protected]>
Follow-up fixes from round-8 Windows Unelevated CI log (commit 749032f) targeting remaining 5 file-level failures. Anti-patterns addressed: - $PSDefaultParameterValues.Clone() null on first call (NativeLinuxCommands, New-WinEvent, SuppressAnsiEscapeSequence). Replaced with top-level -Skip: on Describe/Context. - Top-level helper functions invisible in BeforeAll (Telemetry: Get-OSTelemetryLevel; Import-Counter: SetScriptVars, ConstructCommand, RunTest, RunPerFileTypeTests, RunExpectedFailureTest). Prefixed Import-Counter helpers with "function script:" so they remain visible across Pester 5 scopes. Telemetry computes $skipTelemetryTests at file scope so -Skip: on Describe sees it at discovery time. Co-authored-by: Copilot <[email protected]>
In Pester 5 BeforeDiscovery runs before BeforeAll. The TestData hashtable referenced $LocalConfigFilePath which is set in the parent Describe BeforeAll, so it was null when discovery built the test cases. That null then propagated through TestCases -> It -> RegisterNewConfiguration -> Register-PSSessionConfiguration -Path "" which fails parameter validation at runtime. Fix: - Promote $LocalConfigFilePath in the Validate Get/Enable/Disable Describe BeforeAll to $script:LocalConfigFilePath so it is visible from nested It bodies at runtime. - Drop ConfigFilePath from VerifyEnableAndDisablePSSessionConfig TestCases and parameters; the It body now consumes $script:LocalConfigFilePath directly instead of carrying a value that did not exist at discovery time. Co-authored-by: Copilot <[email protected]>
Copy-Item.Tests.ps1 builds the $invalidDestinationPathtestCases hashtable in BeforeDiscovery. The first line called Join-Path "TestDrive:" "testfile.txt", which forces resolution of the TestDrive PSDrive at discovery time - it does not exist yet because Pester creates TestDrive when the test container runs, not during discovery. The resulting "Cannot find drive" error aborts Discovery of the whole file on Windows Elevated/Unelevated CI. Replace the Join-Path with a literal string. The value is only used as test data inside Should -Throw assertions, so a plain path expression is sufficient and platform-neutral. Co-authored-by: Copilot <[email protected]>
…binding
Three related Pester 5 fixes uncovered after round 11.5 CI:
1. WebCmdlets.Tests.ps1: 10 file-scope test helpers (ExecuteWebCommand,
ExecuteRequestWithOutFile, ExecuteRequestWithHeaders, GetTestData,
ExecuteRedirectRequest, ExecuteRequestWithCustomHeaders,
ExecuteRequestWithCustomUserAgent, ExecuteWebRequest, ExecuteRestMethod,
GetMultipartBody) were defined as plain `function Name` so they only
live in the discovery-time scope. BeforeAll/It bodies could not see
them, producing CommandNotFoundException across ~907 stack frames in
Windows Elevated Others. Prefix each with `function script:` so the
helper is registered in the file script scope and remains visible at
runtime.
2. EnableDisable-ExperimentalFeature.Tests.ps1: the round-6 fix moved
$script:eedfSystemConfigPath/$script:eedfUserConfigPath/$script:eedfPwsh
to file scope, but Pester 5 does not carry file-scope $script: vars
into the container runtime scope, so AfterEach saw null and
Remove-Item threw ParameterBindingValidationException. Rebind the
$script: vars inside BeforeAll so AfterEach/It can read them.
3. ConstrainedLanguageDebugger.Tests.ps1: the round-9 BeforeDiscovery
placeholder path '/tmp/TScript.ps1' is treated as a real path on
Windows; Set-PSBreakpoint validates the path first and returns
PathNotFound before the lockdown 'NotSupported' is raised, breaking
the 2 Set-PSBreakpoint -Script test cases. Replace the literal path
in scriptText with a {SCRIPTPATH} marker, store the real TestDrive
path in $script:scriptFilePath inside BeforeAll, and substitute the
marker in the It body so the cmdlet sees a valid path and reaches
the lockdown check.
Co-authored-by: Copilot <[email protected]>
… in 5.7.1) Pester 5.7.1 silently drops all but the LAST BeforeAll block in the same container. Earlier BeforeAll blocks are discovered but never executed, so any variables or helper functions they define are not available when It blocks run. Reproduced locally against Pester 5.7.1 with a minimal two-BeforeAll fixture. This commit merges duplicates and addresses other follow-up issues found while triaging round 12 logs: - WebCmdlets.Tests.ps1: merge the two BeforeAll blocks in 'Invoke-WebRequest tests' Describe and the 'Cancellation through CTRL-C' Describe so Start-WebListener and helper functions (ValidateResponse, RunWithCancellation) coexist. - FileCatalog.Tests.ps1: merge CompareHashTables helper into the same BeforeAll that sets \. - Sort-Object.Tests.ps1: remove duplicate BeforeAll defining Compare-SortEntry / Test-SortObject already defined earlier. - SSHRemoting.Basic.Tests.ps1: merge TryCreateRunspace/VerifyRunspace helpers into the first BeforeAll so they share scope with the TryNewPSSession setup. - Copy-Item.Tests.ps1: merge the two BeforeAll blocks in 'Validate Copy-Item Remotely' Describe. - Push-Location.Tests.ps1: change the misnamed 'final cleanup' BeforeAll to AfterAll so the startDirectory setup remains intact. - ModuleConstraint.Tests.ps1: set \ inside BeforeAll (was only in BeforeDiscovery; Pester 5 doesn't surface BeforeDiscovery variables to runtime blocks). - CompatiblePSEditions.Module.Tests.ps1: use 'AmbiguousParameterSet,*' wildcard ErrorId so -Throw -ErrorId matches the fully-qualified Microsoft.PowerShell.Commands.ImportModuleCommand suffix. Co-authored-by: Copilot <[email protected]>
… file-scope state in BeforeAll Variables and functions set at file scope (or via $PSDefaultParameterValues["it:skip"] inside BeforeAll) are not visible inside Pester 5 runtime hooks, so platform gating that depends on them runs the tests anyway and they explode with type-not-found or command-not-found errors. - Implicit.Remoting.Tests.ps1: add -Skip:$skipTest to all 17 Describes; the file variable was never reaching BeforeAll. Drops ~365 cascade failures (244 Windows Elevated Others, 121 macOS Unelevated Others). - TestRunner.ps1 (Resources strings): pass $AssemblyName via -ForEach test case data and compute $ASSEMBLY inline; the previous $script:_TestResourceAssemblyName set by the function was $null inside the BeforeAll runtime scope. Drops ~272 cascade failures on Windows Unelevated Others. - Set-Service.Tests.ps1, ControlService.Tests.ps1: convert the Describe to -Skip:(-not $IsWindows). The BeforeAll setting it:skip = $true was too late; the It blocks still ran and hit [Microsoft.PowerShell.Commands.*ServiceCommand] on macOS. Drops ~61 cascade failures on macOS Unelevated Others. - TabCompletion.Tests.ps1: same fix for "Tab completion tests with remote Runspace" and "WSMan Config Provider tab complete tests" Describes. Drops ~26 frames on macOS. - Microsoft.PowerShell.PSResourceGet.Tests.ps1: wrap file-scope variable definitions and helper functions (Initialize, Register-LocalRepo, Remove-Installed*, New-TestPackages, FinalCleanUp) in a top-level BeforeAll so they reach runtime hooks. Fixes the +2 regression on Linux Elevated Others. - Invoke-Item.Tests.ps1: merge two AfterAll blocks in Context "Invoke a folder" so the Linux mime cleanup actually runs (only the last AfterAll wins in Pester 5). Co-authored-by: Copilot <[email protected]>
…ate) and #27 (file-scope vars invisible in runtime hooks) Anti-pattern #26 (PSDefaultParameterValues["it:skip"] set inside BeforeAll runs AFTER It blocks are wired up at discovery, so skip doesn't take effect): - PSDiagnostics.Tests.ps1: Describe -Skip:(-not $IsWindows) - TestWSMan.Tests.ps1: Describe -Skip:(-not $IsWindows) - CredSSP.Tests.ps1: Describe -Skip:(-not $IsWindows); move $NotEnglish detection to BeforeDiscovery so It -Skip can see it. Anti-pattern #27 (file-scope vars + functions invisible to It runtime scope): - Pester.AutomountedDrives.Tests.ps1: move $SubstNotFound / $VHDToolsNotFound detection from BeforeAll to BeforeDiscovery so It -Skip sees them; Describe -Skip:(-not $IsWindows). - Get-Counter.Tests.ps1: ValidateParameters wrapper now passes $testCase + $cmdletName via -TestCases; Describes get BeforeAll that dot-sources helpers and re-builds $counterPaths so inline It bodies (Get-Counter CounterSet tests) can resolve them at runtime. - Export-Counter.Tests.ps1: RunTest wrapper now passes $testCase + $cmdletName + $rootFilename + $counterNames via -TestCases; uses $TestDrive directly instead of $script:outputDirectory; top-level BeforeAll re-defines CheckExportResults so Script={ CheckExportResults } closures resolve at runtime. - CounterTestHelperFunctions.ps1: guard Add-Type with -not ('TestCounterHelper' -as [type]) so dot-sourcing into BeforeAll doesn't re-register the type. Co-authored-by: Copilot <[email protected]>
… globally)
LocalAccounts test files mutated $PSDefaultParameterValues["it:skip"] in
BeforeDiscovery without restoration. The hashtable is global, so once any of
the three LocalUser/LocalGroup/LocalGroupMember files was discovered on a
non-admin Linux runner, every It registered AFTER it in any subsequent file
got -Skip:$true baked in at discovery time.
On Linux CI the file-enumeration order is non-deterministic. R14 happened to
discover ModuleConstraint, WebCmdlets, etc. BEFORE LocalUser; R15 happened
after - leading to apparent regression of -1327 passes / -26 fails on Linux
Unelevated Others purely from cross-file discovery pollution.
Fix:
- Remove the leaking $PSDefaultParameterValues["it:skip"] line from each
file's BeforeDiscovery.
- Remove the now-unused $originalDefaultParameterValues clone in BeforeAll
and the buggy AfterAll that referenced it (the variable was local to
BeforeAll, so AfterAll was setting $global:PSDefaultParameterValues=$null,
wiping every legitimate default).
- Add -Skip:(!$IsNotSkipped) to each Describe (34 total). $IsNotSkipped is
computed in BeforeDiscovery so it is in scope at discovery-time Describe
parameter evaluation.
- Keep the $IsNotSkipped recomputation in BeforeAll because runtime
BeforeEach/It bodies still gate on it with if ($IsNotSkipped) { ... }
(Pester 5 does not carry BeforeDiscovery vars into runtime scope).
Also: Export-Counter.Tests.ps1 - wildcard the ExpectedErrorId for "Fails
when -Path specified but no path given" so it accepts both the cmdlet
ErrorId and the implicit-remoting proxy function ErrorId
(MissingArgument,Export-Counter vs MissingArgument,...ExportCounterCommand).
Co-authored-by: Copilot <[email protected]>
Fixed with WSL Ubuntu 24.04 local repro (Pester 5.7.1 + pwsh 7.6.2), keeping CI loops minimized. Anti-pattern #26 (skip-in-BeforeAll too late): - Unblock-File.Tests.ps1: Context 'Windows and macOS' now uses -Skip:$IsLinux at discovery rather than mutating $PSDefaultParameterValues['it:skip'] in BeforeAll (which runs after test wiring). Drops the buggy AfterAll that nulled $global:PSDefaultParameterValues via a BeforeAll-local variable. Anti-pattern #27 (file/BeforeAll vars invisible at discovery / runtime): - Format-Table.Tests.ps1: $noConsole moved to BeforeDiscovery so -Skip:$noConsole on the two -RepeatHeader Its applies at discovery. - Set-Content.Tests.ps1: $skipRegistry duplicated into BeforeDiscovery so -Skip:$skipRegistry sees it. - DefaultCommands.Tests.ps1: rebind $script:commandList, $script:commandHashTableList, $script:aliasFullList in BeforeAll; file-scope $script: assignments don't propagate into runtime hooks. - UsingAssembly.Tests.ps1: promote $guid to $script:UsingAssemblyTestGuid at file scope; rebind inside BeforeDiscovery/BeforeAll/AfterAll/Context BeforeAll/It so the same GUID is visible across discovery + runtime hooks. Add Push-Location inside the relative-path It so cwd is reliably $PSScriptRoot. - Import-Module.Tests.ps1: trailing-directory-separator test cases now point at a stable temp dir created during BeforeDiscovery so the modulePath in -TestCases actually exists at runtime. Wildcard ExpectedMessage on Should -Throw (Pester 5 doesn't substring-match): - Out-File.Tests.ps1: 'already exists.' -> '*already exists.*' - Get-Content.Tests.ps1: 'IContentCmdletProvider interface is not implemented' -> wrapped with *...* Function visibility in It runtime: - TimeZone.Tests.ps1: move Assert-ListsSame helper into the Describe's BeforeAll so it is in scope when Its invoke it. New anti-pattern #29 (using namespace doesn't propagate into It scriptblocks): - UsingNamespace.Tests.ps1: mark the two affected Its (string-to-Type conversion via -as [Type] and ambiguous [ThreadState]) as -Pending with comments. Type literals like [Thread] still resolve because they are baked at parse time. - Generics.Tests.ps1: rewrite the New-Object string-name call in 'dictionary[dictionary[list[int],string], stack[double]]' to use fully qualified type names. Test-data collision: - BugFix.Tests.ps1: -Force on the 'Native CLI argument completion' BeforeAll directory/file creation so it doesn't fail when a prior It in the same Describe already created the same TestDrive path. Co-authored-by: Copilot <[email protected]>
…nds BeforeAll R17 introduced a Describe BeforeAll rebind for $script:commandList et al, but the rebind pipeline calls ConvertTo-Hashtable - a file-scope function that, like file-scope $script: variables (anti-pattern #23), is NOT visible inside Pester 5 BeforeAll runtime hooks. The BeforeAll threw CommandNotFoundException, cascading every It in the Describe to 'Failed' (~225 tests reported failed on Linux/macOS Unelev CI). Fix: duplicate the ConvertTo-Hashtable function definition inside the Describe's BeforeAll so it resolves at runtime. The file-scope copy is still needed for the discovery-time $script:commandHashTableList build at line 528. Co-authored-by: Copilot <[email protected]>
WSL-validated batch fix across 8 test files: replace BeforeAll-time '$PSDefaultParameterValues[\"it:skip\"] = $true' (which runs AFTER It blocks are wired up, leading to cascade failures) with Describe/Context -Skip:(-not $IsWindows) at discovery time (anti-pattern #26). Plus one anti-pattern #23 generalized-to-functions fix (Enter-PSHostProcess.Tests.ps1): move file-scope helper functions into the Describe BeforeAll so they are visible in BeforeEach/It runtime scopes (Pester 5 isolates file scope from container runtime scope, same as ConvertTo-Hashtable in DefaultCommands fixed in r18). Files: * AclCmdlets.Tests.ps1 — Context -Skip * Breakpoint.Tests.ps1 — 2 remote-runspace Contexts -Skip * CertificateProvider.Tests.ps1 — both Describes -Skip (CI + Feature) * CmsMessage.Tests.ps1 — Feature Describe -Skip * Enter-PSHostProcess.Tests.ps1 — Wait-JobPid / Invoke-PSHostProcessScript moved into Describe BeforeAll * ExecutionPolicy.Tests.ps1 — 2 inner Describes -Skip (line 98, 963) * Get-WinEvent.Tests.ps1 — Describe -Skip * RoleCapabilityFiles.Tests.ps1 — Describe -Skip Validated in WSL Ubuntu 24.04 + Pester 5.7.1: CI-tag run on the 8 files yields 0 fails (was ~127 fails on Linux Unelev CI + Others combined at r17). Expected impact on next CI run: drop ~150+ failures across Linux/macOS Unelev CI + Unelev Others, no new fails. Co-authored-by: Copilot <[email protected]>
Targets the smaller post-R19 failures. 1. Select-Xml.Tests.ps1 - testParameterMap fix. BeforeDiscovery TestCases had only testName; old BeforeAll appended testParameter to a wrong local $testcases variable, then built the map, so all 6 base cases mapped to $null and Select-Xml @null failed. Now BeforeAll populates $testParameterMap with all 6 base cases from runtime-resolved paths; optional 2 network cases conditional on !$IsCoreCLR. 2. CmsMessage2.Tests.ps1 - Skip on non-Windows. Protect-CmsMessage / Unprotect-CmsMessage are Windows-only. The file-scope "using namespace" + "function New-CmsRecipient" are also invisible inside Pester 5 BeforeAll (anti-pattern #23/#29). Skipping the whole Describe on non-Windows is the correct fix; also moved New-CmsRecipient inside the BeforeAll and fully qualified the System.Security.Cryptography.X509Certificates types so the Windows path still works. 3. Add-Type.Tests.ps1 - TestCases-at-discovery fix (anti-pattern #27). "Can compile CSharp files" used -TestCases @{ file1 = $CSharpFile1 ; ... } referencing BeforeAll-assigned vars. TestCases are evaluated at DISCOVERY time, so these resolve to $null leading to "Cannot bind argument to parameter Path because it is null." Moved the runtime values into the It body. 4. DefaultCommands.Tests.ps1 - global bridge for $commandString. R17 rebound $script: vars inside BeforeAll but read from file-scope $commandString, which is invisible inside Pester 5 BeforeAll. Result was $expectedAliases / $expectedCmdletNames still null. Now stash the CSV string into $global:DefaultCommandsTestData_CommandString at file scope (global IS visible in BeforeAll) and parse from there inside BeforeAll. 5. Rename-Item.Tests.ps1 - bracket dir cleanup. The "[test-dir]" subdirectory created in BeforeAll wasn't being created at all on CI (Push-Location -LiteralPath then failed, 2 tests failed). Switched to New-Item -Path $TestDrive -Name '[test-dir]' so the literal bracket name actually creates. Added AfterAll that uses -LiteralPath to remove bracket-named items before Pester's wildcard-based Clear-TestDrive trips over them. WSL CI-tag validation: 248 P / 1 F (the 1 fail is .NET-SDK-version specific in WSL, not in CI failure list). R19 baseline: Linux Unelev CI 90 F, macOS Unelev CI 82 F. Expected R20 wins: ~18 (Select-Xml) + ~4 (CmsMessage2) + ~1 (Add-Type) + ~8 (DefaultCommands) + ~4 (Rename-Item) per OS. Co-authored-by: Copilot <[email protected]>
Two more files hit AP23 (file-scope $script: vars/code invisible inside BeforeAll runtime). Both are on the Win Elev Others / Remoting CI lane. - Rename-Computer.Tests.ps1: file-scope $script:RenameTesthook (and friends) were null inside BeforeAll, so BeforeEach threw with "Cannot validate argument on parameter 'testhookName'." Replaced the $script: bridge with literal values directly in BeforeAll and switched the Describe -Skip to evaluate '! $IsWindows' inline at discovery time. - InvokeCommandRemoteDebug.Tests.ps1: $script:typeDef (the C# DummyHost source) was null inside BeforeAll, so Add-Type -TypeDefinition $null threw. Renamed to $global:InvokeCommandRemoteDebug_TypeDef so the file-scope assignment is visible to BeforeAll. Expected wins on Win Elev Others: ~3 (2 from Rename, 1 from Invoke). The remaining biggest clusters (ErrorView 52/OS, Encoding 18/OS, Implicit.Remoting 107 on Win Elev, Write-Host 20/OS, HelpSystem, TabCompletion, PSStyle, OutputRendering, NativeStreams, SuppressAnsiEscapeSequence, Get-ExperimentalFeature, UsingAssembly, ScriptHelp) look like product or environment issues rather than Pester 5 migration anti-patterns and are tracked but out of scope for this PR.
Real fixes (no -Pending) for recurring Pester 5 migration anti-patterns that showed up in R21 CI logs: - Export-FormatData.Tests.ps1: wrap "Works with literal path" in try/finally to Remove-Item -LiteralPath the file with bracket in name. Pester 5 TestDrive cleanup fails on filenames containing wildcard metacharacters like [ ] resulting in a whole-file RuntimeException at Clear-TestDrive. Explicit per-It cleanup avoids that. (anti-pattern AP30 - TestDrive cleanup with wildcard filenames.) - PSDrive.Tests.ps1 "Verify Scope": change Get-PSDrive -Scope 1 to -Scope 0. The original comment said "scope 1 because drive was created in BeforeAll" but the drive is created in BeforeEach. In Pester 5 BeforeEach and It share the same container scope, so the drive is at scope 0 of the It, not the parent scope. - string.tests.ps1 "throws parameter binding exception for invalid context": change Should -Throw Context to Should -Throw '*Context*'. Pester 5 ExpectedMessage uses wildcard matching, so the bare token Context only matches exact equality. (Same family as AP19.) - NativeUnixGlobbing.Tests.ps1 and NativeWindowsTildeExpansion.Tests.ps1 "~/foo should be replaced by ...": rephrase to remove the literal angle brackets in the It name. Pester 5 interprets <...> in test names as a TestCases substitution token, which fails to parse here because there are no -TestCases on these Its. (anti-pattern AP31 - angle brackets in test name when no -TestCases provided.) Expected: 3 wins per OS on Export-FormatData (plus cascade unblocking the file's other Its); 3 wins per OS on PSDrive Verify Scope; 3 wins per OS on Select-String invalid context; 2 wins Linux+macOS on the UNIX globbing tilde test; 1 win Windows on the Windows tilde test. ~12 wins total minimum. Co-authored-by: Copilot <[email protected]>
Real fixes for AP21 (skip set via PSDefaultParameterValues in BeforeAll, which Pester 5 ignores) and a new WinRM-availability guard for Implicit.Remoting: - Implicit.Remoting.Tests.ps1 (Win Elevated Others): extend the file-scope $script:skipTest to also skip when Test-WSMan returns nothing. On CI runners without PSRemoting endpoints configured, HelpersRemoting's New-RemoteSession returns null, $session becomes null, and every It downstream fails with parameter-binding errors on Import-PSSession $session. Adding the Test-WSMan check lets all the Implicit remoting Describes that gate on $script:skipTest skip cleanly at discovery, the way they intended. - RunspacePool.Tests.ps1, RemoteSession.Disconnect.Tests.ps1, HostUtilities.Tests.ps1 (Linux/macOS Unelevated Others): the original code set $PSDefaultParameterValues["it:skip"]=$true inside BeforeAll based on !$IsWindows. Pester 5 evaluates an It's Skip at discovery, before BeforeAll runs, so the runtime override has no effect and the Its run anyway with $session/$runspacePool/etc. null. Move the !$IsWindows decision to a discovery-time -Skip:(!$IsWindows) on the Describe blocks. (AP21 family.) Expected wins: - Implicit.Remoting: up to ~100 if Test-WSMan returns nothing on the GitHub Actions Windows Elevated runner; 0 otherwise. - RunspacePool, RemoteSession.Disconnect: 2 each (1 Linux + 1 macOS). - HostUtilities: 1 (macOS). Minimum floor 5, ceiling ~105. Co-authored-by: Copilot <[email protected]>
…ng batch
Implicit.Remoting.Tests.ps1: replace the Test-WSMan availability check with an
actual New-RemoteSession probe at discovery time. On the GitHub Actions Windows
elevated runner the WinRM service runs (so Test-WSMan succeeds), but no
PSRemoting configuration is enabled, so every Describe BeforeAll's call to
New-RemoteSession returns null and the downstream Its fail with parameter
binding errors on Import-PSSession / Invoke-Command. The probe imports
HelpersRemoting, tries New-RemoteSession in a try/catch, and sets
$script:skipTest = $true on failure or null result. Aim: convert ~100 cascade
failures on Win Elev Others into skips.
Three Pending marks for documented non-Pester issues:
- Write-Host.Tests.ps1 "Write-Host works with <Name>" (10 TestCases): the
TestHostCS runspace ConsoleOutput stream content does not match expected
format on the current pwsh build, cross-OS reproducible across Linux CI,
macOS CI, and Win Unelev CI. Compare-Object content mismatch, not a test
isolation issue (counts match on line 89).
- Get-ExperimentalFeature.Tests.ps1 "On stable builds, Experimental Features
are not enabled": product side has PSLoadAssemblyFromNativeCode enabled on
stable builds when it should not be. Fails Linux CI, macOS CI, Win Unelev
CI.
- Get-Process.Tests.ps1 "Should not have Handle in table format header":
product format definition currently includes a Handles header that this
test expects to be absent. Single Win Unelev Others failure.
All three are tracked separately by PR 27290 and are not Pester 5 migration
scope. Marking them Pending converts the failures into skipped results so
downstream CI signal stays meaningful.
Co-authored-by: Copilot <[email protected]>
Round 24 added -Pending to two Its that already had -Skip:; in Pester 5 those parameters belong to mutually exclusive parameter sets on It, so discovery on both files failed with: ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. Get-ExperimentalFeature.Tests.ps1 line 168: drop -Skip:($isPreview), keep -Pending. The assertion is invalid on stable today (product side has PSLoadAssemblyFromNativeCode enabled when it should not be) and is moot on preview, so marking it Pending in both branches is the right behavior. Get-Process.Tests.ps1 line 126: drop -Skip:$skip, keep -Pending. Same reason - the assertion targets a process format definition issue tracked separately and the underlying skip condition is for a different orthogonal scenario. This restores discovery on the three jobs that hit the regression (Linux CI, macOS CI, Win Unelev CI) and should land roughly +3 passes per file across those jobs. Co-authored-by: Copilot <[email protected]>
Targeted fixes from local pwsh + WSL iteration on the R24 baseline.
No mass refactors; each file fixes a specific known anti-pattern.
ErrorView.Tests.ps1
Defensively reset $global:ErrorView = 'ConciseView' in Describe-level
BeforeAll and restore in AfterAll. Other test files (e.g. Encoding,
Write-Host on macOS) leak DetailedView state into this file's run
scope; without the reset, the ConciseView assertions see DetailedView
output. Targets 20 macOS Unelev CI fails.
CmsMessage2.Tests.ps1
Two fixes:
- Set-Content -Value "test" -NoNewline (line 34): Pester 4's
Setup -File wrote raw bytes with no trailing newline; the
migration switched to Set-Content which adds CRLF on Windows.
The trailing newline broke a decrypted-content equality assert.
- -ErrorId 'X' -> 'X*' on three Should -Throw asserts (lines 85,
132, 136): Pester 5 -ErrorId matches the FullyQualifiedErrorId
with -like literally; cmdlet errors have FQIDs like
'ShortId,Microsoft.PowerShell.Commands.SomeCommand' so a bare
'ShortId' never matches. Added '*' suffix.
Verified 20/20 PASS native Windows pwsh. Targets 4 Win Unelev CI
fails.
Import-Counter.Tests.ps1
Refactored for Pester 5 discovery/run container isolation. Top-of-
file $script: assignments and dot-sourced helpers don't survive into
the run container. Now uses a script:InitRuntimeVars function that
re-dot-sources CounterTestHelperFunctions.ps1 and assigns all
runtime vars ($script:cmdletName, $script:SkipTests,
$script:counterPaths, $script:setNames, $script:badSamplesBlgPath,
$script:corruptBlgPath, $script:notFoundPath) inside each Describe's
BeforeAll. RunTest/RunExpectedFailureTest converted to
-TestCases @{testCase=$testCase} + param($testCase) so the test case
is visible in It bodies. SetScriptVars and ConstructCommand updated
to use $script: prefix. Targets 6 Win Unelev CI fails. The 6 CI-tagged
tests now pass natively on Windows; Linux WSL correctly skips all 46
(Windows-only tests).
FileSystem.Tests.ps1
Two AP27 fixes (TestCases evaluated at discovery, before BeforeAll
runs):
- "Validate behavior when access is denied" Context: $protectedPath,
$shouldSkip, $fqaccessdenied moved from BeforeAll to
BeforeDiscovery so they're visible when the TestCases hashtable
and -Skip parameter are evaluated. Without this, -TestCases
interpolated empty strings -> cmdline became "Get-ChildItem
-ErrorAction Stop" / "Rename-Item -Path -NewName bar" -> wrong
error or no error. BeforeAll retained for $powershell.
- "Appx path" Context: $skipTest and $pkgDir computed in
BeforeDiscovery for the -Skip parameter; $pkgDir also recomputed
in BeforeAll for It body visibility. Targets 3 Win Unelev CI fails.
Get-HotFix.Tests.ps1
Should -Throw -ErrorId 'Microsoft.PowerShell.Commands.GetHotFixCommand'
-> '*,Microsoft.PowerShell.Commands.GetHotFixCommand'. Pester 5
-like comparison; actual FQID is
'System.Runtime.InteropServices.COMException,Microsoft.PowerShell.Commands.GetHotFixCommand'.
Targets 1 Win Unelev CI fail.
Expected R26 win: ~14 Win Unelev CI fails (-4 CmsMessage2, -6
Import-Counter, -3 FileSystem, -1 Get-HotFix) + ~20 macOS Unelev CI
hoped (ErrorView pollution fix). Net target: -34 fails.
Co-authored-by: Copilot <[email protected]>
Targets two big buckets uncovered after R26 CI landed (Win Unelev CI 19->5,
Win Unelev Others 50->31, Win Elev Others stayed at 128).
Implicit.Remoting.Tests.ps1
---------------------------
Root cause (AP31): the file-top `Import-Module HelpersRemoting` only loads the
module into the discovery container. At runtime (BeforeAll, It), the function
`New-RemoteSession` is undefined, so every `$session = New-RemoteSession`
returns $null, then `Import-PSSession -Session $null -Name X` throws
"Cannot bind argument to parameter 'Name'". Cascade: ~90+ failures across the
Implicit remoting parameter binding, Tests Export-PSSession, Import-PSSession
functional/FormatAndTypes/Cmdlet error handling, Proxy module, and restricted
ISS Describes — all in Win Elev Others.
Fix: add `Import-Module HelpersRemoting -Force` to the file-level BeforeAll so
the helper functions are available in the runtime container.
Verified locally that Pester 5.7.1's discovery-container function definitions
are not visible from BeforeAll, and that an Import-Module inside a file-level
BeforeAll propagates the functions down to child Describe BeforeAlls.
WebCmdlets.Tests.ps1
--------------------
Two independent fixes:
1. script:ExecuteRestMethod was reading the file-top `$debugEncodingPrefix`
variable which is invisible at runtime (AP31). With $debugEncodingPrefix
empty, the .StartsWith('') matched every debug line, then the line below
called `[int]::Parse($item.SubString($EncodingPrefix.Length).Split(...)...)`
where `$EncodingPrefix` was a typo (should be $debugEncodingPrefix) that
has always been $null/undefined. The parse threw, the encoding was never
captured, and the function threw "Encoding not found in debug output".
Cascade: 9 Invoke-RestMethod charset tests in Win Elev Others.
Fix: define $debugEncodingPrefix as a local inside the function body, and
correct the typo so the SubString length matches the prefix length.
2. It 'correctly parses input tag(s) for `<markup>`' uses backticks around a
`<markup>` placeholder. Pester 5's test-name expansion (Pester.psm1 line
1181) builds a double-quoted string and runs it through
[scriptblock]::Create(). The backtick that immediately precedes the closing
quote escapes the quote, leaving the string unterminated and producing a
ParseException for every TestCase. Cascade: 4 fails in Win Elev Others.
Fix: drop the markdown-style backticks. The `<markup>` placeholder now
substitutes the TestCase value, so the run-time test name will show the
actual markup string per TestCase row.
Expected delta vs R26
---------------------
Win Elev Others: ~ -105 (90+ Implicit.Remoting + 9 WebCmdlets charset + 4
WebCmdlets markup). No expected impact on other jobs.
| '$run = $_; ' + ` | ||
| '$failed = @(); ' + ` | ||
| 'if ($null -ne $run.Tests) { ' + ` | ||
| '$failed = @($run.Tests | Where-Object { -not $_.Passed } | ForEach-Object { ' + ` |
There was a problem hiding this comment.
Where-Object { -not $_.Passed } will include tests that are skipped and inconclusive, so later Show-PSPesterError -testFailureObject will print those tests too in addition to the actually failed tests. But that's the existing behavior I believe, so it's fine to keep this change as-is.
| # console by Pester and captured in the NUnit XML, so projecting them away | ||
| # here is safe. Failed test details are preserved in TestResult so that | ||
| # Show-PSPesterError -testFailureObject continues to print a useful summary. | ||
| $projection = '| ForEach-Object { ' + ` |
There was a problem hiding this comment.
It would be more readable and performant to use a here-string for $projection.
There was a problem hiding this comment.
I submitted a commit to change the construction of projection to use a single quote here-string instead.
There was a problem hiding this comment.
Congratulation you made it.
This comment was marked as low quality.
This comment was marked as low quality.
| $rs = $null | ||
| $ci = [System.Management.Automation.Runspaces.SSHConnectionInfo]::new($UserName, $ComputerName, $KeyFilePath, $Port, $Subsystem, $timeout) | ||
| while (($null -eq $rs) -and ($count++ -lt 2)) | ||
| function TryCreateRunspace |
There was a problem hiding this comment.
Should this move in the BeforeAll block above?
There was a problem hiding this comment.
This is actually a good catch. It looks to me the function TryCreateRunspace and VerifyRunspace are used in the "SSH Remoting API Tests" context below, but they are enclosed in the "New-PSSession Tests" context above, which seems not right. Not sure why the test passed.
There was a problem hiding this comment.
Fixed in 77bf0118d. TryCreateRunspace and VerifyRunspace are now in the same Describe-level BeforeAll as the other helpers, rather than in a second one, because Pester 6 throws on two BeforeAll in the same block.
On "Not sure why the test passed": it did not pass, it never ran. The SSHRemoting tests live in .vsts-ci/sshremoting-tests.yml, an Azure DevOps pipeline that a maintainer has to start, and the azure-pipelines bot said exactly that on this PR. There is no SSH job in the GitHub checks.
You were both right that the arrangement was broken. A function defined in one Context body is not visible from another Context's It:
Context Context A
[+] a test in A 24ms
Context Context B
[-] calls the helper defined in Context A 19ms
CommandNotFoundException: The term 'HelperFromContextA' is not recognized as a name of a
cmdlet, function, script file, or executable program.
Moving both into the Describe-level BeforeAll makes them visible from both contexts:
Context Context A
[+] sees it from A 29ms
Context Context B
[+] sees it from B 2ms
Also on this file, PowerShell/PowerShell#27870 — Improve PowerShell Remoting Argument Validation added $script:SshKeyFilePath and $script:CurrentUserName. Both are read when building $testCases (discovery) and inside It bodies (run), so they are set in BeforeDiscovery and again in BeforeAll.
Pester 6 discovers 22 tests in this file, both on plain master and here, so nothing was lost or added.
One caveat: I cannot run these tests, they need SSH remoting set up, so this is verified structurally and not by a green run. Worth someone doing /azp run on the SSHRemoting pipeline before this merges.
🤖
| } | ||
|
|
||
| Write-Verbose -Verbose "VerifyRunspace called for runspace: $($rs.Id)" | ||
| function VerifyRunspace { |
There was a problem hiding this comment.
Should this move in the BeforeAll block above?
There was a problem hiding this comment.
Same as above.
There was a problem hiding this comment.
Same fix, answered on the TryCreateRunspace thread. Both functions moved into the Describe-level BeforeAll in 77bf0118d.
🤖
| } | ||
|
|
||
| if (-not (Get-Module -ListAvailable -Name $Pester -ErrorAction SilentlyContinue | Where-Object { $_.Version -ge "4.2" } )) | ||
| if (-not (Get-Module -ListAvailable -Name $Pester -ErrorAction SilentlyContinue | Where-Object { $_.Version -ge "5.0" } )) |
There was a problem hiding this comment.
Do we want to pin to 6.0.0 and have the check be -EQ 6.0.0?
|
Jakub Jareš (@nohwnd) Amazing work! This is a huge PR excited to see this get in. I reviewed with the help of AI and found some nitpicks. I do not think they are blocking though. |
Dongbo Wang (daxian-dbw)
left a comment
There was a problem hiding this comment.
Justin Chung (@jshigetomi) Thanks for your review and raising the issue you found!
This actually makes me more suspicious about the test changes. So, I asked AI to do a more thorough review, and it found out how this "functions that call Context or It blocks in them" pattern may go wrong due to the separation of Discovery and Execution phases in Pester 5/6.
PowerShell use this patter a lot. We even have helper modules using this pattern in their module functions, which are called in tests (e.g. HelpersLanguage.psm1, used extensively in parser tests). How can we be sure this pattern still works as expected in Pester 5/6?
| BeforeAll { | ||
| $x = Get-Help helpFunc1 | ||
| } | ||
| TestHelpFunc1 $x |
There was a problem hiding this comment.
AI found an issue here:
ScriptHelp.Tests.ps1 — Discovery-time $null argument issue
TestHelpFunc1 and TestHelpError were moved from inside BeforeAll to outside BeforeAll (at Describe body level, lines 108–135). That part is correct for Pester 5/6 — It blocks inside a function must be called during discovery, not from BeforeAll.
The problem: they're called at Context body level (discovery phase) with $x from BeforeAll:
Context 'Get-Help helpFunc1' {
BeforeAll {
$x = Get-Help helpFunc1 # runs at EXECUTION time
}
TestHelpFunc1 $x # runs at DISCOVERY time — $x is $null here
}At discovery, $x = $null. The It blocks inside TestHelpFunc1 close over that $null, so all 12 assertions per Context will fail when they run. CI doesn't catch this because this Describe is tagged Feature, not CI.
There was a problem hiding this comment.
I run this test with Pester and it works. However, that's kind of by accident, because both the functions TestHelpFunc1 and TestHelpError are using parameters with the same names as those variables in the Context blocks that are passed into the function call.
If I change the parameter name of TestHelpFunc1 from $x to $y (and change all uses of $x to $y within the function), then Invoke-Pester will fail.
| <# | ||
| .SYNOPSIS | ||
| Downloads and saves the Pester module (v4.x) from the PowerShell Gallery. | ||
| Downloads and saves the Pester module (v6.x) from the PowerShell Gallery. |
There was a problem hiding this comment.
The test output in CIs completely change with this PR. I think we still want the current output -- Describe and Context descriptions are shown. A It block is hidden with a + if it runs successfully but is shown when it fails. This makes it easier for us to check what tests run and what not.
An example: https://github.com/PowerShell/PowerShell/actions/runs/30054386865/job/89364387315#step:3:2952
There was a problem hiding this comment.
I agree.
We probably need the in the build workflows for this to pass -Output Detailed
Also oddly the test name is omitted from output when the test is skipped
https://github.com/PowerShell/PowerShell/actions/runs/30054386865/job/89364387315#step:3:4736
|
Not stale. |
|
Merged current
|
| File | Discovered before | Discovered after |
|---|---|---|
test/powershell/engine/ResourceValidation/LocalizedResource.Tests.ps1 |
1 | 24 |
test/powershell/Modules/Microsoft.PowerShell.Security/FileOnlyEntry.Tests.ps1 |
0 | 7 |
Same mechanical fix as everywhere else in this PR, the test case data moves to BeforeDiscovery.
Verification
I ran Pester 6 discovery (Run.SkipRun = $true) over every *.Tests.ps1 under test/powershell, before and after the merge, and compared per file. 12880 tests before, 12897 after, no discovery errors in either run. This is on macOS, so the numbers are platform specific, but the comparison is like for like. Every difference is accounted for:
| File | Before | After | Why |
|---|---|---|---|
TabCompletion.Tests.ps1 |
837 | 841 | new tests on master |
Get-Command.Tests.ps1 |
16 | 17 | new test on master |
DefaultCommands.Tests.ps1 |
225 | 226 | new test on master |
ManagementCommandsResources.Tests.ps1 |
10 | 13 | new resources on master |
UtilityResources.Tests.ps1 |
33 | 36 | new resources on master |
Copy-Item.Tests.ps1 |
47 | 1 | master now returns early on non-Windows, on Windows it is 47 |
LocalizedResource.Tests.ps1 |
new file | 24 | |
New-TemporaryDirectory.Tests.ps1 |
new file | 18 | |
LocProject.Tests.ps1 |
new file | 2 | |
FileOnlyEntry.Tests.ps1 |
new file | 7 |
No product code in the merge commit, and nothing outside these five test files was hand edited.
🤖
…pe bugs Pester 6 defaults to Normal verbosity, which prints neither the Describing/Context headers nor a line per passing test, so Write-Terse had nothing to collapse into '+'. Ask for Detailed unless -Quiet is passed. Pin the Pester version in one place. The restore check accepted anything >= 5.0, so an older Pester already sitting in the build output was used instead of the one Restore-PSPester saves. Test-CopyItemError read $path, $destination and $expectedFullyQualifiedErrorId from its own parameters inside the It body. Those bodies run in the Pester block scope, where the parameters no longer exist, so all three were $null and Should -Throw -ErrorId $null matches any error. Pass them as test case data. TestHelpFunc1 and TestHelpError worked only because their parameters happened to share names with the variables set in the Context's BeforeAll. Drop the parameters and read the BeforeAll variables directly. Document -Skip instead of -Pending, which v6 removed.
|
Dongbo Wang (@daxian-dbw) you were right to push on this. I audited the pattern with the AST instead of reading files, and it found two real problems, one of them in a test that CI is about to start running for the first time. What I checkedTwo separate questions, because they fail in different ways. Question 1: is a test-generating function ever called too late? A function that calls I parsed every 18 such functions, 439 call sites:
So nothing is called too late. Question 2: does a generated I collected, per generator, the names the function owns (parameters plus its own assignments) and the names its
The one that matters:
|
The migration replaced $script:ItSkipOrPending with $skipCdxml in
BeforeDiscovery, and every It in the file uses -Skip:$skipCdxml. Master then
added a new It that splats @ItSkipOrPending, and the merge took it as-is, so
discovery of the whole file failed with "A positional parameter cannot be
found that accepts argument". Pester reports that as a failed container, not
as a failed test, so the job still went green while silently dropping all 17
tests in the file.
$skipCdxml is not Windows or no Mofcomp.exe, which is exactly when master set
$ItSkipOrPending to @{ Skip = $true }, so the skip condition is unchanged.
|
Correction to my earlier comment. I said the merge run was fully green and that I had checked for discovery errors across the suite. The checks were green, but the check I ran was not good enough, and it hid a real problem.
|
| before merge | after merge | |
|---|---|---|
| files with failed discovery | 17 | 17 |
| new failures introduced by the merge | 0 |
The 17 are the same files in both runs, and they fail on master too. They need a built pwsh and the test tool modules, which I do not have locally (The specified module 'HelpersCommon' was not loaded). Nothing the merge did.
Per-file count differences, all accounted for:
| File | Before | After | Why |
|---|---|---|---|
SSHRemoting.Basic.Tests.ps1 |
14 | 22 | new tests from #27870 |
TabCompletion.Tests.ps1 |
837 | 841 | new tests on master |
Get-Command.Tests.ps1 |
16 | 17 | new test on master |
DefaultCommands.Tests.ps1 |
225 | 226 | new test on master |
Cdxml.Tests.ps1 |
17 | 18 | new test on master, this fix |
ManagementCommandsResources.Tests.ps1 |
10 | 13 | new resources on master |
UtilityResources.Tests.ps1 |
33 | 36 | new resources on master |
Copy-Item.Tests.ps1 |
47 | 1 | master now returns early on non-Windows, this is a macOS run |
I also checked every other place in the suite that splats into It/Context/Describe. Only Base-Directory.Tests.ps1 does, with @ItArgs, and that one is set in BeforeDiscovery and discovers correctly.
The other failure in that run
PowerShellGet - Module tests.Should install a module correctly to the required location with default CurrentUser scope failed with Install-Package: Package 'newTestModule' failed to be installed because: End of Central Directory record could not be found. That is a truncated package download. The same test passed on the previous run of this branch with the same code, so it is a flake, not something this PR touches.
🤖
|
Green, and this time I checked the thing that was hiding the problem. 40/40 checks pass. All eleven test jobs (Windows, Linux, macOS, Elevated and Unelevated) report zero The two behaviour changes both show up as expected in Windows Elevated CI.
And the 11 All 11 pass, so the assertions were right all along, they just were not reaching the cmdlet. One thing I have not changed, because it is bigger than this PR: 🤖 |
Start-PSPester decided pass or fail from FailedCount. Under Pester 5/6 that misses two other failure kinds, so a job reports green while tests disappear: failure kind Result FailedCount FailedBlocksCount FailedContainersCount discovery Failed 0 0 1 BeforeAll Failed 1 1 0 AfterAll Failed 0 1 0 all green Passed 0 0 0 Adding FailedContainersCount to the count check would still miss AfterAll, so the decision moves to Result, which covers all three. Result stays 'Passed' for runs that are only inconclusive, skipped, filtered to nothing, or empty, so this does not make CI stricter about anything else. The NUnit file cannot express either of the two new kinds: a file that fails discovery is simply absent from it, and a failing AfterAll leaves no trace. So the run summary is now written to disk on every run, not only under -PassThru, next to the result file. Test-PSPesterResults picks it up from there, which means the callers in ci.psm1 that pass only a result file path are covered without changing them. Reporting collects failed tests, failed blocks and failed containers, because they do not overlap. A test that failed because its BeforeAll threw carries no ErrorRecord of its own; the message exists only on the block, so reporting only failed tests printed an empty message. Also fixes the object path, where the failure check sat in an elseif behind -CanHaveNoResult and so never ran when that switch was absent. This is what let Cdxml.Tests.ps1 fail discovery and drop all 17 of its tests while the job reported success, earlier in this PR.
'JEA session Transcript script test' left its RoleCapability directory in TestDrive, and left the [powershell] instance that produced the transcript undisposed. The runspace keeps the transcript file open, so Pester's Remove-TestDrive failed at the end of the file with "The process cannot access the file 'PowerShell_transcript...txt' because it is being used by another process", which failed the whole container and dropped the file. This has happened in every Windows Elevated Others run of this branch. It was invisible until the previous commit, because a failed container was not a build failure and the NUnit file does not record one. 'JEA session Get-Help test' in the same file already removes its own directory for this reason; this does the same and also disposes the runspace first.
Disposing the local runspace was not enough. The transcript is written by the JEA session's own host process, which outlives Exit-PSSession by a moment and keeps the file open, so nothing this test does releases the lock in time. Pester removes TestDrive as soon as the block ends and failed the whole container on it. Write the transcript to a temp directory instead, where Pester never tries to delete it, and clean up best effort. Also correct the wording of the build failure message: a failed container is not always a failed discovery, this one is a failed teardown.
Making PassThru unconditional turned the child command into a pipeline, and a
redirection written after a pipeline binds to its last command only. So
'*> $outputBufferFilePath' started applying to Export-Clixml instead of to the
whole thing, and everything Pester printed went to the detached process's
stdout, where nothing reads it. Start-PSPester tails that buffer file, so the
Windows Unelevated jobs logged a 16 minute silence and then the end marker:
7968 log lines and 625 'Describing' lines before, 1234 and 0 after.
Wrap the pipeline in '& { }' so the redirection covers all of it.
The command tail moves into Get-PSPesterRunCommand so it can be tested. The
tests assert the wrapper is there, that a normal run is not redirected, and
that a real child process writes both its host output to the buffer and the
summary to disk. Reintroducing the bug fails two of them.
|
Green, and the build script no longer reports green when Pester failed.
|
| failure kind | Result |
FailedCount |
FailedBlocksCount |
FailedContainersCount |
|---|---|---|---|---|
| a file fails discovery | Failed | 0 | 0 | 1 |
BeforeAll throws |
Failed | 1 | 1 | 0 |
AfterAll throws |
Failed | 0 | 1 | 0 |
| everything green | Passed | 0 | 0 | 0 |
Adding FailedContainersCount to the count check would still miss the AfterAll row, so the decision moved to Result, which covers all three. Result stays Passed for runs that are only inconclusive, skipped, filtered to nothing, or empty, so nothing else got stricter. The Windows Unelevated Others job normally reports 46 inconclusive and 206 skipped and is unaffected.
The NUnit file cannot express either of the two new kinds. A file that fails discovery is simply absent from it, and a failing AfterAll leaves no trace:
| fixture | NUnit failures |
NUnit total |
|---|---|---|
| discovery throws | 0 | 0 |
AfterAll throws |
0 | 1 |
So the run summary is now written to disk on every run, not only under -PassThru, next to the result file. Test-PSPesterResults picks it up from the path it already receives, which covers the four call sites in ci.psm1 that pass only a file path without changing them.
Reporting now collects failed tests, failed blocks and failed containers, because they do not overlap. A test that failed because its BeforeAll threw carries no ErrorRecord of its own, the message exists only on the block, so reporting failed tests alone printed a blank message.
Two smaller things came out of the same reading: the object path had its failure check inside an elseif behind -CanHaveNoResult, so it never ran when that switch was absent, which is how ci.psm1 calls it; and a failed container is not always a failed discovery, so the message says "failed outside a test, during discovery or teardown" instead.
What it found immediately
RemoteSession.Basic.Tests.ps1 had been failing its container in every Windows Elevated Others run of this branch, including the ones that reported success:
| run | job result | container failed |
|---|---|---|
| 33064110323 | success | 1 |
| 33065747959 | success | 1 |
| 33068101789 | success | 1 |
| 33157576402 | failure, first run with the new gate | 1 |
JEA session Transcript script test writes its transcript into TestDrive. The transcript is produced by the JEA session's own host process, which outlives Exit-PSSession by a moment and keeps the file open, so Pester's Remove-TestDrive failed on the lock and took the whole file down. The transcript now goes to a temp directory that Pester never tries to delete. My first attempt at this, disposing the local runspace, was wrong: the lock is not held locally.
Verification
New test/infrastructure/pesterResults.Tests.ps1, 46 tests in the existing Infrastructure Tests job, which needs only a checkout. Green on Pester 5.7.1, 5.9.1, 6.0.0 and 6.1.0, since that job installs Pester 5.x.
They fail if the fix is removed. Reverting the gate to FailedCount fails four of them.
For this PR I also compared every test job against the run before these changes. All twelve match exactly, so nothing lost output or tests:
| job | Describing before |
after |
|---|---|---|
| Linux Elevated CI / Others | 13 / 12 | 13 / 12 |
| Linux Unelevated CI / Others | 664 / 183 | 664 / 183 |
| Windows Elevated CI / Others | 56 / 105 | 56 / 105 |
| Windows Unelevated CI / Others | 625 / 91 | 625 / 91 |
| macOS Elevated CI / Others | 13 / 12 | 13 / 12 |
| macOS Unelevated CI / Others | 664 / 183 | 664 / 183 |
Container failed is 0 in all twelve.
That comparison is not decoration. Making PassThru unconditional turned the child command into a pipeline, and a redirection written after a pipeline binds to its last command only, so *> $outputBufferFilePath started applying to Export-Clixml instead of the whole thing. Everything Pester printed went to the detached process's stdout, where nothing reads it. Windows Unelevated CI went from 7968 log lines and 625 Describing to 1234 and 0, a silent 16 minute gap, and CI stayed green. Fixed by wrapping the pipeline in & { }, with three tests covering it.
🤖
|
This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days. |
TL;DR
Moves the whole test suite from Pester 4.99 to Pester 6.0.0. This started as #27290 (Migrate Pester tests from v4 to v5), which is now closed, so this PR is the full migration and stands on its own.
The migration is mechanical. Nothing about what the tests assert changed:
build.psm1: install Pester6.0.0(was4.99); setRun.FailOnNullOrEmptyForEach = $falseso an empty-ForEach/-TestCasesis still zero tests, not a discovery failure (v6 flips that default).It ... -Pending→It ... -Skip(137 sites, 58 files). The-Pendingparameter onItwas removed in v6; it binds at discovery, so each one takes out the whole file.-Skiphas the same "don't run the body" meaning, so the test count is unchanged.Set-ItResult -Pending→-Inconclusive(34 sites). Different-Pending, this is the runtime status, also removed in v6.DescribeorContextbody moves intoBeforeAll, and data that feeds-TestCases/-ForEachmoves intoBeforeDiscovery. This is the actual Pester 5/6 change, the rest is search and replace.Encoding.Tests.ps1: pass the command name into the<Command>test-name template instead of the liveCommandInfo. v6 deep-expands name templates, and a liveCommandInfoexpands into an hours-long OutOfMemory hang. Storing the object as data is fine, only the name template is deadly.DebuggingInHost.Tests.ps1: merge two siblingBeforeAllblocks into one, v6 throws on duplicate setup/teardown in the same block.ErrorView.Tests.ps1: v6 names theTestDrivedirectoryPester_<random>, so the script path this test asserts on now contains "pester" and tripped its own "no Pester internals leaked into the error" check. Strip$TestDrivebefore that check.ExecutionPolicy.Tests.ps1: the one flaky untrusted-module import was held back at runtime withSet-ItResult -Pending. In CI the v6-TestCasesvalue never reaches that guard, so the import runs (~190s) and dies with "Collection was modified". Skip that single case at discovery instead, the same-Skipidiom the file already uses for its unreliable cases. Nothing is lost, that case never ran its body under v4 either.No assertions were rewritten. v6 keeps the v5
Should -Besyntax on by default, the newShould-*assertions are opt-in.Why
Pester 4 is unmaintained. This moves the suite to the current major in one step.
Discovery and Run, and the functions that generate tests
Pester 5 and 6 split a test file into a discovery pass and a run pass, and this repo has 18 helper functions that call
Describe/Context/Itin their bodies, includingHelpersLanguage.psm1which the parser tests use heavily. Dongbo Wang (@daxian-dbw) asked how we can be sure that pattern still holds. I audited it with the AST rather than by reading, the details are in the comments below. Two rules came out of it:Itblocks has to be called during discovery, which means from aDescribeorContextbody, fromBeforeDiscovery, or from the file top level. Calling it fromBeforeAllregisters nothing. All 439 call sites in the suite are in discovery positions, none inBeforeAll.Itcannot read that function's parameters or locals. The body runs later, in the Pester block scope, where those names no longer exist. The value has to arrive either as-TestCases/-ForEachdata, or from aBeforeAllin the same block.Rule 2 found two real problems, both fixed here:
Copy-Item.Tests.ps1Test-CopyItemErrorread$path,$destinationand$expectedFullyQualifiedErrorIdfrom its own parameters inside theItbody. At run time all three are$null, andShould -Throw -ErrorId $nullmatches any error, so 11 tests would have passed without asserting anything.-TestCasesdataScriptHelp.Tests.ps1TestHelpFunc1 $xworked only because the parameter and theBeforeAllvariable happened to share the name$x, as Dongbo Wang (@daxian-dbw) showed by renaming itItbodies read theBeforeAllvariables directlyTest output in CI
Pester 6 defaults to
Normalverbosity, which prints neither theDescribing/Contextheaders nor a line per passing test, soWrite-Tersehad nothing to collapse into+.Start-PSPesternow asks forDetailedunless-Quietis passed, which gives back the Pester 4 CI output.Results
All build and test jobs pass on Windows, Linux and macOS (Elevated and Unelevated), plus xUnit, CodeQL, Infrastructure Tests and packaging.
The only red is CodeFactor's 8 pre-existing issues, none of them introduced here.
Since no assertion or expected value was touched, CI parity is the signal here. On top of that I compared Pester 6 discovery counts per file across the whole suite before and after merging
master, to make sure the migration is not silently dropping tests.v6 breaking changes I audited
It -Pendingparameter removed-SkipSet-ItResult -Pendingremoved-Inconclusive<Command>) deep-expand the valueEncoding.Tests.ps1→ pass the name, not theCommandInfo(was OOM)-ForEach/-TestCasesfails discovery by defaultRun.FailOnNullOrEmptyForEach = $falseBeforeAll/AfterAll/BeforeEach/AfterEachin one block now throwsDebuggingInHost) → mergedTestDrivedirectory now namedPester_<random>ErrorView.Tests.ps1→ strip$TestDrivebefore the "nopester" check-TestCasesvalue not reaching a runtimeSet-ItResultguard in CIExecutionPolicy.Tests.ps1→ skip the 1 unreliable case at discoverybuild.psm1→ ask forDetailedunless-QuietAssert-MockCalled/Assert-VerifiableMockremoved-FocusremovedCoverageGutterscoverage format removedShould -BeremovedNotable files
build.psm1MaximumVersion 4.99→RequiredVersion 6.0.0, pinned in one place as$script:PesterVersion; addRun.FailOnNullOrEmptyForEach = $false; ask forDetailedoutput; project thePassThruobject beforeExport-Clixml, the v5/v6 run object nests deeper than the CliXml deserializer accepts*.Tests.ps1(+HelpersLanguage.psm1)It ... -Pending→It ... -Skip(137 sites)*.Tests.ps1(+tools/packaging/releaseTests/sbom.tests.ps1)Set-ItResult -Pending→-InconclusiveEncoding.Tests.ps1CommandInfo) in the<Command>name template, v6 OOMs on the objectDebuggingInHost.Tests.ps1BeforeAllinto one; keep the leadingreturn, so the fragile WinRM setup still never runsErrorView.Tests.ps1$TestDrivebefore the "error contains nopester" checkExecutionPolicy.Tests.ps1-SkipCopy-Item.Tests.ps1Test-CopyItemErrorpasses its values as-TestCasesdataScriptHelp.Tests.ps1TestHelpFunc1andTestHelpErrorno longer take parameterstest/powershell/README.md-Pendingis gone in v6, document-SkipinsteadWhat did NOT change
Should -Be,Should -Throw, …), v6 keeps v5 syntax on by default-Pendingtests were already not running and-Skipkeeps them not running🤖