[release/v7.5] Add GitHub Actions annotations for Pester test failures#26836
Conversation
…6789) Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: TravisEz13 <[email protected]> Co-authored-by: Travis Plunk <[email protected]>
There was a problem hiding this comment.
Pull request overview
This pull request backports GitHub Actions annotations for Pester test failures from #26789 to the release/v7.5 branch. When Pester tests fail in GitHub Actions workflows, the changes automatically generate clickable file annotations that link directly to the failing test code and workflow run logs, improving debugging efficiency and visibility in pull requests.
Changes:
- Added
Get-PesterFailureFileInfohelper function to build.psm1 for parsing stack traces from both Pester 4 and Pester 5 formats - Enhanced process-pester-results.ps1 to generate GitHub Actions workflow annotations for test failures with file paths, line numbers, and direct links to logs
- Implemented cross-platform path handling to convert absolute paths to workspace-relative paths for proper annotation display
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| build.psm1 | Adds Get-PesterFailureFileInfo function with regex patterns to extract file paths and line numbers from Pester 4 and 5 stack traces |
| .github/actions/test/process-pester-results/process-pester-results.ps1 | Generates GitHub Actions error annotations for each test failure with file location, line number, and log links |
| foreach ($testfail in $failures) { | ||
| $description = $testfail.description | ||
| $testName = $testfail.name | ||
| $message = $testfail.failure.message | ||
| $stack_trace = $testfail.failure.'stack-trace' | ||
|
|
||
| # Parse stack trace to get file and line info | ||
| $fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace | ||
|
|
||
| if ($fileInfo.File) { | ||
| # Convert absolute path to relative path for GitHub Actions | ||
| $filePath = $fileInfo.File | ||
|
|
||
| # GitHub Actions expects paths relative to the workspace root | ||
| if ($env:GITHUB_WORKSPACE) { | ||
| $workspacePath = $env:GITHUB_WORKSPACE | ||
| if ($filePath.StartsWith($workspacePath)) { | ||
| $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\') | ||
| # Normalize to forward slashes for consistency | ||
| $filePath = $filePath -replace '\\', '/' | ||
| } | ||
| } | ||
|
|
||
| # Create annotation title | ||
| $annotationTitle = "Test Failure: $description / $testName" | ||
|
|
||
| # Build the annotation message | ||
| $annotationMessage = $message -replace "`n", "%0A" -replace "`r" | ||
|
|
||
| # Build and output the workflow command | ||
| $workflowCommand = "::error file=$filePath" | ||
| if ($fileInfo.Line) { | ||
| $workflowCommand += ",line=$($fileInfo.Line)" | ||
| } | ||
| $workflowCommand += ",title=$annotationTitle::$annotationMessage" | ||
|
|
||
| Write-Host $workflowCommand | ||
|
|
||
| # Output a link to the test run | ||
| if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) { | ||
| $logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)" | ||
| Write-Host "Test logs: $logUrl" |
There was a problem hiding this comment.
The foreach loop does not check if $failures is null or empty before iterating. While the XPath query will return an empty collection rather than null in most cases, if the test-results XML structure is malformed or unexpected, the SelectNodes call could potentially return null, causing errors.
Add a null/empty check before the foreach loop to handle edge cases where no failures node collection is returned, similar to the pattern used in Test-PSPesterResults function at line 2010 which checks the failure count before iterating.
| foreach ($testfail in $failures) { | |
| $description = $testfail.description | |
| $testName = $testfail.name | |
| $message = $testfail.failure.message | |
| $stack_trace = $testfail.failure.'stack-trace' | |
| # Parse stack trace to get file and line info | |
| $fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace | |
| if ($fileInfo.File) { | |
| # Convert absolute path to relative path for GitHub Actions | |
| $filePath = $fileInfo.File | |
| # GitHub Actions expects paths relative to the workspace root | |
| if ($env:GITHUB_WORKSPACE) { | |
| $workspacePath = $env:GITHUB_WORKSPACE | |
| if ($filePath.StartsWith($workspacePath)) { | |
| $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\') | |
| # Normalize to forward slashes for consistency | |
| $filePath = $filePath -replace '\\', '/' | |
| } | |
| } | |
| # Create annotation title | |
| $annotationTitle = "Test Failure: $description / $testName" | |
| # Build the annotation message | |
| $annotationMessage = $message -replace "`n", "%0A" -replace "`r" | |
| # Build and output the workflow command | |
| $workflowCommand = "::error file=$filePath" | |
| if ($fileInfo.Line) { | |
| $workflowCommand += ",line=$($fileInfo.Line)" | |
| } | |
| $workflowCommand += ",title=$annotationTitle::$annotationMessage" | |
| Write-Host $workflowCommand | |
| # Output a link to the test run | |
| if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) { | |
| $logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)" | |
| Write-Host "Test logs: $logUrl" | |
| if ($failures -and $failures.Count -gt 0) { | |
| foreach ($testfail in $failures) { | |
| $description = $testfail.description | |
| $testName = $testfail.name | |
| $message = $testfail.failure.message | |
| $stack_trace = $testfail.failure.'stack-trace' | |
| # Parse stack trace to get file and line info | |
| $fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace | |
| if ($fileInfo.File) { | |
| # Convert absolute path to relative path for GitHub Actions | |
| $filePath = $fileInfo.File | |
| # GitHub Actions expects paths relative to the workspace root | |
| if ($env:GITHUB_WORKSPACE) { | |
| $workspacePath = $env:GITHUB_WORKSPACE | |
| if ($filePath.StartsWith($workspacePath)) { | |
| $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\') | |
| # Normalize to forward slashes for consistency | |
| $filePath = $filePath -replace '\\', '/' | |
| } | |
| } | |
| # Create annotation title | |
| $annotationTitle = "Test Failure: $description / $testName" | |
| # Build the annotation message | |
| $annotationMessage = $message -replace "`n", "%0A" -replace "`r" | |
| # Build and output the workflow command | |
| $workflowCommand = "::error file=$filePath" | |
| if ($fileInfo.Line) { | |
| $workflowCommand += ",line=$($fileInfo.Line)" | |
| } | |
| $workflowCommand += ",title=$annotationTitle::$annotationMessage" | |
| Write-Host $workflowCommand | |
| # Output a link to the test run | |
| if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) { | |
| $logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)" | |
| Write-Host "Test logs: $logUrl" | |
| } |
| $testName = $testfail.name | ||
| $message = $testfail.failure.message | ||
| $stack_trace = $testfail.failure.'stack-trace' | ||
|
|
There was a problem hiding this comment.
The code assumes that $testfail.description, $testfail.name, $testfail.failure.message, and $testfail.failure.'stack-trace' will always be present and not null. If the XML structure is missing any of these elements (e.g., a test failure without a description or message), this could cause issues when constructing the annotation.
Consider adding null checks or using null-coalescing/default values to handle cases where these properties might be missing from the XML, similar to defensive programming patterns seen elsewhere in the codebase.
| # Provide sensible defaults if any of the expected fields are missing | |
| if (-not [string]::IsNullOrWhiteSpace($description)) { | |
| $description = [string]$description | |
| } | |
| else { | |
| $description = "(no description)" | |
| } | |
| if (-not [string]::IsNullOrWhiteSpace($testName)) { | |
| $testName = [string]$testName | |
| } | |
| else { | |
| $testName = "(unnamed test)" | |
| } | |
| if (-not [string]::IsNullOrWhiteSpace($message)) { | |
| $message = [string]$message | |
| } | |
| else { | |
| $message = "(no failure message)" | |
| } | |
| # Skip annotation if there is no stack trace to parse | |
| if ([string]::IsNullOrWhiteSpace($stack_trace)) { | |
| continue | |
| } |
| $annotationTitle = "Test Failure: $description / $testName" | ||
|
|
||
| # Build the annotation message | ||
| $annotationMessage = $message -replace "`n", "%0A" -replace "`r" | ||
|
|
||
| # Build and output the workflow command | ||
| $workflowCommand = "::error file=$filePath" | ||
| if ($fileInfo.Line) { | ||
| $workflowCommand += ",line=$($fileInfo.Line)" | ||
| } | ||
| $workflowCommand += ",title=$annotationTitle::$annotationMessage" |
There was a problem hiding this comment.
The annotation title and message construction follows the same escaping pattern as the existing Write-Log function (line 2920 in build.psm1), which only escapes newlines and carriage returns. However, according to GitHub Actions workflow command documentation, additional characters should be escaped for complete safety: percent signs (%), commas (,), and colons (:) have special meaning in workflow commands.
The title on line 73 concatenates $description and $testName which may contain these special characters. The message on line 76 uses the same incomplete escaping pattern as Write-Log.
While this matches existing codebase patterns, it could cause annotations to be malformed or fail to display correctly when test names or messages contain special characters. Consider enhancing the escaping to include: % → %25, , → %2C, and : → %3A for more robust handling, or document this as a known limitation.
| $workspacePath = $env:GITHUB_WORKSPACE | ||
| if ($filePath.StartsWith($workspacePath)) { | ||
| $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\') | ||
| # Normalize to forward slashes for consistency | ||
| $filePath = $filePath -replace '\\', '/' | ||
| } |
There was a problem hiding this comment.
The path handling logic on lines 65-66 uses StartsWith and Substring operations which are case-sensitive on Unix-like systems but case-insensitive on Windows. On case-sensitive filesystems, if $workspacePath ends with a trailing slash but $filePath doesn't start with exactly the same case, the path won't be converted to relative form, potentially causing annotations to fail or point to wrong locations.
Additionally, there's no validation that the resulting relative path doesn't escape the workspace directory (e.g., through path traversal sequences like "../"). While this is likely safe since the paths come from stack traces, it's a defense-in-depth consideration.
Suggested fix: Normalize path separators before comparison and ensure consistent trailing separator handling. Consider using [System.IO.Path]::GetRelativePath() if available in the PowerShell version used.
| $workspacePath = $env:GITHUB_WORKSPACE | |
| if ($filePath.StartsWith($workspacePath)) { | |
| $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\') | |
| # Normalize to forward slashes for consistency | |
| $filePath = $filePath -replace '\\', '/' | |
| } | |
| $workspacePath = [System.IO.Path]::GetFullPath( | |
| $env:GITHUB_WORKSPACE.TrimEnd([System.IO.Path]::DirectorySeparatorChar, [System.IO.Path]::AltDirectorySeparatorChar) | |
| ) | |
| $fileFullPath = [System.IO.Path]::GetFullPath($filePath) | |
| $relativePath = $null | |
| try { | |
| $relativePath = [System.IO.Path]::GetRelativePath($workspacePath, $fileFullPath) | |
| } | |
| catch { | |
| # If GetRelativePath fails for any reason, fall back to manual logic below. | |
| } | |
| $useRelativePath = $false | |
| if ($relativePath -and -not [System.IO.Path]::IsPathRooted($relativePath)) { | |
| $parentTraversalPrefix = '..' + [System.IO.Path]::DirectorySeparatorChar | |
| if ($relativePath -ne '..' -and -not $relativePath.StartsWith($parentTraversalPrefix)) { | |
| $useRelativePath = $true | |
| } | |
| } | |
| if ($useRelativePath) { | |
| $filePath = $relativePath | |
| } | |
| else { | |
| # Fallback: if the file is under the workspace (case-insensitive), compute a safe relative path. | |
| if ($fileFullPath.StartsWith($workspacePath, [System.StringComparison]::OrdinalIgnoreCase)) { | |
| $filePath = $fileFullPath.Substring($workspacePath.Length).TrimStart('\', '/') | |
| } | |
| else { | |
| # As a last resort, keep the normalized absolute path. | |
| $filePath = $fileFullPath | |
| } | |
| } | |
| # Normalize to forward slashes for consistency | |
| $filePath = $filePath -replace '\\', '/' |
| # Try pattern: ", <path>:123" (Pester 5 format) | ||
| # This handles both Unix paths (/path/file.ps1:123) and Windows paths (C:\path\file.ps1:123) | ||
| if ($StackTraceString -match ',\s*((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1):(\d+)') { | ||
| $result.File = $matches[1].Trim() | ||
| $result.Line = $matches[2] | ||
| return $result | ||
| } | ||
|
|
||
| # Try pattern: "at <path>:123" (without comma) | ||
| # Handle both absolute Unix and Windows paths | ||
| if ($StackTraceString -match 'at\s+((?:[A-Za-z]:)?[\/\\][^,]+?\.ps[m]?1):(\d+)(?:\r|\n|$)') { | ||
| $result.File = $matches[1].Trim() | ||
| $result.Line = $matches[2] | ||
| return $result | ||
| } | ||
|
|
||
| # Try pattern: "<path>: line 123" | ||
| if ($StackTraceString -match '((?:[A-Za-z]:)?[\/\\][^,]+?\.ps[m]?1):\s*line\s+(\d+)(?:\r|\n|$)') { | ||
| $result.File = $matches[1].Trim() | ||
| $result.Line = $matches[2] | ||
| return $result | ||
| } | ||
|
|
||
| # Try to extract just the file path if no line number found | ||
| if ($StackTraceString -match '(?:at\s+|in\s+)?((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1)') { | ||
| $result.File = $matches[1].Trim() | ||
| } |
There was a problem hiding this comment.
The regex patterns on lines 1901, 1909, 1916, and 1923 use non-greedy matching (.+?) to capture file paths. However, these patterns could fail or capture incorrect paths in edge cases:
-
Line 1901: The pattern
,\s*((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1):(\d+)will match the first.ps1or.psm1it encounters, which could be incorrect if the stack trace contains multiple file paths on the same line. -
Line 1923: The fallback pattern
(?:at\s+|in\s+)?((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1)without line number matching could incorrectly extract paths from multiline stack traces. -
All patterns: Paths with special characters, spaces, or unicode characters might not be handled correctly, especially on Unix systems where file paths can contain almost any character except null and forward slash.
While these edge cases may be rare in practice, they could cause annotations to point to wrong files or fail to be generated when tests fail in files with unusual names.
Backport of #26789 to release/v7.5
Triggered by @daxian-dbw on behalf of @app/copilot-swe-agent
Original CL Label: CL-Test
/cc @PowerShell/powershell-maintainers
Impact
REQUIRED: Choose either Tooling Impact or Customer Impact (or both). At least one checkbox must be selected.
Tooling Impact
Customer Impact
GitHub Actions workflow annotations are now automatically generated when Pester tests fail, providing clickable file annotations with direct links to the failing test code and workflow logs. This improves debugging efficiency and visibility in PRs.
Regression
REQUIRED: Check exactly one box.
This is not a regression.
Testing
Comprehensive testing performed including absolute/relative path conversion for Unix and Windows paths, Pester 4 and 5 format parsing, multiple test failures, backward compatibility, and PSScriptAnalyzer compliance. Includes demonstration test using actual CI code path.
Risk
REQUIRED: Check exactly one box.
Changes add new functionality to test result processing without modifying existing behavior. Backward compatible - Show-PSPesterError is unaffected. Only adds annotations when running in GitHub Actions. Tested with both Pester 4 and 5.