Skip to content

[release/v7.5] Add GitHub Actions annotations for Pester test failures#26836

Merged
daxian-dbw merged 3 commits intoPowerShell:release/v7.5from
daxian-dbw:backport/release/v7.5/26789-4db414ca3
Feb 17, 2026
Merged

[release/v7.5] Add GitHub Actions annotations for Pester test failures#26836
daxian-dbw merged 3 commits intoPowerShell:release/v7.5from
daxian-dbw:backport/release/v7.5/26789-4db414ca3

Conversation

@daxian-dbw
Copy link
Member

Backport of #26789 to release/v7.5

Triggered by @daxian-dbw on behalf of @app/copilot-swe-agent

Original CL Label: CL-Test

/cc @PowerShell/powershell-maintainers

Impact

REQUIRED: Choose either Tooling Impact or Customer Impact (or both). At least one checkbox must be selected.

Tooling Impact

  • Required tooling change
  • Optional tooling change (include reasoning)

Customer Impact

  • Customer reported
  • Found internally

GitHub Actions workflow annotations are now automatically generated when Pester tests fail, providing clickable file annotations with direct links to the failing test code and workflow logs. This improves debugging efficiency and visibility in PRs.

Regression

REQUIRED: Check exactly one box.

  • Yes
  • No

This is not a regression.

Testing

Comprehensive testing performed including absolute/relative path conversion for Unix and Windows paths, Pester 4 and 5 format parsing, multiple test failures, backward compatibility, and PSScriptAnalyzer compliance. Includes demonstration test using actual CI code path.

Risk

REQUIRED: Check exactly one box.

  • High
  • Medium
  • Low

Changes add new functionality to test result processing without modifying existing behavior. Backward compatible - Show-PSPesterError is unaffected. Only adds annotations when running in GitHub Actions. Tested with both Pester 4 and 5.

…6789)

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: TravisEz13 <[email protected]>
Co-authored-by: Travis Plunk <[email protected]>
@daxian-dbw daxian-dbw requested a review from a team as a code owner February 15, 2026 23:48
@daxian-dbw daxian-dbw added the CL-Test Indicates that a PR should be marked as a test change in the Change Log label Feb 15, 2026
Copilot AI review requested due to automatic review settings February 15, 2026 23:48
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request backports GitHub Actions annotations for Pester test failures from #26789 to the release/v7.5 branch. When Pester tests fail in GitHub Actions workflows, the changes automatically generate clickable file annotations that link directly to the failing test code and workflow run logs, improving debugging efficiency and visibility in pull requests.

Changes:

  • Added Get-PesterFailureFileInfo helper function to build.psm1 for parsing stack traces from both Pester 4 and Pester 5 formats
  • Enhanced process-pester-results.ps1 to generate GitHub Actions workflow annotations for test failures with file paths, line numbers, and direct links to logs
  • Implemented cross-platform path handling to convert absolute paths to workspace-relative paths for proper annotation display

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File Description
build.psm1 Adds Get-PesterFailureFileInfo function with regex patterns to extract file paths and line numbers from Pester 4 and 5 stack traces
.github/actions/test/process-pester-results/process-pester-results.ps1 Generates GitHub Actions error annotations for each test failure with file location, line number, and log links

Comment on lines +49 to +90
foreach ($testfail in $failures) {
$description = $testfail.description
$testName = $testfail.name
$message = $testfail.failure.message
$stack_trace = $testfail.failure.'stack-trace'

# Parse stack trace to get file and line info
$fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace

if ($fileInfo.File) {
# Convert absolute path to relative path for GitHub Actions
$filePath = $fileInfo.File

# GitHub Actions expects paths relative to the workspace root
if ($env:GITHUB_WORKSPACE) {
$workspacePath = $env:GITHUB_WORKSPACE
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
}

# Create annotation title
$annotationTitle = "Test Failure: $description / $testName"

# Build the annotation message
$annotationMessage = $message -replace "`n", "%0A" -replace "`r"

# Build and output the workflow command
$workflowCommand = "::error file=$filePath"
if ($fileInfo.Line) {
$workflowCommand += ",line=$($fileInfo.Line)"
}
$workflowCommand += ",title=$annotationTitle::$annotationMessage"

Write-Host $workflowCommand

# Output a link to the test run
if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) {
$logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)"
Write-Host "Test logs: $logUrl"
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The foreach loop does not check if $failures is null or empty before iterating. While the XPath query will return an empty collection rather than null in most cases, if the test-results XML structure is malformed or unexpected, the SelectNodes call could potentially return null, causing errors.

Add a null/empty check before the foreach loop to handle edge cases where no failures node collection is returned, similar to the pattern used in Test-PSPesterResults function at line 2010 which checks the failure count before iterating.

Suggested change
foreach ($testfail in $failures) {
$description = $testfail.description
$testName = $testfail.name
$message = $testfail.failure.message
$stack_trace = $testfail.failure.'stack-trace'
# Parse stack trace to get file and line info
$fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace
if ($fileInfo.File) {
# Convert absolute path to relative path for GitHub Actions
$filePath = $fileInfo.File
# GitHub Actions expects paths relative to the workspace root
if ($env:GITHUB_WORKSPACE) {
$workspacePath = $env:GITHUB_WORKSPACE
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
}
# Create annotation title
$annotationTitle = "Test Failure: $description / $testName"
# Build the annotation message
$annotationMessage = $message -replace "`n", "%0A" -replace "`r"
# Build and output the workflow command
$workflowCommand = "::error file=$filePath"
if ($fileInfo.Line) {
$workflowCommand += ",line=$($fileInfo.Line)"
}
$workflowCommand += ",title=$annotationTitle::$annotationMessage"
Write-Host $workflowCommand
# Output a link to the test run
if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) {
$logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)"
Write-Host "Test logs: $logUrl"
if ($failures -and $failures.Count -gt 0) {
foreach ($testfail in $failures) {
$description = $testfail.description
$testName = $testfail.name
$message = $testfail.failure.message
$stack_trace = $testfail.failure.'stack-trace'
# Parse stack trace to get file and line info
$fileInfo = Get-PesterFailureFileInfo -StackTraceString $stack_trace
if ($fileInfo.File) {
# Convert absolute path to relative path for GitHub Actions
$filePath = $fileInfo.File
# GitHub Actions expects paths relative to the workspace root
if ($env:GITHUB_WORKSPACE) {
$workspacePath = $env:GITHUB_WORKSPACE
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
}
# Create annotation title
$annotationTitle = "Test Failure: $description / $testName"
# Build the annotation message
$annotationMessage = $message -replace "`n", "%0A" -replace "`r"
# Build and output the workflow command
$workflowCommand = "::error file=$filePath"
if ($fileInfo.Line) {
$workflowCommand += ",line=$($fileInfo.Line)"
}
$workflowCommand += ",title=$annotationTitle::$annotationMessage"
Write-Host $workflowCommand
# Output a link to the test run
if ($env:GITHUB_SERVER_URL -and $env:GITHUB_REPOSITORY -and $env:GITHUB_RUN_ID) {
$logUrl = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)"
Write-Host "Test logs: $logUrl"
}

Copilot uses AI. Check for mistakes.
$testName = $testfail.name
$message = $testfail.failure.message
$stack_trace = $testfail.failure.'stack-trace'

Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code assumes that $testfail.description, $testfail.name, $testfail.failure.message, and $testfail.failure.'stack-trace' will always be present and not null. If the XML structure is missing any of these elements (e.g., a test failure without a description or message), this could cause issues when constructing the annotation.

Consider adding null checks or using null-coalescing/default values to handle cases where these properties might be missing from the XML, similar to defensive programming patterns seen elsewhere in the codebase.

Suggested change
# Provide sensible defaults if any of the expected fields are missing
if (-not [string]::IsNullOrWhiteSpace($description)) {
$description = [string]$description
}
else {
$description = "(no description)"
}
if (-not [string]::IsNullOrWhiteSpace($testName)) {
$testName = [string]$testName
}
else {
$testName = "(unnamed test)"
}
if (-not [string]::IsNullOrWhiteSpace($message)) {
$message = [string]$message
}
else {
$message = "(no failure message)"
}
# Skip annotation if there is no stack trace to parse
if ([string]::IsNullOrWhiteSpace($stack_trace)) {
continue
}

Copilot uses AI. Check for mistakes.
Comment on lines +73 to +83
$annotationTitle = "Test Failure: $description / $testName"

# Build the annotation message
$annotationMessage = $message -replace "`n", "%0A" -replace "`r"

# Build and output the workflow command
$workflowCommand = "::error file=$filePath"
if ($fileInfo.Line) {
$workflowCommand += ",line=$($fileInfo.Line)"
}
$workflowCommand += ",title=$annotationTitle::$annotationMessage"
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The annotation title and message construction follows the same escaping pattern as the existing Write-Log function (line 2920 in build.psm1), which only escapes newlines and carriage returns. However, according to GitHub Actions workflow command documentation, additional characters should be escaped for complete safety: percent signs (%), commas (,), and colons (:) have special meaning in workflow commands.

The title on line 73 concatenates $description and $testName which may contain these special characters. The message on line 76 uses the same incomplete escaping pattern as Write-Log.

While this matches existing codebase patterns, it could cause annotations to be malformed or fail to display correctly when test names or messages contain special characters. Consider enhancing the escaping to include: %%25, ,%2C, and :%3A for more robust handling, or document this as a known limitation.

Copilot uses AI. Check for mistakes.
Comment on lines +64 to +69
$workspacePath = $env:GITHUB_WORKSPACE
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The path handling logic on lines 65-66 uses StartsWith and Substring operations which are case-sensitive on Unix-like systems but case-insensitive on Windows. On case-sensitive filesystems, if $workspacePath ends with a trailing slash but $filePath doesn't start with exactly the same case, the path won't be converted to relative form, potentially causing annotations to fail or point to wrong locations.

Additionally, there's no validation that the resulting relative path doesn't escape the workspace directory (e.g., through path traversal sequences like "../"). While this is likely safe since the paths come from stack traces, it's a defense-in-depth consideration.

Suggested fix: Normalize path separators before comparison and ensure consistent trailing separator handling. Consider using [System.IO.Path]::GetRelativePath() if available in the PowerShell version used.

Suggested change
$workspacePath = $env:GITHUB_WORKSPACE
if ($filePath.StartsWith($workspacePath)) {
$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'
}
$workspacePath = [System.IO.Path]::GetFullPath(
$env:GITHUB_WORKSPACE.TrimEnd([System.IO.Path]::DirectorySeparatorChar, [System.IO.Path]::AltDirectorySeparatorChar)
)
$fileFullPath = [System.IO.Path]::GetFullPath($filePath)
$relativePath = $null
try {
$relativePath = [System.IO.Path]::GetRelativePath($workspacePath, $fileFullPath)
}
catch {
# If GetRelativePath fails for any reason, fall back to manual logic below.
}
$useRelativePath = $false
if ($relativePath -and -not [System.IO.Path]::IsPathRooted($relativePath)) {
$parentTraversalPrefix = '..' + [System.IO.Path]::DirectorySeparatorChar
if ($relativePath -ne '..' -and -not $relativePath.StartsWith($parentTraversalPrefix)) {
$useRelativePath = $true
}
}
if ($useRelativePath) {
$filePath = $relativePath
}
else {
# Fallback: if the file is under the workspace (case-insensitive), compute a safe relative path.
if ($fileFullPath.StartsWith($workspacePath, [System.StringComparison]::OrdinalIgnoreCase)) {
$filePath = $fileFullPath.Substring($workspacePath.Length).TrimStart('\', '/')
}
else {
# As a last resort, keep the normalized absolute path.
$filePath = $fileFullPath
}
}
# Normalize to forward slashes for consistency
$filePath = $filePath -replace '\\', '/'

Copilot uses AI. Check for mistakes.
Comment on lines +1899 to +1925
# Try pattern: ", <path>:123" (Pester 5 format)
# This handles both Unix paths (/path/file.ps1:123) and Windows paths (C:\path\file.ps1:123)
if ($StackTraceString -match ',\s*((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1):(\d+)') {
$result.File = $matches[1].Trim()
$result.Line = $matches[2]
return $result
}

# Try pattern: "at <path>:123" (without comma)
# Handle both absolute Unix and Windows paths
if ($StackTraceString -match 'at\s+((?:[A-Za-z]:)?[\/\\][^,]+?\.ps[m]?1):(\d+)(?:\r|\n|$)') {
$result.File = $matches[1].Trim()
$result.Line = $matches[2]
return $result
}

# Try pattern: "<path>: line 123"
if ($StackTraceString -match '((?:[A-Za-z]:)?[\/\\][^,]+?\.ps[m]?1):\s*line\s+(\d+)(?:\r|\n|$)') {
$result.File = $matches[1].Trim()
$result.Line = $matches[2]
return $result
}

# Try to extract just the file path if no line number found
if ($StackTraceString -match '(?:at\s+|in\s+)?((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1)') {
$result.File = $matches[1].Trim()
}
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regex patterns on lines 1901, 1909, 1916, and 1923 use non-greedy matching (.+?) to capture file paths. However, these patterns could fail or capture incorrect paths in edge cases:

  1. Line 1901: The pattern ,\s*((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1):(\d+) will match the first .ps1 or .psm1 it encounters, which could be incorrect if the stack trace contains multiple file paths on the same line.

  2. Line 1923: The fallback pattern (?:at\s+|in\s+)?((?:[A-Za-z]:)?[\/\\].+?\.ps[m]?1) without line number matching could incorrectly extract paths from multiline stack traces.

  3. All patterns: Paths with special characters, spaces, or unicode characters might not be handled correctly, especially on Unix systems where file paths can contain almost any character except null and forward slash.

While these edge cases may be rare in practice, they could cause annotations to point to wrong files or fail to be generated when tests fail in files with unusual names.

Copilot uses AI. Check for mistakes.
@daxian-dbw daxian-dbw merged commit d5d02a1 into PowerShell:release/v7.5 Feb 17, 2026
43 of 47 checks passed
@daxian-dbw daxian-dbw deleted the backport/release/v7.5/26789-4db414ca3 branch February 17, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-Test Indicates that a PR should be marked as a test change in the Change Log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants