Add GitHub Actions annotations for Pester test failures by Copilot · Pull Request #26789 · PowerShell/PowerShell

Copilot · 2026-02-12T17:57:01Z

Add GitHub Workflow Annotations for Pester Test Failures ✅

Summary

Successfully implemented GitHub Actions workflow annotations for Pester test failures. When tests fail in GitHub Actions, clickable file annotations are now automatically generated with direct links to the failing test code and workflow logs.

Implementation Checklist

Changes Made

Files Modified:

build.psm1 - Get-PesterFailureFileInfo helper function
.github/actions/test/process-pester-results/process-pester-results.ps1 - Annotation generation logic
test/powershell/Modules/DemoGitHubAnnotations.Tests.ps1 - NEW demonstration test

New Function:

Get-PesterFailureFileInfo (in build.psm1): Parses stack traces to extract file path and line number
- Supports Pester 4 format: at line: X in <path>
- Supports Pester 5 format: at X, /path:line
- Handles Unix and Windows paths

Modified Script:

process-pester-results.ps1: Now generates annotations when processing test result XML files
- Parses all test failure XML nodes from result files
- Extracts file location from stack traces using Get-PesterFailureFileInfo
- Converts absolute paths to relative paths using $env:GITHUB_WORKSPACE
- Creates file annotations with clickable links to test code
- Includes direct links to workflow run logs
- Normalizes paths to forward slashes

Demonstration Test:

Created test/powershell/Modules/DemoGitHubAnnotations.Tests.ps1
Uses actual CI code path: Pester test execution → process-pester-results.ps1 → annotations
Tagged with 'CI' so it runs in normal CI workflows
Intentionally fails to demonstrate the annotation feature in CI output
DELETE THIS FILE after verifying the feature works in production CI

Features

✅ Pester 4 Support (Primary version used in PowerShell repo)
✅ Pester 5 Support
✅ Cross-Platform (Unix and Windows paths)
✅ Relative Paths (Correctly formatted for GitHub Actions)
✅ Backward Compatible (Show-PSPesterError unaffected)
✅ No Configuration Required (Automatic)
✅ Correct Location (Annotations generated in process-pester-results.ps1)

Example Output

::error file=test/powershell/Modules/DemoGitHubAnnotations.Tests.ps1,line=27,title=Test Failure: Should fail to demonstrate GitHub Actions annotations - REMOVE THIS TEST / ...::Expected strings to be the same...
Test logs: https://github.com/PowerShell/PowerShell/actions/runs/12345

Testing Performed

✅ Absolute path to relative path conversion (Unix paths)
✅ Absolute path to relative path conversion (Windows paths)
✅ Pester 4 and 5 format parsing
✅ Relative paths remain unchanged
✅ Multiple test failures
✅ Backward compatibility
✅ PSScriptAnalyzer compliance
✅ Annotations generated in correct location (process-pester-results.ps1)

Benefits

🚀 Faster Debugging: Click annotations to jump directly to failing test code
👁️ Better Visibility: Annotations appear in PR Files Changed tab
🔗 Complete Context: Direct links to full test run logs
📊 Multiple Failures: Each failure gets its own annotation
🔄 Backward Compatible: No impact on existing code paths
📍 Centralized Processing: Annotations generated where test results are processed

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

- Added Get-PesterFailureFileInfo helper function to parse stack traces - Modified Show-PSPesterError to emit GitHub workflow error annotations with file/line info - Added links to test logs when running in GitHub Actions - Maintains backward compatibility when not in a GitHub workflow Co-authored-by: TravisEz13 <[email protected]>

- Added support for Pester 5 stack trace format (e.g., "at X, /path:line") - Handle both Unix and Windows paths in all regex patterns - Verified parsing with multiple test cases including real Pester output Co-authored-by: TravisEz13 <[email protected]>

…tackTraceString - Renamed parameter to avoid conflict with PowerShell automatic variable - Updated function call to use new parameter name - Verified Pester 4 and Pester 5 format support Co-authored-by: TravisEz13 <[email protected]>

TravisEz13 · 2026-02-12T18:48:37Z

@copilot
The example, and therefore the code isn't right

::error file=/home/runner/work/PowerShell/PowerShell/test/example.tests.ps1,line=42,title=Test Failure: Parser Tests / Should parse correctly::Expected 'true' but got 'false'
Test logs: https://github.com/PowerShell/PowerShell/actions/runs/12345

the paths should be relative to the root of the repo

- GitHub Actions expects paths relative to workspace root, not absolute paths - Added path conversion logic using GITHUB_WORKSPACE environment variable - Normalizes paths to use forward slashes for consistency - Handles both Unix and Windows absolute paths - Relative paths remain unchanged Co-authored-by: TravisEz13 <[email protected]>

Copilot · 2026-02-12T18:51:47Z

@copilot
The example, and therefore the code isn't right

...

Fixed in 9b85763. Paths are now converted to be relative to the repository root using $env:GITHUB_WORKSPACE.

Examples:

Absolute: /home/runner/work/PowerShell/PowerShell/test/example.tests.ps1
Converted to: test/example.tests.ps1

The conversion handles both Unix and Windows paths and normalizes to forward slashes.

TravisEz13 · 2026-02-12T19:09:47Z

@copilot add a test that fails to demonstrate this feature with the plan on removing it once we are sure the code works.

- Added test in ciModule.Tests.ps1 to demonstrate the annotation feature - Test intentionally fails to show annotations in CI output - Verifies relative path conversion and test log links - Can be removed after feature is verified in actual CI runs Co-authored-by: TravisEz13 <[email protected]>

Copilot · 2026-02-12T19:13:55Z

@copilot add a test that fails to demonstrate this feature with the plan on removing it once we are sure the code works.

Added in 790ef44. The test "Should generate GitHub Actions annotation with relative file path - DEMONSTRATION TEST (will fail)" intentionally fails to demonstrate the annotation feature in CI.

When run, it generates:

::error file=test/infrastructure/ciModule.Tests.ps1,line=42,title=Test Failure: Demo Context / Demo test that fails::This is a demonstration failure to show GitHub Actions annotations
Test logs: https://github.com/PowerShell/PowerShell/actions/runs/12345

The test verifies:

Absolute paths are converted to relative paths
Annotations include file location and line number
Test log links are generated

Remove the test after confirming the feature works in production CI.

test/infrastructure/ciModule.Tests.ps1

- Removed test from ciModule.Tests.ps1 that manually called Show-PSPesterError - Created DemoGitHubAnnotations.Tests.ps1 in test/powershell/Modules/ - New test runs through normal Pester execution and Test-PSPesterResults - Demonstrates actual code path: Pester test fails -> Test-PSPesterResults -> Show-PSPesterError -> annotations - Tagged with 'CI' so it runs in normal CI workflows - DELETE DemoGitHubAnnotations.Tests.ps1 after verifying feature in production Co-authored-by: TravisEz13 <[email protected]>

TravisEz13 · 2026-02-12T21:27:54Z

@copilot

I think printing the error is in the wrong place.
job-logs.txt

https://github.com/PowerShell/PowerShell/blob/master/.github/actions/test/process-pester-results/process-pester-results.ps1
seems to be the right place based on the log.

GitHub
PowerShell/.github/actions/test/process-pester-results/process-pester-results.ps1 at master · PowerShell/PowerShell
PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

- Moved GitHub Actions annotation logic from Show-PSPesterError to process-pester-results.ps1 - process-pester-results.ps1 is the central place where test results are processed in CI - Removed redundant GITHUB_WORKFLOW check (already in workflow context) - Show-PSPesterError now only displays error info (backward compatible) - Get-PesterFailureFileInfo remains in build.psm1 for reuse Co-authored-by: TravisEz13 <[email protected]>

Copilot · 2026-02-12T21:32:02Z

@copilot

I think printing the error is in the wrong place.
...

Fixed in b490e52. Moved annotation generation from Show-PSPesterError to process-pester-results.ps1.

Changes:

Annotations now generated in process-pester-results.ps1 where test results are centrally processed
Removed redundant $env:GITHUB_WORKFLOW check (already in GitHub Actions context)
Show-PSPesterError simplified to only display error info (backward compatible)
Get-PesterFailureFileInfo remains in build.psm1 for reuse

Annotations are now generated at the correct point in the CI flow, after all test XML files are collected.

Copilot

Pull request overview

Adds GitHub Actions workflow annotations for Pester test failures by extracting file/line information from NUnit XML stack traces and emitting ::error commands during test-results processing.

Changes:

Added Get-PesterFailureFileInfo helper in build.psm1 to parse Pester stack traces for file/line.
Updated process-pester-results.ps1 to iterate failed test cases and emit GitHub Actions error annotations (and a run log URL).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`build.psm1`	Introduces a helper to parse stack traces and return file/line info for annotations.
`.github/actions/test/process-pester-results/process-pester-results.ps1`	Generates `::error` annotations from failed test cases in NUnit XML and attempts workspace-relative path normalization.

Copilot · 2026-02-12T22:11:39Z

.github/actions/test/process-pester-results/process-pester-results.ps1

+            $annotationTitle = "Test Failure: $description / $testName"
+
+            # Build the annotation message
+            $annotationMessage = $message -replace "`n", "%0A" -replace "`r"


$annotationMessage = $message -replace "n", "%0A" -replace "r" is missing the replacement value for the second -replace, which will cause a PowerShell parse error at runtime. Provide an explicit replacement (e.g., strip \r or encode it) so the action can run.

Suggested change

$annotationMessage = $message -replace "`n", "%0A" -replace "`r"

$annotationMessage = $message -replace "`n", "%0A" -replace "`r", "%0D"

Copilot · 2026-02-12T22:11:40Z

.github/actions/test/process-pester-results/process-pester-results.ps1

+            $workflowCommand = "::error file=$filePath"
+            if ($fileInfo.Line) {
+                $workflowCommand += ",line=$($fileInfo.Line)"
+            }
+            $workflowCommand += ",title=$annotationTitle::$annotationMessage"


The GitHub Actions workflow command needs escaping for %, \r, \n, and for property values also : and ,. As written, $filePath, $annotationTitle, or $annotationMessage containing these characters can produce a malformed ::error ...::... command (or truncate fields). Add proper workflow-command escaping before concatenating file=, title=, and the message payload.

Copilot · 2026-02-12T22:11:40Z

.github/actions/test/process-pester-results/process-pester-results.ps1

+                if ($filePath.StartsWith($workspacePath)) {
+                    $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
+                    # Normalize to forward slashes for consistency
+                    $filePath = $filePath -replace '\\', '/'
+                }
+            }
+


The workspace-relative path conversion is brittle across platforms: StartsWith($workspacePath) is case-sensitive and doesn't normalize path separators, so Windows paths (or mixed \ vs /) may not match and will stay absolute, which GitHub annotations typically won't link correctly. Consider normalizing both paths (full path + separator normalization, OrdinalIgnoreCase on Windows), and always normalize to forward slashes even for already-relative paths.

Suggested change

if ($filePath.StartsWith($workspacePath)) {

$filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')

# Normalize to forward slashes for consistency

$filePath = $filePath -replace '\\', '/'

}

}

try {

$workspaceFullPath = [System.IO.Path]::GetFullPath($workspacePath)

}

catch {

$workspaceFullPath = $workspacePath

}

try {

$fileFullPath = [System.IO.Path]::GetFullPath($filePath)

}

catch {

$fileFullPath = $filePath

}

# Normalize separators for comparison

$normalizedWorkspace = $workspaceFullPath.TrimEnd([System.IO.Path]::DirectorySeparatorChar, [System.IO.Path]::AltDirectorySeparatorChar)

$normalizedFile = $fileFullPath.Replace([System.IO.Path]::AltDirectorySeparatorChar, [System.IO.Path]::DirectorySeparatorChar)

$comparison = if ($IsWindows) {

[System.StringComparison]::OrdinalIgnoreCase

}

else {

[System.StringComparison]::Ordinal

}

$workspacePrefix = $normalizedWorkspace + [System.IO.Path]::DirectorySeparatorChar

if ($normalizedFile.StartsWith($workspacePrefix, $comparison) -or $normalizedFile.Equals($normalizedWorkspace, $comparison)) {

$relativePath = if ($normalizedFile.Equals($normalizedWorkspace, $comparison)) {

''

}

else {

$normalizedFile.Substring($normalizedWorkspace.Length + 1)

}

if ($relativePath) {

$filePath = $relativePath

}

}

}

# Normalize to forward slashes for consistency (required by GitHub annotations)

$filePath = $filePath -replace '\\', '/'

.github/actions/test/process-pester-results/process-pester-results.ps1

…6789) Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: TravisEz13 <[email protected]> Co-authored-by: Travis Plunk <[email protected]>

Initial plan

739b0cc

Copilot AI assigned Copilot and TravisEz13 Feb 12, 2026

Copilot started work on behalf of TravisEz13 February 12, 2026 17:57 View session

Copilot AI and others added 3 commits February 12, 2026 18:01

Copilot AI changed the title ~~[WIP] Add comments to test code with failure details~~ Add GitHub Actions annotations for Pester test failures Feb 12, 2026

Copilot AI requested a review from TravisEz13 February 12, 2026 18:09

Copilot finished work on behalf of TravisEz13 February 12, 2026 18:09

TravisEz13 added the CL-Test Indicates that a PR should be marked as a test change in the Change Log label Feb 12, 2026

Copilot started work on behalf of TravisEz13 February 12, 2026 18:49 View session

Copilot finished work on behalf of TravisEz13 February 12, 2026 18:52

Copilot started work on behalf of TravisEz13 February 12, 2026 19:10 View session

Copilot finished work on behalf of TravisEz13 February 12, 2026 19:14

TravisEz13 reviewed Feb 12, 2026

View reviewed changes

test/infrastructure/ciModule.Tests.ps1 Outdated Show resolved Hide resolved

Copilot started work on behalf of TravisEz13 February 12, 2026 20:25 View session

Copilot AI requested a review from TravisEz13 February 12, 2026 20:29

Copilot finished work on behalf of TravisEz13 February 12, 2026 20:29

Copilot started work on behalf of TravisEz13 February 12, 2026 21:28 View session

Copilot finished work on behalf of TravisEz13 February 12, 2026 21:33

Delete test/powershell/Modules/DemoGitHubAnnotations.Tests.ps1

0366a8b

TravisEz13 marked this pull request as ready for review February 12, 2026 22:06

TravisEz13 requested review from a team and jshigetomi as code owners February 12, 2026 22:06

Copilot AI review requested due to automatic review settings February 12, 2026 22:06

Copilot started reviewing on behalf of TravisEz13 February 12, 2026 22:06 View session

adityapatwardhan approved these changes Feb 12, 2026

View reviewed changes

Copilot AI reviewed Feb 12, 2026

View reviewed changes

TravisEz13 enabled auto-merge (squash) February 12, 2026 22:12

TravisEz13 added BackPort-7.4.x-Consider BackPort-7.5.x-Consider BackPort-7.6.x-Consider labels Feb 12, 2026

TravisEz13 merged commit 4db414c into master Feb 12, 2026
48 checks passed

TravisEz13 deleted the copilot/add-comments-on-test-failures branch February 12, 2026 22:53

TravisEz13 mentioned this pull request Feb 12, 2026

[release/v7.4] Add GitHub Actions annotations for Pester test failures #26800

Merged

9 tasks

TravisEz13 added Backport-7.4.x-Migrated and removed BackPort-7.4.x-Consider labels Feb 12, 2026

daxian-dbw mentioned this pull request Feb 15, 2026

[release/v7.5] Add GitHub Actions annotations for Pester test failures #26836

Merged

9 tasks

daxian-dbw added Backport-7.5.x-Migrated and removed BackPort-7.5.x-Consider labels Feb 15, 2026

daxian-dbw mentioned this pull request Mar 9, 2026

[release/v7.6] Add GitHub Actions annotations for Pester test failures #26969

Merged

9 tasks

daxian-dbw added Backport-7.6.x-Migrated and removed BackPort-7.6.x-Consider labels Mar 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GitHub Actions annotations for Pester test failures#26789

Add GitHub Actions annotations for Pester test failures#26789
TravisEz13 merged 9 commits intomasterfrom
copilot/add-comments-on-test-failures

Copilot AI commented Feb 12, 2026 •

edited

Loading

Uh oh!

TravisEz13 commented Feb 12, 2026

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

TravisEz13 commented Feb 12, 2026

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

Uh oh!

TravisEz13 commented Feb 12, 2026 •

edited by unfurl-links bot

Loading

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Copilot AI Feb 12, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

	$annotationMessage = $message -replace "`n", "%0A" -replace "`r"
	$annotationMessage = $message -replace "`n", "%0A" -replace "`r", "%0D"

-                if ($filePath.StartsWith($workspacePath)) {
-                    $filePath = $filePath.Substring($workspacePath.Length).TrimStart('/', '\')
-                    # Normalize to forward slashes for consistency
-                    $filePath = $filePath -replace '\\', '/'
-                }
-            }
+                try {
+                    $workspaceFullPath = [System.IO.Path]::GetFullPath($workspacePath)
+                }
+                catch {
+                    $workspaceFullPath = $workspacePath
+                }
+                try {
+                    $fileFullPath = [System.IO.Path]::GetFullPath($filePath)
+                }
+                catch {
+                    $fileFullPath = $filePath
+                }
+                # Normalize separators for comparison
+                $normalizedWorkspace = $workspaceFullPath.TrimEnd([System.IO.Path]::DirectorySeparatorChar, [System.IO.Path]::AltDirectorySeparatorChar)
+                $normalizedFile = $fileFullPath.Replace([System.IO.Path]::AltDirectorySeparatorChar, [System.IO.Path]::DirectorySeparatorChar)
+                $comparison = if ($IsWindows) {
+                    [System.StringComparison]::OrdinalIgnoreCase
+                }
+                else {
+                    [System.StringComparison]::Ordinal
+                }
+                $workspacePrefix = $normalizedWorkspace + [System.IO.Path]::DirectorySeparatorChar
+                if ($normalizedFile.StartsWith($workspacePrefix, $comparison) -or $normalizedFile.Equals($normalizedWorkspace, $comparison)) {
+                    $relativePath = if ($normalizedFile.Equals($normalizedWorkspace, $comparison)) {
+                        ''
+                    }
+                    else {
+                        $normalizedFile.Substring($normalizedWorkspace.Length + 1)
+                    }
+                    if ($relativePath) {
+                        $filePath = $relativePath
+                    }
+                }
+            }
+            # Normalize to forward slashes for consistency (required by GitHub annotations)
+            $filePath = $filePath -replace '\\', '/'

Conversation

Copilot AI commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add GitHub Workflow Annotations for Pester Test Failures ✅

Summary

Implementation Checklist

Changes Made

Features

Example Output

Testing Performed

Benefits

Uh oh!

TravisEz13 commented Feb 12, 2026

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

TravisEz13 commented Feb 12, 2026

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

Uh oh!

TravisEz13 commented Feb 12, 2026 • edited by unfurl-links bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI commented Feb 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Copilot AI commented Feb 12, 2026 •

edited

Loading

TravisEz13 commented Feb 12, 2026 •

edited by unfurl-links bot

Loading