Remotely create a combined trace

Why combined remote tracing?

Sometimes the machine you need to investigate is physically inaccessible, hung, or its GUI has become unresponsive. In such cases, you can still capture rich diagnostic data—kernel events, network traffic, and user‑mode system calls—by driving tracing entirely from a separate “Admin PC.” Combining multiple trace types into one workflow lets you correlate, for example, CPU scheduling stalls with network latency or user‑mode I/O activity.

1. Preparing Your Admin PC

Your Admin PC acts as the controller: it hosts the tracing tools, launches them remotely, and later pulls the data for analysis.

1. Download Procmon (Process Monitor)
Procmon is part of Sysinternals and lets you capture file, registry, thread, and DLL activity in real time. Grab it here: https://live.sysinternals.com/Procmon.exe

2. Download and install the Windows ADK
The Windows Assessment and Deployment Kit (ADK) includes the Windows Performance Toolkit (WPT), which provides xperf.exe for high‑resolution kernel and user‑mode ETW tracing. Get it from here: https://aka.ms/adk

3. Verify installation of ADK and set path variable
After installation, confirm the WPT path. From an elevated PowerShell on your Admin PC, run:
$wpt = (Get-ItemPropertyValue -Path "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows Kits\Installed Roots" -Name "KitsRoot10") + "Windows Performance Toolkit" This gives you the full folder path where xperf.exe and its DLLs reside.

Note: Always leave this Powershell window open. Use it for all commands following later in this post.

2. Staging the Remote (Investigated) PC

To avoid needing physical access, we’ll copy the tracing binaries over PowerShell Remoting:

1. Establish a remote session
On your admin PC, switch to the open Powershell window and run following command, replacing TargetComputer with the machine’s name or IP:
$remote = New-PSSession -ComputerName targetComputer

2. Copy Tracing Tools
Using the session, push xperf.exe, perfctrl.dll, and Procmon.exe into the Remote PC’s C:\Windows\System32 (so they’re on the PATH and can be launched by any user):
Copy-Item "$wpt\xperf.exe" -ToSession $remote -Destination C:\Windows\System32 Copy-Item "$wpt\perfctrl.dll" -ToSession $remote -Destination C:\Windows\System32 Copy-Item "$HOME\Downloads\Procmon.exe" -ToSession $remote -Destination C:\Windows\System32

3. Starting multiple Traces

Back in your Admin PC’s elevated PowerShell, enter the remote session, then kick off three concurrent traces:

1. Enter the Remote PowerShell Session
Enter-PSSession $remote

2. Kernel & User‑Mode ETW Trace with xperf
This captures kernel dispatcher events, context switches, profile sampling, process creations, and image loads:
& xperf -on Diag+Dispatcher+Profile -stackwalk Profile+CSwitch+ProcessCreate+ImageLoad -buffersize 1024 -maxfile 1024 -start UserSession -on Microsoft-Windows-Win32k+Microsoft-Windows-WinLogon
– on Diag+Dispatcher+Profile: Enables diagnostic, dispatcher and profiling ETW providers.
– stackwalk: Records stack traces for profiling and context switches
– start UserSesstion: Creates a separate user-mode ETL file.

3. Network Trace with netsh
Captures low‑level network packets and connection events:
& netsh trace start capture=yes scenario=netconnection maxsize=2048 tracefile=c:\net.etl

4. Process Monitor (Procmon) Trace
Gathers file, registry, and process/thread activity:
Start-Process "Procmon.exe" -ArgumentList "/accepteula /backingfile C:\Procmon.pml /quiet"
– accepteula: Automatically accepts the license.
– backingfile: Destination .pml file.
– quiet: Suppresses the GUI window.

At this point, all three tracing engines are running on the Remote PC, writing to:
C:\Procmon.pml (Procmon log)
C:\kernel.etl (xperf kernel data)
C:\user.etl (xperf user data)
C:\net.etl (netsh network ETL)

4. Reproduce the Issue

With tracing in progress, switch to the Remote PC (via RDP, KVM, or in‑person if possible) and trigger the problem scenario. Whether it’s a hung application, a service crash, or a network failure, perform the exact steps that cause the issue so the traces capture the relevant events.

5. Stopping Traces & Merging Results

Once you’ve reproduced the issue, return to your Admin PC’s PowerShell and terminate each trace in turn:

1. Terminate Procmon
Start-Process "Procmon.exe" -ArgumentList "/terminate"
This cleanly finalizes the .pml file.

2. Stop the xPerf session
& xperf -stop UserSession -stop
This writes out kernel.etl and user.etl.

3. Stop the Network Trace
& netsh trace stop
This can take a while until all network events are merged.

4. Merge xPerf and NetSH ETLs
& xperf -merge C:\kernel.etl C:\user.etl C:\net.etl C:\trace.etl
To produce a single trace (trace.etl) combining kernel, user, and network data.

Tips & Best Practices

• Firewall & Remoting: Ensure WinRM is enabled on the Remote PC (Enable-PSRemoting -Force) and firewalls allow WS‑Management (TCP 5985/5986).
• Disk Space: Tracing can generate large ETL files—ensure both machines have sufficient free space.
• Separate ETLs: Splitting kernel/user sessions (-start UserSession) reduces file sizes and makes merging more flexible.
• Time Synchronization: Make sure both PCs’ clocks are synced (e.g., via AD) so ETW timestamps align.

In Enterprises,remoting is configured with Group Policy Objects (GPO). Enable following GP settings:

• Computer Configuration\Policies\Windows Settings\Security Settings\System Services\Windows Remote Management: Automatic
• (optional) Computer Configuration\Policies\Windows Settings\Security Settings\System Services\Windows Remote Management: Automatic
• Computer Configuration\Policies\Windows Settings\Security Settings\Windows Defender Firewall with Advanced Security\Inbound Rules: Windows Remote Management (HTTP-In): Allow