Red Team Research - Runspace Debugging

Tldr; Runspace debugging is a cool feature added into the .NET framework that allows developers to attach to a PowerShell Host (PSHost) process to debug a script. Once a process instantiates the PSHost, it never goes away, even if the runspace object is disposed. I submitted it as a bug to Microsoft and hopefully it will be fixed in the future. This can be used as a neat way to execute PowerShell code under someone else’s legitimate process or PowerShell session.

On the Red Team, we have two primary goals. We perform adversary simulations and conduct vulnerability research. In a nutshell, our team conducts vulnerability assessments where our activities mimic one or more tactics, techniques, and procedures (TTPs) of legitimate threat actors. Additionally, we use creative ways to understand the technology stack and processes that we are typically facing during any given assessment to achieve our objectives. We do all of this to challenge assumptions about security posture and help our defenders train. The results of these assessment lead to improving our controls, processes, and shared knowledge.

What is Runspace Debugging

Last year I remember seeing an interesting talk posted on Twitter from Lee Holmes (@lee_holmes). Lee is a Principal Security Architect with Microsoft. Among other things, he is a PowerShell guru and has been involved with many open source projects and security conferences wielding his PowerShell knowledge. The topic he had presented at PowerShell Conference EU 2019 was titled ‘HoneyBotting: Extracting confessions from client-side implants’. I highly recommend watching the recording (https://youtu.be/1S9YNJpktBM).

The basis of the talk was definitely geared towards defenders. Essentially, defenders could use PowerShell debugging to interact with malicious PowerShell implants and control what an adversary might be returned via command and control channel (he demoed with Empire). What caught my attention specifically was being able to interact with another PowerShell session and access object/variable values. This was the first time I had even heard of PowerShell debugging. I remember watching, pausing, and testing multiple times to really grasp how this worked and what it was doing under the hood.

A .NET project that uses the System.Management.Automation.dll and incorporates a runspace object from that namespace (or simply just running PowerShell), will invoke a PSHost object for the lifetime of the calling process. This has been used by processes to accomplish tasks using PowerShell code inline and external to the .NET assembly. When a runspace is instantiated, the PSHost object provides a remote (local inter-process communication or RPC over SMB) debugging capability over a single instance named pipe.

For what it’s worth, Microsoft may refer to this feature as PowerShell Runspace Debugging, but I removed PowerShell from my reference to it because of a bug I discuss later.

Practice 1

If you would like to follow along, here are some steps to understand what is going on.

• Open up PowerShell.exe on Windows (you can also use PowerShell_ISE.exe or pwsh.exe, but the results below may vary slightly).
• At the prompt, enter the following string and hit ENTER:

[System.IO.Directory]::GetFiles("\\.\pipe\")

• This command will effectively list all of the named pipes available on your system, pending adequate access of course.
• Your current PowerShell process will have an associated named pipe server with the following format:
  • PSHost.<StartTimestampTicks>.<ProcessID>.DefaultAppDomain.powershell
  • If you see multiple entries with this format, meaning multiple PowerShell sessions, enter $PID into your session to match your process to your named pipe.
• There will be some more commands later, so leave this session running.

So what does this mean? When you start PowerShell, by default a named pipe server or inter-process communication (IPC) listener is started. Why? There may be other reasons beyond my understanding at this point, but a great reason for this listener is to enable the remote debugging of your PowerShell script.

Now, what is a runspace? In simple terms and to the degree that I can explain is similar to process threading. In .NET applications, you have one or more Application Domains (AppDomains). AppDomains can provide isolation between functionally different segments of a process. Each AppDomain has one or more runspaces where work is carried out. As such, when the .NET Common Language Runtime (CLR) is initialized upon the start of a new .NET process, the Default AppDomain is started which in turn starts a default runspace. A task worker or runner section of the application may be launched in a separate AppDomain or split into multiple runspaces for the sake of multithreading the work.

The Bug

Later during an assessment, I decided to enumerate processes on a host we were operating from. In there we saw some custom binaries, in fact, custom C# binaries being executed as Windows services. Any time I see custom C# binaries, I get excited for two reasons.

First, custom applications are typically not given the scrutiny or visibility that commercial applications are given, leaving opportunity for vulnerabilities. Second, C# can be easily decompiled into next to exact source code with tools like dnSpy.

I decompiled one of the binaries and discovered this service was being used to install applications on the host. The way it was doing this was taking install parameters (strings the user controls) and passing them to a separate method. This second method then instantiated a runspace and passed the install parameters to PowerShell. Finally, the PowerShell and runspace objects were disposed and the method returned. I thought I might be able to somehow inject some code into the parameters, so I returned to the host to try some things. While on the host, I ran the following command using PowerShell:

Get-PSHostProcessInfo

The output returned two lines. The first line was my PowerShell process. The second was the custom binary Windows service. I thought this was really interesting because I hadn’t initiated the command injection testing. Determined to figure out what was being installed, I opened up the Windows Event Viewer to look at PowerShell logs. To my surprise, nothing had been executed recently and there appeared to be no active PowerShell processes other than my own actually running code.

I attempted to connect to the PowerShell session using Enter-PSHostProcess, but got an access denied error. After reading through the Microsoft docs on PowerShell debugging (https://docs.microsoft.com/en-us/powershell/module/Microsoft.PowerShell.Core/Enter-PSHostProcess and https://devblogs.microsoft.com/powershell/powershell-runspace-debugging-part-1/), I learned that the runspace being connected to either needs to be running in the same user context as the current user or the current user must be an administrator on the system. Given this, the process therefore must be running as someone other than me. This is primarily due to the permissions set on the named pipe server mentioned above. To verify this, run the SysInternals tool AccessChk against your pipe name from above. You’ll see the executing user and BUILTIN\Administrators have read and write access to the object.

Using a separate privileged account, I ran the same set of commands and this time was able to connect to the other process. Curious to know who was running the PowerShell Host I ran [System.Environment]::UserName which returned SYSTEM. At this point, I did see a 4104 event for the PowerShell script being running as SYSTEM under the target PSHost process.

I went back to the decompiled source code. I copied out the basic logic of the method that was using the runspace into a new project.

using System;
using System.Management.Automation;
using System.Management.Automation.Runspaces;
 
namespace RunPowerShell
{
    class Program
    {
        static void Main(string[] args)
        {
            CallPS();
            Console.WriteLine("Hit any key to exit");
            Console.ReadKey();
        }
 
        static void CallPS()
        {
            Console.WriteLine("Creating Runspace...");
            using (Runspace rs = RunspaceFactory.CreateRunspace())
            {
                Console.WriteLine("Creating PowerShell...");
                using (PowerShell pshell = PowerShell.Create())
                {
                    pshell.AddCommand("Get-Service");
                    Console.WriteLine("Invoking PowerShell...");
                    pshell.Invoke();
                }
                Console.WriteLine("PowerShell disposed");
            }
            Console.WriteLine("Runspace disposed");
        }
    }
}

I compiled the code and executed it as myself. The output was as expected: it ran the method, returned, and was awaiting user input at the ReadKey() invocation. Next I ran Get-PSHostProcessInfo and sure enough there was a runspace for RunPowerShell even though the method had executed, disposed of the object and returned to Main. In my haste, I would suggest that the issue is that the named pipe server object is not properly disposed leaving the PSHost active/orphaned inside the process until the process is terminated (once a PSHost process, always a PSHost process).

Luckily, many if not all PowerShell components are now open-sourced on GitHub (https://github.com/PowerShell/PowerShell). I read through lines and lines of the source code, but eventually gave up. Open sourcing PowerShell, by the way is a great opportunity to learn how components of PowerShell work. I had considered submitting this as an issue on GitHub, but was slightly fearful this could be abused.

Even though this didn’t feel like an actual security vulnerability, I submitted it through Microsoft Security Response Center (MSRC). After analysis, MSRC determined it did not qualify as a security vulnerability, although it was handed off to the PowerShell team who was going to add it to their backlog to be investigated. Being a huge PowerShell nerd, if this does get fixed in a future release, I will be very happy.

To be fair, I call this a bug because either the PSHost is not properly disposed or the developer documentation on PSHost does not mention this behavior. If you have projects that include the System.Management.Automation.dll library and explicitly use the Runspace or PowerShell objects, be aware of the possibility that this PSHost is available for remote debugging beyond the developer’s intended usage of the runspace.

Named pipe server security from PowerShell project on GitHub

Offensive Use Case

A few months passed and my team was performing another assessment. I had gotten into a situation where we had found a set of credentials which gave us administrator access to a server. The security administrators of this system had done a lot to lock down what the account could access and how the account could be used. The only logon type that was permitted was Network logon (type 3) and everything else was explicitly denied. Using PowerShell remoting and WinRM I could connect to the server with the account and run commands in a remote interactive PowerShell session.

I started enumerating the box to see what I could see. Being familiar with PowerShell debugging, I ran my favorite runspace enumeration command (Get-PSHostProcessInfo). There happened to be two other processes with PSHosts, both of them PowerShell_ISE. They were running under two different users. I attempted to connect to both processes and was successful at doing so. The first account was not very privileged, but nonetheless allowed me to move off of the system and access things the restricted account could not. The second account was another administrator of the system. It had access to other privileged systems as well.

Here’s the cool thing about this: the second account was a member of Protected Users. If you aren’t familiar with this, go check it out. It’s 100% worth your time putting your administrator accounts in here. Long story short, among many other things, this prevents the Kerberos double hop. Another way to say that is it prevents lateral movement off the original system where the Kerberos ticket/session originated by only allowing it to be valid for that session.

https://docs.microsoft.com/en-us/windows-server/security/credentials-protection-and-management/protected-users-security-group

Referencing the Microsoft documentation on legitimate uses for PowerShell runspace debugging, I used the following commands to access the active runspace of these system administrators' PowerShell_ISE sessions:

Get-PSHostProcessInfo
Enter-PSHostProcess -Id <PID>
Get-Runspace
Enable-RunspaceDebug -RunspaceId <RunspaceID>
Debug-Runspace -Id <RunspaceID>
Get-Variable *

After running these commands, I had access to the script that was currently being executed by the privileged account as well as all the values being stored in memory by the script.

Effectively what I can do now is run PowerShell code under a legitimate process using another highly privileged account. It’s much like code injection, but’s done using native tools on the platform. Additionally I could use it to connect to a compromised account’s existing PowerShell session and execute code. This can give you access to values stored in objects such as secure strings ($creds.GetNetworkCredential().Password). Also worth noting, once the process is terminated, the Host goes away with it, so another thing I might do is spawn a new PowerShell process (start powershell) to live in beyond the lifetime of the original process.

Practice 2

To simulate this, return to that PowerShell session from up above. Run the following commands to setup this session to be debugged:

$myVar = "DebugMe"
$creds = Get-Credential "myFakeAccount" # at the prompt enter a fake password
for($i=0; $i-lt100; $i++){$i; Start-Sleep -s 120}

Output in PowerShell session 1

Start a new PowerShell.exe session. In the new PowerShell window, use the above list of commands starting with Get-PSHostProcessInfo and continuing on replacing the PID and RunspaceID with valid values from your environment. You will notice once your first session runs another command such as completing the sleep and incrementing the counter, it will hit a breakpoint. You can look at values such as the $i counter or $creds to see the valid data is available from your debugging session in the newer PowerShell process.

Output in PowerShell session 2

Detection and Prevention

As I stated earlier, this is not an actual vulnerability according to Microsoft. When I was researching the use of runspace debugging, there aren’t a lot of stackoverflow articles or other forums mentioning developers actually using this. There also didn’t seem to be a lot of research by the security community on it either. Because of these facts, simply monitoring the use of the mentioned PowerShell debugging cmdlets in PowerShell logs might be a good start. To prevent certain activities, an administrator could leverage Constrained Language Mode or Just Enough Admin (JEA) focusing on the PowerShell commands and underlying APIs. Another way you can determine if someone is connected to a PSHost process is by looking at the current runspaces by running Get-Runspace. If you see RemoteHost as one of the runspace names, there’s a good chance this PSHost is being debugged.

Runspaces without and with a debug session attached

I am not an expert at either the detection or prevention of this technique, so I would suggest figuring out what works best in your environment with the experts you trust. That being said, if an adversary wanted to get around those controls, there are other methods for accessing these PSHost processes that doesn’t use the native PowerShell runtime. In this case, there may be more advanced ways of detecting the abuse of this such as Microsoft Sysmon event ID 18 (PIPE CONNECTED) events. The tools and data you have access to and the creativity of your experts are your only limits.

Sysmon Event ID 18

Until Next Time

PowerShell has been used and abused for years. Runspace debugging, at least to me, is a relatively new (PowerShell and Windows Management Framework version 5) concept that doesn’t appear to have the exposure it deserves. I learned two things from this research that I wanted to share here. First, the bug in the Automation library leaves a PowerShell Host available from the executing process even after the runspace has been disposed. Second, runspace debugging is a great way to run PowerShell code in another process under the radar and without the need for anything other than your own PowerShell code. There will always be challenges depending on the environment you find yourself in, but taking advantage of existing tooling and knowledge you have available are key.

To learn more about technology careers at State Farm, or to join our team visit, https://www.statefarm.com/careers.