As mentioned in a recent blog post, our team is once again offering in-person training, and we have substantially updated our course for this occasion. Over the next several weeks, we will be publishing a series of blog posts, offering a sneak peek at the types of analysis incorporated into the updated Malware & Memory Forensics training course.
Introduction
To begin the series, this post discusses a new detection technique for hidden services on Windows 7 through 11. Since not all readers will be familiar with hidden services and the danger they pose on live systems, we will start with some brief background. We will then walk through how services.exe stores service information, and how we can recover it in an orderly manner. This will lead to how we developed two new Volatility 3 plugins to help automate detection of hidden services.
The power of these plugins will be showcased against the powerful GhostEmperor APT rootkit that was discovered in the wild by researchers at Kaspersky. GhostEmperor employs a kernel mode rootkit and a userland DLL to maintain persistence and control the victim system. This DLL operates as a service that is hidden from live analysis and DFIR triage tools, and it interacts directly with the rootkit driver in kernel memory. As will be demonstrated, by automatically detecting the hidden service of GhostEmperor through memory analysis, we can quickly find the rest of its components, including those hidden on the live system.
Services Background
Services are a powerful feature of Windows that allow malware to run in one of three possible forms. The first allows malware to register a DLL that will be loaded into a shared svchost.exe process, hiding it amongst other DLLs loaded inside the same process, as well as the many svchost.exe instances that run on a normal system. The second form allows malware to run as its own process. The third, and most dangerous, form is when malware creates a service to load a kernel driver (rootkit).
When services are created and started using standard methods, a few artifacts are left behind for investigators to find. The first is a set of registry keys and values under CurrentControlSet\Services\<service name>. The second is the service’s entry within a linked list maintained by services.exe. This list is enumerated when system APIs, such as EnumServiceStatus{A,W,Ex}, and tools, such as sc.exe query, are used to enumerate services on the running system.
Given the power of services, malware often abuses the ability to create or hijack services for its own purposes. This leads to the inspection of services on a running system by endpoint detection and response solutions (EDRs) and threat hunting teams to look for any suspicious signs. To avoid detection while keeping a service active, malware has historically targeted both sources of artifacts—the registry keys and the services.exe list—with registry keys being targeted in two ways: deleting or hiding them.
In the first approach, malware will delete its registry keys while running, and then rewrite them before system shutdown or reboot. This has a major disadvantage though, as sudden system crashes or service stops prevent the malware from re-registering its persistence.
This deficiency led to the current approach malware takes, including by GhostEmperor, which is to simply hide its keys from the running system. The following screenshot shows Kaspersky’s report on the malware’s approach:
As discussed in Kapersky’s report, the CmRegisterCallback usage effectively allows the malware to hide its service’s keys from tools on the live system. Detecting this malicious callback is possible with Volatility’s callbacks plugin though, and there are also EDRs capable of enumerating callbacks from within kernel memory. To avoid these EDRs, some rootkits found during recent APT campaigns have implemented a completely new method of registry key hiding, known as GetCellRoutine hijacking, that we will cover in an upcoming post along with another new Volatility 3 plugin.
Beyond the registry, malware also wants to hide its malicious service from tools on the live system that query services.exe to enumerate running services. To accomplish this, malware will inject code into the services.exe process, and then unlink the malicious service of interest. This will effectively hide the service from live DFIR triage tools and built-in Windows commands. It’s the detection of these unlinked services using new memory forensics capabilities that we cover further in this blog post.
| Note: Chapter 12 in The Art of Memory Forensics is devoted to discussion of Windows services, ways malware abuses them, and several historical methods of detection. If you would like a complete treatment of the subject after reading this blog post, then we suggest reading this chapter. |
Detecting Unlinked Services
As mentioned, a wide variety of malware samples will unlink their malicious services for anti-forensics purposes. The following screenshot from the Kaspersky report on GhostEmperor describes this for the malware sample:
In our analyzed memory sample, the name of the hidden service is “msdecode”, which is one of the possibilities listed in the report.
The Art of Memory Forensics details one method for detecting unlinked services with Volatility. This method relies on scanning physical memory for services records, and then drawing a dot graph of how each service is linked to other services. This linkage is based on the previous and next pointers of the doubly linked list. During normal operations, each service record should have one service’s forward pointer referencing it, and one service’s backwards pointer referencing it. In the case of an unlinked service, the hidden service will have no services that reference it. The following image from The Art of Memory Forensics shows how this detection logic is applied to an unlinked wscsvc service:
As can be seen, all services other than wscsvc have previous and next pointers (green and red arrows) pointing to them from other services. This is a direct visual indication that the wscsvc service is unlinked.
Unfortunately, this detection method is no longer viable for two main reasons. First, the Connected Devices Platform subsystem creates a wide variety of temporary services during system operations. This means that smear (changes to memory during acquisition) will often cause these services to appear as unlinked when using the method that detected wscsvc. The second reason is that scanning physical memory will find copies of service records relating to services that have since changed state (restarting, start<->stop). The ability to recover these historical records is a powerful aspect of memory forensics, but unfortunately clutters the results for this particular use case, as the historical records are no longer tracked by services.exe.
To illustrate these issues, we created a Volatility 3 plugin, svclinks, that reports a text-based version of the visual graph. We ran svclinks against our memory sample infected with GhostEmperor. This plugin reports only services that it thinks are unlinked, and the results are shown below:
As can be seen, while our target service, Msdecode, is in the reported list of unlinked services, so are several other services, all of which are false positives. Given the inability to rely on our old method, we needed to develop a new one.
Replicating services.exe‘s Enumeration of Services
Knowing that the live system uses the list inside of services.exe to report services, along with the fact that malware takes great effort to hide from this list, we chose to use it as a source for detecting unlinked services. This detection relies on cross-comparing the services found through scanning, which Volatility 3 already supports, versus the list walking performed in our new plugin. This is similar to using pslist and psscan (or psxview) to detect unlinked processes within the kernel.
Over the years, Microsoft has made substantial changes to the methods services.exe uses to track services, but, luckily for us, we only have to be concerned with changes in the data structure layout and the name of global variables. The following screenshots show how the services.exe database is declared across Windows versions:
Windows 10+:
Windows 7:
For data structure layouts, Volatility 3 already contained definitions for most of the types needed. All we had to add was the CServiceDatabase type and the offset to the first service record. Luckily, this was at a constant offset for all versions tested.
Enumerating the Service List in Volatility 3
To automate detection of unlinked services, two Volatility 3 plugins were developed. The first, svclist, locates and then enumerates the list of services maintained by services.exe. The second, svcdiff, compares the services obtained from scanning with the services obtained from walking the list. We will now discuss how these plugins are implemented with several screenshots of code. If you would like to read a nearly line-by-line breakdown of implementing a Volatility 3 plugin that performs similar actions, please see our post on detecting the skeleton key attack of Mimikatz.
Finding the Service Database
Obtaining the address of the service database inside of a particular memory sample is easy, since Volatility 3 supports automatic symbol resolution through PDB files. This tells us our plugin precisely where to find the database within the memory sample.
To start this recovery, Volatility’s process enumeration API is used to find the _EPROCESS object for services.exe. Next, the following code is used to automatically download and parse the PDB file for the executable, and then search the variations of the service database’s symbol name:
The end result of this code is that the svclist plugin will automatically know where to find the services database, which then tells the plugin how to find the beginning of the list.
Enumerating Services from the List
Once the list is found, Volatility’s traverse API for services can be used to walk the list; svclist then has little work left to do, as the existing svcscan plugin already contains a get_record_tuple API that gathers the information about a service (name, path, PID, etc.) to report to the analyst:
Using this, the output from our plugin then looks the same as when svcscan runs.
Detecting Unlinked Services in Volatility 3
Our detection of unlinked services in the new svcdiff plugin is based on comparing the set of services generated by the svcscan plugin and our new svclist plugin. In particular, each of these plugins is programmatically run, and then the names of any service found through scanning—but not through list walking—is reported.
By keying in on the name, we work around the issues found when linked-list pointers are used. This fix works because even if a service is stopped and restarted (which creates multiple data structures in memory), the name will be the same between runs. The name-based approach also removes the chance of false positives from the temporary services generated on Windows 10+.
The following screenshot shows the core of this plugin and how easy it is to leverage existing APIs in Volatility 3 to produce powerful new capabilities:
In this code, the services_scan API is first used to gather the names of services based on scanning. As shown in the get_tuple_record screenshot, the name of the service is the sixth entry. Next, service_list from our new plugin is used to gather services like services.exe does on the live system. Finally, a simple set difference is used to determine names found from scanning that were not found in the list. These are then reported to the output rendering API.
Detecting Ghost Emperor
With our new plugin available, automatically detecting GhostEmperor’s unlinked service is as simple as one Volatility invocation:
In this invocation, svcdiff reports only one service, Msdecode, which we know is the one hidden by GhostEmperor.
Exploring the Hidden Service
With this information in hand, we can investigate further by determining other components and actions of this service. To start, we can look at the list of DLLs inside of the process (reported as PID 4756 by svcdiff):
In this abbreviated output, we see DLLs inside of system32, as well as the msdecode.dll of the malware. By applying the –dump option to dlllist, the plugin will extract all of a processes DLLs to disk. Looking at the strings output of this extracted file shows the name of several APIs used for gathering sensitive system information and anti-forensics purposes:
The extracted DLL file can then be loaded into your reverse-engineeering (RE) tool of choice, scanned with YARA signatures, and other static analysis techniques.
Kernel Mode Components
After examining loaded DLLs, we can then examine the handles of the process to determine which system resources it is accessing. Since we know a kernel rootkit is involved, we search for any references to Device files within the handles output. Device files are created by drivers to allow userland processes to “speak” directly with the driver. This is the most commonly used interface by rootkits to allow the controlling process to specify filenames, registry keys, and processes to hide, as well as actions like enabling privilege escalation.
Looking at the Device files being accessed by the Msdecode service process shows us an interesting entry to a device named dump_audio_codec0; the other entries are present on all Windows systems:
Attempting to investigate dump_audio_code0 further instantly shows that it is malicious in nature. The following screenshot shows the output of the modules and driverscan plugins of Volatility 3 while searching for the driver:
As seen, the driver does not appear in modules output, which only happens when anti-forensics techniques are used. This is verified by the output of driverscan that shows the module’s base address and size have both been set to “0”. This is a common anti-forensics technique to hide a module on a live system and prevent its direct extraction from memory.
This technique is, in fact, so common that Volatility has a special-purpose plugin called drivermodule to detect discrepancies between module and driver data structures:
In this output, two modules are reported. The first, RAW, will trigger in all nearly all memory samples but, as seen in the first column, it is reported as a known exception. However, dump_audio_code0 is not, and as we verified multiple times already, this driver is definitely worthy of deep investigation.
Between the usage of our new svclist plugin, along with the drivermodule plugin, we have directly detected both the userland and kernel components of the rootkit, and we have done so without any existing IOCs specific to GhostEmperor. As demonstrated, memory forensics continues to be a necessary component to accurately detect modern rootkits and malware.
Conclusion
In this blog post we have demonstrated a new memory-forensics technique to detect hidden services in a smear-resistant manner. Given the number of malware samples that hide services from the live system, as well as the danger posed by these services, it is essential that malware can be detected in a reliable manner.
If you have any questions about this blog post, please let us know! You can email us, or find us on Mastodon and Twitter. We also have our own Slack Server. If you enjoyed this content, then be sure to check out the announcement of our updated training class. During the course, students are taught how to detect modern malware, such as the sample discussed in this blog post, as well as gain significant hands-on experience through many real-world labs.
Finally, we will be presenting new research on triaging modern Windows rootkits at BSidesCharm in Baltimore in a few weeks, so please come say hello if you will be there!
(c) Volatility Labs
