This article is meant to give you an overview of how sandboxes are used and why malware writers try to avoid their code to run on them. It will also list some methods that are in use to make malware sandbox-sensitive. Without going into much detail it will also give you an idea of the countermeasures that are possible.
What's a sandbox and what's a VM?
A sandbox is a secluded environment on a computer, where you can run untested code or malware to study the results without having any ill effects on the rest of your software. A virtual machine is the most commonly used example of a sandbox, since it emulates a complete computer, called a guest operating system, on the main machine (called the host).
How do malware researchers use VMs?
In malware research sandboxes are used to study the behavior of malware. As you can imagine this is a lot safer than doing it on a computer that you may need for other matters. And it is not only safer, it's also a lot quicker to restore an image of a previous state of the guest machine (a matter of seconds most of the times) as opposed to the worst case scenario for a real computer, re-formatting and re-installing the software that you need (which could take hours).
Why do the bad guys try to prevent that?
In the beginning this may have started as a way to discourage amateur researchers and generally make investigations harder for everyone. But in the ongoing "arms race" between malware writers and security professionals, malware writers started to add routines to their programs, that check if they are running in such a virtual environment. When they detect they are running in a sandbox or on a VM they throw an error or just stop running silently.
These days they are mainly trying to avoid automated analysis, which is used by many security software vendors. If the sample does not perform any noticeable actions, the sample may be discarded as one of many samples that are not considered malicious.
If they are successful in doing so they will be able to postpone detection by these vendors. And every day a commercial malware can postpone this, it allows it to bring in more money for the creator.
And how do the malware writers go about trying to avoid sandboxes?
There are a few methods for a process to determine if it is running in a virtual environment. The most obvious one would be to check for running processes that a VM uses, like for example VBoxTray.exe (VirtualBox). A similar method is to compare the services that are running to a list of known services in use by virtualization software. Or check for drivers like vmmouse.sys (VMWare).
Another method would be to check for virtual hardware like a network interface. Like any network interface, they are assigned a unique MAC address that includes the manufacturer's identification number. Given the limited number of manufacturers checking for these processes and interfaces does not require a lot of code. This could change in the future however.
The same kind of checks can be done for GUIDs and other unique identifiers used by virtualization software.
The next level is a bit more complicated. It has to do with the method that is used to avoid conflicts between the descriptor tables the processor uses for the host and the client. You do not want an interrupt for the client to land at the host and vice versa. To avoid this the IDTR, GDTR and LDTR for the guest OS are moved to a different place to separate them from the ones the host OS is using.
Another action often performed is to check for extra debuggers. These are not necessarily a sign for running a VM, but do often yield the desired result since researchers use debuggers to follow "target programs". The program that wants to avoid being debugged checks the port number for the debugger of the process. A value other than 0 indicates that the process is being run through a debugger handled by the user.
Then there are some techniques in use that specifically try to fool the automated threat analysis methods. One of them is to lay dormant for a while. As you can imagine an automated threat analysis tool does not wait forever, they have more work to do. At an estimated million new variants each day, servers can’t wait for many minutes when they can normally create a full report in a few minutes. So the malware waits a while before it actually starts running its main routine. Or it waits for a certain API call that an automated threat analysis machine would normally never make, for example the malware waits for the mouse to be used.
Although there are some methods that can be used to circumvent VM-awareness, these typically require knowledge and skills that most amateur researchers do not have. Examples are editing source code of the sample to skip the test or overwrite the test with instructions that have no influence on the rest of the code (NOP instructions).
VM processes can be renamed and instructions redirected, but this could influence the results and interfere with the host computer. So, usually it is much easier and more convenient to use a "real" machine and flatten it after the testing is done, by restoring an earlier hard drive image.
Sources where you can find the more technical details: