No malware author wants an analyst snooping around their code, so they employ tricks to inhibit analysis.
Along with visualization technology like VMware, debuggers are also targeted by malware. This is because if a debugger is attached to the running malware, it's more than likely being analyzed.
While nothing presented here is really new, it might help you to identify what's happening if you find yourself in one of these traps.
1. VMware I/O port
VMware tools is a software package users can install on their VMware virtual machines to increase their functionality. For example, one thing it allows for is drag-and-drop functionality between the host and guest, and vice versa. Competitors such as Oracle Virtualbox offers a similar package for their virtual machines known as Virtualbox Guest Additions.
VMware Tools uses a special I/O port to communicate data to/from the host and virtual machine. Malware takes advantage of this functionality and implements it using only a few lines of Assembly code.
If this code is running under a VMware virtual machine, it will execute successfully, and place the magic number into the EBX general purpose CPU register. This is a very efficient way of telling if a system is running inside a VMware virtual machine.
2. Virtual PC instructions
The x86 instruction set has a limited number of instructions that your CPU can understand. Every now and then, you may encounter a situation where your CPU doesn't understand an instruction, and therefore won't be able to process it correctly.
The above instructions are part of an anti-debugging technique for Microsoft Virtual PC, as was documented by researchers at Symantec. Here is another view using Ida Pro that gives us another clue we're dealing with a Virtual PC instruction that is otherwise illegal in normal processors:
If the target system is not running Virtual PC, an exception is generated and captured by the malware. However, if Virtual PC is running, no exception would be generated indicating that the user is in fact running Microsoft Virtual PC.
3. Check Descriptor Table Registers
There is one Local Descriptor Table Register (LDTR), one Global Descriptor Table Register (GDTR), and one Interrupt Descriptor Table Register (IDTR) per CPU. These have to be moved to a different location when a guest operating system is running to avoid conflicts with the host. Ocassionally, you'll see malware check for these by using the ASM instructions SLDT, SGDT, and SIDT to get the value of these registers.
This is perhaps one of the easiest identifiable anti-debug methods, where the malware scans its own process to look for particular dynamic-link libraries (DLLs) that may be associated with analyst tools. The targeted dlls here can be anything related to debuggers or tools that may inject special DLLs into the malware's process (i.e. sandboxes).
5. Product ID check
Checking the Window Product ID found within the registry can yield clues to what kind of System you are running. In the past, many Sandboxes used hardcoded product IDs in their Operating System environment. While most Sandboxes and other automated analysis systems use randomly generated product IDs, you can still occasionally find these checks.
These are just a few examples of tricks used to inhibit analysis of malicous code. Keeping aware of these methods can make your analysis process more effective.