Malware today is so numerous and diverse that security professionals have known for some time that signature based solutions would no longer be able to cut it alone. Not only are there too many new malware files each day, some of them are able to change their shape and signature as they go along. But,if you can’t recognize something by its looks, you might be able to categorize it by its behavior. This is where methods like HIPS (Host Intrusion Prevention System) come into play.
By definition HIPS is an installed software package which monitors a single host for suspicious activity by analyzing events occurring within that host. In other words a Host Intrusion Prevention System (HIPS) aims to stop malware by monitoring the behavior of code. This makes it possible to help keep your system secure without depending on a specific threat to be added to a detection update.
Historically HIPS and firewalls are closely related. Where a firewall regulates the traffic to and from your computer based on a rule set, HIPS do more or less the same, but for the major changes made on your computer.
The major changes that can be allowed for a program when creating a rule-set
HIPS solutions protect the computer against known and unknown malicious attacks. In case of attempted major changes by a hacker or malware, HIPS blocks the action and alerts the user so an appropriate decision about what to do can be made. What does the HIPS consider major changes? I made a list of possible major changes and why malware might want to make them. The list is far from complete, but more like a bare minimum of what your HIPS should be guarding:
Take control of other programs. For example sending a mail using the default mail client or sending your browser to a certain site to download more malware.
Trying to change important registry keys, so that the program starts at certain events.
Ending other programs. For example your virus scanner.
Installing devices or drivers, so that they get started before other programs
Interprocess memory access, so it can inject malicious code into a trusted program.
What can you expect of a good HIPS?
At the very least it should have the power (authority) to stop active malware. If it is unable to stall another program while waiting for your decision, the battle is already lost. Additionally it should have a basic set of rules that any user can apply until he is more familiar with the software and/or the need for more elaborate rules emerges. Adapting or creating new rules should be possible (there are always exceptions to be made) and it should be user friendly to do so. For one thing it has to be very clear to the user what the consequences of his changes are, or he will find himself wondering at some point why this or that no longer works. For these cases and other help, I would also check out if there are forums (or other places) where you can find help in individual cases. A knowledge base is not always enough to find all the answers.
The normal method of a HIPS is runtime detection. It intercepts actions when they occur, but some HIPS also offer pre-execution detection. This means that the nature of an executable is analyzed before it runs, to check for suspicious behavior.
Are there any risks?
Risks associated with HIPS are false positives and wrong user decisions. HIPS respond to certain changes that other software wants to make on your system. For example any HIPS will keep an eye on the registry key HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run and others like that, from where programs are started automatically when Windows boots up. But obviously there are many legitimate programs that use this key as well. So when a change is made to the content of that key (an extra value is added), the user will be presented with a choice, Block or Allow. For this key there are many online resources on which you can base an informed choice, but most users will hit Allow, especially if they are in the process of installing something. Some HIPS will let you know what other users have decided in this particular case, but especially when numbers are still small, this can be deceiving and it is not really a decision based on relevant information. You are only hoping the majority of the users before you was right. The system is only as good as the responses of the user to the popup alert. Even if the HIPS software correctly identifies a threat, the user may inadvertently approve the wrong action and the PC could still become infected.
Conclusions: HIPS can be a valuable part of a layered defense, but I would advise to add at least one detection based security solution. While HIPS should be for everyone, it requires at least a decent knowledge of computing to use them effectively.