How does Data Execution Prevention work?

Data Execution Prevention aims to protect your computer by making it harder for hackers to silently execute their programs in your PC.

As Windows runs, its Virtual Memory Manager maps addresses in RAM to locations on the hard disk (in the pagefile or swapfile). At the same time, hardware DEP inserts a special bit into the disk version of an address, marking it as non-executable.

If a hacker program attempts to write code to such a location and then execute it, a DEP-enabled processor detects the exploit and registers an error. If so, Windows can shut down the problem application or, if the hacked code is in an area used by Windows, halt a portion of the operating system itself.

Windows XP Service Pack 2 (SP2) has a software-only version of DEP, which is not as effective as the hardware version. Fortunately, Vista provides support for both software DEP and hardware DEP. In either case, you'll want to turn on those DEP settings that you can benefit from. Vista users should read on, while XP users can skip down to the section entitled "Turning on DEP."

Does my system support DEP?

Follow the steps below to find out if the processor in your Vista computer supports hardware DEP:

Step 1. In Vista's Windows Explorer application, launch the System Properties dialog box by right-clicking Computer in Explorer's folder list.

Step 2. Choose Properties, or launch the System icon in Control Panel's System and Maintenance category.

Step 3. Click Advanced System Settings in the task bar on the left.

Step 4. Click Continue, if prompted by User Account Control.

Step 5. Under Performance, click Settings.

Step 6. In the Performance Options dialog box, click the Data Execution Prevention tab. If your processor supports this feature, a sentence to that effect appears in the lower part of the dialog box.

Here's a fast way to get to the same dialog box using only the keyboard, with minimal mouse clicking:

Step 1. In Vista, press Win+R to open the Run dialog box.

Step 2. Type SystemPropertiesDataExecutionPrevention and press Enter.

Step 3. Click Continue, if prompted by User Account Control.

Are all of my applications using DEP?

As the Performance Options dialog box suggests, DEP is turned on by default for most Windows services and programs - but not all. Vista users can see which applications aren't covered by taking these steps:

Step 1. Right-click an empty area of the taskbar and choose Task Manager (or press Ctrl+Shift+Esc).

Step 2. Click the Processes tab and choose View, Select Columns.

Step 3. Scroll to the bottom of the Select Process Page Columns dialog box and check Data Execution Prevention.

Step 4. Click OK.

The new column shows you which processes have DEP enabled (most of them) and which do not - notably Explorer (explorer.exe) and Internet Explorer (iexplore.exe). If you happen to have Windows Media Player (wmplayer.exe) or Outlook 2007 (outlook.exe) running, you'll notice DEP is disabled for these applications as well. You may also see some IE plug-ins listed here, like Java (jusched.exe) or the Google toolbar (GoogleToolbarNotifier.exe).


Figure 1. Windows Task Manager can show you which applications are using DEP.

If DEP is so useful, why is it disabled for important applications like Outlook 2007 and IE 7? The answer is that many developers disable DEP to maintain backward compatibility with other products, such as add-ons or plug-ins. For example, although plug-ins such as Adobe's Acrobat Reader and Flash Player now work with DEP enabled for IE, as of this writing, the Google toolbar and Sun Microsystem's Java plug-in do not.

How to turn on DEP

Both Vista and XP let you turn on DEP globally, while allowing you to make exceptions for applications that have problems. To do that, you need to return to the Performance Options dialog:

In Vista, click Start, type SystemPropertiesDataExecutionPrevention, and press Enter. Click Continue in the User Account Control dialog box.

In XP, click Start, Run, then type sysdm.cpl and press Enter. Click the Advanced tab. In the Performance box, click Settings. Click the Data Execution Prevention tab.

In both XP and Vista, select Turn on DEP for all programs and services except those I select.

In Vista only, take time now to specify a few of the programs you saw listed in Task Manager earlier to keep DEP disabled for them. To do that, click Add and browse for the .exe file of a program you know normally does not use DEP (for example, explorer.exe, wmplayer.exe, outlook.exe). Select the filename and click Open. Click OK to acknowledge the risk of turning off DEP for that application. Repeat for each application that normally doesn't use DEP.

The strategy here is to enable DEP for these applications one at a time over an extended period to see if they can live with this feature. Start by unchecking one of the boxes for an app you added to the exception list. Click OK (and OK again to acknowledge the restart prompt) and restart your system. If the unchecked application runs well for a few days, return to the Performance Options dialog box, and uncheck another app. Repeat until everything is running with DEP - or until you find one or more apps that need DEP disabled to run properly.

XP users have no way to spot applications that don't use DEP by default, but they can start with Outlook 2007 and Windows Media Player 11. If Windows closes an application with a Data Execution Prevention error message (or any serious error on a regular basis), you can add that application to the exclusion list, as explained above. If you're lucky, the error message will contain a Change Settings button to get you to the dialog box more quickly.

Note that the Data Execution Prevention tab of the Performance Options dialog box only lets you adjust DEP settings for 32-bit applications. If you have the 64-bit version of Vista installed (which can run both 32- and 64-bit apps), you're covered: Windows applies DEP to all 64-bit services and programs. In fact, if you try to add a 64-bit application to the exclusion list, Vista displays an error telling you it can't be done.

Working around the IE exception

Contrary to what you might expect, one type of program in particular ignores the settings in the Data Execution Prevention tab - namely, browsers such as Internet Explorer 7. The only way to enable DEP for IE 7 is in the Internet Options control panel in Vista. XP users apparently have no way to activate DEP for IE 7.

To get a DEP-enabled IE browser in Vista, begin by disabling most or all of your IE add-ons. From the IE command bar, choose Tools, Manage Add-ons, Enable or Disable Add-ons. In the Manage Add-ons dialog box, select a helper application in the list and click the Disable button below. Repeat for all items in the list, except those you know to be safe (such as Adobe Acrobat and Flash). Click OK.

Now let's turn on DEP for Internet Explorer:

Step 1. Click Start, type inetcpl.cpl, and press Ctrl+Shift+Enter to open the dialog with administrative privileges. (If you don't run this dialog as an administrator, the option in question will be greyed out.)

Step 2. Click Continue in the User Account Control prompt.

Step 3. Click the Advanced tab and scroll to the bottom of the Settings list.

Step 4. Check Enable memory protection to mitigate online attacks.

Step 5. Click OK.

Now, restart Internet Explorer (if it was running). If everything seems to go smoothly, return to the Manage Add-ons dialog box. Enable one of the plug-ins, click OK, and restart IE again.

As with the applications you specified earlier, you'll want to use IE for a while to make sure everything works as desired. If IE won't start or you see errors with some Web sites, you may need to disable the problem plug-in. If you can't live without a DEP-intolerant plug-in, you may have to turn off DEP for IE altogether.

Other apps that are DEP exceptions

IE 7 is not the only program that ignores Windows global DEP settings. Even with DEP turned on globally, Task Manager shows that neither Mozilla Firefox nor Opera support DEP.

If DEP is important to your sense of Internet security, IE 7 is the only major browser that supports it - until the other applications provide support for this feature.

Managing installer and application problems

Although the DEP is supposed to display a message indicating when it has shut down an errant program, some sources claim that the messages don't always appear, and that DEP can sometimes even prevent programs (especially installers) from launching. These sources go so far as to recommend turning off DEP entirely.

Such advice is like throwing out the proverbial baby with the bathwater. If you do have problems with applications that end abnormally or won't run, you can always return to the Performance Options dialog to turn off DEP temporarily as a test. This can help you get your software installed, for example, if an installer won't run.

Overall, you're much better off making exceptions for a few problem programs (and reporting the difficulty to the manufacturer) than shutting down DEP entirely.

Finally, you should look at DEP as only one weapon in your security arsenal. DEP adds an important layer of protection, but it isn't a reason to give up your other security tools.