Hardware DEP has a backdoor

Introduction :
Hardware DEP (Data Execution Prevention) is a feature that protects Windows XP SP2 from buffer overruns. When a compatible CPU is present, Windows XP SP2 DEP interface offers 2 options. The default one protects only the core Windows components and the ones manually added by the user (OPTIN). The most secure one protects all the programs on your systems except the ones manually added by the user (OPTOUT).
For more informations on how to activate hardware DEP, visit my DEP to keep buffer overrun at bay blog post.

Investigating hardware DEP :
I activated DEP on my 2 compatible PCs in mid 2006, it was protecting my PCs in the background so I forgot about it. Recently the security guru Steve Gibson of grc.com in it's Security Now podcasts mentioned that IrfanView refused to run in ALWAYSON DEP mode. While it would run perfectly in in OPTOUT mode. Theoretically OPTOUT with an empty exclusion list should behave exactly like ALWAYSON. The only possible explainations were either IrfanView was doing a nasty trick to evade DEP or Microsoft lied about the definition of OPTOUT.

I ran a bunch of queries on Google to find more information on the subject... nothing really impressive. Then a contributor on the GRC security newsgroup posted about packed executable disabling DEP. Back in late 2005, when the WMF exploit surfaced Kaspersky pointed out that IrfanView and XnView were running without protection even in OPTOUT mode. Leaving their users with their pants on the ankles. It clearly stated:

“However, when testing the latest builds of third party image viewers like Irfanview and XnView HW DEP didn’t prevent exploitation, even with HW DEP enabled for all programs. This is because both Irfanview and XnView are packed with ASPack and Windows disables HW DEP for ASPack packed files.”
— Roel Schouwenberg, Kaspersky

ASPack allows to compress executable files (EXE, DLL, OCX,...) so their footprint on the hard disk is smaller. The file is decompressed on the fly at runtime. The CPUs are faster than hard disk so in the end running these apps should be faster. I never used DriveSpace/DoubleSpace (back in the DOS days) or RAID0... mainly for low reliability and lack of decent performance boost. So I have a hard time believing that saving 2MB on my laptop 160GB hard disk is a useful trick. Especially when it comes at the expense of DEP protection.

So this new keyword unleashed the doors of very interesting stuff on Google (query ASPack DEP Kaspersky). I quickly found out that IrfanView was not using any trick to evade DEP... Microsoft just coded a backdoor used only in OPTOUT. Bascially Microsoft checks the executable header for a section matching one of the 3 strings. If one these strings is found, DEP will be turned OFF for this application by windows. To find these strings all you have to do is use a hex editor to open ntdll.dll found in your sytem32 directory.

The 3 backdoor strings in ntdll.dll :

By order of appearance, the strings are:

  • aspack : from ASPack. Common programs that use ASPack are the 2 image viewers IrfanView and XnView... but also the web browser Opera.
  • pcle : (source not yet identified) I put my 0.02€ on Pinnacle System for their long history of crashing apps and their NASDAQ ticker.
  • sforce : from Star-Force the highly unpopular game/app copy protection. The list of current Star-Force protected games can be found on the Boycott StarForce website.

The easiest way to identify the DEP status of an application is to run Process Explorer. In order to display the DEP status you'll need to go to the View menu, click on Select columns and then tick the "DEP status" checkbox.

Process Explorer window :

The compressed programs appear in purple. You can see that packed apps are common but only the ASPack ones disable DEP. Both IrfanView and Opera run with DEP off. In this case, I had Skype (proprietary packer???) and FastStone Capture (UPX). I also use uTorrent which uses PECompact.

From my point of view the worst case is Opera. By definition a web browser is on the front line it has to be backed up by DEP. I would barely tolerate that a local widget that displays an analog clock disables DEP... but a browser is way beyond my threshold. If the widget requires DEP to be turned off to run, I want to be in charge of doing so. Just like I want to control the opened ports in my routers and therefore turn off UPnP, I want to have the control of my hardware DEP protection.

Conclusion :
I want to believe that Microsoft will eventually release a properly named OPTOUT mode. If the current backdoored version was to stay it should be named with clear unsecured in its description. If some spin doctor wanted to pull "Hybrid" out of his sleeves he should take this PR stunt and *censored*. On the programs side, it would be a good time for the ASPack users to switch to UPX which is open source and DEP compliant. Or simply dump packed executables for good. I don't care if my web browser takes 200 extra milliseconds to launch. But I do care to have DEP blindly protecting my programs from undisclosed buffer overruns.

Some will say that ALWAYSON locks this backdoor... but in my case being able to OPTOUT one program in beta stage is priceless. Therefore I'd rather replace the strings in ntdll.dll by some random gibberish so that no other app can slip thru my fingers. Last year, for speed reasons, I switched from IrfanView to FastStone Image Viewer. I'll now use Opera only as the last resort and for testing purpose on known websites.

Edits :

    • IrfanView 4.x+ now uses UPX packer so it's fully hardware DEP compliant.
    • A bunch of PhotoShop compatible plugins require switching DEP off for the hosting app (PhotoShop, Paint Shop Pro or others).
    • These plugins don't trigger a DEP interception, they just fail to properly run and crash the host application. Here are some plugins that crash: Xero Graphics, PhotoWiz and plugins built using FilterMeister. The FilterMeister devs are aware of the issue and are working on a solution.
    • Opera from version 9.50 alpha and up is now DEP compliant (FYI: it uses UPX to pack its executable).

    Useful links :