CODEGATE 2012 – Binary 200

This challenge requires us to retrieve a plaintext string which would be eventually printed by the provided DLL. Our routine identification attempt tells us that the DLL has been packed using PEtite v2.1.

The packer is easily defeated using the well-known ESP trick: a few instructions after the DLL’s entrypoint a PUSHAD instruction is executed to preserve the register state so the packer can do its magic. Before jumping to OEP it will pop the stored registers, thus we can easily get to OEP by executing the PUSHAD instruction and placing a hardware on-access bp on on the first dword on the stack. Hit run (ignore any exceptions) and you should land right after a POPAD. Three instructions onwards we find a JMP to OEP. Execute the jump, dump the process and subsequently repair the IAT using Imprec.

The interesting part of the unpacked DLL is the x export, especially the following part:

Sleep(v10, v2, 1000);
index = 0;
key5 = 0xECu;
v13 = sub_100023B3(0);
v14 = sub_10002253(&v13);
if ( *(_DWORD *)(v14 + 8) == 6 )
  index = *(_DWORD *)(v14 + 8);
crypt_v0 = crypt1_ptr[2 * index];
crypt_v1 = crypt2_ptr[2 * index];
key_setup((int)&key, (int)&key0);
XTEA((int)&key, 0, (int)&crypt_v0, (int)&crypt_v0);
hProcess = GetCurrentProcess(7, &v21, 4, 0);
result = p_NtQueryInformationProcess(hProcess);
v19 = result;
if ( !result )
  if ( v21 )
fp = fopen(path_buf, "a");
fwrite(&crypt_v0, 1, 8, fp);
sub_10002088("\n", fp);
result = fclose(fp);

Apparently the DLL is meant to be run as a service, but static disassembly suffices to obtain the flag. We observe a modified XTEA algorithm is used to decrypt a ciphertext determined by an index into an array. As the code explicitly checks the value of index (against the integer literal ‘6’), we assume this indicates the ciphertext we need to decrypt.

Another interesting observation is that the key0-pointer passed to key_setup is already initialized as the DLL is loaded. This, however appears to be a trick to fool the analyst – throughout the DLLs executed code all but two key bytes are replaced (sometimes depending on arbitrary checks, eg. VMware detection). Luckily IDA allows the same analyst to quickly find all cross-references to the key bytes, so it’s easy to determine what the actual key bytes ought to be.

As for the XTEA algorithm, the decipher routine was slighly tweaked by adding the delta each round (instead of substracting it) and both delta and sum are initialized using different constants (compared to original XTEA).

The code below decrypts the correct word from the ciphertext array, yielding the flag: &I%W=K)l (which I initially assumed to be a bad decrypt, expecting actual text instead of ASCII garbage):

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>

void do_decipher(unsigned int num_rounds, uint32_t v[2], uint32_t const key[4]) {
    unsigned int i;
    uint32_t v0=v[0], v1=v[1], delta=0x61C88647, sum=0xC6EF3720;

    for (i=0; i < num_rounds; i++) {
        v1 -= (((v0 << 4) ^ (v0 >> 5)) + v0) ^ (sum + key[(sum>>11) & 3]);
        sum += delta;
        v0 -= (((v1 << 4) ^ (v1 >> 5)) + v1) ^ (sum + key[sum & 3]);
    v[0]=v0; v[1]=v1;

void decipher(uint32_t v[2], uint32_t const key[4]) {
        v[0] = __builtin_bswap32(v[0]);
        v[1] = __builtin_bswap32(v[1]);

        do_decipher(32, v, key);

        v[0] = __builtin_bswap32(v[0]);
        v[1] = __builtin_bswap32(v[1]);

int main(int argc, char *argv[])
    uint32_t key[4];
    uint32_t data[2];

    uint8_t out[9];

    uint32_t table[] = { 0x04A92C695, 0x734DC7E1,
                         0x0C1C00E04, 0x448ADBC1,
                         0x0CEDC98C1, 0xA48CC62C,
                         0x011710F01, 0x82D8415D,
                         0x05EA3F1AF, 0x77172654,
                         0x07DA3C860, 0xBDFA5FDB,
                         0x0C7A1AE1E, 0x026D50B3,
                         0x0E433615A, 0x8A13F05B};

    uint8_t k[16];

    int valid;

    int i;
    for(i = 1; i < 7; i++) {
        /* red herring key */
        uint8_t k1[] = {0xA7, 0x67, 0x25, 0x5A, 0xD9, 0x72, 0xE1, 0xAB, 0x84, 0x18, 0x4F, 0x93, 0x18, 0x0C, 0x8D, 0x83};

        /* actual key */
        uint8_t k2[] = {0x1E, 0xa0, 0xf5, 0xc6, 0xd9, 0xec, 0x02, 0xf6, 0x59, 0x18, 0x7c, 0x2e, 0x6f, 0x85, 0x5d, 0xde};

        int j;
        for(j = 0; j < 4; j++) {
            key[j] = __builtin_bswap32(((uint32_t*)k2)[j]);

        data[0] = table[2*i];
        data[1] = table[2*i+1];

        decipher(data, key);

        memset(out, 0, sizeof(out));
        memcpy(out, data, 8);

        valid = 1;
        j = 0;
        while(valid && j < 8) {
            valid = out[j] >= ' ' && out[j] <= '~';
            printf("%s\n", out);

    return 0;
$ ./bin200 

Comments are closed.