PE Encryptor / Decryptor

Discussion in 'Programmer's Corner' started by maple23, Jul 11, 2008.

  1. maple23

    Thread Starter New Member

    Apr 23, 2008
    6
    0
    I am working on a PE encryptor / decryptor which doesn't use a static key. The encryptor generates a random number (1-255) to use as its key. Knowing that the first character in all PE files (excluding VxDs and 16-bit DLLs) is "M" (starting the DOS MZ header), the decryptor will keep trying different keys until the first byte of the PE is "M". Once we find the key, we decrypt the rest of the file. What I have written looks like this:

    Code ( (Unknown Language)):
    1.     lea    esi, offset FILE_CONTENTS    ; ESI = contents of encrypted file
    2.     xor    edi, edi            ; EDI = 0 (encryption key)
    3.  
    4. FIND_KEY:
    5.     inc    [edi]                ; EDI++
    6.     xor    byte ptr [esi], [edi]        ; FILE_CONTENTS[ESI] = XOR FILE_CONTENTS[ESI], EDI
    7.     cmp    byte ptr [esi], "M"        ; did we find the key?
    8.     jne    find_key            ; if not, check again
    9.  
    10.     mov    ecx, FILE_SIZE - 1        ; ECX = FILE_SIZE - 1 (we already decrypted the first byte)
    11.  
    12. DECRYPT:
    13.     xor    byte ptr [esi], [edi]        ; encrypt byte
    14.     inc    esi                ; get next byte
    15.     loop    DECRYPT                ; loop until ECX = 0
    Unfortunately, this doesn't work. I get the error "Operand types do not match" in both of the XOR lines. Does anyone have any input or ideas on how to solve this problem?
     
  2. Mark44

    Well-Known Member

    Nov 26, 2007
    626
    1
    How about instead of this:
    xor byte ptr [esi], [edi] ; encrypt byte

    you try this:
    xor byte ptr [esi], byte ptr [edi] ; encrypt byte

    If you make this change in both of your xor lines that are giving you the error. I haven't tried it, but I think it will work, as now the assembler will be able to ascertain the sizes of the objects being pointed to.

    BTW, you mentioned that the first bytes of the PE are 'M' and 'Z'. The reason for these characters are that they are the initials of a Microsoft dev and architect who came up with this file format: Mark Zbikowski. There's an entry in Wikipedia on him.
     
  3. maple23

    Thread Starter New Member

    Apr 23, 2008
    6
    0
    Sorry for the late response, but no - still get the same errors:

    Assembling file: PE.ASM
    *Warning* PE.ASM(76) Argument needs type override
    **Error** PE.ASM(77) Illegal memory reference
    **Error** PE.ASM(84) Illegal memory reference
    Error messages: 2
    Warning messages: 1
    Passes: 2
     
  4. Mark44

    Well-Known Member

    Nov 26, 2007
    626
    1
    These aren't the same errors that you gave in your first post. In that post you said the error was "Operand types do not match" in the two lines.

    What you're going to have to do now is find the lines in your code that are giving the errors. There's a warning in line 76, and error in line 77, and another error of the same kind in line 84.

    See if you can find these lines of code. You know if you have them if you can comment them out and then assemble your code. If you have commented the right lines, the errors will go away. Your program won't run, but you have at least identified the right lines.

    After that, post the part of the code that's giving you problems, and I'll take a look.
    Mark
     
  5. maple23

    Thread Starter New Member

    Apr 23, 2008
    6
    0
    I don't think just giving you error messages will help much. This is the re-written code (using byte-size registers for encryption):
    Code ( (Unknown Language)):
    1. .386p      ; at least a 386 required
    2. .model flat     ; 32 bit registers
    3. EXTRN GetFileSize :PROC   ; needed APIs
    4. EXTRN GlobalAlloc :PROC
    5. EXTRN GlobalLock :PROC
    6. EXTRN CreateFileA :PROC
    7. EXTRN ReadFile :PROC
    8. EXTRN WriteFile :PROC
    9. .DATA      ; static variables
    10.  szFileToEncrypt db "test.exe", 0
    11.  szOutputFile db "encrypted.exe", 0
    12. .DATA?      ; dynamic variables
    13.  BYTES_READ dd ?
    14.  BYTES_WRITTEN dd ?
    15.  FILE_CONTENTS dd ?
    16.  FILE_SIZE dd ?
    17.  TEMP_MEMORY dd ?
    18.  hReadFile dd ?  ; file-handles
    19.  hWriteFile dd ?
    20. .CODE
    21. MAIN:      ; our entrypoint
    22.  xor eax, eax
    23.  push eax
    24.  push 00000080h   ; FILE_ATTRIBUTE_NORMAL
    25.  push 00000003h   ; OPEN_EXISTING
    26.  push eax
    27.  push eax
    28.  push 80000000h   ; GENERIC_READ
    29.  push offset szFileToEncrypt  ; "test.exe"
    30.  call CreateFileA   ; open the file for reading
    31.  cmp eax, -1    ; error opening file?
    32.  je OPEN_ERROR   ; get out of here!
    33.  mov hReadFile, eax   ; save file-handle
    34.  xor eax, eax
    35.  push eax
    36.  push [hReadFile]   ; handle to "test.exe"
    37.  call GetFileSize
    38.  mov eax, [FILE_SIZE]  ; save file size
    39.  push [FILE_SIZE]
    40.  push 00000002h   ; GMEM_MOVEABLE
    41.  call GlobalAlloc   ; create a buffer on the fly
    42.  mov [TEMP_MEMORY], eax  ; save handle to memory
    43.  xor eax, eax
    44.  cmp [TEMP_MEMORY], 0  ; error creating memory?
    45.  je OPEN_ERROR   ; get out
    46.  push [TEMP_MEMORY]
    47.  call GlobalLock   ; lock memory
    48.  mov [FILE_CONTENTS], eax  ; FILE_CONTENTS = buffer
    49.  xor eax, eax
    50.  push eax
    51.  push offset BYTES_READ
    52.  push [FILE_SIZE]   ; size of "test.exe"
    53.  push offset FILE_CONTENTS  ; pointer to memory
    54.  push [hReadFile]   ; pointer to file-handle
    55.  call ReadFile   ; read the contest of "test.exe"
    56.  cmp eax, 0    ; error reading?
    57.  je OPEN_ERROR   ; get out
    58.  lea esi, [FILE_CONTENTS]  ; ESI = contents of encrypted file
    59.  xor al, al    ; AL = 0 (encryption key)
    60. ;
    61. ; the program continutes to loop forever starting here
    62. ;
    63.  
    64. FIND_KEY:
    65.  inc al    ; AL++
    66.  xor byte ptr [esi], al  ; encrypt byte
    67.  cmp byte ptr [esi], "M"  ; did we find the key?
    68.  jne FIND_KEY   ; if not, check again
    69.  mov ecx, FILE_SIZE - 1  ; ECX = FILE_SIZE - 1 (we already decrypted the first byte)
    70. DECRYPT:
    71.  xor byte ptr [esi], al  ; encrypt byte
    72.  inc esi    ; get next byte
    73.  loop DECRYPT    ; loop until ECX = 0
    74. ;
    75. ; this code never gets executed - the above keeps looping
    76. ;
    77.  xor eax, eax
    78.  push eax
    79.  push 00000080h   ; FILE_ATTRIBUTE_NORMAL
    80.  push 00000002h   ; CREATE_ALWAYS
    81.  push eax
    82.  push eax
    83.  push 40000000h   ; GENERIC_WRITE
    84.  push offset szOutputFile  ; "encrypted.exe"
    85.  call CreateFileA   ; create new file for writing
    86.  cmp eax, -1    ; error creating file?
    87.  je OPEN_ERROR   ; get out
    88.  mov [hWriteFile], eax  ; save file-handle
    89.  xor eax, eax
    90.  push eax
    91.  push offset BYTES_WRITTEN
    92.  push [BYTES_READ]   ; from reading the file
    93.  push esi    ; encrypted file contents
    94.  push [hWriteFile]   ; pointer to file-handle
    95.  call WriteFile   ; write encrytped file contents to "encrypted.exe"
    96. OPEN_ERROR:
    97.  ret     ; poor-man's ExitProcess(0)
    98.  
    99. ;-------------------------------------------------------------------------------
    100. gh: EXTRN MessageBoxA :PROC
    101.  push 0
    102.         call pushGH
    103.         call pushGH
    104.  push 0
    105.  call MessageBoxA
    106.  ret
    107. pushGH:
    108.  call $ + 15
    109.   db "got here.", 0
    110.  ret
    111. ;-------------------------------------------------------------------------------
    112. END MAIN
     
  6. Mark44

    Well-Known Member

    Nov 26, 2007
    626
    1
    Please reread my previous post. I wasn't asking for the error messages: you gave them to me already. I was asking which lines of code generated the errors you showed. They are at lines 77 and 84.
     
Loading...