I am working on a PE encryptor / decryptor which doesn't use a static key. The encryptor generates a random number (1-255) to use as its key. Knowing that the first character in all PE files (excluding VxDs and 16-bit DLLs) is "M" (starting the DOS MZ header), the decryptor will keep trying different keys until the first byte of the PE is "M". Once we find the key, we decrypt the rest of the file. What I have written looks like this:

    lea    esi, offset FILE_CONTENTS    ; ESI = contents of encrypted file
    xor    edi, edi            ; EDI = 0 (encryption key)

FIND_KEY:
    inc    [edi]                ; EDI++
    xor    byte ptr [esi], [edi]        ; FILE_CONTENTS[ESI] = XOR FILE_CONTENTS[ESI], EDI
    cmp    byte ptr [esi], "M"        ; did we find the key?
    jne    find_key            ; if not, check again

    mov    ecx, FILE_SIZE - 1        ; ECX = FILE_SIZE - 1 (we already decrypted the first byte)

DECRYPT:
    xor    byte ptr [esi], [edi]        ; encrypt byte
    inc    esi                ; get next byte
    loop    DECRYPT                ; loop until ECX = 0

Unfortunately, this doesn't work. I get the error "Operand types do not match" in both of the XOR lines. Does anyone have any input or ideas on how to solve this problem?

 

How about instead of this:
xor byte ptr [esi], [edi] ; encrypt byte

you try this:
xor byte ptr [esi], byte ptr [edi] ; encrypt byte

If you make this change in both of your xor lines that are giving you the error. I haven't tried it, but I think it will work, as now the assembler will be able to ascertain the sizes of the objects being pointed to.

BTW, you mentioned that the first bytes of the PE are 'M' and 'Z'. The reason for these characters are that they are the initials of a Microsoft dev and architect who came up with this file format: Mark Zbikowski. There's an entry in Wikipedia on him.

 

Sorry for the late response, but no - still get the same errors:

Assembling file: PE.ASM
*Warning* PE.ASM(76) Argument needs type override
**Error** PE.ASM(77) Illegal memory reference
**Error** PE.ASM(84) Illegal memory reference
Error messages: 2
Warning messages: 1
Passes: 2

 

Sorry for the late response, but no - still get the same errors:

Assembling file: PE.ASM
*Warning* PE.ASM(76) Argument needs type override
**Error** PE.ASM(77) Illegal memory reference
**Error** PE.ASM(84) Illegal memory reference
Error messages: 2
Warning messages: 1
Passes: 2

These aren't the same errors that you gave in your first post. In that post you said the error was "Operand types do not match" in the two lines.

What you're going to have to do now is find the lines in your code that are giving the errors. There's a warning in line 76, and error in line 77, and another error of the same kind in line 84.

See if you can find these lines of code. You know if you have them if you can comment them out and then assemble your code. If you have commented the right lines, the errors will go away. Your program won't run, but you have at least identified the right lines.

After that, post the part of the code that's giving you problems, and I'll take a look.
Mark

 

I don't think just giving you error messages will help much. This is the re-written code (using byte-size registers for encryption):

.386p      ; at least a 386 required
.model flat     ; 32 bit registers
EXTRN GetFileSize :PROC   ; needed APIs
EXTRN GlobalAlloc :PROC
EXTRN GlobalLock :PROC
EXTRN CreateFileA :PROC
EXTRN ReadFile :PROC
EXTRN WriteFile :PROC
.DATA      ; static variables
 szFileToEncrypt db "test.exe", 0
 szOutputFile db "encrypted.exe", 0
.DATA?      ; dynamic variables
 BYTES_READ dd ?
 BYTES_WRITTEN dd ?
 FILE_CONTENTS dd ?
 FILE_SIZE dd ?
 TEMP_MEMORY dd ?
 hReadFile dd ?  ; file-handles
 hWriteFile dd ?
.CODE
MAIN:      ; our entrypoint
 xor eax, eax
 push eax
 push 00000080h   ; FILE_ATTRIBUTE_NORMAL
 push 00000003h   ; OPEN_EXISTING
 push eax
 push eax
 push 80000000h   ; GENERIC_READ
 push offset szFileToEncrypt  ; "test.exe"
 call CreateFileA   ; open the file for reading
 cmp eax, -1    ; error opening file?
 je OPEN_ERROR   ; get out of here!
 mov hReadFile, eax   ; save file-handle
 xor eax, eax
 push eax
 push [hReadFile]   ; handle to "test.exe"
 call GetFileSize
 mov eax, [FILE_SIZE]  ; save file size
 push [FILE_SIZE]
 push 00000002h   ; GMEM_MOVEABLE
 call GlobalAlloc   ; create a buffer on the fly
 mov [TEMP_MEMORY], eax  ; save handle to memory
 xor eax, eax
 cmp [TEMP_MEMORY], 0  ; error creating memory?
 je OPEN_ERROR   ; get out
 push [TEMP_MEMORY]
 call GlobalLock   ; lock memory
 mov [FILE_CONTENTS], eax  ; FILE_CONTENTS = buffer
 xor eax, eax
 push eax
 push offset BYTES_READ
 push [FILE_SIZE]   ; size of "test.exe"
 push offset FILE_CONTENTS  ; pointer to memory
 push [hReadFile]   ; pointer to file-handle
 call ReadFile   ; read the contest of "test.exe"
 cmp eax, 0    ; error reading?
 je OPEN_ERROR   ; get out
 lea esi, [FILE_CONTENTS]  ; ESI = contents of encrypted file
 xor al, al    ; AL = 0 (encryption key)
;
; the program continutes to loop forever starting here
;
 
FIND_KEY:
 inc al    ; AL++
 xor byte ptr [esi], al  ; encrypt byte
 cmp byte ptr [esi], "M"  ; did we find the key?
 jne FIND_KEY   ; if not, check again
 mov ecx, FILE_SIZE - 1  ; ECX = FILE_SIZE - 1 (we already decrypted the first byte)
DECRYPT:
 xor byte ptr [esi], al  ; encrypt byte
 inc esi    ; get next byte
 loop DECRYPT    ; loop until ECX = 0
;
; this code never gets executed - the above keeps looping
;
 xor eax, eax
 push eax
 push 00000080h   ; FILE_ATTRIBUTE_NORMAL
 push 00000002h   ; CREATE_ALWAYS
 push eax
 push eax
 push 40000000h   ; GENERIC_WRITE
 push offset szOutputFile  ; "encrypted.exe"
 call CreateFileA   ; create new file for writing
 cmp eax, -1    ; error creating file?
 je OPEN_ERROR   ; get out
 mov [hWriteFile], eax  ; save file-handle
 xor eax, eax
 push eax
 push offset BYTES_WRITTEN
 push [BYTES_READ]   ; from reading the file
 push esi    ; encrypted file contents
 push [hWriteFile]   ; pointer to file-handle
 call WriteFile   ; write encrytped file contents to "encrypted.exe"
OPEN_ERROR:
 ret     ; poor-man's ExitProcess(0)
 
;-------------------------------------------------------------------------------
gh: EXTRN MessageBoxA :PROC
 push 0
        call pushGH
        call pushGH
 push 0
 call MessageBoxA
 ret
pushGH:
 call $ + 15
  db "got here.", 0
 ret
;-------------------------------------------------------------------------------
END MAIN

 

Please reread my previous post. I wasn't asking for the error messages: you gave them to me already. I was asking which lines of code generated the errors you showed. They are at lines 77 and 84.