I am working on a PE encryptor / decryptor which doesn't use a static key. The encryptor generates a random number (1-255) to use as its key. Knowing that the first character in all PE files (excluding VxDs and 16-bit DLLs) is "M" (starting the DOS MZ header), the decryptor will keep trying different keys until the first byte of the PE is "M". Once we find the key, we decrypt the rest of the file. What I have written looks like this:
Unfortunately, this doesn't work. I get the error "Operand types do not match" in both of the XOR lines. Does anyone have any input or ideas on how to solve this problem?
Rich (BB code):
lea esi, offset FILE_CONTENTS ; ESI = contents of encrypted file
xor edi, edi ; EDI = 0 (encryption key)
FIND_KEY:
inc [edi] ; EDI++
xor byte ptr [esi], [edi] ; FILE_CONTENTS[ESI] = XOR FILE_CONTENTS[ESI], EDI
cmp byte ptr [esi], "M" ; did we find the key?
jne find_key ; if not, check again
mov ecx, FILE_SIZE - 1 ; ECX = FILE_SIZE - 1 (we already decrypted the first byte)
DECRYPT:
xor byte ptr [esi], [edi] ; encrypt byte
inc esi ; get next byte
loop DECRYPT ; loop until ECX = 0