This is the circuit to interface between PC and atmega128. In this circuit I use max233 to convert signal from RS232 to TTL and vice versa. However when I use Hercules software to simulate COM port and send a massage to atmega128 but the massage when I connect virtual terminal isn't what I sent. I don't know why, hope anyone who has experience help me.

 

You have two receivers connected together and two transmitters connected together. What you need to establish is that the TxD on the PC is connected to RxD on the ATmega128, AND that RxD on the PC is connected to TxD on the ATmega128.

A DTE (aka a "Terminal") follows this rule "Terminals Transmit on Two". The "Two" in the previous quote refers to pin 2 of the DB-9 connector. Therefore a DCE (aka a computer or modem) will Receive on pin 2.

On any serial device you can establish which pin is which by measuring the voltage with respect to GND. The TxD output will be at a negative voltage when the line is idle. The RxD input will be undriven and close to GND. That way you can ge the connections correct.

Google "Null Modem" for another solution to rewiring your board.

 

Ok, thanks for help!
Now this problem is solved, I can send data from compim => MAX233 => RX (ATMEGA128).
As you can see in the picture bellow the data in virtual terminal is exact what I sent.
Now I have a c code to receive and transmit data from ot to USART0 of atmega128, you can see code below. But the problem is that when I connect RX of virtual terminal to TX of atmega128, I don't receive any signal at all. Could you help me with this code, what is wrong in the code or anything wrong in my circuit?

#include <avr/io.h> 


#define F_CPU 16000000 
#define BAUD 9600 

unsigned char usart_receive(void) 
{ 
   while ( !(UCSR0A & (1<<RXC)) ); 
   return UDR0; 
} 

void usart_transmit(unsigned char data) 
{ 
   while ( !( UCSR0A & (1<<UDRE)) ); 
   UDR0 = data; 
} 

int main(void) 
{ 
   char k; 
   UBRR0H = 0x00; 
   UBRR0L = F_CPU/16/BAUD-1; 

   UCSR0B = ((1<<RXEN)|(1<<TXEN)); 

   while(1) 
   { 
      k = usart_receive (); 
      usart_transmit(k); 
   } 
}

 

You have two problems.

1) You do not need inverters between the ATmega and MAX233.

2) Pins 2 and 3 at the 9-pin connector should be reversed.

 

In proteus you can also connect directly to the virtual comport from the MCU.