Hi we are using 2 GSM modems with 2 external omnidirectional Antennas when placing this antenna should i consider minimum clearance between each other or placing near to each other won't affect the performance?
Yes, you should keep the antennas out of each other's near-field. When placed in too close proximity, mutual coupling can actually detune the antennas possibly having a critical impact on performance if your signal is marginal, as well as causing greater power consumption as the radios increase output in an effort to achieve a reliable connection.
The near-field is roughly \(λ\over{2π}\) or about 50mm for 900MHz and 25mm for 1800MHz—so you want to be outside that area. Additionally, there is potential for desense (lose of sensitivity due to the transmitter of one modem swamping the front end of the front end of the other) that would suggest putting the antennas at least \(λ\over4\), or about 800mm for 900MHz and 400mm for 1800MHz.
If this can't be done, you could try placing the antennas at 90° to each other, such as in an X formation. This would make their signals cross polarized and might help reducing the RSS from one to the other.
Another consideration is ground plane. If these are just monopole antennas, like simple \(λ\over4\) whips, they will need a good ground plane. A substantial metal enclosure would work, or a metal plate for the purpose could be added if the enclosure is non-conducting. Remember for proper performance the image antenna must be \(λ\over4\), like the radiator, and so the plate or enclosure needs to be at least that large.
If this is not the case, use an antenna that doesn't require an external ground plane, such as a \(λ\over2\) design, or dipole.