Vortex,
Your information given is VERY vague,,:( Does your email server reside on the same machine that osTicket is running on? If these are two different machines can you telnet port 25 from the osTicket machine to email server ip,or FQDN? If you can in fact telnet to your email server ip number do you have an actual username of the (admin@osticket.mydomain.net) for example on the actual email server?
Take Care