Hi
My suggestion is configuring your own smtp server (if you have the choice) in the computer which is hosting the website.
In that case you need to configure it as a SMTP server. You will either have to install a SMTP server, like how we install Apache,IIS and others as web servers.
Please note even IIS can act as a smtp server but there are many of them available for free. Also SMTP servers usually runs on port 25.
If you do not have the SMTP server installed on that PC (cause you want to use your provider´s smtp server), make sure by contacting them that the system e-mail address defined on osTicket is already registered and able to send/receive email traffic.
After that go to your php.ini and look for the smtp settings
; For Win32 only.
SMTP = //your SMTP server IP or DNS
smtp_port = 25
Also you can create a new php file and access it via webbrowser(e.g: testmail.php) so you can check if emails are being sent out...
<?php
mail("your_emaill@yourdomain.com", "Testing", "Line 1\nLine 2\nLine 3", "From: osticket_system_email@yourdomain.com") or die("email error");
?>
Hope this helps, ;)