I just checked the wiki, there are steps just for Gmail!
Gmail Configuration
In order to use Gmail, your host must support SSL, so osTicket can negotiate the secure connection, and you must enable IMAP or POP in your GMail or GApps account. Configure in osTicket (the easy part)
Under Admin panel -> Emails -> Emails -> Sending email via SMTP For most people, enter either ssl://smtp.gmail.com with port 465 or tls://smtp.gmail.com with port 587.
note: If you have a google apps/G-Suite account, this might change, see below.
Select "Authentication Required". Leave Header Spoofing unchecked. Make sure your username is your full email and password are set correctly in the Email Login Information
If you test at this point and it doesn't work, continue reading.
You may need to consult your PHP error log (the location varies by OS and personal preferences so consult your php.ini to determine its location). The PHP error log often contains more information as to why something is not working correctly.
http://osticket.com/wiki/Email_settings#Gmail_Configuration