You say http://localhost/osTicket/api/auth/oauth2
then you say it brings you to http://localhost/osticket/
. Is your webserver case sensitive? Also, please post a screenshot of the URL when you are submitting the popup from osTicket, a screenshot of your osTicket OAuth2 config, and a screenshot of your Redirect URI configuration in the App Registration.
Cheers.