Issue resolved, thanks Kevin. More info below in case anyone else runs into this issue.
The key troubleshooting tip was that osTicket falls back on PHPMail/Sendmail after a failed SMTP attempt. The answer was in the log, but every time I would test SMTP and fail, I kept looking at the latest log entry, which had the PHP mail error. I needed to look for the error immediately before it showing the first SMTP attempt that failed:

Once I realized that, the error message (and some Googling) indicated that I had one more step to do with the MailerSend SMTP service I was using before the account would work. I completed that step on the MailerSend end, and now I can send emails from osTicket.