Having the exact same issue. Followed the setup guide to the letter; OAth for remote mailbox is working fine. When I go to set up outgoing SMTP, I try "same as remote mailbox" initially, which fails. Then configure Microsoft OAuth, even tried with its own secret and it fails every time - 5.7.3 Authentication unsuccessful.
I've ensured that the account has authenticated SMTP enabled. I've even added the azure application for OSTicket as a service principal for exchange and assigned to the account - no joy 🙁
I have patched includes/class.mail.php as I was just getting error 500 after the upgrade to 10.17 from 10.17.x branch if that is relevant.
It DOES display the green OAuth successful banner after completing the config and signing in/consenting, and a valid token is displayed in the OAuth config, but it displays "5.7.3 Authentication unsuccessful" when I click "Save Changes"