Not real sure what you mean by "outside".
Should these people be able to send emails to your ticket system?
If not then you could block them, or their domain. (Admin panel -> Emails -> Banlist)
If they shouldn't have accounts, you can also make your registration method private, or or require registration to create tickets. (Admin panel -> Settings -> Users)
Are they replying to the original email thread, and not the email from the ticket system?
Weird things can happen if they reply to the thread, as it would generate new message-ids.
And like you mentioned already you can disable Accept Email Collaborators.
(Admin panel -> Emails -> Settings).