Email threading doesnt work. Replies make new tickets.

unpaidIntern

Hi all,

I'm working on turning replies to a ticket confirmation email into comments on the original ticket, thing is they're turning into new tickets.

I've looked at old posts and checked the usual:

The email headers match. The original's Message-ID matches with the reply's In-Reply-To ID.
The ticket ID's are in the subjects of both emails. I'm using the built in templates so they shouldn't have changed. All I've done was to add "AUTO-REPLY TEMPLATE" into the body to know which template I'm getting.

We're using v1.18.3.

For testing purposes I'm an Agent, I'm making tickets via the agent interface using my email as the owner of the ticket, then replying to those tickets from my email when I get the new ticket notification.
It also does not work when I'm using the client interface, also as an agent.

When I reply to the auto-reply, it makes a new ticket. But when another IT tech replies to the auto-reply created by that new ticket, it becomes a comment on that new ticket. The original ticket stays untouched.

The current flow is:
A ticket is created.
Based on the help topic, a team is assigned using the filter.
A plugin assigns a task to the topic.
The email is sent out (somewhere after the creation of the ticket.)
The owner replies to the email.
A new ticket is made.
It repeats.

Is there anything else I'm missing?

KevinTheJedi

unpaidIntern

Is the Ticket set to a closed status that does not allow re-open?

Cheers.

unpaidIntern

KevinTheJedi No, its open

KevinTheJedi

unpaidIntern

Then that doesn't really make sense.. if an In-Reply-To or References entry matches an existing Message-ID in the system then it will thread; unless it's set to a closed status that doesn't allow reopen. I would need to see a screenshot of a thread_entry_email record from the db with mid equaling a valid Message-ID then a full raw export of an email with headers (in EML/MSG format) that references that Message-ID in the In-Reply-To and/or References headers and created a new Ticket instead of threading to the corresponding Open Ticket.

Cheers.

unpaidIntern

KevinTheJedi
Apologies, the header does not match for some reason.

It seems like the first auto-response emails that get sent to the user after a ticket is created don't have their Message_Id's inputted into the thread_entry_email table. But the replies to that email get their Message_ID's recorded, and it works fine afterwards.

Another thing to note is that the headers in the original email and the reply do not match. Neither the original Message_ID nor the reply's In-Reply-To/Reference ID are recorded in the table.

In fact, the table has a bunch of ID's that look similar but I cant find any actual email tied to them.
(I found this out by doing a number of SELECT mid FROM ost_thread_entry_email WHERE mid = "xyz"; queries)
Only the replies' Message_ID are recorded.

But even then, shouldn't the subject line be enough to tack it on as a reply?
I barely touched the template, only moving the ID portion to the beginning of the subject line.
That is the subject line of the Auto-Response of the ticket created by the reply.

This is the subject line of the Auto-Response of the original ticket where the reply created the ticket above.

This is the reply itself.

That's the template.

KevinTheJedi

unpaidIntern

Emails sent FROM the helpdesk are not recorded in the thread_entry_email table. Only incoming mail gets recorded there. Emails sent FROM the helpdesk have a unique Message-ID that has information encoded in the string to help us decode and match when someone replies to it. If a user replies to that email its Message-ID should be added as an In-Reply-To and/or References header so when the helpdesk processes it we see that, decode it, and find possible matches.

The subject is not always enough. Even if it contains the Ticket number it still has to match other criteria.

I guess I need to see a full flow of what you are doing with exports of the emails so I can see headers, etc. to be able to tell you what's happening.

Cheers.

unpaidIntern

KevinTheJedi
Here's the 3 headers.

The full flow is:
I sign into the agent interface.
I make a new ticket.
I assign myself as the owner.
I fill it out and wait for the Auto-Response email.
While waiting, the email gets sorted and assigned to the IT Team by the built in filter.
Upon creation, a plugin I wrote automatically assigns a task to that ticket.
Once the email comes, I reply to that email from the same email I put into the ticket.

*At this point the ticket is still open, no one has touched it.

After 10 minutes a new ticket with the subject line of the reply pops up in the ticket list.
I then get an Auto-Response for that new ticket, in the subject line there's a new Message_ID.
At this point I can continuously reply to this new email and it becomes comments on that new ticket.

KevinTheJedi

unpaidIntern

The headers match up.. so I'm not sure what's going on in your case. I'd say retest this with fully vanilla install with no mods and no custom plugins to see if it works there. Just from what I'm gathering from what you're saying and the headers it doesn't really make sense or match up.

Cheers.

unpaidIntern

KevinTheJedi
A couple of random things:
We're using NGINX and Outlook, helpdesk has its own 365 email subscription, and we're using MariaDB for the database server.

Is there an incompatibility with any of this?

KevinTheJedi

unpaidIntern

No, that's all fine. None of that would change the processing of emails and thread matching. Unless outlook modifies the raw emails or something but that's hardly ever the case.

Cheers.

unpaidIntern

KevinTheJedi I found an older post where you explained how the threading works.
Can you explain it further? You said the system checks:

The In-Reply-To and Reference fields to match it to an already existing Message_ID.
The Message_ID to check if it originated from the system. (Can you explain this further?)
The subject line to see if it has a valid Ticket Number.

What tables would these be checking against?
If ost_thread_entry_email only logs incoming emails, what table does the first test crossreference? Where are the original Message_ID's stored?
What other criteria needs to be fulfilled if the Ticket Number isn't enough to connect them?

KevinTheJedi

unpaidIntern

The code is open source and publicly available for review: https://github.com/osTicket/osTicket

When an email is sent from the system it generates a unique message ID with references encoded within. When the system gets a reference or in-reply-to value that came from the system we can decode it and lookup the references embedded within.

The Ticket number has to match, the sender has to belong to the Ticket, etc. You can follow the entire pipeline in the codebase.

If checking references encoded within message-id generated by the system then it checks the thread by id. Maybe entry id as fallback (I think, you'd have to double check code). If checking references from header that's not generated by the system then it checks the thread_entry_email table as I mentioned earlier.

There is no table that stores outbound message-ids as I mentioned above.

Cheers.

unpaidIntern

KevinTheJedi
I was looking in class.mailer.php and I found decodeMessageID().
It looks like it wants a MID like this: <Bxxxxx-12345-BASE64TAG@my-domain>
But mine look like this: SA1PR02MB8694A474F7ED35REDACT@REDACT.namprd02.prod.outlook.com

Would this cause any issues? I'm a bit confused how the code specifies that format, but the threading works for the replies to the new ticket made by the reply to the original autoresponse ticket.

I also updated to 1.18.4 and the issue still persists, it also still persists even if I disable my custom plugin.

KevinTheJedi

unpaidIntern

The one you posted with outlook is not a message-id generated by our system so we cannot decode it. That would be used in the lookup stage for thread/entry matching.

Cheers.

unpaidIntern

KevinTheJedi I've noticed that the first post in the new ticket thread is the original ticket's auto-response and attached to that is my reply to it, is this normal behavior?

KevinTheJedi

unpaidIntern

Can you show me what you mean?

Cheersl

unpaidIntern

KevinTheJedi
The two tickets:

Original ticket:

Reply to the autoreply ticket:

That was all of the thread, it was too big so I just split it into parts.

The first part of the autoreply ticket is the original auto-response sent out by the original ticket's creation, and when I reply to it the auto-response becomes a ticket and my reply becomes a comment.

The red blocks are me, and the green is just our domain. Helpdesk is the helpdesk email.

KevinTheJedi

unpaidIntern

Something weird is definitely going on. Why no user message or response in the initial ticket? Shouldn't matter, just curious. As long as the Ticket owner or CC sends a message to the alert sent from the system it should thread through the existing Ticket.

So basically, when the Ticket is created it should send out a new Ticket auto-response to the User and associated CCs (if enabled). That auto-response email should have subject like "[#xxx] Subject Here" (judging by your screenshots - these can be modified in the templates). When a User or CC replies to that alert email its Message-ID (that should be in osTicket Message-ID format as it was sent from our system) would then be added to the In-Reply-To and/or References header in their email. Then when osTicket processes that email it should see the Message-ID from the In-Reply-To/References headers, decode it, lookup the matches, ensure everything checks out, then processes and threads the mail.

Also, the fact that you have a message with a white/gray bar means that that email came in and was associated but the system couldn't determine how it ended up there. Which indicates another potentially related issue with how you have your mail routing or something.

Cheers.

unpaidIntern

KevinTheJedi So the outgoing email should have an OSTicket message ID, it seems like somewhere along the way the header gets altered with an Outlook message ID and thats whats messing it up?

I looked at the routing rules and didn't see anything, I'm currently running a message trace so I'll get back to you with that.

KevinTheJedi

unpaidIntern

I have zero clue what’s going on in your case so I’m not sure exactly. You can validate this yourself by creating a ticket, inspecting the new ticket auto-response’s raw headers, and making sure its Message-ID is an osTicket generated one. If so then when you click Reply/Reply All and reply as a user/CC the message you send should include the auto-response's Message-ID in the In-Reply-To/References header.

Cheers.

KevinTheJedi

unpaidIntern

Judging by the screenshots this is the flow I see:

A ticket is created for Minh
Helpdesk Bot is added as CC
Org CCs are added
Team gets assigned

Then the subsequent new ticket is from a “helpdesk” email. Does the Helpdesk Bot user have the same email address as the system email you are fetching from? If so that’s your issue.

Cheers.