Issues with historical import from gmail

devaps

osticket-admin-control-panel-info.pdf

86kB

Hello,

I am a new user. A month ago, I started looking at osTicket for my single-user, low-volume helpdesk system and have been experimenting with the product since.

See attached for my installation info. I am using the latest available version. osTicket is installed inside an Ubuntu 24.04 VM running on my MacBook M2 laptop.

I setup the admin user and email during the installation setup. I then added a single agent in osTicket. I setup the system email as my gmail account which has historically served as my helpdesk. I turned off all alerts, imported all my clients as users, setup my gmail as my Remote and Outgoing mailbox, with the fetch folder as Helpdesk and Fetch Folder Action to Archive/Move to Folder: Tickets. Helpdesk and Tickets are gmail labels I use to place the conversations in Helpdesk folder and after osTicket processes them, get them moved to Tickets folder.

With this setup, I have done hundreds of experiments to fetch my gmail conversations and expecting them to get converted to 1 ticket per conversation. But it's not consistent. Sometimes, it works as expected, but most of the times, it creates duplicate tickets for the same conversation. And the duplicates are not the same number either. Sometimes 2, sometimes 3 and so on.

So I tried to bypass the default fetch sequence by setting fetch interval to 1 (minute) and fetch size to 1 (email). And I did the following steps:

Place exactly 1 email message from a conversation in chronological sequence in Helpdesk.
Manually invoke cron.php from the command line.
Check ticket has been created
Wait 1 minute for fetch interval to lapse
Ensure the message has moved from Helpdesk to Tickets (in Gmail)
Place the next chronologically correct message from the same conversation into Helpdesk
Manually invoke cron.php again
osTicket creates a second ticket instead of attaching to the already existing one

Very rarely, it does it correctly by attaching to the same ticket and I couldn't find any consistent way to reproduce the circumstances under which it processes emails correctly.

I have poured over the archives and found this issue reported many times from a decade ago, but didn't find any concrete resolution to the problem, so I am seeking help here again.

Mine is a low volume, single user scenario (at most 2-3 tickets a month), but I do have 20 years of service records sitting in Gmail that I would like to import first, before starting to use the GUI for future tickets.

With the help of chatGPT, I created 21 versions of the importer script and nothing seemed to work. Tried invoking the cron for 1 message at a time, tried invoking the TicketController() api, tried manipulating ThreadEntry directly by injecting the ticket number into ThreadEntry calls and nothing worked. chatGPT's latest version 21 even went so far as manipulating the sequence to fetch the ticket number after the first message, then modify the subject for subsequent messages (within the same conversation) to prepend the ticket number based on the format from a test ticket created via GUI. Even that didn't work!

Then there's other inconsistencies in how emails get processed. Consider the following scenarios:
Scenario 1: Use GUI to create ticket
When I go to New Ticket form, I am asked to fill out User and Collaborators, then subject and message. Then choose the system email id (not agent email id) to post the ticket. The ticket that is created has From: showing the User on the Open Tickets tab, which is what I expect.

Scenario 2: Import 1 email from Helpdesk
When the first email in the conversation thread was sent from my gmail account and this gets imported, it does 2 unexpected things. First, it creates that email as a user in osTIcket even though it is clearly registered as a system email (and all settings reflect outgoing mails to use system emails, not agent emails). And second, it shows From: as the newly created user (for system email) and the actual users are attached as collaborators to the ticket. Only time the From: is shown correctly is when the customer initiated the conversation first and I replied to that as a consequence.

These are only the start of my problems. I also noticed that all imported emails are being created as tickets with today's timestamp, when I would expect them to create tickets with timestamps from the gmail thread. But I found a script from the archives (parseDatesFromImapEmailHeaders.sql) that seems to fix this after the fact. But I haven't gotten to this step yet because I am still struggling with getting osTicket to import my gmails correctly.

Am I missing something basic in my setting? Seems like this email import is something everyone would want to see function properly before migrating massive amounts of historic data.

Please help. I am now at the point of giving up!

Cheers.

KevinTheJedi

devaps

Provide screenshots, examples, headers, etc. Via email we match headers just like any mail provider. First In-Reply-To, then References, then any message-id’s in the body, then lastly fallback to ticket number in subject.

Is the system email an alias or anything? I mean we have this running for thousands (if not millions) of people and it works flawlessly. 9/10 it’s a mail setup/routing issue or user issue or something along those lines. Not saying that’s what’s happening here but if it was a real bug then I would expect way more reports and the application to be unusable for more than just you.

Cheers.

ntozier

devaps So I tried to bypass the default fetch sequence by setting fetch interval to 1 (minute) and fetch size to 1 (email). And I did the following steps:
Place exactly 1 email message from a conversation in chronological sequence in Helpdesk.
Manually invoke cron.php from the command line.
Check ticket has been created
Wait 1 minute for fetch interval to lapse
Ensure the message has moved from Helpdesk to Tickets (in Gmail)
Place the next chronologically correct message from the same conversation into Helpdesk
Manually invoke cron.php again
osTicket creates a second ticket instead of attaching to the already existing one

I presume you still have those emails, can you:
Inspect the (raw) headers, do the message-id fields match?
(you can do this either using a text editor to open the file and look at the email directly, or in gmail using show original)

If they do not match that is likely the culprit.
osTicket tries to match based on a number of things, but if the message-id is different... it really breaks most of it.

From another post:

KevinTheJedi We first check the References and In-Reply-To headers for any relevant Message IDs matching any existing thread entries. If a matching Message ID found we thread through that Ticket. If none found we move on to the Message ID itself to see if we can parse it (ie. if our system sent it out). Lastly, we check the Subject line for any Ticket Numbers in the format of #number. If a number is found we attempt to look it up and if available we thread through that Ticket.

devaps

Thank you for your prompt reply. I have now made some progress, but still having issues.

I made the following changes inside the Ubuntu VM:

Synced all timezones to use local time instead of UTC. This meant installing QEMU so that my MacBook's time syncs with VMs time. I also set the VM to explicitly set timezone to Asia/Kolkata. Then updated mariadb's timezone to use SYSTEM as its timezone. Then I changed php.ini in both apache2 and cli directories to use local time.
In apache2's php.ini, I changed post_max_size from 8M to 30M, upload_max_filesize from 2M to 25M, memory_limit from 128M to 256M
In cli's php.ini, I changed post_max_size from 8M to 30M, upload_max_filesize from 2M to 25M and retained memory_limit as -1.
In osTicket GUI, Admin Panel->Settings->Tickets, I disabled Claim on Response and Auto-refer on Close.

With these settings, when I tried to process one year's worth of mails comprising 157 messages from 60 conversations, it failed again. But when I process 2-3 conversations (~ 6 messages in total) in a batch, it's working now. It is correctly converting 1 conversation into 1 ticket.

Once I successfully process an entire year's batch I am happy to share the script chatGPT wrote. From what I can tell, it seems to be doing what I designed it to do.

Now my question is why did those settings matter and which one or ones in particular I should change so that my entire year's batch can get processed? If I increase some or all of those settings to be in GB instead of MB, will it matter and is it an overkill to process just 150-odd messages. These are normal back and forth conversations and some have attachments, but nothing alarming in size - just a few PDFs and JPEGs shared over email.

Thank you.

Cheers.

devaps

I spoke too soon. What seems to work when I last reported, has again stopped working. So I’m back to duplicate tickets. It seems like a clean start on a fresh database setup always ends up with duplicates, but after many runs deleting incorrect tickets, it suddenly seems to work. I have already spent too much time trying o figure out why it works sometimes, so I’ll close this thread for now and wait for 2.0 to see if that fixes things.

Cheers.

ntozier

I'm going to guess that you're not doing anything with the emails once they are processed. (leave alone)
Since I'm sure you do not want to delete, have you considered moving processed emails to a folder?

I'm guessing that there is something wrong with the message-id comparison when checking if the email needs to make a ticket.

KevinTheJedi

ntozier

My thinking is the post-fetch settings mixed with process ordering (which post-fetch settings and mailbox volume can affect).

Cheers.

devaps

As indicated in my original post -
"My fetch folder is Helpdesk and Fetch Folder Action is to Archive/Move to Folder: Tickets"
I have tried changing the setting to Delete as well, but that didn't work either.

I didn't really understand Kevin's response. Mine is a very low volume (max 200 tickets a year), single user scenario. I am not exposing this to the entire customer base. They continue to write to my gmail as they have done before. Instead I run it on localhost in my browser and fetch tickets as the conversations get closed in gmail (hoping to convert each conversation into a single ticket)

Cheers..

KevinTheJedi

devaps

Re-reading your OP I may have found something that I missed the whole time. In your OP you said you set up YOUR email as a system email. Are you also using that as the user and/or agent email?

Cheers.

devaps

No, that's not the case.
My system email = gmail
My admin email = x.y@example.com
My agent email = yahoo

However, as stated in my post, I only imported all my clients as users. But as soon as I process even one conversation, my system email gets created as a user. I have "Allow Any User to send email" setting turned on. If I turn this off, my system email doesn't get created as a user, but it also skips messages from the conversation when I am the sender.

Cheers.

KevinTheJedi

devaps

Then it sounds like a mail routing issue or something along those lines. The system uses the FROM address as the user email. Your email address should be the FROM address in the emails you are sending. If the system is creating a user for the system email then that means the system email is in the FROM field; not your email address. That would definitely cause issues. We do have a filter action that allows you to use the Reply-To address as the user email for instance just like this where your system email address is the sender (FROM) and the real user address is in the Reply-To. Can you post full raw headers of an example where system email is created as user instead of your address?

Cheers.

devaps

KevinTheJedi Your email address should be the FROM address in the emails you are sending. If the system is creating a user for the system email then that means the system email is in the FROM field; not your email address. That would definitely cause issues.

Ok this is confusing me. My whole support desk is a single email address (which is a primary gmail account - no alias). But since osTicket wants the admin email to be different (I could be wrong, does it?), I created my admin and agent profiles with different email ids than the support email (gmail).

ntozier I presume you still have those emails, can you:
Inspect the (raw) headers, do the message-id fields match?
(you can do this either using a text editor to open the file and look at the email directly, or in gmail using show original)

If they do not match that is likely the culprit.
osTicket tries to match based on a number of things, but if the message-id is different... it really breaks most of it.

I don' t think this is the issue. All my conversation threads are clean and they group nicely into a conversation in gmail. My script also prints the message headers in the logs and I don't see any broken headers.

Anyway, per Kevin's request, I fired up osTIcket again after a long pause today on a clean pristine database. And today, it's processing everything correctly, so I am not able to send screenshots of the discrepancies I noted earlier. My gmail history for 2026 has 157 messages in 60 conversations and so far it has processed around half of them. I am going to let it run till the end to see if it breaks.

But in the meantime, I am attaching screenshots of my setup. my questions are -

If I am to use this as a single user, running on localhost instance, can I do away with Agent profile and use the sole Admin profile to manage tickets?
If that's the case, does my admin email have to be different from my main support/system email (which is my gmail)? I really need only my gmail to fetch and respond to tickets.
Ideally, I only want to setup one single email (my gmail) and use that for admin/agent email (From) as well as my fetch and outgoing mailbox. Is this possible? How should my setup be if I have to achieve this single user scenario?

Please see some screenshots from the successful run that's still ongoing today. As you can see, my system email is my gmail. My admin and agent emails are all different. My support email (gmail) being created as a user and all my tickets that are processed so far have my support id in the From column in Open Tickets view. This is because I initiated all those conversations in gmail. Only time From is shown as the actual user is when the first message in the conversation was initiated by the user instead of me (see tickets nos. 506112 and 821668).

I am unable to attach all the files and screenshots, so here's the link to the folder containing all relevant pages -

https://mega.nz/folder/0G4kBRIR#j65WPNf-0MssIDi03xO_IA

Hopefully you can access and view the files in that folder.

Cheers.

KevinTheJedi

devaps

I will not go to a third party site for security reasons. You can upload everything directly to this forum. Zip everything up and upload the zip if you are otherwise restricted.

As for the emails needing to be different, absolutely. Your system email should be unique and should not exist as a user or agent email address nor the admin email address. That would cause the scenario you are experiencing (and more).

Yes, you can use the sole admin agent to manage tickets. Admin is also an agent; just with extra privileges and access.

Cheers.

devaps

KevinTheJedi I will not go to a third party site for security reasons. You can upload everything directly to this forum. Zip everything up and upload the zip if you are otherwise restricted.

ost-gmail.zip
915kB

Sorry, I didn't know links to external sites are against protocol. Yesterday, it kept saying I am not authorised to perform the action or something to that effect. Today, I am able to upload the zip file, so I have attached it here.

So regarding that entire year's batch that hadn't completed as I was writing to you yesterday, it again failed after successfully processing 53 conversations. I have attached eml files or 2 conversations that have failed in the past (from recent runs). The SIP restart-.eml are one set and HDFCMF transaction-.eml is the other set. The latter set are the ones that end up with duplicate tickets in the current run. But there's no consistency in which ones fail and which ones succeed.

KevinTheJedi As for the emails needing to be different, absolutely. Your system email should be unique and should not exist as a user or agent email address nor the admin email address. That would cause the scenario you are experiencing (and more).

This is not my setup currently. My admin/agent/system emails are all different right now. But if what you say is true, then it is apparently creating the system email as a new user in order to process tickets. The zip file contains screenshots of various sections of osTicket showing this to be the case.

Thank you for clarifying that I can run the app with just admin user alone. When I run a clean install next, I will get rid of the agent profile to eliminate conflicts due to settings on that profile. The zip file contains agent, admin and system profiles that were used in the last run.

Also, thanks to you and @ntozier for helping me trying to find a fix for this. Appreciate your time.

Cheers.