Increase the speed of gmail email fetch

skulblock

Hi all,We have been running Osticket for last four years as our primary support portal. We have over 100 agents and daily around 4-5k tickets gets generated. Recently the volume of tickets have increased and we have spotted some issues. The fetch is via cron, I've set it to run every 10 minutes since every minute 10s of email land in our inbox. There was a volatile day recently and every minute the emails count was doubled. The cron still runs it fetches but it ignores few emails here n there. We thought initially it was some filter / ticket issue, but they are not having any issues per say, many where reply emails for old tickets. What could have caused this issue? If the ticket count doubles in forth coming months, how can we increase the speed of email fetch? We have around 5-6 filters also running. Specs attached. Outgoing emails are via an external SMTP server and not using the gmail smtp relay.

skulblock

Attachment went missing for some reason.. attaching again the specsAdditionally, we have 12 core CPU and 32gb RAM.

ntozier

"We have over 100 agents and daily around 4-5k tickets gets generated."Those are some impressive statistical numbers. Thanks or sharing them."What could have caused this issue?"I'm honestly not sure. I'll ask the devs to take a look at this thread but they may want an example email (with full headers, but you could truncate the body) to test with."If the ticket count doubles in forth coming months, how can we increase the speed of email fetch?"I'm not aware of any way to speed up transferring emails from remote server to osTicket. I imagine that its governed by your internet speed, the latency between the two, any speed throttling that the mail server does, and any timeout in the protocol, or lastly in the implementation of the mail collection library used in the project. Again I'll ask the devs to take a look at this thread. If you can reduce spam to the mail box some how, or utiliHow many emails are you fetching per cron run? I wonder if the emails you saw not getting fetched right away were because of a cron calling collection before the last collection had completed.

KevinTheJedi

There could be many reasons why some emails didn't fetch but I have a feeling it is due to the Email Per Fetch Limit. Say you have the limit set to 10 and the Fetch Frequency is 10 minutes. The system will check the mailbox every 10 minutes, look at the top 10 emails, and grab all that are unread. If you had the Top 20 emails as Unread then 10 of those would not be fetched. If more are stacking on top of those they would never be fetched (this is why we suggest moving them to a new folder upon fetching). So if you have a lot of email traffic I would suggest bumping up the emails per fetch limit.The way to speed up the cron is to up the Fetch Frequency limit and the cron run time. You can also look into Email Piping as an alternative.I hope this helps. Cheers.

skulblock

Hi @[deleted] and @[deleted],Thanks for the response, somehow missed them until now. We have set max fetch to 120 emails per fetch. It seems to be the max allowed by osticket to be set? And fetch cron frequency is 10 minutes." I'll ask the devs to take a look at this thread but they may want an example email (with full headers, but you could truncate the body) to test with." -- That day after few hours all emails were fetched, if it happens again, i'll take a sample email which is not fetched and paste its headers here. " The system will check the mailbox every 10 minutes, look at the top 10 emails," -- that makes sense, so the fetch is not FIFO? If so, that could be the reason why this is happening. Also the support admins had set like 8-9 filters which im assuming could also slow down the fetch time.

Grizly

If it's a GSuite account, you can use the admin console to split delivery to a local email server, like one on your ticket system, then pipe them to tickets as they arrive.

https://support.google.com/a/answer/2685650

http://osticket.com/wiki/Email_Piping

Bit more complicated to configure, but should handle more email per second than fetching.