Massive email bug- shutdown all systems, including emails and scripts- 1000's emails

DDUser

Holy cow! Found a MASSIVE bug that generated thousands of emails, eventually shutting down the email server (too many messages per hour) and throttling CPU for over usage of script.

A user entered a ticket with username and email similar to the following:

Name: postmaster@exchange.xyzserver.co

Email: postmaster@exchange.xyzserver.com

(Yes, no "m" on the first entry, and I've changed it to a generic "xyzserver" instead of the actual server)

The ticket was entered into a running osTicket system. The standard notices were sent to admin, dept heads, etc.

The Confirmation email (normally sent to the user, with the ticket number and such) was bounced from the postmaster account, which generated an additional ticket (mail delivery error).

A terrible loop was created where a constant stream of failed messages created new tickets, which (I think) also bounced. These tickets also went out to admin, dept heads, and to the alerts email address.

This quickly flooded our email sending system (the one configured in osTicket), which stopped accepting SMTP emails, which generated additional system errors, which generated additional tickets.

The system was so busy trying to process and send the tickets that our host went into automatic CPU throttling, slowing down the system.

We could not access the admin panel (I wanted to add the postmaster@... account to the ban list). I eventually got the email added to the banlist via direct SQL database access.

This stopped the emails from going out, but the tickets continued to be generated, now as errors.

This went on for a day and a half, now all the tickets started generating stale ticket alerts today.

I went into direct SQL database access and deleted all tickets with "Email delivery system" in the Name field. Over 3000 tickets were deleted.

In summary:

1. A single ticket started a flood of new tickets and emails

2. These processes overwhelmed the CPU of the host, blocking access to the admin banlist section

3. I couldn't "stop" osTicket, nor could my host.

4. This activity triggered automatic flooding controls which shut down our SMTP server

5. Thousands of tickets were generated.

6. A day or so later (we are set to 24hrs for stale ticket alert), each ticket generated a stale ticket alert and began the loop over again

7. Only by direct manipulation of the SQL database could I delete the tickets and stop the storm, including removing the original, offending ticket.

Had we hosted either of the domains used for osTicket or for the email system serving osTicket (two separate domains) on our main domain, we would have been out of business for a few days as a result of a single bad email address.

The closest forum post/topic I found is:

http://osticket.com/forums/showpost.php?p=16844&postcount=1(http://osticket.com/forums/showpost.php?p=16844&postcount=1)

1. How can we prevent this in the future?

2. Is there some way to limit the retries to 5 attempts, and stop the system from generating new tickets with each error?

3. Is there some way to validate email addresses in the ticket entry form?

4. Can this happen with any email address, or is it the "unlimited power" of the postmaster@.... account that contributed to this problem?

5. Is there an "emergency stop" that can be run on the system to stop the processing of all tickets (take the system offline) so only the admin can make changes?

Please help, as we had to take this offline until this is resolved.

Thanks!

- DD

Kelli

1 & 2: You can stop it by going to admin panel -> settings and setting a 'max open tickets' value.

3: it does validate the email address format, but .co is a valid TLD.

4: I honestly can't really remember right now. I know it's come up before, but I can't remember what triggered it specifically. I'll have to go back and look.

5: You can go into admin panel -> settings and change the helpdesk status to offline. If you can't get into the admin panel, you can edit it directly from the db, by changing the is_online value in the ost_config table. Note that if you log out of the system when it's offline, you won't be able to log back in and would have to manually change it from the db again.

DDUser

Thanks for tips, any way to prevent the bug?

1 & 2: You can stop it by going to admin panel -> settings and setting a 'max open tickets' value.
3: it does validate the email address format, but .co is a valid TLD.
4: I honestly can't really remember right now. I know it's come up before, but I can't remember what triggered it specifically. I'll have to go back and look.
5: You can go into admin panel -> settings and change the helpdesk status to offline. If you can't get into the admin panel, you can edit it directly from the db, by changing the is_online value in the ost_config table. Note that if you log out of the system when it's offline, you won't be able to log back in and would have to manually change it from the db again.

Thank you for your prompt reply. I have some additional question/comments about your reply.

If I limit the number of open tickets, won't that prevent valid tickets from being created, too?

3 & 4: I did ask about validating the email input. Perhaps I should have asked if it can TEST and determine if it is a valid email account? Of course, that wouldn't help if the account won't accept email, such as the postmaster account.

Maybe that's the bug? The email address is proper, would test as exists, but bounces mail back to the sender. (I could see this happening if the osTicket sending email address is placed on a spam list, with the emails returned to sender...?)

5: If I manually turned it offline, would osTicket stop creating new tickets and sending out emails? Great tip, by the way! Also excellent advice about not logging out of the admin panel when offline- good to know.

We REALLY like osTicket! I hate that my first post is a complaint/bug. I've been lurking and reading for some time, but finally registered so I could post this problem.

I plan to make some mods, which I can post up once they are ready.

But for now, we have to find a way to prevent the system from going nuts and killing the domains.

Is there a way to prevent the system from generating new tickets on system errors, or at least prevent all the duplicates?

Thanks!

- DD

peter

What version of osTicket do you have?

Are you using email piping or polling (IMAP/POP)?

Did you set maximum open tickets (it's important for loop control)?

What FROM email address was used on the bounced messages (post full email headers if possible)?

DDUser

Peter,

Thank you for your questions. I put my answers below.

I will private send you some very specific info that I don't want to post in the open, but will put enough in this thread to help others who may have a similar issue.

What version of osTicket do you have?
1.6.0
Are you using email piping or polling (IMAP/POP)?
POP/IMAP Fetch.
Did you set maximum open tickets (it's important for loop control)?
No, it is 0 for unlimited.
What FROM email address was used on the bounced messages (post full email headers if possible)?
I'll have to look for the actual message, I was deleting very quickly and it's mixed in with others.

Thanks!

- DD