KevinTheJedi

Yep, I see that I am still running php 7.4 for my automated cron job (when we should be running 8.0)... I need to fix that in my windows scheduled task...

However when running with php 8.0 I get the same thing (doesn't matter)...

C:\>"C:\Program Files (x86)\PHP\v8.0\php.exe" C:\inetpub\wwwroot\Helpdesk\api\cron.php
string(9) "Subject: "
string(22) "Test ticket from yahoo"
string(8) "Poster: "
NULL
string(8) "Alerts: "
bool(true)
string(13) "Autorespond: "
bool(true)

KevinTheJedi

Also, when I send a reply in from my yahoo account, I get no output from running the cron job from the command prompt... Nothing at all.. And I get no agent alert or confirmation response back to yahoo..

... Yet the reply does seem to process into the ticket ... I can see it there in the portal ...

So it looks like this code is never hit, when I reply from my yahoo account (but is hit when I reply from my home account)... And it looks like the email is being processed for the ticket ...

Is that helpful??

... Again, I appreciate this so much, thank you!

Oye, it seems I must have hit some kind of posting limit, and now my most recent posts are "awaiting approval"....

Anyhow, hopefully you will be able to see my posts...

And to be clear, when I did get the output back, when I posted from my home account, I did get the agent alert and the message confirmation... So that did work...

I got no output when replying from the yahoo mail user, and did not get the agent alert or the message confirmation...

KevinTheJedi

Okay,

So I have done a lot of digging into the postMessage function, and after many var_dumps and emails going back and forth, I discovered that the issue boils down to the $vars being passed into the function with $vars['autorespond'] set to true on some emails replies coming in, and $vars['autorespond'] set to false on others…

I can prove this because at the very top of the function I put this line:

var_dump('VarsAutoRespondAtTop:', isset($vars['autorespond']) ? $vars['autorespond'] : 'Autorespond var not set');

and when running the cron job manually at a command prompt, I would get true or false depending on where the email reply was coming from… (vars[‘autorespond’] was always set, I would never get “var not set” from that)…

… Now earlier, I mentioned that the yahoo mails coming in were not showing anything in the output… Somehow, when I was working with it later in the night, the yahoo mails WERE showing output… I think what might have been happening is that even though I turned off the chron job (or the Windows scheduled task), the helpdesk was still seeing activity from the other member in our support group. Because of that, I think emails were still being fetched (because of the activity?), and it was (I guess) perfectly correlated with when I was testing yahoo mail and when I was testing my home mail … At least that is what I think was going on because when I was testing this off-hours I was getting responses from the var_dumps each time I sent in an email…

One other thing about the home mail…. I have several portals for that email, I can use my Thunderbird email client, or I can use the email client on my cell phone, or the zoho web mail portal for my account… So when I reply to a helpdesk email from the Thunderbird client or my email client off my cell phone, I would get agent alerts every time… When I reply to a helpdesk email from the zoho web mail portal, I would not get an agent reply every time…. So, even from the same email account, just the email client I am using does seem to make a difference… I must have tested this at least 10 times from each client, replying to the same exact agent response email, and I get the same results every time… And, of course, email responses generating from the yahoo web mail client, or the gmail web mail client, never result in an agent alert in any of the tests…

Anyhow, I spent a while testing var_dumps for everything, in and around the area of code we were testing before, and I found that after this code is run:

        $reopen = $autorespond; // Do not reopen bounces
        if ($autorespond && $message->isBounceOrAutoReply())
            $autorespond = $reopen= false;
        elseif ($autorespond && isset($vars['autorespond']))
            $autorespond = $vars['autorespond'];

The $autorespond would show true or false depending on the email source… Then I narrowed it down to the $vars['autorespond']; , and from there I noticed that the function was just being passed the vars set that way right in the vars parameter to the function… (the IsBounceOrAutoReply would always return “False” in all my tests, so that is not the issue here)..

Anyhow, clearly there is something going on with emails depending on the source and client being used to make the replies… I don’t know if this can be found in the headers, or somewhere else, and from here I am not sure where to go…

The good news here, I think, is that I can finally narrow the issue down to some very specific circumstances… Before it all seemed kind of random and we would sometimes get alerts and sometimes not, and this drove us crazy… Now I can reliably reproduce the issue and I get the same results every time I take certain actions.. Also, I think I know (at least in a general sense) where in the PHP code this issue lies, even though I have no idea what to check for next or how to fix it…

This is a lot more than I knew just yesterday…

So what do you think??

If you would like, I can send you headers again, except this time from the same email account (just from different portals), as we are getting different results depending on the portal… Would that be helpful??

@KevinTheJedi

Hello Kevin,

.. Or for anyone else that comes across this ...

So what I did was I commented out some of the code in /include/class.ticket.php so it would ignore the value in $vars['autorespond']... So my code there looks like this:

        $reopen = $autorespond; // Do not reopen bounces
        if ($autorespond && $message->isBounceOrAutoReply())
            $autorespond = $reopen= false;
        //elseif ($autorespond && isset($vars['autorespond']))
        //    $autorespond = $vars['autorespond'];

And did some more testing, and indeed now I am getting agent alerts from ticket replies from all of my accounts (yahoo, gmail, personal email via all portals, etc..).... Hopefully this will translate into alerts for all of our other helpdesk replies as well (I don't see why it shouldn't)...

So removing that check does help the issue... Although this really isn't a fix, its more like I am just masking the symptoms here and creating more risk and side effects... I'm guessing that probably puts me at a higher risk of an auto-respond infinite loop of ping-ponging emails... We have had that happen to us before, a few times, with Spiceworks... It was not pretty...

How much of a risk is it really?? It seems there are other catches in the code for that sort of thing...

Anyhow, I hope all is well.. I also hope that those headers I provided are at least somewhat helpful (are they?), and I would be happy to provide more if that would help..

Thanks much ....
...

So, put another way, would the following be a legitimate fix for OSTicket, in class.thread , line 465:

From:

        $vars = array(
            'mid' =>    $mailinfo['mid'],
            'header' => $mailinfo['header'],
            'poster' => $mailinfo['name'],
            'origin' => 'Email',
            'source' => 'Email',
            'ip' =>     '',
            'reply_to' => $entry,
            'recipients' => $mailinfo['recipients'],
            'thread_entry_recipients' => $mailinfo['thread_entry_recipients'],
            'to-email-id' => $mailinfo['to-email-id'],
            'autorespond' => !isset($mailinfo['passive']),
        );

To:

        $vars = array(
            'mid' =>    $mailinfo['mid'],
            'header' => $mailinfo['header'],
            'poster' => $mailinfo['name'],
            'origin' => 'Email',
            'source' => 'Email',
            'ip' =>     '',
            'reply_to' => $entry,
            'recipients' => $mailinfo['recipients'],
            'thread_entry_recipients' => $mailinfo['thread_entry_recipients'],
            'to-email-id' => $mailinfo['to-email-id'],
            'autorespond' => ((!isset($mailinfo['passive']))  || (isset($mailinfo['passive']) && ($mailinfo['passive']))) ,
        );

So doing that seems to fix the problem (and the 2-line comment-out workaround is no longer needed)....

So what we are doing is checking if "passive" is set. IF not set, then set autorespond to true, and if it is actually set, then set autorespond to the true/false value of "passive"...

However is this a legitimate fix to OSTicket to solve this problem, and should be submitted to the source code repository?? Or does this break your intended purpose for the "passive" / "autorespond" flag ??

Seems reasonable to me, however I don't know what the real intention of "passive" here is...

... Anyhow, if this is legitimate, can we use that fix for future versions ?

@KevinTheJedi

Okay, so if that is the case, then really the fix belongs somewhere here:

class.thread.php -> lookupByEmailHeaders -> Line 1490

        // Passive threading - listen mode
        if (count($possibles)
                && ($entry = ThreadEntry::objects()
                    ->filter(array('email_info__mid__in' => array_map(
                        function ($a) { return "<$a>"; },
                    $possibles)))
                    ->first()
                )
         ) {
            $mailinfo['passive'] = true;
            return $entry;
        }

... Because incoming emails that are evaluated sometimes hit this logic when lookupByEmailHeaders is called, but lookupByEmailHeaders is not called on every email...

When lookupByEmailHeaders is not called, then passive is not set, and so we don't have a problem (autorespond will evaluate to true).

When lookupByEmailHeaders is called, then passive is set here (seemingly every time, at least every time I have tested web based email clients so far), and the agent alerts do not go out.

... So I am trying to evaluate what is happening with that conditional logic there, and I think that is where I get a little stuck (I just cannot follow what it is doing there) ....

So what if I did a variable dump of $possibles, and $entry, and $mailinfo right before the return $entry, for web-based emails that fall into this conditional, and submitted that, would that be enough to figure out what is happening here for these emails??

Emails that work (do trigger an agent alert) never fall into this code because lookupByEmailHeaders is never called to begin with (and so passive is never set)... Is that a problem too??

... Thanks much

Oh yes... I am sending the email back from a alert that was sent out from the agent reply to a ticket via the UI.... That email alert would get sent out, once to yahoo mail, and once to my personal email. I would respond to each from the yahoo web mail, and from the android client for my personal email respectively. Agent would receive alert from the personal mail response, and not receive alert from the yahoo mail response... This happens consistently each time (at least for this particular ticket)..

So here is what I did...

I will attach a archive with a bunch of files here... I've tried to obfuscate my name (which is also in my email addresses) in the email as best I could... IF this obfuscation is causing issues (I don't think it would) then I can provide the original files instead if you like..

I reverted all files back to OSTicket stock 1.16.3 (fyi the only plugin we are using is the filesystem attachments), and then modified the lookupByEmailHeaders functions in the class.thread.php. I modified the code to dump out various variables to files at various points in the code, including a backttrace at the top of the function calls...

I then ran two examples, once with a response from yahoo mail (which fails at alerts), and once with a response from my android client personal email (which succeeds at alerts)...

... And my apologies ... It looks like I got confused by the two definitions for lookupByEmailHeaders in the code, and it seems that the lookupByEmailHeaders (the second one, with two parameters) is always being called for all emails... I am so sorry for that... That was wrong information by me... IT turns out that the second one (with two parameters) is the only one being called, at least with the two examples I am providing, but it is being called every time an email is processed, however it is exiting the function differently depending on the email client being used to respond (and so the files provided will be slightly different for the 2 examples)....

Rather than explain each file and what I captured, I will also include my modified class.thread.php file, which will show you each file I am dumping to and what I captured...

So there is some very interesting data in there, but what I think is the most notable is the match_message file (for each) showing the results of the matching to the "possibles"... This shows empty for the web client and yet it finds stuff from the android client...

Anyhow, here are the two file sets, and the modified source file..

Does this help??

I am happy to put in some more var dumps if you would like to see more stuff...

... Thanks much

... Howard

KevinTheJedi

Actually, now that I am looking at this a bit closer, I can see how the emails are routed here, and when you bring up forwards and mail lists, etc... I think I have have spotted a potential issue here...

See, when we bring in emails, we have OSTicket go directly to our google account via IMAP... Technically, we are under a google apps umbrella account, and the OSTicket general helpdesk email is an account under our google apps account with its own identity...

... However ...

When we send email out, rather than go through google, we have a localized Microsoft SMTP server here, and so all of our outbound SMTP traffic flows through there, which in turn smart-hosts it over to our SMTP relay provider (MailJet)... Mailjet then delivers the email to the recipients...

So I think somewhere in that process, the headers might be modified or messed with (??) either by our SMTP server or by Mailjet ?? Could that be the cause of our issue here? If so that could explain why this appears to not affect a lot of folks (Although, using a smarthost/edge server or SMTP relay provider for outgoing email is probably not all that uncommon)...

Anyhow, I believe in the past we tried to set up OSTicket to mail out directly from google apps, however we had some deliverability issues... However now I know that we need to turn on that "less secure apps" setting in google for this account. With that setting we should be allowed to send directly out using google and SMTP, routing through there instead of our local SMTP and MailJet...

Could this be the source of our problem?? Has this sort of thing caused these sorts of issues??

Perhaps we should give that a try...

Hello Hans,

So what fixed it was simply setting up our outbound SMTP server to go through google apps, rather than through our 3rd party SMTP provider (which we use generally for system generated outbound emails)..

I don't like setting up google as a SMTP provider, and Google goes out of its way to prevent it (although there are work-arounds at Google to make it work), but eventually we could get that to work, and it seemed to alleviate most of the e-mail notification issues..

I think the combination of receiving emails via google apps (where our domain email is hosted) and sending through a different SMTP provider for outbound (in this case, MailJet) created some discrepancies in the headers that OSTicket was unable to parse, and caused the autorespond variable to set like that... I tried very hard (at that time) to walk through the code to see where that was happening, but it got so complicated (especially without formal debugging tools) that I eventually just gave up and resigned to work around the problem as I mention above...

There are still issues with email alerts (at least for agents) in the software. For example, no matter what we did, when we tried to use Teams to set up assignments for our 2-person helpdesk, we just could not get email alerts to work properly (and notify the two of us), so we ditched using teams. Instead now what we do is just assign the other person as a collaborator on the ticket when we need to, and so we get notified that way (yes, its icky)... This is sort of a kludge and is somewhat inconvenient (because now and agent is seen as an end-user rather than an agent), but I think its the best we can do with it...

If there is a better way to do this and have email alerts work, I would love to hear it... With our prior helpdesk, we were able to just assign multiple agents (or have a primary agent, and a CC agent) to the ticket... Here we can "sort-of" do the same thing by using the collaborator work-around...

.... It should be noted that we are still on an old version (1.16.3), and so some of this might have been improved since the last time I tried to work on this ... I suppose I should try to update and re-visit this ...

... Not sure if this helps or not ...

... Howard