Empty ticket body from email in ISO-8859-2

MaciejS

osTicket v1.14.1 / nginx/1.16.1 / MySQL 5.7.28 / PHP 7.3.12

Every email coded in ISO which contains polish letters (śćńęął...):

in subject - is trimmed from polish letter
in body - is empty

I'm using Exchange Online which uses ISO-8859-2 as the default encoding for Poland.
I did a lot of tests and when email encoding is UTF-8, the tickets are fine.
Cut out from the ISO header:

Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: =?iso-8859-2?Q?Rafa=B3_?= <rafal@domain.pl>
To: =?iso-8859-2?Q?Zam=F3wienia_?= <zamowienia@domain.pl>
Subject: =?iso-8859-2?Q?zam=F3wienie_?=
Thread-Topic: =?iso-8859-2?Q?zam=F3wienie_?=

Regards,
Maciej

KevinTheJedi

@MaciejS

Please post a full raw email that is causing you issues so I can use it to attempt to replicate your issue.

A full raw email is the raw email containing all headers and full raw body/content.

Cheers.

MaciejS

Hi,
Here You are:

Received: from VI1P190MB0400.EURP190.PROD.OUTLOOK.COM (2603:10a6:10:120::37) by DB6P190MB0439.EURP190.PROD.OUTLOOK.COM with HTTPS via DB8PR06CA0063.EURPRD06.PROD.OUTLOOK.COM; Thu, 27 Feb 2020 08:27:33 +0000 Authentication-Results: domain.pl; dkim=none (message not signed) header.d=none;domain.pl; dmarc=none action=none header.from=domain.pl; Received: from VI1P190MB0736.EURP190.PROD.OUTLOOK.COM (10.186.158.209) by VI1P190MB0400.EURP190.PROD.OUTLOOK.COM (10.165.198.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2772.14; Thu, 27 Feb 2020 08:27:31 +0000 Received: from VI1P190MB0736.EURP190.PROD.OUTLOOK.COM ([fe80::10dc:9d88:fbab:3248]) by VI1P190MB0736.EURP190.PROD.OUTLOOK.COM ([fe80::10dc:9d88:fbab:3248%5]) with mapi id 15.20.2772.012; Thu, 27 Feb 2020 08:27:31 +0000 Content-Type: application/ms-tnef; name="winmail.dat" Content-Transfer-Encoding: binary From: Janusz Testowy <j.testowy@domain.pl> To: =?iso-8859-2?Q?Zam=F3wienia_domain?= <zamowienia@domain.pl> Subject: =?iso-8859-2?Q?Wiadomo=B6=E6_testowa?= Thread-Topic: =?iso-8859-2?Q?Wiadomo=B6=E6_testowa?= Thread-Index: AQHV7Ue/m2Z8pjF0JU+/AO7buFTGeg== Date: Thu, 27 Feb 2020 08:27:31 +0000 Message-ID: <VI1P190MB0736A1E0EFC56C445C5CCFFDEBEB0@VI1P190MB0736.EURP190.PROD.OUTLOOK.COM> Accept-Language: pl-PL, en-GB, en-US Content-Language: pl-PL X-MS-Has-Attach: X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: <VI1P190MB0736A1E0EFC56C445C5CCFFDEBEB0@VI1P190MB0736.EURP190.PROD.OUTLOOK.COM> MIME-Version: 1.0 X-MS-Exchange-Organization-MessageDirectionality: Originating X-MS-Exchange-Organization-AuthSource: VI1P190MB0736.EURP190.PROD.OUTLOOK.COM X-MS-Exchange-Organization-AuthAs: Internal X-MS-Exchange-Organization-AuthMechanism: 04 X-Originating-IP: [149.156.1.3] X-MS-Exchange-Organization-Network-Message-Id: 325dca8c-dd4d-4d83-8767-08d7bb5ee3ba X-MS-PublicTrafficType: Email Return-Path: j.testowy@domain.pl X-MS-Exchange-Organization-ExpirationStartTime: 27 Feb 2020 08:27:32.0654 (UTC) X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmit X-MS-Exchange-Organization-ExpirationInterval: 1:00:00:00.0000000 X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmit X-MS-Office365-Filtering-Correlation-Id: 325dca8c-dd4d-4d83-8767-08d7bb5ee3ba X-MS-TrafficTypeDiagnostic: VI1P190MB0400: X-MS-Oob-TLC-OOBClassifiers: OLM:1728; X-Forefront-Antispam-Report: SFV:SKI;SFS:;DIR:INB;SFP:;SCL:-1;SRVR:VI1P190MB0400;H:VI1P190MB0736.EURP190.PROD.OUTLOOK.COM;FPR:;SPF:None;LANG:pl; X-Microsoft-Antispam: BCL:0; X-MS-Exchange-AntiSpam-MessageData: lRZQCWhugogFKks6fUIQTvvayZvy9d/do1dipXLbT+PQPZFHffzRXy+wwoDHgifTeRx0b8qEAL2W+IlWjxdXdS2DpFQEDw62iGP/EuB1DGY0zTM3R1B8ol4z6DQaqUM9lK/4L7VPyeVQiUGFYXoP5A== X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Feb 2020 08:27:31.6598 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 6e1d6934-556b-4c39-a28e-371187eee58a X-MS-Exchange-CrossTenant-Network-Message-Id: 325dca8c-dd4d-4d83-8767-08d7bb5ee3ba X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: E9wEULJT09kDZCfkXWB5ahw2feK4plYLtRLICp3bexxhwq94dgF2Jrrxls/VxNe/awcHN0HjPoX3Pj2iOO4S3g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1P190MB0400 X-MS-Exchange-Transport-EndToEndLatency: 00:00:01.9301027 X-MS-Exchange-Processed-By-BccFoldering: 15.20.2750.019 X-Microsoft-Antispam-Mailbox-Delivery: ucf:0;jmr:0;ex:0;auth:0;dest:I;ENG:(750127)(520011016)(706158)(944506383)(944626516); X-Microsoft-Antispam-Message-Info: +0Kph/G4reUhM7TJc4twpy3Q9WcmMk2YxPHZtjU5PVGQaPBN409q9FNBcHynu8kHeKEMGwU/kn9CEqURB9ji+sSQuPnkBcEKk+EgG/pdv8FzbIohrNi+yHbNAloiKfH2O3mB9Q0euMxxVGDEz4HF2nNN1wP8bw8iXkMo3FyzKqf3cBHN8lWp7J/FYrnLXo2rrZTgpluUdtwKjro0hTxvhHT1vfDIl74aEvdmhTbWR0vyUbDipM6WRMUWOtm8Ny6bzwn9+F39GlPr4g9OfqCpK7d7nMG6UyzRpY31zk6tbRT1C1CuyANHTGiQ4FKee+Evu1BesjlMUCLtWJfkcSnaF+siqbgTEj2UdpzXBYtKh4zdcyEyihw9hItMeeZh7Qs4C+NdN380s51mp3H1oJ3OQEjSs+dGcgm8Pe+WblOseOTWxEmFrf1R3LY+raP5M11BnFitqe18fW6uoBs1CtBUgQ== X-EsetId: 37303A299CB3D569607D60

Is it OK?

Regards

MaciejS

Any update?

Regads

KevinTheJedi

@MaciejS

You just posted the headers. We need the headers AND raw body. The body is the most important part as that's where the issue is arising.

Cheers.

zajc

I have exactly the same problem (osTicket v1.14.1 / nginx/1.16.1 / MySQL 5.7.28 / PHP 7.3.12) with Slovenian characters (ISO-8859-2). It works only if email encoding is UTF8. Here is the source mail.

Delivered-To: bla.bla@siol.si
Received: by 2002:a4a:d02c:0:0:0:0:0 with SMTP id w12csp431243oor;
        Fri, 1 May 2020 02:26:52 -0700 (PDT)
X-Google-Smtp-Source: APiQypJnoy24xkbmtQW186il9sN1ceez2VY4UJdyoFJoFX5Wq6zO6SzHbC6yWt91x0hxhO+zLym8
X-Received: by 2002:a17:906:bcec:: with SMTP id op12mr2356056ejb.245.1588325212696;
        Fri, 01 May 2020 02:26:52 -0700 (PDT)
ARC-Seal: i=2; a=rsa-sha256; t=1588325212; cv=pass;
        d=google.com; s=arc-20160816;
        b=UipsYlthXOWZ0Pw1W9G7jlCR9VV8Okc0el4w6Sd+GtE/0iV8bSmwpGrVMR/yde6v3s
         AtCXeo+zmEDiFqSQyfsw3Z+BQn2H2dJ45119ySUWA7GEUucQXLrJxJ36ljs4oHFVFpMI
         kKSLIA9gsY+xs6t5u8hpoJ09h54EScon+J28hrm7lTWb9W9mQMW5kw82DO5BBikutNMA
         zdKl8xPwLUy5HIb34JazTvRMRweyj7PkrIxBMYdjMcz21JyrmM+WG7GQcG1hJ6qUPMfU
         XSNyaGtnzaY310Yu4OxqGqXjWRkrTfpToyyCRtDatTXxeMpwJBhgPBLKQ94DpVCt3jKb
         TVPQ==
ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
        h=mime-version:content-language:accept-language:message-id:date
         :thread-index:thread-topic:subject:to:from:dkim-signature;
        bh=ZIPFY26fxJfLG0x25BtWrm246txMIPQR1S7hRV8tBRY=;
        b=Sr7ILZcU6YpJlmffMa0BCFyXHA2jvHrbM8wpXyHdu3S/s4/NpcErPyj1IBXfZzdjwp
         thbdOCW6G2w4OPd57/2RhPNzbTm/OErJuGt84TbJ6QJuGVQcpjp6TRid9nRyCDi3jOhW
         uwCXiAmY/jj47os4H5iMzm68WmF9t7aDU3g1W3HpQcA+Xn87MiCIVsqPzOkgzVGhm7sC
         fSsI+fA8jeEoemcgFZAahjWqbvoMpCz4HE9x+owdX1f5j2JtVGEM7iMCGh4I7Pm1SYoH
         BuKvck7tGBz3BAiI9wtWDEyIcSH/xf2jixm0PeqKOwPN47kL93kzsj4TECSqbQTKwPtx
         /QNA==
ARC-Authentication-Results: i=2; mx.google.com;
       dkim=pass header.i=@hotmail.com header.s=selector1 header.b=owYkIFWf;
       arc=pass (i=1);
       spf=pass (google.com: domain of jesenice@hotmail.com designates 40.92.73.31 as permitted sender) smtp.mailfrom=jesenice@hotmail.com;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hotmail.com
Return-Path: <jesenice@hotmail.com>
Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-oln040092073031.outbound.protection.outlook.com. [40.92.73.31])
        by mx.google.com with ESMTPS id rs10si1405036ejb.288.2020.05.01.02.26.52
        for <bla.bla@siol.si>
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 01 May 2020 02:26:52 -0700 (PDT)
Received-SPF: pass (google.com: domain of jesenice@hotmail.com designates 40.92.73.31 as permitted sender) client-ip=40.92.73.31;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@hotmail.com header.s=selector1 header.b=owYkIFWf;
       arc=pass (i=1);
       spf=pass (google.com: domain of jesenice@hotmail.com designates 40.92.73.31 as permitted sender) smtp.mailfrom=jesenice@hotmail.com;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hotmail.com
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
 b=M8VEPynm3sH5L1KXoKbX28bS0xk+FT0iewCuV7V9LNNSOPHVSqdH5mvwiY3V++LAZQ2ukmkq0h87tePRtZ+90kKJmWflv4RJW19io+up48vKxW0viDyoJak5YuIglkuOajgBm+ilNYkEhXuTJ8wwRD5aEqAT4gosDODvTzixlZyct2Zp9nN+QrwBddxUr/bA0gCwoX6UPBgDecQ6IAkovRTFYQjtc29ND6FC6qqmI8m8FSBnH40GFMWI+/AkpYxfL2sP9lT3Nc9HcD7z0m+6EYBeO6hhXjUTNJwYTxiphoUMpMeZnUMiVzhdIhbDkqVp/4nx/JHMUSPX895gZh9aCg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
 s=arcselector9901;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=ZIPFY26fxJfLG0x25BtWrm246txMIPQR1S7hRV8tBRY=;
 b=HYV5FGVlCLT4pKRpgSvA51D6v726TER9S8gGzlmAKGUNxcYuMmiQfmMhLUvszsE1cVUYUACJbZcXkkwrrAYmKLchpase8ZVIF7UG9bWglibdT22Spd0lCzZ/J9bsXL8FTdZX9blHlUMWESVzSo/mwm65X+6bchtZnfsbrlvpfef2LYbIiyeL8JTiD2YGvlEn4Wcvzs7ETlY3H/JDsYO8abpesYSAz8iUtHYIYk2nWHqGo1hMHPqvXsL9fZZOvyriQE09BaDMhAzlDn2qKABnXEKlfGoEMNFc6dcBwPHRbeOXUnViyS0kpsDMLlHmXtkQY7ARwwKvd1NexdTw8w/mXA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none;
 dkim=none; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com;
 s=selector1;
 h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
 bh=ZIPFY26fxJfLG0x25BtWrm246txMIPQR1S7hRV8tBRY=;
 b=owYkIFWfCC1NRzcBToTlVZie3ltYSXG/mLtdn0xvufDwJJtC8Yfsi7xi6+4zmj23IQJv8yqlPaiAOlFSNYKj4e6fyXCNIu7qZakTdATdSrKgkKRXtTxkb+Upf2vxZOVXzEVpMmy2cCxvpkjNT1U7YV1UFTU5vaWEljPQqjQY3Y/DemJkCBMe5iMTsyn0NbY2lI7L/5BIcoCHMTgGUSaT2IymveTFdu2TlGILKrm4hs9HvNHYJeTk3ge7M81UGkEDmG79AGf+46UngfuGbJ3uR6f/a0l5AKp8ZcMblc0kozozaHc5bwm82vW4u2YBKfgCMUnLIO5/qt3DPD868k+UUg==
Received: from VI1EUR04FT019.eop-eur04.prod.protection.outlook.com
 (2a01:111:e400:7e0e::52) by
 VI1EUR04HT146.eop-eur04.prod.protection.outlook.com (2a01:111:e400:7e0e::394)
 with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2937.15; Fri, 1 May
 2020 09:26:51 +0000
Received: from DBBPR03MB5222.eurprd03.prod.outlook.com
 (2a01:111:e400:7e0e::48) by VI1EUR04FT019.mail.protection.outlook.com
 (2a01:111:e400:7e0e::251) with Microsoft SMTP Server (version=TLS1_2,
 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.19 via Frontend
 Transport; Fri, 1 May 2020 09:26:51 +0000
Received: from DBBPR03MB5222.eurprd03.prod.outlook.com
 ([fe80::8430:b96b:beae:6811]) by DBBPR03MB5222.eurprd03.prod.outlook.com
 ([fe80::8430:b96b:beae:6811%3]) with mapi id 15.20.2937.028; Fri, 1 May 2020
 09:26:51 +0000
From: zajc cjaz <jesenice@hotmail.com>
To: "bla.bla@siol.si" <bla.bla@siol.si>
Subject: =?iso-8859-2?B?qXVtbmlraSDIxq6p0Ojmvrnw?=
Thread-Topic: =?iso-8859-2?B?qXVtbmlraSDIxq6p0Ojmvrnw?=
Thread-Index: AQHWH5qlBsvWxopuUE6OanryTKR1zA==
Date: Fri, 1 May 2020 09:26:51 +0000
Message-ID:
 <DBBPR03MB5222E1EFE0BBF05A9A724AF2D7AB0@DBBPR03MB5222.eurprd03.prod.outlook.com>
Accept-Language: en-GB, en-US
Content-Language: en-GB
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-incomingtopheadermarker:
 OriginalChecksum:293DEE23BB57AA27C521741D9A874C4FBFE4EADA8ABD8A7BAAF939F072622324;UpperCasedChecksum:87E79A55B3FD235CCC02F02663DB48839B9BAD0E8F3F134E418EC60309C057EE;SizeAsReceived:6694;Count:42
x-tmn: [fJayhbUQ3dY3yeq5Zy83521N/EHHjBuZ]
x-ms-publictraffictype: Email
x-incomingheadercount: 42
x-eopattributedmessage: 0
x-ms-office365-filtering-correlation-id: c10c9868-2167-45f3-f0fc-08d7edb1c814
x-ms-traffictypediagnostic: VI1EUR04HT146:
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
 FuurofOpTl+bO0+efQgdgTcpS7bUCgCx9D4YzGXMQrS9ySkHpHZgOAyKeWeIIaNvQKIQO2pca/nDcp+RczCyiUjHccLiGwj7p6uqAuTCLxQfrcgVi1Zj9rPTdGBuYz0o4YRud620YN2oMuO/tsMV1gxipxQFjGCeKzJJjXwgv03Ph1ntNldokD11I/B4tlJz7AySZXnwaw3DeT5OP+7OjQ==
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:sl;SCL:0;SRV:;IPV:NLI;SFV:NSPM;H:DBBPR03MB5222.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFTY:;SFS:;DIR:OUT;SFP:1901;
x-ms-exchange-antispam-messagedata:
 +TbN+gZQkYut4lx1FiHioF0poaCqMv96TQTObeSl1ebHJ+cknJHn5fLNYz/IXctjMgWayx02DzrWQEpGWCugJSmD31qYGRMEn8dlq38F1re3zmoEBvRDxo5t+WBWFOPy4u2A/eBkute/+FTcc0H0vQ==
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative;
	boundary="_000_DBBPR03MB5222E1EFE0BBF05A9A724AF2D7AB0DBBPR03MB5222eurp_"
MIME-Version: 1.0
X-OriginatorOrg: hotmail.com
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-Network-Message-Id: c10c9868-2167-45f3-f0fc-08d7edb1c814
X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 May 2020 09:26:51.7484
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1EUR04HT146

--_000_DBBPR03MB5222E1EFE0BBF05A9A724AF2D7AB0DBBPR03MB5222eurp_
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable

=A9umniki=C8=C6=AE=A9=D0

--_000_DBBPR03MB5222E1EFE0BBF05A9A724AF2D7AB0DBBPR03MB5222eurp_
Content-Type: text/html; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-=
2">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
 color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
=A9umniki=C8=C6=AE=A9=D0</div>
</body>
</html>

--_000_DBBPR03MB5222E1EFE0BBF05A9A724AF2D7AB0DBBPR03MB5222eurp_--

KevinTheJedi

@zajc

I piped your example into my local install of v1.14.2 and it worked just fine (see image below).

Can you try upgrading to the latest release of v1.14.2? Also, can you try piping that email into your helpdesk to see if it works that way? If so, then it has to be something with fetching specifically.

Cheers.

zajc

I have found what the problem is.

/include/class.charset.php line 62
$text = iconv($from, $to.'//IGNORE', $text);

if I change to
$text = iconv($from, $to, $text);

it works. Now I must find out why //IGNORE is not working in my setup.

zajc

I guess the problem is that in my environment the locales are not set properly and this is known iconv PHP bug.

I change in class.charset.php

if (function_exists('iconv')) $text = iconv($from, $to.'//IGNORE', $text); elseif (function_exists('mb_convert_encoding')) $text = mb_convert_encoding($text, $to, $from);

if (function_exists('mb_convert_encoding')) { ini_set('mbstring.substitute_character', 'none'); $text = mb_convert_encoding($text, $to, $from); }