Ok, I think I have this all figured out - and I noticed that the switch to Windows Authentication (instead of "Anonymous authentication", using the built in username/password form in Osticket) was also causing problem for the "attachments on the filesystem plugin..."
On IIS 10 when you have a website with Anonymous authentication and default application pool settings (which is how I originally had it set up) file reads/writes to the filesystem are performed under the context of the built in "IUSR" account, this also applies to temp files it tries to create in C:\Windows\Temp or whatever temp directory you have specified, and also in your nominated directory for the attachments on the filesystem plugin. If you check files in these directories you'll see they're owned by "IUSR".
However when you enable Windows (HTTP) Authentication in IIS in conjunction with HTTP Passthru authentication plugin everything changes.
By default the user authenticated via the browser (which is a domain user account) is now also the user account that IIS uses to read and write files, and if all of your users don't have read/write access to your temp directory and file attachment directory things break....and just giving full control to all users in those directories is probably a bad idea.
So here's what I did to fix it. I don't know if this is the best way to do it (not an IIS expert as I keep saying, so take my bad advice with a grain of salt...) but it worked.
What we want to do is have IIS read/write files on the filesystem as the same user no matter who authenticates via HTTP authentication - emulating the behaviour of Anonymous authentication. This leaves it up to OSticket to decide who has access to what file attachments/temp files etc. However you can't choose the IUSR account for this as it is a "special" built in account without a password.
So I created a local (not domain) user account called "OSTicket" on the server with a strong random password, then in IIS you can specify that this is used for file access instead of the individual user accounts:

Change physical path credentials to the account you created also entering the password:


From now on this user account is used for all file reads/writes performed on behalf of OSTicket users, so permissions should now be set for temp and file attachment directories.
To do this I added the local OSTicket user account to the local IIS_IUSRS group which is designed for this purpose, then adjusted permissions on the folders for this group.
The Temp directory (C:\Windows\Temp for a default install) should already have full permissions for CREATOR OWNER, and you should make sure that the IIS_IUSRS group has special permissions "list folder / read data", "create files / write data", "create folders / append data".
This allows the user to create files and modify/delete etc only files which it has created, similar to a sticky /tmp directory on Linux.
In my file attachments directory for the file attachments plugin I simply gave "full control" to IIS_IUSRS.
This seems to have fixed the issues with both file attachments and the temp file error I was seeing when pasting images into the editor. If you create a new attachment on a knowledgebase article then find it in the filesystem you'll find it's now created as being owned by the new user account you created rather than IUSR (anonymous authentication) or the requesting user. (windows authentication prior to this configuration change)
The moral of the story is - if you are going to use the HTTP Passthrough plugin with Windows Authentication in IIS you have some extra configuration work to do in IIS to make it use only one fixed user account for file access/ownership and also need to adjust the permissions on your temp directory and file attachments directory to have things working properly with the user account you create.