The 3rd party wysiwyg editor doesn't handle images from the clip board gracefully most of the time (or at least it didnt used to) so I have always used the insert image button on the text window which has always worked reliably for me.
The undefined 500 sounds like you are trying to upload a filetype that you haven't defined (in osTicket) or that your webserver does not know how to handle the attachment.