Unfortunately there is no developer guide at the moment.
Beside that let me mention some things related to custom modifications:
To be able to update/upgrade in the future to newer osTicket version you should e.g. use git and create a repo to merge your changes with new osTicket versions in the future. Since the developers work hard and fast to implement new features, fix bugs and improve existing stuff, keeping your osTicket installation up to date is really important! And I am not just saying that out of fun, it's the truth which some learned the hard way... ;)
But the better way is definitely to propose your changes via pull requests at github, so the developers can include them into the core of osTicket. Maybe they will tell you that they can not include your changes into the project for some reasons or that they don't like your modifications, but you can work/discuss with them to get the things into a state that they are ok with the changes and then they will definitely merge your mods to the core. But keep in mind that they don't just say "no" since you or someone else wrote the code, no they just need to make sure that e.g. your mods don't conflict with there plans or break existing functions and so on.
Btw here is the github repo: http://github.com/osTicket/osTicket-1.8/
