Is there a way to set which departments an agent should be able to transfer a ticket to?

The closest way I've found was to give the agent extended access to these departments and uncheck in his Miscellaneous permissions "Ability to see all Departments". --> This way, the agent will see in the department transfer list only those he's got extended access to.

The problem with this is that the agent will be able to see/access all tickets from these departments...