Previously I was using Deluge as the torrent client, and that had a plugin readily available for unzipping/unraring downloaded files. Unfortunately Transmission seems a little less mature in this regard so I had to come up with my own solution.
The end goal was to have any torrents with RAR files automatically decompressed to one central location. That location would still be in the Downloads folder, but not in the torrent directory itself.
Transmission has an option to run a script upon torrent download completion. So that was my way to implement this requirement. My script relies on the unrar command to be available, so there is a little bit of extra work required if you want to use it. First you need to install Homebrew and after that's done, type in the following (and press enter after) into the Terminal.app...
Below are the contents of the script. It's a Bash script and should be saved in the home directory i.e. in Finder you can open the 'Home' folder from the Go menu, Home option. You can save the script below on your desktop and then drag it over to Home. Make sure the name of the file is txunrar.sh.
Then in Transmission, go to the preferences Transfers > Management > When download completes option. It's possible to pick a script to run, which I set to txunrar.sh in the home directory for the logged in user.
Now whenever Transmission completes a torrent, it will run the script, which will check to see if the torrent is a RAR file or if it contains any RAR files inside it. The contents of the RAR files will be extracted into the Downloads > Extracted folder in your Home folder..
There is also a log file created and every time a torrent is processed, log entries are written to it. If anything goes wrong you will see it in the log file. This log file is called expand.log and is found in the Downloads > Extracted folder in your Home folder.
You can learn more about Transmission scripts from its wiki page.