I checked the Security & Privacy settings, and IntelliJ was listed as allowed to accept incoming network connections there.
This is what I ended up doing. Open the Terminal.app and type in the text below. Make sure that IntelliJ is not running while you do this.
This will re-sign the app and the prompt will disappear. Note: You will get the prompt one more time after doing this, however it won't come up the next time.
You will see something like this in the terminal: