This was the error I was getting - "The private key that you've seleted does not appear to be valid."
I did some searching around and it appeared that my key had to be converted first. The header of my key started with:
That was a sign that Google wouldn't accept that key. Luckily openssl can easily convert the key to an expected format with the following command:
Now after loading the second, converted, RSA key, Google accepted it without error!