Google Cloud service account authorization without OAuth

Google’s OAuth documentation goes in length about how to sign a JWT with the service account key in order to call their token endpoint https://oauth2.googleapis.com/token to get an OAuth token so that you can call actual Google Cloud APIs, only to mention at the end in a small addendum that you can skip the token endpoint step altogether and just use your self-signed JWT directly. 😬

In this blog post we’ll develop this last step, because it’s so much more convenient, reliable, and there’s a few undocumented things about it.

The normal flow

But before, let’s quickly look at the “normal” recommended OAuth flow. Borrowing this diagram from their documentation:

1. Create a self-signed JWT using your service account key.
2. Use it to authenticate to https://oauth2.googleapis.com/token to request an OAuth token.
3. Use that OAuth token to call Google APIs.

This is not bad, but having to go over the network to authenticate and refresh tokens before they expire adds extra overhead, delay, error handling, retry logic, and in general just an extra few things that can go wrong.

And I don’t like things out of my control that can go wrong.

The better flow

On the other hand, the poorly documented “service account authorization without OAuth method” consists in:

1. Create a self-signed JWT using your service account key.
2. Use it directly to call Google APIs.
3. Profit.

Same amount of steps, but you can imagine why I like this method better.

Implementing direct authorization from scratch

Typically, the Google Cloud SDK in the language of your choice takes care of this for you (and most of the time uses this self-signed method, because they too realize it’s a much superior method). But in some cases, you have to reimplement the authorization step, for example when running on Cloudflare Workers, which I wrote about in details in that article.

As of today their documentation mentions the JWT must have the following header and payload:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "SERVICE_ACCOUNT_PRIVATE_KEY_ID"
}
.
{
  "iss": "SERVICE_ACCOUNT_EMAIL",
  "sub": "SERVICE_ACCOUNT_EMAIL",
  "aud": "https://SERVICE.googleapis.com/",
  "iat": 1511900000,
  "exp": 1511903600
}

Where the parts in all caps are variables to adapt to your situation. Then the JWT can be signed with RS256 (RSA signature with SHA-256), and used in a Authorization: Bearer header against the service you included in the aud field.

And it does work most of the time (again check out my post to see the vanilla JavaScript implementation), but in some cases like with Google Cloud Storage, it breaks down.

When it breaks down

With Google Cloud Storage, when using the documented method with a aud field of https://storage.googleapis.com/, we sadly get an error when calling the API, e.g. when trying to get a file:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>AuthenticationRequired</Code>
  <Message>Authentication required.</Message>
</Error>

Or when trying to upload a file:

{
  "error": {
    "code": 401,
    "message": "Invalid Credentials",
    "errors": [
      {
        "message": "Invalid Credentials",
        "domain": "global",
        "reason": "authError",
        "locationType": "header",
        "location": "Authorization"
      }
    ]
  }
}

But the exact same code to generate a JWT works seamlessly with Pub/Sub, Datastore and other services! Why is that? Should we fall back to using the OAuth endpoint for those problematic services?

No.

The new, undocumented JWT payload

It turns out that you need to remove the aud field and replace it with a scope field, akin to the one we would pass to the main OAuth token endpoint.

In the case of Google Cloud Storage, our JWT payload would now look like this:

{
  "iss": "SERVICE_ACCOUNT_EMAIL",
  "sub": "SERVICE_ACCOUNT_EMAIL",
  "scope": "https://www.googleapis.com/auth/iam https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/devstorage.full_control",
  "iat": 1511900000,
  "exp": 1511903600
}

You can find the full list of OAuth scopes in the Google Cloud OAuth 2.0 documentation.

It turns out the scope field is also accepted by Pub/Sub and other services that were working fine with aud, so we can just make our generic implementation use the scope field and be done with it. Sweet!

Bonus: how did I find out about that?

This is the story about this answer I posted on the Stack Overflow question I linked earlier.

First, I dug in the Google Cloud Node.js SDK to see how they implemented the service account authentication.

It turns out they do use the self-signed JWT method, in their shared auth library, but it’s conditional to a variable useJWTAccessWithScope being set to true by the client SDK. For example, this is where Pub/Sub sets it, and this is where GCS doesn’t set it (as of today).

But what if we force this variable to true?

import { Storage } from '@google-cloud/storage'

const storage = new Storage()

storage.authClient.useJWTAccessWithScope = true

const file = await storage.bucket('bucket').file('file').get()

By running this script with NODE_DEBUG=https, we can see that without the useJWTAccessWithScope line, the client makes a call to https://www.googleapis.com/oauth2/v4/token first, then calls https://storage.googleapis.com/storage/v1/b/bucket/o/file, but with useJWTAccessWithScope, it skips the first token call (and everything works still)!

We can also notice that the token from the OAuth token endpoint contains hundreds of dots (.) at the end, whereas the self-signed token is just the usual 3 parts Base64URL JWT. Not an useful information, but interesting.

Either way, this proved that despite not working with the method in the documentation, self-signed authentication was effectively supported by Google Cloud Storage. So how did that SDK-generated token differ? The easiest is to copy that token from our NODE_DEBUG=https output and parse the payload segment:

pbpaste | cut -d. -f2 | base64 --decode

Or in Node.js:

Buffer.from(token.split('.')[1], 'base64').toString()

There we see they use a scope parameter as opposed to aud.

We can track it down to the code of the authentication library, and we can also see where the Google Cloud Storage client defines the necessary OAuth scopes.

Conclusion

With some investigation in the Google Cloud Node.js SDK source code and some NODE_DEBUG=https debugging, we can dissect their implementation of the self-signed service account authentication, and replicate it on our side.

This enables us to use this simpler and superior mechanism that Google uses internally, instead of the method that’s widely documented of calling their OAuth endpoint.

I hope that you learnt something thanks to this article, and that it helps you build great things! And as always, keep hacking! 🚀

Want to leave a comment?

Join the discussion on Twitter or send me an email! 💌
This post helped you? Buy me a coffee! 🍻