Previously on Journey to a distributed PDS, I discovered that when one node generates the DPoP nonce that a client uses as part of a request and another node then receives that as part of a request to validate, it would fail because node one would have generated it differently than node 2. This was due to Cocoon using an incremental counter on a timer which would get used to generate and then validate a nonce. Each node would have a different counter and so they would both calculate different values.

I had a couple ideas on how to solve this, but wasn't really feeling them and thankfully Devin suggested a really good idea.

devin ivy 🐋's avatar
1mo

another approach is to use a technique similar to TOTP to statelessly compute the same nonces on different nodes over time. requires having some amount of clock syncing, but it doesn't have to be very precise.

Once I saw this I knew immediately that this would probably be the best solution and I was annoyed that I didn't think of it myself.

So off I went to implement it and it turned out to be pretty damn easy to do and it worked perfectly.

There's the PR that I used and it's relatively straight forward.

Some things to note are:

  • A new env which contains the nonce secret (so that all nodes can generate the same nonce). Previously this was just a randomly generated value that was then stored to a local file so that it was consistent across restarts. No harm in making it an env.

  • A new nonce generator that instead of using an incremental counter, used a rounded time value. Currently it's hardcoded to 15 minute intervals because I need to do some testing on how small I can make that interval. The smaller the interval, the less time that a nonce value is valid for which tightens up the secureness of the value.

I've been using it like this for a few weeks and haven't had any issues so far. The past few days I've attempted to test with a small interval and managed to get it down to 1 minute which is what Cocoon was using for it's rotation before, so I expect I'll PR that soon.

I was really impressed how this solution turned out because I was actually dreading getting into the OAuth trenches to figure it out.

Side note:

I created a new fork of Cocoon and called it distributed-pds because I wanted to have everything all in one place and not clutter up the fork I had for Cocoon which I like to contribute from time to time. Feel free to give it a star and follow the progress. I'm currently in the middle of adding some documentation on how to run it in distributed mode.