... and I'm back! An unfortunate sequence of events (involving ancient account details) lead to me failing to renew my personal domain, so it broke DNS... which broke my self-hosted Mastodon as well
I did get a new TIL out of it - I moved to simonwillison.org for 24 hours and now I'm using Cloudflare to redirect that back to simonwillison.net https://til.simonwillison.net/cloudflare/redirect-whole-domain
I wonder how much of an impact this had on the invisible Mastodon infrastructure...
Every time someone I follow posted in the past 24 hours their instance would have tried and failed to deliver to my instance - how hard do these things get retried?
Am I about to see a massive flood of incoming deliveries that finally work?
@simon by default, sidekiq jobs that throw an error will be retried 25 times over the course of 20 days https://github.com/sidekiq/sidekiq/wiki/Error-Handling#automatic-job-retry
it's possible mastodon has customized this but I haven't specifically looked. would expect you'll get more traffic than usual but shouldn't be all at once
Here is the retry algorithm, if you were down a day. They will trickle in vs a massive flood
(0..16).each do |count|
delay = (count**4) + 15
jitter = rand(0.5 * (count**4))
total_delay = delay + jitter
puts "Retry #{count}: #{total_delay} seconds"
end
Retry 0: 15.713002155688077 seconds
Retry 1: 16.48915648567112 seconds
Retry 2: 31 seconds
Retry 3: 124 seconds
Retry 4: 382 seconds
Retry 5: 938 seconds
Retry 6: 1319 seconds
Retry 7: 2617 seconds
Retry 8: 6006 seconds
Retry 9: 9468 seconds
Retry 10: 12612 seconds
Retry 11: 19804 seconds
Retry 12: 26167 seconds
Retry 13: 31795 seconds
Retry 14: 55286 seconds
Retry 15: 67782 seconds
Retry 16: 89823 seconds
@JsonCulverhouse @clifff @simon May I ask why you add a jitter?
@demiurg @JsonCulverhouse @clifff at a guess it's to avoid accidentally having a whole bunch of retries fire at the exact same time - better to spread them out with a little bit of jitter
@simon @JsonCulverhouse @clifff Yes, makes sense! Thank you!