Integromat Webhooks will randomly stop working for 24-72 hours at a time

I need to toss my name in the hat. We use Zapier integration and it’s critical to our pipeline. I just noticed that our automations have not been running since 12/30. We’re now 8 days out of sync and missing data. This is frustrating because it seems like this happens every single month. Can anyone at Patreon please put the API and webhooks back on top priority?

@CodeBard any suggestions on why these webhooks have not been triggering for 8-9 days?

1 Like

There may be an issue with 1st of month causing issues with the api, likely the api catching up late. This may be affecting integrations like Zapier which rely on processing members/pledges at a given date and not retrying them. We are going to investigate the 1st of month issue in the next 1-2 weeks.

1 Like

Much appreciated. I’ve also setup a new Zapier automation that will toggle any Patreon workflows on/off/on every Sunday. Hopefully it’s just that and not needing to re-authenticate the account. Will report back near end of the month.

2 Likes

APIs are down again. Three new subscribers this morning and none of them were sent to Integromat. I confirmed nothing is wrong on the Integromat end.

API is working again. As always, it was nothing I did on my end. It looks like, indeed, this is likely an issue related to the 1st of the month, although it is not limited to only the 1st. I would love to know what the result of the investigation into this was. I ended up losing a number of new subscribers this month alone. Every time this happens I lose new patrons, which of course, are your patrons too.

The first of month issue is being investigated. Its a complicated and delicate issue, so resolving it may take a while.

Coding integrations in a failsafe manner is always a good idea and can resolve that. For now, you could use a cron that pulls your patrons’ list regularly from Patreon instead of solely relying on webhooks.

Additionally, the reality of webhooks are that they are subject to any kind of situation in between the origin and the target - from network congestion to any security scheme that enters into the equation at any point. Our webhooks retry 7 times with exponential delay.

https://docs.patreon.com/#robust-retries

This also means that if the first 2 attempts that happen within first 4 hours fail for whatsoever reason, the next retry will be a day late. If that also fails, next retry is in 3 days.

Therefore having a regular cron or a backup utility to force-sync everything you need just in case is always a good idea.

I say this with complete respect to you for trying to help, but I don’t think asking me to hire someone to program a cron or backup redundancy to make up for Patreon’s technical issues is an acceptable answer. I run webhooks through Integromat for dozens of things, and in the past year not a single one of them has failed, except for Patreon, which has gone down for at least 24 hours a half dozen times.

Can you imagine if credit card payments would completely stop working every couple months, and the answer to that issue was to accept Paypal payments from patrons? This is no different, except there is no money involved so it is not treated as a high priority. And agreed, it is NOT as important as payments, but it’s still pretty important and doesn’t deserve a band-aid fix.

Again, I don’t say this with disrespect toward you, but this problem occurring so frequently is just not acceptable.

I was also told that this is caused by the 1st day issue but it happens every day. After a new patron pledges it takes around 3-4 hours to appear in the relationship manager and the webhook is only called after that :confused: this way I cannot grant the rewards promised to the patrons in time.

1 Like

If I might say, “holy SHIT” the API has been down since the 4th (9 days) and I had no idea. Not a single new supporter has been processed correctly. I couldn’t figure out why all of my new sign-ups removed their pledges the same day. When on earth will this be fixed?

Api hasnt been down?

https://status.patreon.com/

Maybe your integration has lost its connection to Patreon? Checking creator access tokens may help.

It started working again a short while after my last post. When I ran the check from Integromat it said everything was connected properly. In addition, all of my other APIs within Integromat continued to function even when Patreon’s integration wasn’t.

1 Like

Im wondering whether Patreon api working from behind Cloudflare IPs may have an effect on this. Can you ask whomever is hosting your integration (Integromat or someone else) whether this could be a concern?

I opened a ticket with Integromat and references the outage dates to see if we can get to the bottom of this.

1 Like

APIs are down again, at least for the past 24 hours. I just tested myself with a test account and indeed, they are not firing.

I have been in communication with Integromat and they have confirmed there are no issues on their end. I reported a specific Patreon sign-up from yesterday along with the exact sign-up timestamp, and they are investigating their logs for that timestamp to see what they can find.

If you wouldn’t mind doing the same, here is the timestamp for a new pledge which didn’t get processed:
Date: Sat, 17 Apr 2021 01:18:34 +0000

From Integromat interface: both connections verified working:

The new subscriber scenario is functional, as always, but when I get a new pledge I get the new supporter email but the API doesn’t fire. This was working as recently as 48 hours ago but currently down.

EDIT: Update, I tried a test again and it’s now working again for now. Would still like to know what happened with the timestamp from yesterday.

There doesnt seem to have been any api downtime at the moment you reported it. It could be a connectivity issue at a point somewhere in between your service provider and Patreon.

Your webhook may have gone stale on api side for such reason. You can try creating a new webhook and checking if it fires off normally.

I sent you a PM 5 days ago related to the specific webhook. Did you get my message?

When i visit that url even now, it gives me a 200 response and says ‘accepted’ regardless of me not submitting any information. There doesnt seem to be any verification of whether the request being made to that url being valid or not either.

Basically anything that accesses that url at this state will likely give out a 200 response. This could mean that even if your app wasnt able to receive a meaningful webhook post and process it properly in the backend, it will still send out 200 response and the webhook will consider its request to be fulfilled.

Im not privy to how integromat works and how is their application logic but this may be something worth checking.

I think what the API specialist from Integromat was asking is if you could look at the Patreon API log for the date: Sat, 17 Apr 2021 01:18:34 +0000 and see if a request for that webhook was made during that time, and if so, what happened. This was a timestamp when I received a sign-up email but no information was sent from Patreon to my Intergromat scenario like it does when everything is functioning properly.

On the Integromat side, they checked the log for this time and determined Patreon didn’t trip the scenario. They have asked if you could check for that specific time to see if an API call was made for that url. Are you able to access the log for that datetime?

I replied via pm. That webhook seems to be active and not failed. Such report granularity in logs for a specific time at that point in time may not be possible. At this point it would be easier if you inform me via email on the day you have observed a webhook fail.

Note that webhook posts may fail for whatsoever reason (including connectivity issues in between two services) and there should be a backup that kicks in in case the functionality provided by webhook cannot find the data provided by the webhook.