You may be receiving duplicate Slack notifications.
This is due to an issue with Slack. Slack is replying with 500 errors to our notifications even though they are clearly processing the messages just fine, our notification system sends multiple retries after receving failures so you will be receiving multiple duplicates and your webhooks will probably be disabled automatically (as they are after too many repeated failures). We will be re-enabling them once the issue is fixed. If your webhook remains disabled, please contact us.
14:17 UTC: We have not received a single 500 error from Slack in 8 minutes. It looks like this may be fixed. Although a broader incident is still ongoing on Slack's end: https://status.slack.com/2021-12/a17eae991fdc437d
14:44 UTC: Webhooks disabled since 12:00 UTC have been re-enabled. Slack status says messaging/notifications part of the incident is resolved, we are not seeing any errors so this incident is now over. If you are experiencing an error or if your webhook has not been re-enabled, please contact us.