Amplemarket - Workflows runs are delayed – Incident details

Workflows runs are delayed

ResolvedDegraded performance1 hour 39 minutes

Affected

Workflows

Operational from 4:34 PM to 5:51 PM, Degraded performance from 5:51 PM to 6:13 PM

  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • IdentifiedUTC

    We’ve confirmed that workflows never stopped running. However, a large backlog built up, which triggered alerts indicating that workflows were not running.

    We are currently increasing the number of Workflow background workers to process the backlog more quickly.

  • UpdateUTC
    We are currently investigating this incident.
  • InvestigatingUTC

    We are currently investigating this incident. Workflow runs are not lost, they will be executed once the issue is resolved.