Amplemarket - Workflows runs are delayed – Incident details

All systems operational

Workflows runs are delayed

Resolved
Operational
Started about 16 hours agoLasted about 2 hours

Affected

Workflows

Operational from 4:34 PM to 5:51 PM, Degraded performance from 5:51 PM to 6:13 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    We’ve confirmed that workflows never stopped running. However, a large backlog built up, which triggered alerts indicating that workflows were not running.

    We are currently increasing the number of Workflow background workers to process the backlog more quickly.

  • Update
    Update
    We are currently investigating this incident.
  • Investigating
    Investigating

    We are currently investigating this incident. Workflow runs are not lost, they will be executed once the issue is resolved.