Slow and expired test runs
Incident Report for Runscope
Resolved
All systems are operating normally.
Posted 11 months ago. Jan 04, 2018 - 20:56 PST
Monitoring
We've deployed some additional measures that have restored test run throughput to near normal levels for newly-initiated test runs. You will see an increased number of 'Expired' test runs as the workers catch up. We are continuing to work on overall stability while monitoring for any regressions or additional hot spots.
Posted 11 months ago. Jan 04, 2018 - 16:06 PST
Update
We are continuing to implement additional measures to attempt to return test run throughput to full capacity. We are seeing marginal ongoing improvements while we try to resolve the issue completely.
Posted 11 months ago. Jan 04, 2018 - 14:36 PST
Update
Test run throughput has improved significantly, but volatility remains as we continue work to mitigate the original issue and the ensuing side effects. You will likely see 'Expired' test runs as we continue to process the backlog of work that was accumulated during the initial slowdown.
Posted 11 months ago. Jan 04, 2018 - 12:29 PST
Update
We have deployed some an update to the system to attempt to mitigate the impact of the issue. You may see some improvement, though it may be temporary while we continue to work on a permanent resolution. We currently do not have an ETA for resolution, but we are working to resolve this incident as quickly as possible.
Posted 11 months ago. Jan 04, 2018 - 11:45 PST
Update
We are continuing to investigate the cause of the delayed and expired test runs. We have identified the affected subsystem, but are continuing work to identify the root cause of the issue.
Posted 11 months ago. Jan 04, 2018 - 11:15 PST
Investigating
We’re investigating an issue with slow and expiring test runs. We will provide more details as they become available.
Posted 11 months ago. Jan 04, 2018 - 10:48 PST