All systems are operating normally.
Jan 4, 20:56 PST
We've deployed some additional measures that have restored test run throughput to near normal levels for newly-initiated test runs. You will see an increased number of 'Expired' test runs as the workers catch up. We are continuing to work on overall stability while monitoring for any regressions or additional hot spots.
Jan 4, 16:06 PST
We are continuing to implement additional measures to attempt to return test run throughput to full capacity. We are seeing marginal ongoing improvements while we try to resolve the issue completely.
Jan 4, 14:36 PST
Test run throughput has improved significantly, but volatility remains as we continue work to mitigate the original issue and the ensuing side effects. You will likely see 'Expired' test runs as we continue to process the backlog of work that was accumulated during the initial slowdown.
Jan 4, 12:29 PST
We have deployed some an update to the system to attempt to mitigate the impact of the issue. You may see some improvement, though it may be temporary while we continue to work on a permanent resolution. We currently do not have an ETA for resolution, but we are working to resolve this incident as quickly as possible.
Jan 4, 11:45 PST
We are continuing to investigate the cause of the delayed and expired test runs. We have identified the affected subsystem, but are continuing work to identify the root cause of the issue.
Jan 4, 11:15 PST
We’re investigating an issue with slow and expiring test runs. We will provide more details as they become available.
Jan 4, 10:48 PST