All systems have been operating normally since 7:30 PDT when our datastore service was restored.
Posted Aug 30, 2016 - 11:55 PDT
Test run throughput has improved to near normal levels as performance of the datastore service has been restored. We are still monitoring closely for regressions in performance but all queues have caught up for the time being.
Posted Aug 30, 2016 - 07:31 PDT
We are seeing improvements in response times from the test results datastore allowing some test runs to resume. Capacity is still significantly reduced. We are continuing to attempt to mitigate the issue with workarounds and additional capacity while we wait for our service provider to resolve the underlying issue.
Posted Aug 30, 2016 - 07:07 PDT
Our primary service provider has acknowledged an issue with the datastore service causing the issue. They are investigating and working on a resolution. We are working to add work arounds for the performance issue as much as possible. Test run processing is very limited at this time.
Posted Aug 30, 2016 - 06:39 PDT
We are investigating a performance issue with our primary datastore for test results. This is causing test runs to slow down and results to not load or load very slowly in the dashboard and API. We are still investigating the full extent of the issue.
Posted Aug 30, 2016 - 06:09 PDT
We're investigating a slowdown in processing test runs.