February 2017

Slow Test Runs
This incident has been resolved.
Feb 17, 03:02-09:04 PST
Test Run Delays
This incident has been resolved.
Feb 14, 06:28-06:50 PST
Test Run Delays
There was a delay in processing test runs for approximately 6 minutes at 11:24am UTC due to an internal queuing system encountering an error. The error was resolved and test runs resumed running normally.
Feb 6, 06:59 PST

January 2017

No incidents reported for this month.

December 2016

DNS Resolution Errors from Ireland (eu1) Location
Our service provider has resolved this issue.
Dec 14, 18:17 - Dec 15, 05:26 PST
Queued Test Runs and Requests
This incident has been resolved.
Dec 12, 07:06-12:26 PST
Extra Notifications
All systems have returned to normal.
Dec 6, 08:24-10:43 PST

November 2016

No incidents reported for this month.

October 2016

No incidents reported for this month.

September 2016

No incidents reported for this month.

August 2016

Test Runs Slow
All systems have been operating normally since 7:30 PDT when our datastore service was restored.
Aug 30, 05:44-11:55 PST
Incorrect URL Character Encodings
We have rolled back a deploy that was causing request URLs in tests to be double encoded or encoded improperly between 22:08 and 22:51 UTC. This caused tests to fail for some customers. We are adding additional precautions to prevent this error from recurring.
Aug 11, 15:58 PST

July 2016

Initial Scripts Broken
There was a 7-minute period around 18:05 UTC where initial scripts defined in an environment were not being processed correctly causing some tests to fail. This was caused by a failed deploy that was rolled back as quickly as possible, resolving the issue. No further issues are expected.
Jul 29, 11:44 PST

June 2016

Site Unavailable
All systems are operating normally. We are continuing to monitor for any regressions or degradations in performance and adding additional capacity where necessary.
Jun 10, 06:56-10:27 PST
Test Runs Delayed
All affected systems are back to normal and test runs continue to operate normally.
Jun 10, 01:10-01:35 PST

May 2016

No incidents reported for this month.

April 2016

No incidents reported for this month.

March 2016

API Test Run Failures
All systems continue to operate normally.
Mar 2, 14:15-14:44 PST

February 2016

Runscope Scheduled API Tests Delayed
All services are continuing to operate normally.
Feb 11, 13:59-16:13 PST

January 2016

API Test run results delayed or expired
All services are operating normally.
Jan 26, 10:25-17:13 PST
Runscope API Tests Not Running
All services are operating normally.
Jan 12, 20:15-20:48 PST

December 2015

Live Traffic Alerts Not Being Processed
*This incident did not affect the processing of test runs or notifications for API tests, which have been operating normally.* Between Dec 23 0:00 UTC and Dec 28 17:30 UTC, Live Traffic Alerts were not being processed, failing to trigger notifications for any matching requests. Matching request counts and details were also not stored during this time. Our internal monitoring system failed to detect a stoppage in the processing of new requests. Once detected, we were able to quickly resolve the issue and begin processing new requests. We are working to improve existing internal monitoring and implementing additional checks to catch similar failures in the future.
Dec 28, 11:49 PST

November 2015

Runscope API Test Runs Not Running
All services are continuing to operate normally.
Nov 4, 06:37-08:18 PST

October 2015

Scheduled maintenance in progress
Our database upgrade has been completed.
Oct 27, 20:01-20:20 PST
Scheduled maintenance in progress
Our database upgrade has been completed.
Oct 26, 21:03-21:13 PST
API Tests Not Running
All systems have returned to normal operation.
Oct 22, 13:25-14:43 PST
Test Results Slow Down
All services have returned to normal operation and API tests continue to run at normal speed.
Oct 19, 14:58-15:48 PST

September 2015

Runscope API Test Performance Degraded
All services have returned to normal operation, historical test data is available and all tests are running again on normal schedules without delay.
Sep 23, 06:23-17:56 PST
Runscope API Tests Not Running
This incident has been resolved.
Sep 20, 02:48-11:45 PST
Runscope API Test Performance Degraded
This incident has been resolved.
Sep 16, 15:54-17:39 PST
Runscope API Tests Not Running
All infrastructure is running again with backups in place. API Test runs have continued to operate normally since infrastructure failovers completed.
Sep 5, 09:13-09:44 PST

August 2015

API Test Results unavailable
This incident has been resolved.
Aug 10, 02:12-09:47 PST

July 2015

Delayed or expired API test results in EU1 region
All systems are operating normally. Between 1:45 PM and 2:30 PM PDT, API test steps were not executing in the Ireland (EU1) region. We identified the root cause and systems quickly returned to normal by 2:35 PM PDT.
Jul 21, 14:09-15:19 PST

June 2015

Delayed or expired API test results, some delays in Gateway traffic display in Dashboard
All systems are operating normally. For a brief period of time this morning, a growing backlog of work to be processed caused a number of API test results to expire. Gateway traffic experienced a delay in showing up in Dashboard but continued to be relayed without incident. We deployed a fix and services began recovering immediately. API test results are no longer expiring, and Gateway traffic is being displayed without delay.
Jun 30, 09:19-10:48 PST
Disappearing test steps
All test data has been restored and tests are running on their normal schedules. We are reaching out to any customers that may be missing test step modifications made between 2:00pm PT / 21:00 UTC and 3:05pm PT / 22:05 UTC.
Jun 12, 15:26-17:43 PST
Runscope Radar tests not setting variables correctly
Data is restored, all API tests should be running with correct settings now
Jun 10, 11:22-12:00 PST

May 2015

No incidents reported for this month.

April 2015

Availability issues
Test runs have been performing normally for the past 36 hours. Capacity and reliability improvements are ongoing. We will open a new incident if issues resurface. If you have any questions, please feel free to contact us at help@runscope.com
Apr 25, 12:11 - Apr 29, 08:58 PST
Slow running tests
All systems are fully operational. Radar tests are running on normal schedules across all regions. Ultimately we found that some Radar tests did not run between 1:15 AM and 1:38 AM Pacific time. You may see some missing or "expired" test runs during that time.
Apr 25, 01:32-02:13 PST
Slow running and expired tests
The issues have been resolved and the system has stabilized. The backlog of traffic was processed, and Radar traffic is available again in the Traffic Inspector. We are performing a full root cause analysis on the underlying issues and putting additional monitoring and infrastructure in place to reduce the likelihood of recurrences of this issue.
Apr 23, 09:37-14:52 PST
Dashboard and Radar pages loading inconsistently
All systems are operating normally, a post-mortem analysis will be after we complete a thorough investigation and determine corrective actions.
Apr 22, 15:55-18:07 PST
Radar Instability
All systems are operational, Radar tests are running on normal schedules across all regions
Apr 20, 15:26-17:37 PST
Errors viewing test results
This incident has been resolved.
Apr 13, 09:49-11:10 PST

March 2015

Radar tests slow to report results
All systems have returned to normal we will conduct a post-mortem and publish the results as soon as they are available.
Mar 19, 07:43-10:45 PST
Slow load times and intermittent failures on Radar test list and Radar test results pages
Slow radar test results are no longer an issue. No data was lost, and tests continued to execute and send notifications in a timely fashion, however Runscope's website was behind on reporting updates to test runs. A postmortem analysis as well as preventative steps will be posted tomorrow.
Mar 16, 11:34-21:42 PST

February 2015

Metrics Display Performance
This incident has been resolved.
Feb 19, 11:18 - Feb 20, 11:44 PST
Radar test run notification delays
Outbound Runscope Radar notifications have returned to normal volumes and are no longer delayed. We've added additional monitoring to our outbound notifications latency as well as put in place performance improvements to make sure that notifications are delivered in a timely manner.
Feb 11, 22:45 - Feb 12, 16:28 PST
Passageway Performance Issues
We've identified the source of performance problems with Passageway and have updated our infrastructure to resolve the issue.
Feb 11, 15:19-15:55 PST

January 2015

Expired Test Runs
This incident has been resolved.
Jan 26, 17:02-20:15 PST
Expiring Test Runs
This incident has been resolved.
Jan 26, 09:27-10:02 PST
Live Search Outage
Live Search indexes have been rebuilt for all of our customers. Custom Stream Views and Search Results should be working correctly for everyone now.
Jan 7, 15:40 - Jan 8, 12:33 PST
Test Runs Stopped
This incident has been resolved.
Jan 5, 02:44-03:37 PST

December 2014

Test Runs from EU1
The issue has been resolved and tests are running from Ireland (EU1) again. You may see test runs marked 'Expired' that were set to run between 9:24 and 9:54pm PT.
Dec 9, 21:43-21:54 PST
Metrics Slow or Unavailable
This incident has been resolved.
Dec 8, 09:55-21:55 PST
Dashboard Unavailability
This incident has been resolved.
Dec 5, 15:17-18:26 PST

November 2014

Data Loading Issue
This incident has been resolved.
Nov 5, 12:03-12:41 PST
Service Unavailable
This incident has been resolved.
Nov 3, 10:59-11:17 PST

October 2014

Service Unavailable
This incident has been resolved.
Oct 20, 13:21-13:38 PST
Dashboard and Radar Issue
All test schedules have been restored and all test runs are operating normally.
Oct 19, 20:24-23:54 PST
HipChat Notification Outage
The issue appears to be resolved on HipChat's side. Will update if issue re-occurs.
Oct 7, 11:02-14:54 PST

September 2014

DNS issue affecting Runscope URL Gateway in all regions
This incident has been resolved.
Sep 18, 04:49-17:17 PST

August 2014

Network Connectivity Issues
This incident has been resolved.
Aug 27, 12:08 - Aug 28, 08:48 PST
Issues with Brazil Service Region
This incident has been resolved.
Aug 26, 16:50 - Aug 27, 09:09 PST
Radar Notification Emails
We have resolved the issue and Radar notification emails for new test runs will be sent. Please contact support (help@runscope.com) if you need further assistance.
Aug 26, 09:19-09:30 PST

July 2014

Radar and Gateway Issues
This incident has been resolved.
Jul 31, 11:28-11:42 PST
Expired Test Runs
Radar tests are running normally.
Jul 16, 12:43-13:03 PST
[Scheduled] US Texas (us4) regional maintenance
The scheduled maintenance has been completed.
Jul 10, 01:00-03:00 PST

June 2014

Intermittent Errors
Service levels are back to normal. We are continuing to investigate the root cause of the issue.
Jun 20, 14:04-14:21 PST

May 2014

Dashboard & Radar Errors
All systems are fully operational.
May 22, 14:59-15:12 PST

April 2014

Heartbleed OpenSSL Vulnerability
To address the Heartbleed vulnerability in OpenSSL, all Runscope servers were patched with the latest version of OpenSSL by April 7th, 9:05pm US Pacific Time. We have reissued all of our SSL certificates and completed the deployment of the updated certificates at April 8th 3:30pm PT.
Apr 8, 15:32 PST