Check out our Nodes Statuspage here ➜
Catchpoint Systems Services
SSL issue with Let's Encrypt certificates
Update - We are continuing to monitor for any further issues.
Oct 20, 10:10 EDT
Update - We are continuing to monitor for any further issues.
Oct 7, 09:45 EDT
Monitoring - Our engineering team has implemented the required fix for monitors that use the old stack, as described in our KB - Emulated and Selenium API. We have tested and deployed this fix to the following nodes while we continue full regression testing and will deploy it to all nodes after testing is completed

Ashburn, VA - AWS Cloud,
Dublin, IE - AWS Cloud,
Washington, US - Azure Cloud,
London, UK - Azure Cloud,
Arlington, VA - Tencent Cloud,
Mumbai, IN - Tencent Cloud

Please contact us at support@catchpoint.com if you have any questions.
Thanks.
Oct 6, 12:21 EDT
Update - We have published a blog with detailed information on this issue
https://www.catchpoint.com/blog/lessons-from-an-internet-outage-issues-caused-by-lets-encrypt-dst-root-ca-x3-expiration

Please be reminded, Catchpoint did not experience outages; however, it did detect that some servers were not properly sending certificates and being impacted
Oct 2, 16:42 EDT
Update - Certificate issue for Object and API - Javascript tests is now resolved, we have finished updating all of our nodes. We are still working on making similar updates for our Emulated and API - Selenium monitors.
Sep 30, 17:54 EDT
Update - Today's issue is caused by an expired root CA certificate, used to sign server certificates issued by Let's Encrypt. This certificate expired at 10AM EDT today, September 30th. There can be two classes of issues which can cause your customers to not be able to connect to your site due to this certificate's expiration. The first is because the client device or OS has not been updated to include updated CA certificates. The second is because the client device or OS relies on underlying libraries which fail as soon as they see the expired CA certificate from the server, regardless of the presence of non-expired alternatives.

Catchpoint has updated all our public nodes (Backbone, Cloud, Last Mile, Wireless) to resolve these certificate errors for Object and API-Javascript tests. We are still working on making similar updates for our Emulated and API-Selenium monitors, which are running in our older HTTP stack. If you are relying on these, you have several options described below. Importantly, make sure to resolve the issue on your server - fixing the certificate chains on the Catchpoint nodes will resolve the test errors, but your clients may still have issues accessing your site, API, or other TLS-based service.

1. Remove the DST Root CA X3 from your server. If you do this, your server will stop sending this as an option to clients, and clients won't attempt to validate it.
2. Enable ignore SSL errors in your Catchpoint tests.
3. Switch tests to API - Javascript or Object - these use newer Catchpoint monitors which rely on different underlying libraries and are already using the updated certificate chain.

Alternately, Catchpoint engineering is working on updating the monitors which rely on our older HTTP stack. We will update on their progress when we have more information.

Some more information on this topic:
https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/211
https://www.bgr.in/features/internet-to-stop-working-for-select-uses-from-today-why-who-will-be-affected-what-to-do-1011330/
https://marketresearchtelecast.com/lets-encrypt-certificates-stuttering-possible-on-september-30th/164576/
https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
Sep 30, 15:17 EDT
Update - We are done updating about 50% of the nodes, we will provide an update when we have completed all of the impacted nodes.
Sep 30, 13:24 EDT
Identified - We have identified a fix for this issue and are working on updating our nodes.
Sep 30, 11:00 EDT
Investigating - We’re investigating an SSL issue caused by another Let's Encyrpt root certificate that was set to expire at 10 AM ET on 9/30, causing Object, emulated and API tests to fail for some of our clients.

We will update here as more information becomes available.
Sep 30, 10:40 EDT

About This Site

The Service Status Page where we will keep you up to date on any current known events, both network outages and scheduled maintenance, which may affect your Catchpoint Services.

Portal Operational
Instant Test Operational
Real User ? Operational
Alerting Operational
Data Pull API Operational
Data Push API Operational
Alerting API Operational
Application Data ? Operational
Catchpoint Help Center Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Database Maintenance – October 23rd 12:00 to 17:00 EDT Oct 23, 12:00-17:00 EDT
On Saturday, October 23rd, Catchpoint will be performing a planned database maintenance from 12:00 pm – 5:00 pm EDT.

During this time the Catchpoint Portal and REST API will be in read-only mode.

You will be able to access and view data, but not create, modify, or delete objects such as tests or contacts.

Please note that our test-monitoring infrastructure will continue to run tests and collect data.

For any questions, please contact Support.
Posted on Oct 13, 10:33 EDT
Past Incidents
Oct 22, 2021
Test runs impacted on select nodes
Resolved - This incident has been resolved.
Oct 22, 12:38 EDT
Update - We have created a KB article with the impacted nodes and will keep it up to date with their status and ETA.

https://support.catchpoint.com/hc/en-us/articles/4408628423067
Oct 19, 15:16 EDT
Update - At 12:00 PM EDT we have deployed a fix on the affected nodes. Approximately 90 have recovered. We are working on addressing the issue on the remainder.
Oct 19, 14:29 EDT
Identified - At 11:00 AM EDT Oct 19, about 160 of our nodes worldwide went offline due to a configuration change causing an unintended impact. We are working on a fix.
Oct 19, 14:28 EDT
Oct 21, 2021
Synthetic data processing delayed
Resolved - This incident has been resolved.
Oct 21, 12:45 EDT
Monitoring - Due to a misconfiguration, data processing was delayed starting 11:30 AM EDT, we have identified the issue and rolled out a fix. Some clients might see a gap in their Synthetic data for up to 40 minutes.

Please contact us at support@catchpoint.com if you have any questions.
Oct 21, 12:15 EDT
Oct 20, 2021

Unresolved incident: SSL issue with Let's Encrypt certificates.

Oct 19, 2021
Perseus Release Notification
Completed - The scheduled maintenance has been completed.
Oct 19, 16:47 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 19, 10:00 EDT
Update - On Oct 19th, we will continue our effort to release Chrome 89 as the stable version which started June, 2021. While we have performed extensive testing, we anticipate a small set of enterprise nodes or websites might not work well with Chrome 89. We are rolling out a new feature with the Perseus release that will give us finer control over Chrome versions which are run by specific nodes, this new feature will help us resolve these incompatibilities if they occur.

For more details on how to choose a Chrome version please go to our KB.

If you have any questions, please contact us at support@catchponint.com.

Thanks.
Oct 15, 15:16 EDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Oct 13, 13:03 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 13, 13:02 EDT
Scheduled - We will be undergoing scheduled maintenance during this time.
Oct 13, 13:02 EDT
In progress - Our Perseus release was originally scheduled to roll out this week. We found an issue in final testing and decided to postpone the release a few days to give our team time to address it properly. The new release date is next Tuesday, 10/19/2021. We're sorry for any inconvenience. We know you value the quality of data we provide and hope you'll find it's worth the wait.
Oct 19, 10:00 EDT
Update - Our next release, Perseus, will roll out on Wednesday, October 13, at 10:00 AM ET.

The Catchpoint Portal (portal.catchpoint.com) will continue to be available throughout the rollout, and your tests will continue to run and collect data. During this time, you might encounter the following:

- Individual clients will have a brief time (about 3-5 minutes) where alerts and charts will be inaccessible or delayed while we upgrade our applications.
- The REST API will experience a short downtime of about 15 minutes. The Alert and Test Data Webhooks will continue to operate throughout the rollout.

During the rollout, support tickets may still be filed via the Help Center or email to support@catchpoint.com.
Oct 11, 15:34 EDT
Scheduled - Our next release, Perseus, will roll out on Tuesday, October 12, at 10:00 AM ET.

The Catchpoint Portal (portal.catchpoint.com) will continue to be available throughout the rollout, and your tests will continue to run and collect data. During this time, you might encounter the following:

- Individual clients will have a brief time (about 3-5 minutes) where alerts and charts will be inaccessible or delayed while we upgrade our applications.
- The REST API will experience a short downtime of about 15 minutes. The Alert and Test Data Webhooks will continue to operate throughout the rollout.

During the rollout, support tickets may still be filed via the Help Center or email to support@catchpoint.com.
Oct 6, 09:04 EDT
Test runs impacted on select nodes
Resolved - At 11:00 AM EDT Oct 19, about 160 of our nodes worldwide went offline due to a configuration change causing an unintended impact. We are working on a fix.
Oct 19, 11:00 EDT
Oct 18, 2021

No incidents reported.

Oct 17, 2021

No incidents reported.

Oct 16, 2021

No incidents reported.

Oct 15, 2021

No incidents reported.

Oct 14, 2021

No incidents reported.

Oct 13, 2021
Oct 12, 2021

No incidents reported.

Oct 11, 2021
Planned Datacenter Network Maintenance - Monday, October 11th from 10PM to 11PM ET
Completed - The scheduled maintenance has been completed.
Oct 11, 23:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 11, 22:00 EDT
Scheduled - Catchpoint will be performing a planned network maintenance at our Las Vegas datacenter on Monday, October 11th from 10PM to 11PM ET

During the maintenance we expect a loss of connectivity of up to 30 minutes.

Catchpoint will continue to collect data globally, but Portal, API and Alerts will be unavailable during the maintenance.

For questions, please reach out to us at support@catchpoint.com
Sep 29, 13:49 EDT
Oct 10, 2021

No incidents reported.

Oct 9, 2021

No incidents reported.

Oct 8, 2021
Issues with multiple ISPs in North America impacting RUM data processing
Resolved - This incident has been resolved.
Oct 8, 09:00 EDT
Update - We are continuing to monitor for any further issues.
Oct 8, 08:28 EDT
Monitoring - We believe the issues are correlated to a widespread routing issue at Telia, a major backbone provide in US and Europe. The issue impacted several key companies that provide cloud and dns services.

The issue has improved. The impact to Catchpoint was limited to portions of our logging infrastructure for Real User Monitoring and Endpoint.
Oct 7, 14:13 EDT
Investigating - We are investigating an issue with our RUM data processing due to issues with multiple ISPs and our managed DNS provider in North America. Some RUM data will be missing for a small set of clients.
We’ll update here as more information is available.

Please contact us at support@catchpopint.com

Thanks.
Oct 7, 13:46 EDT
← Incident History Powered by Atlassian Statuspage