Update - We are continuing to monitor for any further issues.
Oct 20, 10:10 EDT
Update - We are continuing to monitor for any further issues.
Oct 7, 09:45 EDT
Monitoring - Our engineering team has implemented the required fix for monitors that use the old stack, as described in our
KB - Emulated and Selenium API. We have tested and deployed this fix to the following nodes while we continue full regression testing and will deploy it to all nodes after testing is completed
Ashburn, VA - AWS Cloud,
Dublin, IE - AWS Cloud,
Washington, US - Azure Cloud,
London, UK - Azure Cloud,
Arlington, VA - Tencent Cloud,
Mumbai, IN - Tencent Cloud
Please contact us at
support@catchpoint.com if you have any questions.
Thanks.
Oct 6, 12:21 EDT
Update - Certificate issue for Object and API - Javascript tests is now resolved, we have finished updating all of our nodes. We are still working on making similar updates for our Emulated and API - Selenium monitors.
Sep 30, 17:54 EDT
Update - Today's issue is caused by an expired root CA certificate, used to sign server certificates issued by Let's Encrypt. This certificate expired at 10AM EDT today, September 30th. There can be two classes of issues which can cause your customers to not be able to connect to your site due to this certificate's expiration. The first is because the client device or OS has not been updated to include updated CA certificates. The second is because the client device or OS relies on underlying libraries which fail as soon as they see the expired CA certificate from the server, regardless of the presence of non-expired alternatives.
Catchpoint has updated all our public nodes (Backbone, Cloud, Last Mile, Wireless) to resolve these certificate errors for Object and API-Javascript tests. We are still working on making similar updates for our Emulated and API-Selenium monitors, which are running in our older HTTP stack. If you are relying on these, you have several options described below. Importantly, make sure to resolve the issue on your server - fixing the certificate chains on the Catchpoint nodes will resolve the test errors, but your clients may still have issues accessing your site, API, or other TLS-based service.
1. Remove the DST Root CA X3 from your server. If you do this, your server will stop sending this as an option to clients, and clients won't attempt to validate it.
2. Enable ignore SSL errors in your Catchpoint tests.
3. Switch tests to API - Javascript or Object - these use newer Catchpoint monitors which rely on different underlying libraries and are already using the updated certificate chain.
Alternately, Catchpoint engineering is working on updating the monitors which rely on our older HTTP stack. We will update on their progress when we have more information.
Some more information on this topic:
https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/211https://www.bgr.in/features/internet-to-stop-working-for-select-uses-from-today-why-who-will-be-affected-what-to-do-1011330/https://marketresearchtelecast.com/lets-encrypt-certificates-stuttering-possible-on-september-30th/164576/https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
Sep 30, 15:17 EDT
Update - We are done updating about 50% of the nodes, we will provide an update when we have completed all of the impacted nodes.
Sep 30, 13:24 EDT
Identified - We have identified a fix for this issue and are working on updating our nodes.
Sep 30, 11:00 EDT
Investigating - We’re investigating an SSL issue caused by another Let's Encyrpt root certificate that was set to expire at 10 AM ET on 9/30, causing Object, emulated and API tests to fail for some of our clients.
We will update here as more information becomes available.
Sep 30, 10:40 EDT