Affecting Other - Email

• 11/14/2024 17:32 - 11/15/2024 00:25
• Last Updated 11/14/2024 17:35

We have become aware of issues sending emails to AOL, Yahoo, and Verizon.

As an Email Service Provider we have always been on the AOL Feedback Loop which provides us notifications when an AOL user reports a message as spam so that we can address it.  As a part of the AOL and Yahoo merger this tool no longer exists and such notifications are no longer sent.

We have reviewed our mail logs and don't believe there to be a larger than normal or unusual flow of emails although it looks as though AOL/Yahoo/Verizon have made a change on their end that is causing them to see our email flow as 'new' or 'suspicious'.  Thus far our attempts to contact the Postmaster hasn't been super effective as they respond to us as though we're an end-user and not an ISP/ESP.

We will continue communicating with the postmaster in an attempt to resolve this as well as making changes on our end in an attempt to resolve the issue including, but not limited to, spreading out our outbound email flows across numerous IP addresses.

Affecting Server - R4

• 07/28/2024 22:00 - 07/29/2024 02:00
• Last Updated 07/29/2024 03:56

On July 28th we will be updating the operating system of the R4 server [R4.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - R3

• 07/28/2024 22:00 - 07/29/2024 02:00
• Last Updated 07/29/2024 01:04

On July 28th we will be updating the operating system of the R3 server [R3.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - P2

• 07/28/2024 22:00 - 07/29/2024 02:00
• Last Updated 07/29/2024 00:35

On July 28th we will be updating the operating system of the P2 server [P2.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - R2

• 07/27/2024 22:00 - 07/28/2024 02:00
• Last Updated 07/28/2024 03:03

On July 27th we will be updating the operating system of the R2 server [R2.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - R1

• 07/27/2024 22:00 - 07/28/2024 02:00
• Last Updated 07/28/2024 03:02

On July 27th we will be updating the operating system of the R1 server [R1.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - P1

• 06/25/2024 22:00 - 06/26/2024 03:00
• Last Updated 06/26/2024 02:04

Original Notice:

On June 25th we will be updating the operating system of the P1 server [P1.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

CloudLinux 7 reaches End of Life [EoL] on July 1, 2024, so no further security updates or patches will be available after that date. Additionally, cPanel has stopped providing updates for CloudLinux 7, keeping us on a slightly older version lacking some of the newer features they've released recently.

CloudLinux 8 reaches End of Life [EoL] on May 31, 2029, so we do not anticipate having to perform this kind of maintenance again for the next 5 years. We would love to go ahead and move to CloudLinux 9, however, cPanel's support for CloudLinux 9 is experimental and not recommended for production environments.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - S4

• 06/24/2024 22:00 - 06/25/2024 02:00
• Last Updated 06/25/2024 02:07

Original Notice:

On June 24th we will be updating the operating system of the S4 server [S4.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

CloudLinux 7 reaches End of Life [EoL] on July 1, 2024, so no further security updates or patches will be available after that date. Additionally, cPanel has stopped providing updates for CloudLinux 7, keeping us on a slightly older version lacking some of the newer features they've released recently.

CloudLinux 8 reaches End of Life [EoL] on May 31, 2029, so we do not anticipate having to perform this kind of maintenance again for the next 5 years. We would love to go ahead and move to CloudLinux 9, however, cPanel's support for CloudLinux 9 is experimental and not recommended for production environments.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - S3

• 06/21/2024 22:00 - 06/22/2024 02:00
• Last Updated 06/22/2024 01:27

Original Notice:

On June 21st we will be updating the operating system of the S3 server [S3.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

CloudLinux 7 reaches End of Life [EoL] on July 1, 2024, so no further security updates or patches will be available after that date. Additionally, cPanel has stopped providing updates for CloudLinux 7, keeping us on a slightly older version lacking some of the newer features they've released recently.

CloudLinux 8 reaches End of Life [EoL] on May 31, 2029, so we do not anticipate having to perform this kind of maintenance again for the next 5 years. We would love to go ahead and move to CloudLinux 9, however, cPanel's support for CloudLinux 9 is experimental and not recommended for production environments.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - S2

• 06/20/2024 22:00 - 06/21/2024 00:28
• Last Updated 06/21/2024 22:48

On June 20th we will be updating the operating system of the S2 server [S2.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

CloudLinux 7 reaches End of Life [EoL] on July 1, 2024, so no further security updates or patches will be available after that date. Additionally, cPanel has stopped providing updates for CloudLinux 7, keeping us on a slightly older version lacking some of the newer features they've released recently.

CloudLinux 8 reaches End of Life [EoL] on May 31, 2029, so we do not anticipate having to perform this kind of maintenance again for the next 5 years. We would love to go ahead and move to CloudLinux 9, however, cPanel's support for CloudLinux 9 is experimental and not recommended for production environments.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Updates:

Affecting Server - S1

• 06/19/2024 22:00 - 06/20/2024 01:05
• Last Updated 06/20/2024 01:03

On June 19th we will be updating the operating system of the S1 server [S1.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

CloudLinux 7 reaches End of Life [EoL] on July 1, 2024, so no further security updates or patches will be available after that date. Additionally, cPanel has stopped providing updates for CloudLinux 7, keeping us on a slightly older version lacking some of the newer features they've released recently.

CloudLinux 8 reaches End of Life [EoL] on May 31, 2029, so we do not anticipate having to perform this kind of maintenance again for the next 5 years. We would love to go ahead and move to CloudLinux 9, however, cPanel's support for CloudLinux 9 is experimental and not recommended for production environments.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Update:
10:48 PM Full disclosure, the upgrade is going to run late today by about an extra hour. The Centos 7 repositories were randomly not responding correctly today. As that is a critical step in the Upgrade process it did delay the start until it was resolved.
12:30 PM The Server is partly up. Email and sites not requiring MySQL are online.
12:47 PM MySQL is now online.
01:02 PM Doing final reboots.

Affecting Server - S0

• 06/18/2024 22:00 - 06/19/2024 00:04
• Last Updated 06/12/2024 23:00

On June 18th we will be updating the operating system of the S0 server [S0.supportedns.com] from CloudLinux 7 to CloudLinux 8.

We are scheduling downtime on this date starting at 10 PM Eastern Time until 2 AM Eastern Time.  While we expect the upgrade only to take approximately 2 hours, we are scheduling a longer window just in case things don't go as quickly as expected.

CloudLinux 7 reaches End of Life [EoL] on July 1, 2024, so no further security updates or patches will be available after that date. Additionally, cPanel has stopped providing updates for CloudLinux 7, keeping us on a slightly older version lacking some of the newer features they've released recently.

CloudLinux 8 reaches End of Life [EoL] on May 31, 2029, so we do not anticipate having to perform this kind of maintenance again for the next 5 years. We would love to go ahead and move to CloudLinux 9, however, cPanel's support for CloudLinux 9 is experimental and not recommended for production environments.

While services will not be unavailable during the whole maintenance window - there will be long periods during the maintenance window when the server is unresponsive.  We have tested these upgrades in our development and testing cluster and the process has gone smoothly but does take a fair bit of time. We will be taking every step possible to ensure this maintenance does not last any longer than necessary.

Affecting Server - ST1

• 07/29/2022 02:18 - 07/29/2022 03:18
• Last Updated 07/29/2022 02:33

We are aware of issues with sending email via Webmail on the ST1 server and we are investigating and working to resolve them.

At this time it looks as though regular SMTP with a stand-alone mail client [ Outlook, Mac Mail, Thunderbird, etc ] are still working.

Update - 2:23 AM ET

This issue appears to be affecting inbound email as well.  This appears to have started with the nightly cPanel update.  We've engaged cPanel emergency support and are continuing to investigate on our end.

Affecting Other - Lets Encrypt

• 09/29/2021 00:00 - 10/01/2021 09:01
• Last Updated 10/01/2021 09:01

One of Let's Encrypt's intermediate certificates, used by browsers and other software to verify the issued certificates, expired September 29th [yesterday].  This intermediate certificate was used to issue and renew certificates whose expiration dates were after September 29th.

Some browsers and applications, such as mail clients, may report these certificates as expired or invalid.

The solution to this issue is to re-issue any certificates having issues via cPanel -> Let's Encrypt.  It may take up to 15 minutes for the new certificate to take effect.

For more details you can see this thread on Let's Encrypt's community forums: https://community.letsencrypt.org/t/r3-intermediate-certificate-has-expired-it-issued-certs-past-its-expiration-date/

Affecting System - SpamExperts Spam Filtering

• 09/15/2020 14:20 - 09/16/2020 00:19
• Last Updated 09/15/2020 17:28

SpamExperts is experiencing issues and they're aware of them and working to resolve them.  We wish we could provide an ETA but SpamExperts has not provided an ETA.

This is the first issue we've seen with them in nearly a decade of providing spam filtering through them so this is a very unusual incident.  When we have more information we will provide it here.

• 09/21/2018 18:10 - 09/26/2018 01:46
• Last Updated 08/26/2020 10:47

Update 7

All of our attempts to speed this process up haven't been successful.  As it stands we're restoring several dozen accounts per server all at the same time.  If you're seeing a cPanel IP Error page and you're using our DNS/Nameservers - your site isn't restored yet.  If you can't log into your cPanel with your cPanel username and password - your site isn't restored yet.

Nobody wants you back online more than us - I wish we could snap our fingers and everybody was online.  Better yet if we could go back and avoid this issue entirely that would be great.  As it stands - if you're not online yet - we're sorry and we're working on it.  If you are back online now - if you have any issues please let us know.

====
Update 6

We are working on two issues right now:
1. The cPanel Restoration process is not restoring MySQL data although the data is in the backup and verified good.  We're working with cPanel Emergency Support on this.
2. We are working on also getting a secondary copy of our backup data local to the backup server that we're going to be spreading across 3 servers with 10 GBPS links so that we can ideally cut the restoration time down as low as, in theory. 2.5 hours.  In reality I doubt we'll be able to saturate the connections due to cPanel restoration overhead - but we're doing our best to both get a second copy of the data just in case as well as giving us more throughput for restorations.

Ideally we'll have everything back online within 10 to 20 hours at the most - but we're still very much working-in-progress on this.

====
Update 5

It looks like our luck is against us.  The damage to the servers was more extensive than we believed originally.  We have the servers in a state of being ready to restore client data and when we attempted to mount the current data - the data can't be mounted / isn't workable.

We still have an administrator working on recovering this local data, however, it is looking like we're going to have to restore all client data from our latest backups.  We're not any more happy about this than any of our clients.  At this point it looks like it could take up to 30 or 40 hours to restore this data.

Once everything is back online we are going to be doing a complete overhaul on our backup system so that a full restore such as this will take only 4 to 6 hours.  There are some bottlenecks in our current backup system that are going to keep things from going quickly.  We could make some changes to the system now in an attempt to speed things up but as the backups we have now are the last remaining copy of the data we aren't going to be taking any chances.  As it stands while we hate to be offline for an extended period - the risk of migrating our backup drives to another server is too great.

====
Update 4

We are performing final updates  to the servers and software in preparation for bringing accounts back online.  There is the possibility that we may have missed some settings / software and there is some software that isn't a priority, like Softaculous, that we'll focus on once services are restored.

We will be getting started on bringing accounts back online shortly.  We're also working to try and make sure that account IP addresses do not change.  While we can't make any promises - we're doing our best.

====
Update 3

All servers are online - we're working on configuring all software - cPanel, LiteSpeed, MySQL, etc - to get them ready to bring accounts back online.

====
Update 2

We are working to bring online the servers that host all clients.  This is primarily a function of provisioning the guests on the hosts and configuring them to be ready to accept accounts.  Once this is done we will be conducing cPanel restorations to the servers of just the cPanel and MySQL data and re-connecting the accounts to the home directory data from before the outage.

We do have current and up-to-date MySQL data from before the outage, however, we're going to be restoring MySQL data from our backups taken last night and then will be working to restore any MySQL data lost between then and the outage with clients on a one-on-one basis as needed.

We hope to have everything back online tonight, however, we will keep you updated as we progress with the disaster recovery.

Once we are fully back online we will be providing a complete Reason For Outage [RFO] to all affected clients.  This RFO will outline what happened, why it happened, what we're changing to prevent it from happening again, and what we're changing so that should anything like this happen again we can recover from it substantially faster.

Our goal is always to be as open and transparent as we can be - and we will continue this.  Right now we're focused on restoring services and will provide more details once services are back online.

====
Update 1

We are having to bring new servers online to restore services.  We are working hard to get this done as quickly as possible and are going to do our best to do this with as little disruption as possible.

No customer data has been damaged or lost - only system-level data.  We will provide full details as to what caused this outage, what we did to resolve it, and what we're going to do to prevent it from happening again as soon as we have the chance.

====
Initial Message

We are experiencing a major outage across all services at this time. We are aware of the issue and are working to restore services as quickly as possible.

We will provide more detail when we can, however, we are focused on restoring services and diverting all energy to those tasks presently.

Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.