Changelog/2021/June 17

Incident Report

 * Jun 20, 2021 - 13:46 EDT: UPDATE: We have the new hardware configured, but we still need more time for set up and for our external partners to do what they need to do. But you can read more about it on the website > https://hypixel.net/
 * Jun 19, 2021 - 11:44 EDT: UPDATE: A progress bar has been added to the website to show how much more needs to be done before we can get up and running again. Additionally, we posted an update regarding what's going on with the maintenance, so be sure to check it out > https://hypixel.net/   If you are having trouble seeing this new update, make sure to press CTRL and F5 at the same time while on the page to refresh the cache.
 * Jun 17, 2021 - 20:33 EDT: UPDATE: The Hypixel Network will be down for emergency maintenance starting now. We are sorry for any inconvenience and thank you for being patient with us. You can read more at https://hypixel.net
 * Jun 16, 2021 - 19:31 EDT: We are continuing to work with our upstream provider to resolve this. We have resolved the issue of some players being unable to connect. We appreciate your patience and support as we work to resolve these issues.
 * Jun 16, 2021 - 07:30 EDT: IDENTIFIED: We are aware of the issue still existing and are working on a larger solution with our provider. This may take some time and we'll keep this incident updated.
 * Jun 15, 2021 - 21:55 EDT: MONITORING: Upstream has implemented a change and we are monitoring the results.
 * Jun 15, 2021 - 20:38 EDT: IDENTIFIED: We are aware of the connection issues occurring again and are working with our upstream provider.
 * Jun 15, 2021 - 19:38 EDT: MONITORING: Upstream has implemented a change and we are monitoring the results.
 * Jun 15, 2021 - 15:33 EDT: UPDATE: We are continuing to work with the upstream provider to resolve the issues as soon as possible
 * Jun 15, 2021 - 15:03 EDT: IDENTIFIED: We have identified the issue with an upstream provider and are working with them to resolve it as soon as possible.
 * Jun 15, 2021 - 15:00 EDT: INVESTIGATING: We are currently investigating an issue with intermittent connection issues to the Minecraft Server and Forums.

Admin Note and Updates
Hey guys!

We will be taking down the Hypixel Network for emergency maintenance. This maintenance begins now and will last for at least 24 hours. Should we choose to extend this downtime, we will post an update on this news post.

During this time we will be reworking the network to ensure the issues we have been having these last few days don't happen again.

We would like to apologize for the inconvenience the last few days have caused and we hope that our community understands the difficulty of the situation.

The situation is: Our host is currently under large-scale DDoS attacks and they are actively working to resolve the situation. We are assisting them the best we can, which is why we have to put the server on maintenance during this process.

We have already done many network improvements that have lessened the impact of the attacks that are either directed at us or not, but it has proven not to be enough. Our top priority is to mitigate the impact of these attacks as quickly as possible while working on permanent solutions that will shield us better from future attacks and isolate our network.

The good news is that we have dealt with DDoS attacks for well over 8 years now and we have a great understanding of the situation. Each of these attacks is helping us better understand weak points that need to be improved from our side of things and also for our host. As much as this situation is bad for everyone at this time, it is giving us very good data to further understand and lessen the impacts of future attacks.

In reality, about a year ago, these attacks would have had no impact at all, but some recent changes at our host caused a flaw in our setup. so the host is working with us to resolve the situation. We all know what to do to resolve the situation because we have done it before, but it requires quite a bit of work to get to that place again. We’re hoping 24 hours is enough for this upgrade, but please understand it may take more than that.

To be fair, we haven't been as alert or ready as we should have been as it's been about 2 years since our last DDoS attacks. This is definitely a good opportunity for us to set some improvements in motion and deal with this once and for all.

We made sure any time-based services on the network are taken care of. This means pausing boosters, pausing the SkyBlock Auction House and Bazaar timers, pausing timers on MVP++ and daily rewards, etc. We’re hoping not to forget anything, but if we do, we are always open to a chat on our Help Desk.

Furthermore, when we bring the server back online, we will be turning on Triple Coins and Triple XP for at least a week.

As the situation progresses, we will be posting any updates to our status page. Once we are ready to go live again, we will make an update with more details and share as much as we possibly can with our community.

As always, we are extremely thankful for everyone's support in these difficult moments and apologies once again for the issues. We want to reassure our community that the current situation is our top priority.

Update #1 (June 18th - 3:35PM ET): There are still several ongoing large-scale DDoS attacks going on in the Minecraft server scene at this time and we are not quite done with our changes before we can come back online safely. We do not have a precise ETA at this moment, as we are relying on several external partners to help us finalize the DDoS protection and major changes to our network infrastructure. We will do our best to give an ETA by tomorrow morning, but overall we aim to get back online at some time this weekend if all goes well. Once again, we appreciate your support during these difficult times, we will keep you guys updated the best we can.

Update #2 (June 19th - 11:15AM ET): We’ve added a progress bar to give you a better understanding of where we are in regards to bringing the Hypixel Server back to all players. Please note that this is not representative of time, and will be manually updated, so don’t completely rely on it. This will, however, give you all at least some transparency on where we are with the work. It's currently too difficult to give an exact ETA as several external teams are involved. As we are passed updates, we will quantify the % and add it to the progress bar.

This morning, the on-site team moved most of the core hardware to be on the Cloudflare network, and our servers will no longer be on any network of our host. We will be effectively 100% Cloudflare, with full DDoS protection from them. Unfortunately, in order to do this, we have had to move (and still are) quite a lot of hardware from the current host’s network; this has taken a fair bit of time due to the large number of servers and racks we have. However, all critical machines required to begin initial setup and hold a decent player count have been moved.

Today, external network teams are being dispatched to assemble and configure the new network. This new setup will be on Cloudflare, and your ping should technically be somewhat the same. However, because this is the first time we will be using full Cloudflare, there’s a possibility that we may have some issues at first. Despite that, their team is assisting us and we have worked with them for years with great success.

Note that the forums, API and some other things might be broken at first, but will be sorted later on.

We will try to give another update as soon as possible. Keep an eye on the progress bar - we will do our best to update it whenever possible!

Thank you for your support!

Update #3 (June 20th - 12:45PM ET): All of the new hardware has been configured to the new network and all the core servers and minigame servers are working great. We are now entering the final steps of setting up and configuring the last bits of the network and DDoS protection, this is the more tricky part because we heavily rely on external partners to do their parts so we have no clear vision on progress at this time.

We are still seeing continuous large-scale DDoS attacks across the board in the Minecraft server scene, which means our decision to make this major change was the right path forward.

Tonight, if all goes well of course, we will be doing tests assuming our partners meet their deadlines. Please note that these tests will require reboots and downtime - the player count will be severely limited. As soon as the situation is fully back in our hands, you will see more frequent updates from us and then slowly increase the player count to allow all of you in.

Our team has done everything humanly possible, spared no expense and worked with incredible partners to make this happen as quickly as possible for our amazing community that we are grateful for.