Duration: 2 days, 11 hours and 22 minutes

Affected Components: Billing / Ordering Portal

Aug 29, 11:00:00 GMT+0
Investigating - We are currently investigating an incident impacting the console, website, and API..

Aug 31, 22:22:09 GMT+0
Resolved - Hostodo restored the customer panel on clean infrastructure after a 29 August ransomware incident on a control-plane host. Customer VPS disks were NOT the encryption target. Some support tickets after 15 July 2026 were not recoverable; some later invoices may be missing from the panel. We have not found evidence that customer data left our systems; because some logs were altered, we cannot completely rule it out. Additional details will be sent via email..

Duration: 1 hour and 26 minutes

Affected Components: LV01 (Las Vegas, NV)

Aug 26, 14:17:09 GMT+0
Identified - We are continuing to work on a fix for this incident..

Aug 26, 15:43:00 GMT+0
Resolved - The affected traffic has been blackholed and connectivity has been restored..

Duration: 8 hours

Affected Components: LV01 (Las Vegas, NV)

Aug 16, 19:30:00 GMT+0
Resolved - This incident has been resolved..

Aug 16, 11:30:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Duration: 1 day, 18 hours and 1 minute

Affected Components: DET01 (Detroit, MI)

Aug 14, 10:00:00 GMT+0
Identified - Following up from yesterday - we will be performing emergency maintenance to fix the disk issues on EPYCK01.DET01 starting at 6:45PM Eastern time..

Aug 14, 22:47:39 GMT+0
Identified - Maintenance is now in progress..

Aug 15, 00:30:11 GMT+0
Identified - We are migrating the disk images now..

Aug 16, 16:48:27 GMT+0
Completed - Maintenance has completed successfully. All VMs have been migrated to the new disk..

Duration: 1 hour and 36 minutes

Affected Components: DET01 (Detroit, MI)

Aug 13, 22:35:40 GMT+0
Identified - We will be performing emergency maintenance to fix the disk issues on EPYCK01.DET01.

Aug 13, 23:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 13, 23:54:16 GMT+0
Identified - We have a missing piece of equipment to fully complete the job. We are powering off the node for a reseat of the existing drive, while we wait for the additional component to arrive. We will likely schedule another maintenance window tomorrow if reseating does not resolve the issue..

Aug 14, 00:35:41 GMT+0
Completed - We have reseated the existing drive, however we are experiencing errors. We will have another maintenance window tomorrow once the missing component arrives and we are able to install the new drive..

Duration: 8 days, 14 hours and 44 minutes

Affected Components: DET01 (Detroit, MI)

Aug 8, 04:15:53 GMT+0
Investigating - We are currently investigating degraded performance on EPYCK01 in Detroit..

Aug 9, 15:49:46 GMT+0
Identified - We have identified a degraded NVMe drive. We're awaiting shipment of a replacement drive and will be replacing it as soon as it arrives..

Aug 16, 19:00:21 GMT+0
Resolved - This incident has been resolved..

Duration: 11 hours and 47 minutes

Affected Components: DET01 (Detroit, MI)

Jun 24, 17:38:17 GMT+0
Investigating - We are currently investigating this incident..

Jun 25, 05:25:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jun 25, 05:25:01 GMT+0
Resolved - This incident has been resolved..

Jun 25, 01:41:59 GMT+0
Identified - We are continuing to work on a fix for this incident. Our upstream provider, INCX, is facing a DDoS attack..

Duration: 1 day and 33 minutes

Affected Components: LV01 (Las Vegas, NV)

Jun 12, 21:32:46 GMT+0
Resolved - This incident has been resolved..

Jun 13, 21:32:00 GMT+0
Resolved - This incident has been resolved..

Jun 11, 21:00:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 5 hours and 22 minutes

Affected Components: DET01 (Detroit, MI), TPA01 (Tampa, FL)

May 26, 13:41:06 GMT+0
Investigating - We are currently investigating this incident..

May 26, 17:40:37 GMT+0
Identified - We are continuing to work on a fix for this incident..

May 26, 18:44:58 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 26, 19:03:00 GMT+0
Resolved - This incident has been resolved..

Duration: 3 hours and 54 minutes

Affected Components: LV01 (Las Vegas, NV)

Apr 30, 11:59:00 GMT+0
Identified - We are experiencing network and routing issues in our Las Vegas presence. We are working diligently with our upstream network provider on a resolution..

May 2, 14:53:11 GMT+0
Monitoring - FiberHub has implemented a fix and are currently monitoring the result..

May 2, 17:53:33 GMT+0
Resolved - This incident has been resolved..

Duration: 7 days, 7 hours and 35 minutes

Affected Components: Billing / Ordering Portal

Mar 29, 03:15:55 GMT+0
Identified - We are continuing to work on a fix for this incident..

Mar 26, 14:36:13 GMT+0
Investigating - We are currently investigating degraded performance on the website and instance management console..

Mar 31, 22:46:42 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Apr 2, 22:10:57 GMT+0
Resolved - This incident has been resolved..

Duration: 13 hours and 6 minutes

Affected Components: DET01 (Detroit, MI)

Mar 21, 18:25:39 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Mar 21, 05:19:51 GMT+0
Investigating - We are currently investigating a network outage impacting epyck01.det01..

Mar 21, 18:25:40 GMT+0
Resolved - This incident has been resolved..

Duration: 2 days, 15 hours and 56 minutes

Affected Components: LV01 (Las Vegas, NV)

Jan 16, 10:00:00 GMT+0
Investigating - EPYCK01.LV01 is currently down. We are currently investigating this incident..

Jan 17, 00:56:53 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result. We had a power supply failure, and were able to switch over to a redundant PSU. .

Jan 19, 01:55:47 GMT+0
Resolved - This incident has been resolved..

Duration: 24 minutes

Affected Components: LV01 (Las Vegas, NV)

Jan 8, 18:10:38 GMT+0
Investigating - We are currently investigating this incident..

Jan 8, 18:34:18 GMT+0
Monitoring - We are having a BGP related bug that we are working to resolve. We implemented a fix and are currently monitoring the result..

Jan 8, 18:34:19 GMT+0
Resolved - This incident has been resolved..

Duration: 16 hours and 33 minutes

Affected Components: LV01 (Las Vegas, NV)

Jan 3, 17:07:40 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jan 3, 17:08:02 GMT+0
Resolved - This incident has been resolved..

Jan 3, 00:34:56 GMT+0
Investigating - We are currently investigating this incident. A network misconfiguration has impacted 3 IPv4 subnets in LV01\. We are working with our upstream provider to resolve. .

Duration: 39 minutes

Affected Components: DET01 (Detroit, MI)

Dec 23, 18:41:43 GMT+0
Investigating - We are currently investigating this incident..

Dec 23, 18:47:47 GMT+0
Identified - We are continuing to work on a fix for inbound DDoS attacks towards Hostodo infrastructure. At the moment we are stable..

Dec 23, 19:20:13 GMT+0
Resolved - This incident has been resolved..

Duration: 11 hours and 29 minutes

Affected Components: LV01 (Las Vegas, NV)

Dec 16, 20:22:06 GMT+0
Identified - We've identified the issue to be a networking misconfiguration caused by our upstream provider. We are working with them to resolve it ASAP..

Dec 16, 23:13:14 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Dec 17, 02:52:56 GMT+0
Resolved - This incident has been resolved..

Dec 16, 15:24:04 GMT+0
Investigating - We are currently investigating a network outage impacting LV01.

Duration: 3 days, 1 hour and 50 minutes

Affected Components: DET01 (Detroit, MI)

Dec 13, 03:08:10 GMT+0
Resolved - This incident has been resolved..

Dec 10, 01:18:34 GMT+0
Investigating - We are currently investigating this incident..

Duration: 28 minutes

Affected Components: DET01 (Detroit, MI)

Dec 6, 19:17:19 GMT+0
Resolved - This incident has been resolved..

Dec 6, 18:49:11 GMT+0
Identified - We are continuing to work on a fix for this incident..

Dec 6, 18:51:42 GMT+0
Identified - We are experiencing a DDoS attack against our \[DET01\] point of presence. We are working on mitigating the attack. This also impacts <https://console.hostodo.com/>.

Dec 6, 19:17:18 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Duration: 20 hours and 15 minutes

Affected Components: LV01 (Las Vegas, NV)

Nov 26, 03:57:36 GMT+0
Investigating - We are currently investigating this incident impacting some legacy clients on [K1.LV](http://K1.LV) and K4.LV.

Nov 26, 07:03:33 GMT+0
Identified - We are continuing to work on a fix for this incident. Dual PSU failures caused some disk corruption on the boot drives of K1LV and K4LV. We are working on migrating impacted customers to new nodes..

Nov 27, 00:11:38 GMT+0
Monitoring - Impacted customers are being migrated to Gen2\. Your instance will be coming online over the next 2-3 hours..

Nov 27, 00:12:12 GMT+0
Resolved - This incident has been resolved..

Duration: 2 days, 15 hours and 36 minutes

Affected Components: DET01 (Detroit, MI), LV01 (Las Vegas, NV), TPA01 (Tampa, FL)

Aug 26, 22:19:59 GMT+0
Identified - LV01 and TPA01 back online. Working on DET01..

Aug 25, 10:54:03 GMT+0
Investigating - We are currently experiencing a major DDOS attack and are working on mitigating .

Aug 25, 22:04:11 GMT+0
Identified - We are experiencing a major DDOS attack and are working on implementing DDOS protection to mitigate the attack. We are continuing to work on a fix for this incident..

Aug 26, 03:32:32 GMT+0
Identified - We are continuing to work on a fix for this incident. We have DET01 DDoS protection coming online first, followed by LV and TPA..

Aug 26, 23:11:49 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 26, 16:10:04 GMT+0
Identified - We are continuing to work on a fix for this incident. We expect all three locations to be back online today..

Aug 28, 02:29:45 GMT+0
Resolved - This incident has been resolved..

Duration: 8 hours and 36 minutes

Affected Components: LV01 (Las Vegas, NV)

Aug 2, 16:06:46 GMT+0
Investigating - We are currently investigating this incident..

Aug 3, 00:42:57 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 3, 00:42:57 GMT+0
Resolved - This incident has been resolved..

Duration: 51 minutes

Affected Components: LV01 (Las Vegas, NV)

Jul 27, 21:47:34 GMT+0
Investigating - We are currently investigating this incident..

Jul 27, 22:07:11 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 27, 22:38:25 GMT+0
Resolved - We are mitigating DDoS attacks from various sources causing outages in LV01\. This incident has been resolved. .

Jul 27, 22:33:36 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 27, 22:13:39 GMT+0
Resolved - This incident has been resolved..

Duration: 4 hours and 41 minutes

Affected Components: LV01 (Las Vegas, NV)

Jul 20, 01:50:46 GMT+0
Identified - We are continuing to work on a fix for this incident..

Jul 19, 23:33:09 GMT+0
Investigating - We are currently investigating a network outage affecting the IP subnet 66.187.4.0/24.

Jul 20, 03:48:01 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 20, 04:14:01 GMT+0
Resolved - This incident has been resolved..

Duration: 13 hours and 37 minutes

Affected Components: LV01 (Las Vegas, NV)

Jul 8, 13:27:38 GMT+0
Investigating - We are currently investigating this incident..

Jul 8, 13:38:49 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 8, 20:17:19 GMT+0
Monitoring - We are continuing to monitoring this issue..

Jul 9, 03:04:24 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 59 minutes

Affected Components: DET01 (Detroit, MI)

Jul 6, 22:18:00 GMT+0
Investigating - We are currently investigating this incident..

Jul 7, 00:17:00 GMT+0
Resolved - This incident has been resolved..

Duration: 7 hours and 31 minutes

Affected Components: LV01 (Las Vegas, NV)

May 21, 14:58:51 GMT+0
Investigating - We are currently network connectivity issues with a few subnets within LV01..

May 21, 16:46:54 GMT+0
Identified - We are continuing to work on a fix for this incident..

May 21, 22:08:00 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 21, 22:30:00 GMT+0
Resolved - This incident has been resolved..

Duration: 1 hour and 28 minutes

Affected Components: LV01 (Las Vegas, NV)

Mar 14, 15:32:00 GMT+0
Completed - .

Mar 14, 03:01:18 GMT+0
Identified - Our upstream provider is performing network maintenance between 11pm and 1am ET today (March 13, 2025) Downtime is expected to be minimal, expect a brief network outage during this period..

Mar 14, 17:00:00 GMT+0
Identified - Our upstream provider had maintenance last night but looks like there are still some issues. We are in contact with them..

Duration: 5 hours and 29 minutes

Affected Components: LV01 (Las Vegas, NV)

Jan 18, 20:00:01 GMT+0
Identified - We are running a bit behind and will be extending the maintenance period for 2 hours..

Jan 18, 20:00:00 GMT+0
Identified - We are scheduling maintenance on Saturday, January 18th, 2025 at 3pm ET/12pm PT to perform a motherboard swap on EPYCK02LV01 & EPYCK03LV01 to resolve performance issues experienced with the current motherboards. We are scheduling a 4 hour window to perform this work, however, we expect this to result in approximately 1 hour of downtime per node. Please let us know if you have any questions..

Jan 19, 01:29:27 GMT+0
Completed - Maintenance has been completed successfully! Performance issues have been resolved on EPYCK02LV01, EPYCK03LV01, and EPYCK02TPA01.

Duration: 10 hours and 34 minutes

Affected Components: TPA01 (Tampa, FL)

Dec 6, 15:10:00 GMT+0
Identified - We will be performing physical network maintenance on EPYC nodes in TPA01 on Friday, Dec 6th, between 8am and 9am ET. Expect approximately 2-3 minutes of downtime during this period..

Dec 7, 02:09:11 GMT+0
Completed - Maintenance has completed successfully..

Dec 6, 15:30:00 GMT+0
Identified - We were delayed here, our technician is en route to the facility to get started on the maintenance..

Dec 6, 15:35:05 GMT+0
Identified - We have begun..

Duration: 3 hours

Affected Components: DET01 (Detroit, MI)

Nov 22, 02:22:16 GMT+0
Identified - EPYCK01.DET01 hardware upgrade has been completed. VMs are coming back online. We will be taking a short break to prepare for the network upgrade. .

Nov 22, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 22, 03:56:23 GMT+0
Identified - Extended maintenance period for 1 hour. Still working through this..

Nov 22, 02:00:00 GMT+0
Identified - During this maintenance window, we will be performing network maintenance on our core router, installing a redundant 10G network drop to add some redundancy and improve overall network health. We will also be performing scheduled maintenance on EPYCK01.DET01 to upgrade its hardware. The hostnode will be powered off, meaning your instance will also be offline for a period of time of up to 20 minutes while we perform the upgrade. This maintenance is scheduled for Thursday, November 21st 2024 between 9PM and 11PM EST. Please contact us via live chat if you have any questions or concerns..

Nov 22, 03:06:21 GMT+0
Identified - Beginning our network maintenance..

Nov 22, 02:03:38 GMT+0
Identified - Working on this now. EPYCK01.DET01 is coming offline for a hardware upgrade..

Nov 22, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour

Affected Components: TPA01 (Tampa, FL)

Jul 26, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 26, 03:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 26, 02:00:00 GMT+0
Identified - We are planning for a scheduled network maintenance tomorrow, Thursday July 25th 2024, between 10PM-11PM EST to modify some firewall rules. This maintenance will affect TPA01 only. We expect up to 15 minutes of downtime during this period..

Duration: 15 days, 15 hours and 23 minutes

Affected Components: DET01 (Detroit, MI), Billing / Ordering Portal

Jun 8, 15:33:33 GMT+0
Identified - We have deracked all equipment and arrived at the new datacenter. We are unloading and will begin to re-rack the equipment shortly..

Jun 8, 14:50:46 GMT+0
Identified - We are beginning to shutdown our equipment..

Jun 8, 14:36:00 GMT+0
Identified - We have arrived at the datacenter and are beginning to shutdown each server..

Jun 8, 17:22:30 GMT+0
Identified - We have finished racking all of our equipment and are working on bringing everything back online now. .

Jun 8, 20:15:22 GMT+0
Completed - We have completed the move successfully!.

Jun 8, 19:28:21 GMT+0
Identified - We are coming back online now. We appreciate your patience..

Jun 8, 14:00:00 GMT+0
Identified - We will be moving our equipment from our current building to a datacenter operated by the same company across the street. Please expect about 4 hours of downtime during this migration as we migrate our equipment, including all customer servers and hardware powering our website. We appreciate your patience and understanding. We will follow up with more details and information. .

Jun 8, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 1 hour and 31 minutes

Affected Components: TPA01 (Tampa, FL)

Feb 28, 12:30:00 GMT+0
Identified - We are planning for a scheduled maintenance on Wednesday, February 28th, between 7:30am and 10am EST. This maintenance will impact all customers on EPYCK01.TPA01\. We will be powering off this node in order to upgrade the memory and disk capacity. .

Feb 28, 12:30:01 GMT+0
Identified - Maintenance is now in progress.

Feb 28, 14:01:20 GMT+0
Completed - Maintenance has been completed, all instances are rebooting now..

Duration: 2 days, 6 hours and 35 minutes

Affected Components: TPA01 (Tampa, FL)

Nov 14, 19:45:09 GMT+0
Identified - We are shutting down and extracting the servers from Miami now. .

Nov 14, 18:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 14, 18:00:00 GMT+0
Identified - We are planning on migrating our Miami, FL presence to a new datacenter in Tampa, FL between Tuesday, November 14th and Wednesday, November 15th 2023. This migration will involve a few hours of downtime as we physically power off the machines and drive them over to Tampa, and get them racked in our new datacenter space. Please expect a total of approximately 6 hours of downtime during this time period. This move will give us more space to expand and allow us to bring Hostodo Gen2 to the Florida region. Thank you for your understanding and feel free to reach out to us with any questions..

Nov 14, 20:47:23 GMT+0
Identified - We are transporting the nodes back to Tampa now. Expect to be online around 10PM EST. .

Nov 17, 00:35:22 GMT+0
Completed - Maintenance has completed successfully..

Nov 15, 02:50:06 GMT+0
Identified - Servers will be back online momentarily..

Duration: 4 hours and 11 minutes

Affected Components: DET01 (Detroit, MI), LV01 (Las Vegas, NV)

Oct 30, 22:00:00 GMT+0
Identified - Maintenance is still in progress.

Oct 30, 19:00:00 GMT+0
Identified - We are planning to perform upgrades to our core network routers in **LV01** Las Vegas, NV and **DET01** Detroit, MI on **Monday, October 30th **between **3:00PM to 6:00PM EST**. Please expect about 30 minutes of downtime during this period. .

Oct 30, 23:11:10 GMT+0
Completed - Maintenance has completed successfully..

Oct 30, 19:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 6 hours and 24 minutes

Affected Components: LV01 (Las Vegas, NV)

Oct 1, 05:47:12 GMT+0
Completed - K4.LV is now back online. .

Sep 30, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 30, 23:23:05 GMT+0
Identified - We have replaced the failed boot drive causing K4LV to become unresponsive. We are currently working on getting all VMs back online within the next hour..

Sep 30, 16:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during this time. This is a continuation of last nights maintenance period. We will be switching over to the new network on Saturday, Sep 30th 2023 between 9AM to 2PM Pacific We will also be resolving an issue causing K4LV to be unresponsive..

Sep 30, 21:05:17 GMT+0
Identified - Maintenance is still in progress. .

Duration: 6 hours and 6 minutes

Affected Components: LV01 (Las Vegas, NV)

Sep 29, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 30, 02:06:21 GMT+0
Completed - Maintenance will continue tomorrow morning. We are aware of an issue with VPSs on K4LV, we are working on it..

Sep 30, 01:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 29, 20:00:00 GMT+0
Identified - We are planning for a scheduled maintenance on September 29th, 2023. We will be upgrading our network equipment and begin using a new network transit blend which will dramatically improve latency. During this maintenance period we will also be performing maintenance on our Las Vegas dedicated servers. Please expect your dedicated server to be shutdown and rebooted during this time. We will be doing this work between 1:00PM to 6:00PM Pacific time (4:00 PM to 9:00PM EST) .

Duration: 1 day and 14 hours

Affected Components: LV01 (Las Vegas, NV)

Jun 30, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 2, 03:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jun 30, 13:00:00 GMT+0
Identified - **When:** Friday, June 30, 2023 9AM Pacific -> Saturday July 1, 2023 11PM Pacific **Where:** Las Vegas, NV **What:** We will be performing a physical migration of our Las Vegas, NV infrastructure to a new rack within the same datacenter. There will also be some networking changes taking place during this maintenance period. We will be aiming for at most 2 hours of downtime. **How Long:** Each node will be down for approximately 2 hours at some point during this maintenance period..

Jul 1, 19:06:40 GMT+0
Identified - UPDATE Saturday July 1st The maintenance was successful however the network change announced previously was not completed and will be completed at a different time. k3.lv.hostodo.com has some extended downtime but will be fixed shortly (12PM PST) All other nodes should be online..

Duration: 3 hours and 2 minutes

Affected Components: LV01 (Las Vegas, NV)

Dec 1, 05:02:05 GMT+0
Completed - Maintenance has completed successfully..

Dec 1, 03:17:17 GMT+0
Identified - Maintenance is now in progress. We have extended the maintenance period to midnight EST. .

Dec 1, 02:00:00 GMT+0
Identified - We will be performing some maintenance on NVME-K3.LV and NVME-K10.LV between 9PM and 11PM Eastern time. Each node will go down for approximately 20 minutes..

Duration: 7 hours and 23 minutes

Affected Components: LV01 (Las Vegas, NV)

Sep 5, 02:23:16 GMT+0
Completed - Maintenance has completed successfully..

Sep 4, 22:08:09 GMT+0
Identified - All nodes are back online! Maintenance still in progress, so there may be some intermittent downtime, but the core work has been completed..

Sep 4, 20:50:00 GMT+0
Identified - Maintenance is now in progress. .

Sep 4, 19:00:00 GMT+0
Identified - We will be performing datacenter maintenance in our Las Vegas, NV datacenter on Saturday September 4th between 1:00 PM and 5:00 PM PST. We will be moving our nodes to a new rack now that we have used up all of the existing space in our existing rack. We expect total downtime of ~ 30 minutes per node. .