Gemini Partial Outage - September 10, 2026

<p> Incident began at <strong>2026-09-01 07:44</strong> and ended at <strong>2026-09-01 11:52</strong> <span>(all times are <strong>US/Pacific</strong>).</span></p><div class="cBIRi14aVDP__status-update-text"><h2>Incident Report</h2> <h2>Summary</h2> <p>On Tuesday, 1 September 2026, some customers in us-central1 experienced network service degradation and instance isolation for a duration of 4 hours and 11 minutes. To our customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.</p> <h2>Root Cause</h2> <p>The event was triggered by the inadvertent physical disconnection of network fiber-optic cables during routine hardware maintenance. A technician was performing a scheduled capacity upgrade on data center routers that support a fraction of capacity in the us-central1-b zone and a small fraction of capacity in the us-central1-f zone.</p> <p>During the hands-on maintenance, optical fibers connecting the data center routers to the network fabric were unplugged and the optical transceivers were replaced with a transceiver that supports higher-density connectivity. This transceiver was not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber.</p> <p>The network is designed with redundant routers in separate data center rooms within the same building so that failure of one router does not interrupt service. The maintenance was intended to upgrade one router at a time over several days, with traffic diversions at each step and verification between steps that the network has returned to a fully-connected state.</p> <p>However, a procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time. The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption.</p> <p>As a result, the technician sequentially unplugged all fiber paths across the affected devices within 13 minutes. The speed and nature of the error prevented warnings of incorrect action from reaching the engineer before connectivity was lost. Additionally, a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed.</p> <p>This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone.</p> <h2>Remediation and Prevention</h2> <p>The issue was detected immediately by automated network loss monitoring systems as well as proactive probes, which rapidly engaged the network engineering and incident response teams.</p> <p>To mitigate the immediate customer impact, engineering teams actively moved traffic away from the impacted infrastructure.</p> <p>Concurrently, hardware operations technicians on site identified the disconnected optical links and physically re-inserted the original optical transceivers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service.</p> <p>Google is committed to preventing a repeat of this issue in the future and is completing the following actions:</p> <ul> <li>Reinforce training in us-central1 region and globally around safe maintenance techniques, including the mandatory requirement to verify whether light is on every unplugged fiber.</li> <li>Finish the migration of network upgrade workflows to a fully automatically sequenced orchestration system. Most common regional network workflows were completed in 2025, and zonal workflows are in progress.</li> <li>Complete the deployment of work stop alerting for actions resulting in the unintended disconnection of live fiber cables in Google data centers.</li> <li>Complete the deployment of the automated system that will move regional traffic away from faulty zones, reducing the time from approximately 19 minutes in this instance to around 5 minutes.</li> </ul> <h2>Detailed Description of Impact</h2> <p>On Tuesday, 1 September, from 07:41 to 11:52 US/Pacific, a portion of the us-central1-b and us-central1-f zones experienced severe network degradation and resource isolation.</p> <ol> <li>Multiple products affected: The network disruption to/from a portion of us-central1-b and us-central1-f zones impacted all zonal GCP Products for some customers in those zones.</li> <li>Regional impact: Regional products using capacity in the affected data center were affected until traffic diversions were fully in place at 08:00 US/Pacific. Google Kubernetes Engine and Cloud Run / Google App Engine had longer impacts, discussed below.</li> <li>Error rate: Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss.</li> </ol> <p><strong>Affected Services and Features</strong></p> <ul> <li>Google Compute Engine: Inability for users in the affected portion of us-central1-b and us-central1-f zones to access virtual machines externally, and inability for VMs to reach remote resources.</li> <li>Cloud SQL, AlloyDB for PostgreSQL, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure.</li> <li>Cloud Spanner, Cloud Firestore: Elevated Remote Procedure Call (RPC) error rates, write timeouts, and latency spikes for the nam5 multi-region instances due to severed connectivity to components running in the impacted zone.</li> <li>Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone.</li> <li>Google Kubernetes Engine (GKE): Between 07:40 and 09:15 US/Pacific on 2026-09-01, customers experienced errors and timeouts on requests to their GKE cluster control planes (Kubernetes API servers) in us-central1. This could have caused new or updated workloads to fail scheduling and Kubernetes API operations to fail.</li> <li>Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific.</li> </ul> <p><strong>Customer Impact</strong></p> <p>Customers with resources hosted in the specific affected clusters within us-central1-b and us-central1-f experienced a complete loss of network connectivity starting at 07:41 US/Pacific. During this time, virtual machines and associated services became unreachable from the internet and from other Google Cloud regions, and those internal resources could not initiate outbound connections.</p> <p>The issue was detected almost immediately and engineers took traffic diversion actions at 07:45 and 08:00 US/Pacific to reduce impact to regional products. By 08:50 US/Pacific, the majority of physical connections were restored and traffic began recovering, and the traffic diversion actions were removed at 09:19 US/Pacific. Full recovery across nearly all affected services and long-running operations was confirmed by 11:52 US/Pacific.</p> <p>Because the network isolation was strictly limited to specific clusters within us-central1-b and us-central1-f, multi-zonal deployments correctly utilizing redundancy across other zones in us-central1 were largely able to bypass the physical hardware failure and continue serving traffic. Customers with projects in both us-central1-b and us-central1-f would not have seen multi-zonal impact - at most one of their zones would have capacity in the affected data center.</p> </div><hr><p>Affected products: AlloyDB for PostgreSQL, Apigee, Cloud Filestore, Cloud Run, Cloud Spanner, Google App Engine, Google BigQuery, Google Cloud Bigtable, Google Cloud Dataflow, Google Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Hybrid Connectivity, Looker (Google Cloud core), Virtual Private Cloud (VPC)</p><p>Affected locations: Iowa (us-central1)</p>