Resolved -
This incident has been resolved.
Jun 29, 15:56 UTC
Update -
Konnect Middle East Region: Indefinite Outage
The Konnect Middle East Region is undergoing a prolonged outage due to the operational loss of AWS datacenters in the region. The duration of this outage is undetermined. As a result, we have no timeline for service restoration in the ME Region
Customers are encouraged to migrate to one of our other Konnect control plane hosting regions in the EU, Australia, India or the United States. If assistance is required, please reach out to your Kong support representative.
Apr 2, 19:48 UTC
Update -
We are observing increased error rates across services in the Konnect Middle East region. Data plane proxy traffic is not impacted.
We are actively working with AWS to resolve the issue and will provide updates as more information becomes available.
Mar 23, 23:19 UTC
Update -
Customers in the Middle East (Bahrain) region are operating in a reduced Availability Zone (AZ) configuration due to an ongoing AWS infrastructure issue.
The platform remains available, but some customers in this region may experience intermittent performance degradation or elevated error rates. Data plane proxy traffic is not impacted.
We are actively working with AWS to resolve the issue and will provide updates as more information becomes available.
Mar 20, 21:12 UTC
Update -
It was discovered that DNS for KIC reconciliation endpoints were still pointing to the out-of-service region. We have failed over these DNS records now to properly point to our new region. Customers using KIC would have seen 500s' during the reconciliation to Konnect. This was resolved at 10:30 UTC on March 5th.
Mar 5, 13:16 UTC
Update -
Current Status Summary for Konnect Middle East Region:
All services have been moved to our failover region in Middle East
Almost all services are operational, with 2 caveats:
- Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service.
- The Debugger is operational including trace data; however session logs will not be available in this region.
- Observability is operational; however UUID to display name resolution will not be available in this region.
Everything else is back online. The team will update as we make progress on these remaining items.
Mar 3, 08:53 UTC
Update -
Current Status Summary for Konnect Middle East Region:
All services have been moved to our failover region in Middle East
Almost all services are operational, with 2 caveats:
- Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service.
- Debugger is unable to accept incoming trace data. While debugging sessions can be started, none of the trace data will be persisted in Konnect until the datastore for traces is restored.
Everything else is back online. The team will update as we make progress on these remaining items.
Mar 3, 05:35 UTC
Update -
The data store for traces sent to Debugger is still experiencing issues in the Middle East region. Debugging sessions can be started but no spans will be able to be ingested. The team will update as they bring that service online.
Mar 3, 05:13 UTC
Update -
Service remains restored, and Kong engineers will continue to monitor the apps in the new region.
Mar 3, 03:45 UTC
Update -
User permission data is now propagating normally again.
Mar 3, 03:03 UTC
Update -
Status Update (PDP – ME-SOUTH-1)
User permission data replication in ME-SOUTH-1 (Bahrain) region are experiencing extended latencies. Mitigation has been implemented to cap this to 15 minutes.
Updated data should now propagate within a maximum of 15 minutes.
We continue to monitor the situation and will provide further updates as needed.
Mar 2, 22:45 UTC
Update -
The following services have been restored:
- All Konnect UI-accessible products except for Event Gateway and Metering
- Dataplane connections to the Konnect Control Plane for configuration download and analytics upload
Analytics data is delayed several hours. Konnect is accepting and persisting Analytics data, but it is not yet visible or queryable as we continue investigating restoring service.
Mar 2, 21:50 UTC
Monitoring -
The Kong engineering team is now verifying restored services in the Middle East region
Mar 2, 21:39 UTC
Update -
We have restored partial service to Konnect. The UI is working again for Gateway Manager and Dev Portal, as well as all backing APIs.
Dataplane connections to the Control Plane and Telemetry endpoints have been restored for 3 hours and should continue to work.
Mar 2, 21:35 UTC
Update -
We continue to make progress on restoring to a failover region for the Middle East region. Key AWS services to support our failover efforts are beginning to work as of 11am PST. The Konnect engineering team is working to restore and validate all services and data and will update in one hour.
Please note the control plane and telemetry endpoints are already restored, so dataplanes can connect to Konnect.
Mar 2, 20:05 UTC
Update -
Control Plane restored for Config Download and Telemetry
We have restored the control plane and telemetry endpoints. Dataplanes can now connect to the control plane api at `me.cp0.konghq.com` and `me.tp0.konghq.com`
This is a read-only instance set up for this incident. Configuration changes are still blocked. But now customers should be able to connect new dataplanes and download their configuration, as well as send telemetry.
The Admin API for changing control plane configuration remains down.
Mar 2, 17:36 UTC
Update -
We are actively migrating the affected services to operate on other Availability Zones in the Bahrain region (ME-SOUTH-1 Region), but critical AWS services are not responding in that region. Our engineers are working to restore full service functionality and we are communicating with AWS to attempt to determine our failover options. At this time we cannot provide an ETA.
We will provide further updates as they become available. Thank you for your patience.
Mar 2, 17:26 UTC
Update -
We are actively migrating the affected services to operate on the two healthy Availability Zones in the Bahrain region (ME-SOUTH-1 Region). Our engineers are working to restore full service functionality and expect the migration to be completed within the next few hours.
We will provide further updates as they become available. Thank you for your patience.
Mar 2, 15:26 UTC
Update -
The incident is ongoing. We are working closely with our cloud provider to monitor restoration efforts.
Due to significant physical infrastructure damage in both the UAE and Bahrain regions, recovery is expected to take more time.
We appreciate your patience as we work toward a full resolution.
Mar 2, 11:27 UTC
Update -
We are currently observing renewed elevated error rates across multiple Konnect services in ME region related to the AWS outage in that region.
Mar 2, 06:31 UTC
Identified -
We are seeing many services in ME region recovering and some services are in degraded state. Dataplanes are not affected as part of this incident. All other Konnect regions are 100% operational and healthy.
Our engineers are closely monitoring the situation. We will provide additional updates as soon as more information becomes available.
Mar 2, 01:54 UTC
Investigating -
We are currently observing renewed elevated 5XX error rates across multiple Konnect services in ME region. Our engineering team is actively investigating the issue and assessing customer impact.
We will provide additional updates as soon as more information becomes available.
Mar 1, 20:07 UTC
Monitoring -
AWS continues to experience issues some services. However most of Konnect services are now recovered and operating as expected. Our engineers are closely monitoring the services and it's operational efficiency.
Mar 1, 18:41 UTC
Identified -
The increased error rates are associated with the AWS me-central-1 outage. Our engineers are monitoring the situation closely.
Mar 1, 14:45 UTC
Investigating -
We are seeing increased 5xx in the ME Region. Our engineers are currently investigating the issue.
Mar 1, 13:08 UTC