This incident is fully resolved at this point, and the metrics API is functioning normally.
Posted Jul 08, 2023 - 21:52 UTC
Update
Confluent engineering is continuing to monitor the mitigation in place. The vast majority of metrics requests should complete successfully at this point, although a small percentage of requests may continue to return incomplete data. Full mitigation is expected some time in the next 4 hours. We will post an update here again by 00:00 UTC.
Posted Jul 08, 2023 - 18:54 UTC
Update
The mitigation is in place. Metrics API is functional and we have expedited the recovery. We will continue to monitor further.
Posted Jul 08, 2023 - 16:51 UTC
Update
We are continuing to monitor the affected services for any further issues.
Posted Jul 08, 2023 - 14:30 UTC
Update
The mitigation in place has been working well. Metrics API is functional and will take about 4 hours to fully recover. We will continue monitoring the affected services and clusters.
Posted Jul 08, 2023 - 09:11 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 08, 2023 - 09:10 UTC
Monitoring
The mitigation in place has been working well. Health+ and KSQL are functioning normally. Metrics API is functional and will take about 10 hours to fully recover. We will continue monitoring the affected services and clusters.
Posted Jul 08, 2023 - 04:43 UTC
Identified
The issue has been identified and a mitigation is in place. We will provide an ETA on when the affected services are back to normal.
Posted Jul 08, 2023 - 01:35 UTC
Update
We are still actively investigating this issue
Posted Jul 08, 2023 - 00:40 UTC
Update
We are continuing to investigate this issue.
Posted Jul 07, 2023 - 21:27 UTC
Update
Confluent Cloud is experiencing higher than expected latency and error rates as of 1830 UTC. It is affecting some Confluent Cloud Metrics. Queries for recent metrics may return incomplete or no data. Health+ customers are unable to submit telemetry to Confluent Cloud and may observe errors for submission in logs. Additionally, some customers may observe KSQL problems due to authentication failures. We are continuing to investigate the issue.
Posted Jul 07, 2023 - 21:20 UTC
Investigating
Confluent Cloud Metrics are experiencing higher than expected latency and error rates as of 1830 UTC for some Confluent Cloud Metrics. Queries for recent metrics may return incomplete or no data. Additionally, Health+ customers are unable to submit telemetry to Confluent Cloud and may observe errors for submission in logs. We are investigating the issue.