Between 08:00 AM and 4:00 PM UTC on June 3, 2025, all API endpoints experienced temporary failures, returning either 204 or 500 errors. The issue was caused by an incorrect deployment to our server infrastructure. This push introduced changes that could not be rolled back cleanly, requiring us to manually delete and recreate all production instances.

Importantly, this incident was not related to any external data sources — the root cause was strictly internal and due to an infrastructure misconfiguration on our side.

We attempted multiple rollbacks which failed then we manually deleted and recreated our production instances to restore service.

Full service was restored by 04:00 PM UTC.

A faulty push passed our staging environment, which did not accurately reflect our production scale and traffic. The resulting discrepancies caused critical issues once deployed live.

To prevent this from happening again, we are:

• Improving the parity between our staging and production environments.

• Allocating more resources to staging to better simulate real-world load and performance.