On April 30 10:00am UTC, our production infrastructure faced a critical failure due to unhandled rate limiting in our scraping systems. This resulted in all API endpoints returning HTTP 500 errors with no data being served.

For two full days (April 30 to May 2), our API was completely unavailable.

On May 2, we deployed a degraded mode: only the most valuable portion of the data (approximately one-third of the usual results) was returned through fallback logic. While this helped partially restore service, the output remained incomplete until full resolution.

On May 6 at 10:00 UTC, we fully resolved the issue. The API is now back to 100% functionality, returning complete and stable data across all endpoints.

The core issue was an insufficient handling of rate limiting across our scraping infrastructure. As limits were hit, key pipelines silently failed without triggering alerts, cascading into a complete halt in data enrichment.

To fix this we rewrote critical parts of our scraping scripts to handle new rate limiting. Increased our scraping capacity to reduce the risk of overload and added robust monitoring and alerting to detect scraping slowdowns and failures early. We also Improved our fallback system to ensure partial data delivery even during issues from now.

To prevent similar incidents, we’ve put in place:

1. Proactive monitoring on data completion rates and proxy health.
2. Faster internal escalation procedures when API performance degrades.
3. Load testing protocols to anticipate and mitigate proxy failures.
4. Automated switch to degraded mode with transparent communication.

We understand the importance of reliability in your workflows and deeply regret the impact this may have had on your operations. We’re fully committed to making our API more resilient, better monitored, and more transparent.