Previous incidents

We identified an issue beginning at 4:15 PM CET where a limited number of profiles were being returned with empty data fields. We implemented a workaround at 5:35 PM CET that successfully resolved the issue. All profiles are now returning complete data as expected. We apologize for any inconvenience this may have caused.

All services have been fully restored and are operating normally. Thank you for your patience.

We have resolved the creationDate field inconsistencies. We are continuing to investigate and stabilize other data fields.

Yesterday, 4:30 PM and 7:30 PM UTC

We experienced performance degradation on our ScrapIn API during two separate windows last evening. Our team responded quickly to mitigate the impact and stabilize the service.

What happened: Increased query load resulted in elevated response times and brief periods of service instability across certain endpoints.

Our response: We implemented immediate optimizations and load balancing adjustments to restore normal operation.

Current status: Performance h...

After experiencing multiple incidents throughout the day, we have made the decision to temporarily disable activity and job endpoints to focus on comprehensive system stabilization. Profile and company endpoints will continue to operate using cached data only during this period.

We recognize that applying quick fixes to isolated issues is not sustainable. Instead, we are taking the necessary time to implement robust, long-term solutions to prevent future disruptions. This maintenance period ...

We experienced intermittent instability between 1:00 AM and 3:30 AM CET. Our fallback systems were automatically activated to maintain service availability. All services are now stable and operating normally.

The issue has been resolved. Profile endpoints are now returning positions, schools, and skills data as expected. We continue to monitor the situation closely over the next few hours.

We experienced multiple cascading issues over the past 24 hours. Here's the timeline:

Yesterday 3:00 PM - 6:00 PM CET: Our primary data source experienced an outage. Our automatic failover to a secondary source activated, but we quickly reached rate limits on that backup system.

Around 7:30 PM CET: We encountered elevated 429/424 rate limit errors from the secondary source. Our team manually switched back to the primary source after confirming its status had improved.

Around 10:00 PM CET: ...

We're investigating high database load following an internal migration. Our team is actively implementing read replicas and performance optimizations. We're monitoring closely and will update you as the situation evolves.

We've identified and resolved the intermittent request failures that occurred earlier. The issue was caused by elevated traffic spikes overwhelming our reverse proxy, which temporarily prevented some requests from reaching our backend servers. We've implemented improved rate limiting and traffic management on our proxy layer to prevent this from occurring in the future. All systems are now operating normally.

We just published a Post-mortem Post here :
https://www.notion.so/visum-internal/API-Service-Disruption-October-27-2025-29aae6c4117d80f3bbe8d138c2921260

Between 2:00 PM and 3:00 PM UTC, we detected instability in our production environment.
To prevent inconsistent results on persons endpoints, we temporarily placed the API in maintenance mode for one hour.

The production environment is now fully stabilized, and our team continues to closely monitor the situation.

We experienced a production outage caused by an issue with one of our hosting provider, impacting all services from 08:00 UTC to 09:10 UTC.

Our team immediately contacted the provider, restarted all services, and redeployed the full production environment. Once the redeployment was completed, all systems returned to normal and have been fully stable since.

Between 16:57 and 17:31 CEST on 2025-10-03, our services were down because of a migration-relation job which crashed our whole production. Roll-back successful within 30min and fully live again.