On the 10. December 2019 at around 11:30, Pronestor was notified of site outages on one of the servers that host Pronestor Planner. This affected 20% of our customers.
Pronestor investigates the issue and discovers an issue with multiple sites, regarding some corrupted files on the web server (customer data is not affected by this).
Pronestor restarts the server to fix the issue. The issue persists.
At 13:30 Pronestor begins server backup restore procedure, to restore the server to an earlier, functioning state. Restoring the server appears to fix the issue for most sites.
Pronestor changes some configurations of several sites for them to be operational as well.
Pronestor begins process for migrating sites away from the faulty server.
Pronestor will continue to migrate sites away from the faulty server to ensure stability. This will have a minor impact on the up time of the sites that are being moved.
Pronestor is currently in the process of streamlining the backup and restore process to be faster and more reliable. Lessons learned from this incident will be incorporated in the process.