April 24, 2024, 10:20 AM EST: We are aware of the service interruption this morning that affected access to the Ocean Portal for users. The service interruption has since been resolved, and access to the Ocean Portal has been restored for all users.
Initial investigation of the interruption indicates a database connection issue. We have since made changes to resolve the issue, and will continue to test and investigate the root cause.
April 24, 2024, 4:35 PM EST: As part of our investigation, this was the sequence of events during this morning's service interruption:
- 9:22 AM EST: Ocean monitoring began detecting database connection failures. Ocean was up, but some requests would have resulted in errors.
- 9:28 AM EST: The Ocean Operations Team became aware of the error reports and began investigating
- 9:49 AM EST: Ocean monitoring began detecting much larger error volume as normal daily site traffic increased. Many/most requests would have resulted in errors at this time. The Operations Team member on call appropriately escalated the issue, but had already identified the issue and developed an action plan.
- 9:50 AM EST: Ocean monitoring received Ocean Platform down alerts.
- 9:59 AM EST: The Ocean Operations Team successfully deployed a script to correct the issue. The issue was resolved at this time.
- 10:00 AM EST: The Ocean Operations Team continued to monitor and test the system to ensure the fix was fully successful. A postmortem investigation plan was developed, which is now ongoing.
Please email us at support@oceanmd.com if you have any additional questions.