We are currently experiencing a system interruption on the Ocean application due to excessive system load. We are investigating and will update this ticket shortly as more information is available. We apologize for the inconvenience.
UPDATE 5:24PM: the Ocean system is back online after a system restart. Total outage was six minutes. We will be investigating the root cause based on some captured diagnostic information and will update this ticket.
ROOT CAUSE ANALYSIS 2017-03-02 10PM: After analysis, it appears that the initial source of the excessive load was a series of directory searches that exposed a poorly indexed path. Compounding the problem was some system instrumentation designed to log "performance warnings"; due to the volume of requests that triggered this logging, it caused a knock on effect, forcing the hard restart by CognisantMD operations staff.
We are addressing the issue in three ways:
- We have put in place a measure through a patch release tonight to help protect against such events in the future
- We have some index changes planned for the next major release of the Ocean server
- We are adding protective code to reduce redundant performance warning logging in the event of a system wide slowdown