8:19am Our monitoring systems detected a service interruption in the TELUS API affecting a number of TELUS EMRs (both MedAccess and PSS) that started Saturday evening around 8:00pm.
This means that Ocean cannot connect to TELUS EMRs via API integrations at this time. API calls like patient check-ins, note uploads, schedule syncs, etc are affected and currently failing. We are escalating this issue through our support channels with Telus and continue to investigate its impact to Ocean.
We will post updates here as we receive new information from TELUS.
Note: During this outage, email relating to Ocean CloudConnect's Failure to sync appointments can safely be ignored.
11:20am Telus has notified us they discovered an issue in their API management layer that was blocking Ocean requests from completing successfully. They are working to mitigate the issue.
11:45am Telus has provided us an update to say that API integrations are operating normally again. We'll continue to actively monitor the health of our integrations throughout the day.
Please let us know if you experience any further issues with Ocean.
2020-10-07: We received root Root Cause Analysis from TELUS. The issue was related to a deadlock in a TELUS replication database resulting from network latency between two data centres on Friday Sept 19th in the evening. The database eventually failed over the weekend, resulting in an API outage that affected Ocean connectivity for all clients.
The remediation action was restarting the database. Long term resolution is a software fix from an independent software supplier responsible for the replication database software.
As for the extended downtime, a further error was found in the monitoring configuration that resulted in the failure not being detected and resolved until Monday when reported by CognisantMD. TELUS reports that the monitoring configuration has been fixed.