TELUS API Service Interruption November 20, 2019 [RESOLVED]

Our monitoring detected a service interruption in the TELUS API affecting a small number of TELUS EMRs  that started last night around 10:53pm. We are currently investigating.

This means that Ocean cannot connect to TELUS EMRs via the API, so patient check-ins and note uploads are failing. We have reported the issue to TELUS.

For affected clinics, while the API is offline you may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

We will post updates as we receive them from TELUS.

---

9:25 AM ET: TELUS found the issue and fixed it at around 8:55. The API is behaving normally again. The root cause appears to be a bug in TELUS API deployment last night that affected EMR Mobile authenticated clients both for the API and the TELUS mobile app. Most Ocean TELUS clients were not affected because they've been migrated to the AML authentication framework. We will update this ticket with any info we receive from TELUS.


TELUS API Service Interruption Affecting Appointment Syncs Oct 1, 2019 [RESOLVED]

According to our monitoring, the TELUS API stopped responding to appointment sync requests at approximately 3 AM ET last night, with all requests for appointment syncs resulting in general system errors from TELUS. This means that Ocean cannot synchronize appointments with TELUS EMRs via the API. However, patients already in the schedule for today may be marked arrived, notes added etc. as normal. Ocean will not "see" appointment changes but will work from the last known version of the appointment book for a clinic.

PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload patient and appointment information manually; Med Access clinics will not be able to check-in new or revised appointments. Other functionality within the Ocean suite may be affected as well.

TELUS has identified the problem and is working to resolve. More updates will be posted as they become available.

2019-10-01 10:20 AM ET: Although we have not yet heard confirmation from TELUS, the errors stopped at around 10 am ET and so we will mark this as resolved. All appointment delta since the last sync at 3 AM should arrive in Ocean within the next 30 minutes. As usual, we will attach any root-cause-analysis that we receive from TELUS to this ticket.


TELUS API Service Interruption September 16, 2019 [RESOLVED]

According to our monitoring, the TELUS API stopped responding at approximately 3:15 ET this afternoon, with all connection requests timing out. This means that Ocean cannot connect to TELUS EMRs via the API, so patient check-ins and note uploads are failing. We have reported the issue to TELUS.

While the API is offline, PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

We will post updates as we receive them from TELUS.

---

4 ET: The API has started to respond again as of 3:55, and we see the backlog of EMR Actions (requests from Ocean to TELUS API) clearing now. New requests from Ocean (e.g. check-ins, etc.) are succeeding, and the backlog should be clear within an hour. We will pass on any root cause analysis that we receive from TELUS.


TELUS API Service Interruption August 30, 2019 [RESOLVED]

TELUS is reporting an unexpected service interruption in the TELUS API affecting all TELUS EMRs since around 6:50 AM EST this morning. This means that Ocean cannot connect to TELUS EMRs via the API. TELUS is investigating and we will pass along any updates we receive from them.

While the API is offline, PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

More information to come.

---

Update 9:50 AM EST: TELUS has marked the issue resolved. We see successful API responses starting around 9:45. We will attach any root cause analysis that we receive from TELUS.


TELUS PS Suite Ocean Note Overwrite Issue

We are working with TELUS to investigate an issue that appears to occasionally cause an Ocean note to overwrite another Ocean note in the PS Suite patient chart when multiple forms are completed in the same Ocean session (e.g. a patient using a tablet to complete multiple forms). The issue can cause one of the new notes to disappear from the chart. It remains visible in the PS Suite "note change history" for the patient although it appears crossed out as if it had been replaced by a subsequent note. 

The issue has been reported by three independent clinics running PS Suite versions 5.13.406 and 5.13.407 in the past two days so we are treating it as a high severity issue even though it's not fully understood nor has it been reproduced in internal TELUS or CognisantMD test instances. The issue appears to be limited to notes added via the TELUS API (e.g. via Cloud Connect) as opposed to the legacy Ocean custom form. We do not expect any impact on eReferral functionality.

To work around the issue, log into Cloud Connect, click "Edit Settings" and enable "Combine Progress Notes". That will cause a single merged note to be added even if multiple forms were completed in the same session.

We are awaiting analysis from TELUS that will inform next steps. In the meantime, if you have encountered this issue, please apply the workaround described above, and let us know at support@cognisantmd.com.

2019-06-10 5PM EST: According to the latest info from TELUS, the issue is associated with the receipt of multiple notes within a single second for a single patient. Ocean notes are often added in rapid succession upon patient session completion when "combine progress notes" is disabled, often within the one-second threshold and therefore a subset of the notes may not render in the UI due to this bug, which has not yet been traced to a specific PSS version. The workaround "combine progress notes" documented above would only protect new Ocean notes from being affected, but it would not help with previous notes -- a fix from TELUS will be required to make such notes visible again. In order to mitigate the issue based on this information from TELUS, we are planning to patch the behaviour of Ocean Cloud Connect tonight (June 10) so that all new Ocean notes in a single patient session are combined for a given patient in PSS. In other words, we will deploy the workaround for all PSS clients. We will update this ticket when such action is taken.

2019-06-11 4PM EST: The "combine note" workaround patch was deployed yesterday evening and no issues have been reported so far today with the updated behaviour, which would have affected approximately two-thirds of our PSS client base. We plan to revert to previous behaviour (i.e. to respect the setting in the Cloud Connect configuration) once TELUS' fix is in place.


TELUS PSS Layout Crash Issue

A number of Ocean sites have reported an issue in which very recent versions of PS Suite crash on some charts with a "java.lang.StackOverflowError" layout error.

Our theory is that dynamic text in a toolbar triggers a new bug in the PSS layout engine. When the Ocean Toolbar includes a display of pending forms for a patient, which gets longer as forms are added to the patient’s form queue, this can potentially cause this layout error to occur.

The issue is not specific to the Ocean Toolbar (and you may encounter it with other dynamic toolbars) but the Ocean Toolbar seems to trigger it in some instances. TELUS is aware of the issue and we have provided diagnostic information to facilitate a fix in an upcoming version of PS Suite.

In the interim, we have created a special version of the Ocean Toolbar that omits the "pending forms" field to attempt to workaround the bug.

If you are experiencing this issue, please email support@cognisantmd.com if you haven't already so that we can track affected sites, and then try upgrading your Toolbar.cfm to v5.22 or newer.

 


Telus API Issue Impacting OceanConnect Note Uploads for PS Suite [RESOLVED]

Our monitoring system has been receiving error reports this morning indicating that note uploads from OceanConnect are failing in the Telus API.  OceanConnect versions prior to 58 are affected.  After investigation, it appears that a recent change to the Telus API has broken older OceanConnect versions.

To correct the issue, tablets running OceanConnect versions prior to 58 can be updated to the latest version of OceanConnect through Google Play or by direct download. Alternatively, clients can upgrade to Ocean Cloud Connect and disable OceanConnect.  See Upgrading OceanConnect on Tablets for information on how to update to the latest version.  Or see Setting Up Ocean Cloud Connect for information on how to move to Cloud Connect. 

We are working with Telus support to determine the root cause and will update this ticket as new information comes available.

----

Dec 13, 2018: Telus has released a patch to their API server which has rectified the issue.

----

Jan 10, 2019, 4pm: We have seen a recurrence of this issue today. After discussion with Telus support, it appears that it has reappeared "because the API Services 2.37.01 deployed Jan 9th did not to have the emergency change that removed this error in the previous services release."  Telus is planning a patch release to rectify the issue.

----

Jan 11, 2019, 2pm: Telus patched the API servers last night at approximately 7:30pm ET.  The problem is resolved.


Telus API Issue Impacting Ocean Demographics Updates for PS Suite [RESOLVED in PSS 5.12.407]

We have been receiving error reports this morning from our monitoring system indicating that patient demographic field updates from Ocean forms are failing in the Telus API due to an internal system error from the TELUS API.

It's not year clear whether the issue is related to a new PSS version or a change to the API service itself. Note uploads are unaffected, but will include text indicating the demographics changes that were attempted but failed.

As an interim workaround, these values can be manually updated by copying and pasting into the appropriate fields. We're working with Telus support to determine the cause and will update this ticket as new information comes available.

Update 2018-12-19: We have confirmed with TELUS that the issue has been fixed in PSS version 5.12.407. If you are experiencing this issue, please request this patch from TELUS.


TELUS API outage impacting All TELUS Clients [RESOLVED]

The TELUS API server has been unresponsive since 8:25 EST this morning. We've seen connection timeouts (generally indicative of an offline system) returned to all TELUS/Ocean clients, including those on PS Suite and Med Access.

We are not aware of any scheduled or unscheduled downtime planned by TELUS and have escalated the issue to TELUS support for more information and requesting an ETA on the fix. We will update this ticket with more information when we get it.

While the API is offline, PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

More information to come.

--- 

10:00 PM: The TELUS API started responding again at around 9:30 PM. It appears to be responding normally since then. We will continue to monitor. As part of our investigation, we discovered that the TELUS EMR Mobile App was also affected. We'll post any root cause analysis that we receive from TELUS.


TELUS API issue impacting Cloud Connect and OceanConnect users [RESOLVED]

We have been notified that there is an issue with the TELUS National API, affecting Ocean clients using Cloud Connect (or OceanConnect). We are investigating the issue, and we will update this article as we receive more information from TELUS.

Update 12:30 PM EST: the issue appears to have been resolved (we are no longer seeing the same frequent errors from the TELUS API) although we are awaiting an explanation from TELUS. All EMR actions have been processed successfully now and the integration is operating normally again.


Telus API Issue impacting OceanConnect Users

We have been notified by Telus that there is an issue with the PSS API affecting Ocean clients using OceanConnect.  The impact seems to be limited to marking patients arrived, but we are still investigating.  We will update this article as we receive more information from Telus.

---

Update 1:30PM: TELUS has confirmed an issue with the stability of the API connections to PSS instances. They are investigating.

--- 

Update 1:40PM: TELUS says the API connections have stabilized and the issue should be resolved. We will continue to monitor.

---

Update May 30, 12:10 ET: The issue is recurring intermittently today.  Telus is aware of the issue.

---

Update June 1, 10:10 ET: The issue went away for a day or so but has returned and a number of clinics are experiencing the PSS arrival arrow issue again this morning. We are seeing the same error message from the TELUS API from those PSS instances and have escalated to TELUS again. TELUS confirmed that they are actively working on the issue. No root cause available from TELUS yet, other than general API instability.

---

Update June 1, 3:20 ET: TELUS says the API tunnel is stable again and things should be running normally. We will follow up with TELUS to try to understand the root cause of the API instability.

---

Update June 7, 10:30 AM ET: We are seeing the TELUS API failing across multiple sites again in the same manner as previously documented in this ticket. We are escalating to TELUS again. TELUS has acknowledge continuing issues with their API and are investigating. We will update this ticket as information becomes available.

--- 

Update June 8, 11:30 AM ET: It appears that the TELUS API has been stable since approximately 4:30 PM ET yesterday.

---

Update June 21, 10:50 AM ET: We are seeing the TELUS API failing across multiple sites again in the same manner as previously documented in this ticket. We have escalated to TELUS again.  We will update this ticket as information becomes available.

--- 

Update June 21, 10:00 PM ET: It appears that the TELUS API has stabilized.  TELUS is accelerating technical work to reduce the likelihood that the issue will recur.

--- 

Update June 22, 10:50 AM ET: TELUS says they successfully migrated their API endpoint to a new server last night to address the persistent instability.

 


TELUS API Connectivity Issues April 9, 2018

We are seeing errors returned from the TELUS API this morning. TELUS is investigating the issue. TELUS clients that have connected an OceanConnect device to the TELUS API may experience the following:

  • Patients not loaded for kiosk use
  • Patients not marked as arrived
  • Generated Ocean notes not being added to chart
  • Ocean Form Reminders delayed

As a temporary workaround, clinics can download clinical notes as required using the "download" link in the toolbar.

Clinics that are not yet connected to the TELUS API will not be affected.

We will update this ticket as more information becomes available from TELUS.

----

Update 2:55PM from TELUS: "Please note that we continue to experience an issue with the SSH Tunnel service for PS Suite clients. This issue is intermittent and is causing the SSH tunnel connection to drop. This results in a “nTELUS EMR Mobile cannot connect to your EMR Server. Please see the following link for more information.” error message from the API. TELUS has a team currently troubleshooting and have made it a high priority."

----

Update 5:30PM: TELUS believes they have isolated the issue to a change in the latest PSS version, and a ripple effect of a handful of clinics upgrading Sunday morning that resulted in Ocean connectivity disruptions for other clinics (that were not upgraded). They are going to disconnect the upgraded clinics from the API tonight, apply a PSS patch and monitor the API tomorrow. TELUS is contacting clinics directly that need to be disconnected, so if you haven't heard from them, there is no action required.

----

Update April 10, 2018 9:45AM: Although we do not yet have confirmation from TELUS, it appears that the maintenance work done by TELUS last night has helped at least some clinics that have reported success with kiosks this morning. Three clinics were disconnected from the API by TELUS; we assume that TELUS has informed those clinics but will be reaching out to confirm. If you are not one of the "disconnected" clinics, please try your kiosks this morning and contact us if you have any issues.

----

Update April 11, 2018 9:45AM: TELUS has confirmed that the issue was related to a PSS upgrade and patches were applied to affected clinics last night to address the issue. All clinics, including those disconnected from the API yesterday, have been reconnected by TELUS and your Ocean Kiosks, Ocean Online and other Ocean technology that interfaces with a TELUS EMR should be back to normal. Let us know if otherwise.

 

 


Ocean Compatibility for PS Suite 5.4

We are aware of a few sites that are running PS Suite 5.4 and having issues with PS Suite and Ocean, particularly with concurrent access to charts in PS Suite. We are working with TELUS on updating the custom form for 5.4 and have released a few custom form iterations in an attempt to address the issues.

If you are running Ocean and PS Suite without issue, we recommend that you do not upgrade the Ocean custom form until both 5.4 and the Ocean custom form are more stable.

On the other hand, if you are experiencing issues downloading notes in 5.4 (freezing, checkout errors, etc.) OR have already upgraded to Ocean custom form 2.84 or higher, please contact us at ocean.tips/support

Update 2015-12-08: We continue to work on the issue and have built a version 3.0 of the custom form for selective distribution that addresses some of the issues and adds some error handling logic. Please contact us if you would like it; it will be made publicly available at the standard download location later this week.

Update 2015-12-11: We have now released version 3.0.1 to the public download site. If you are running PSS 5.3 or 5.4, you should upgrade to this version as soon as possible. If you are running PSS 5.2, you should hold off upgrading the Ocean custom form for the time being.