TELUS API Maintenance November 1st 2020 Affecting Most Ocean Modules

TELUS Notification:

Please be advised of the following scheduled EMR API maintenance  taking place Sunday November 1st, 2020.  The EMR service will be unavailable during this time.

 

Date: Nov 1st, 2020 12am to 2pm EST 

Impact: Service will be unavailable

 

Ocean Impact:

We don't expect that the TELUS API will be unavailable throughout the maintenance window above, but the following services will be delayed/unavailable during the period(s) that it is down:

  • Patient reminders scheduled during this time period will be sent once the TELUS maintenance completes 
  • Online Booking will be unavailable (i.e. setting up schedules and booking appointments)
  • EMR synchronization will be unavailable
  • Some Kiosk functionality will be unavailable
  • Patient authenticated website forms will be unavailable (for patients to submit a form)
  • Health Report Manager (HRM) messages will be unavailable 
  • Referral import will be unavailable

Use of some of these features will result in error messages to both patients and Ocean users as they attempt to communicate with the TELUS API. Normal functionality will return once the maintenance window completes.


TELUS API DST Glitch Affecting Clinics Open November 1st 2020

October 26 2020- Clinics open on Sunday, November 1st 2020 may be affected by a small glitch in the way Telus reports schedule availability to Ocean.  In particular, it will affect our clients using Ocean's online booking to book appointments for November 1st 2020.

Daylight savings time in Canada ends on November 1st at 2:00 AM. The glitch affects how Ocean receives the schedule availability- namely the start time is reported incorrectly to Ocean (off by 1 hour).

An hour of schedulable clinic time that isn't actually available for booking will be shown to patients as if it is. For example, if your clinic is open at 10am on November 1st 2020, Ocean's online booking will show 9am time slots available. Patients will receive an error when they attempt to book into these unavailable slots. Fortunately the correct booking time is otherwise preserved, e.g. a 2pm appointment will be booked on 2pm on this day.

image__1_.png

You can prevent patients from booking in the incorrect time slot by booking something into the slot like this:

Since the 1-hour affects both the start and the end time, unfortunately there isn't a simple workaround to open up the last available hour on that date. Clinics may have to manually book into those slots, or open them up for all Sundays and then manually block off the ones other than November 1st.

We have escalated this issue to TELUS for further investigation and we will update this article as we learn more. 


TELUS API Service Interruption September 21, 2020 [RESOLVED]

8:19am Our monitoring systems detected a service interruption in the TELUS API affecting a number of TELUS EMRs (both MedAccess and PSS) that started Saturday evening around 8:00pm.

This means that Ocean cannot connect to TELUS EMRs via API integrations at this time. API calls like patient check-ins, note uploads, schedule syncs, etc are affected and currently failing. We are escalating this issue through our support channels with Telus and continue to investigate its impact to Ocean.

We will post updates here as we receive new information from TELUS.

Note: During this outage, email relating to Ocean CloudConnect's Failure to sync appointments can safely be ignored.

--

11:20am Telus has notified us they discovered an issue in their API management layer that was blocking Ocean requests from completing successfully. They are working to mitigate the issue.

--

11:45am Telus has provided us an update to say that API integrations are operating normally again. We'll continue to actively monitor the health of our integrations throughout the day.

Please let us know if you experience any further issues with Ocean.

--

2020-10-07: We received root Root Cause Analysis from TELUS. The issue was related to a deadlock in a TELUS replication database resulting from network latency between two data centres on Friday Sept 19th in the evening. The database eventually failed over the weekend, resulting in an API outage that affected Ocean connectivity for all clients.

The remediation action was restarting the database. Long term resolution is a software fix from an independent software supplier responsible for the replication database software.

As for the extended downtime, a further error was found in the monitoring configuration that resulted in the failure not being detected and resolved until Monday when reported by CognisantMD. TELUS reports that the monitoring configuration has been fixed.


TELUS API Online Booking Service Interruption Jul 31, 2020 [RESOLVED]

Last night the CognisantMD team released a workaround for this issue, which appears to restore the primary provider restriction behaviour for clinics. As a result of this workaround, clinics may find that their online booking is working again.

TELUS continues to investigate the original cause of the problem that led to the primary provider not being returned in some circumstances from the TELUS API. However, the result of this investigation is unlikely to impact clients, now that the functionality has been restored.


TELUS API Online Booking Service Interruption Jul 30, 2020

We have been alerted by some clients using PS Suite that patients have been unable to book online as of about 4PM yesterday, July 29, 2020.

It appears that, when Ocean requests a patient's information for online booking, the TELUS API is no longer reporting the patient's primary provider to Ocean, making it appear that the patient does not have a primary provider at the clinic.

Online booking schedules are usually restricted to show only for patients with certain primary providers, so these schedules are not currently available to the patient as a result.

A temporary workaround that sites may consider involves removing the schedule's restriction for primary providers.

It is unclear whether this issue affects all PS Suite sites and Med Access sites as well.
We have escalated this issue to TELUS as an urgent support concern.


TELUS API Service Interruption March 6, 2020 [RESOLVED]

Our monitoring detected a service interruption in the TELUS API affecting a number of TELUS EMRs  that started this afternoon around 1:00pm. TELUS confirmed this outage at 2:45pm.

This means that Ocean cannot connect to TELUS EMRs via the API, so patient check-ins and note uploads are failing. Telus is aware of the issue and is currently investigating.

We will post updates as we receive them from TELUS.

Note: During this outage, email relating to Ocean CloudConnect's Failure to sync appointments can safely be ignored.

---

2020-03-07 12:49am: Notice received from TELUS HEALTH that the issue impacting the TELUS API is resolved.


TELUS API Service Interruption January 2, 2020 [RESOLVED]

Our monitoring detected a service interruption in the TELUS API affecting a number of TELUS EMRs  that started early this morning around 1:51am. We are currently investigating.

This means that Ocean cannot connect to TELUS EMRs via the API, so patient check-ins and note uploads are failing. We have reported the issue to TELUS.

For affected clinics, while the API is offline you may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

We will post updates as we receive them from TELUS.

...

9:33 AM ET: The API appears to be behaving normally again. Our team will continue to monitor throughout the day and we will provide an update on the root cause or any other information we receive from TELUS as it becomes available.


TELUS API Service Interruption November 20, 2019 [RESOLVED]

Our monitoring detected a service interruption in the TELUS API affecting a small number of TELUS EMRs  that started last night around 10:53pm. We are currently investigating.

This means that Ocean cannot connect to TELUS EMRs via the API, so patient check-ins and note uploads are failing. We have reported the issue to TELUS.

For affected clinics, while the API is offline you may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

We will post updates as we receive them from TELUS.

---

9:25 AM ET: TELUS found the issue and fixed it at around 8:55. The API is behaving normally again. The root cause appears to be a bug in TELUS API deployment last night that affected EMR Mobile authenticated clients both for the API and the TELUS mobile app. Most Ocean TELUS clients were not affected because they've been migrated to the AML authentication framework. We will update this ticket with any info we receive from TELUS.


TELUS API Service Interruption Affecting Appointment Syncs Oct 1, 2019 [RESOLVED]

According to our monitoring, the TELUS API stopped responding to appointment sync requests at approximately 3 AM ET last night, with all requests for appointment syncs resulting in general system errors from TELUS. This means that Ocean cannot synchronize appointments with TELUS EMRs via the API. However, patients already in the schedule for today may be marked arrived, notes added etc. as normal. Ocean will not "see" appointment changes but will work from the last known version of the appointment book for a clinic.

PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload patient and appointment information manually; Med Access clinics will not be able to check-in new or revised appointments. Other functionality within the Ocean suite may be affected as well.

TELUS has identified the problem and is working to resolve. More updates will be posted as they become available.

2019-10-01 10:20 AM ET: Although we have not yet heard confirmation from TELUS, the errors stopped at around 10 am ET and so we will mark this as resolved. All appointment delta since the last sync at 3 AM should arrive in Ocean within the next 30 minutes. As usual, we will attach any root-cause-analysis that we receive from TELUS to this ticket.


TELUS API Service Interruption September 16, 2019 [RESOLVED]

According to our monitoring, the TELUS API stopped responding at approximately 3:15 ET this afternoon, with all connection requests timing out. This means that Ocean cannot connect to TELUS EMRs via the API, so patient check-ins and note uploads are failing. We have reported the issue to TELUS.

While the API is offline, PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

We will post updates as we receive them from TELUS.

---

4 ET: The API has started to respond again as of 3:55, and we see the backlog of EMR Actions (requests from Ocean to TELUS API) clearing now. New requests from Ocean (e.g. check-ins, etc.) are succeeding, and the backlog should be clear within an hour. We will pass on any root cause analysis that we receive from TELUS.


TELUS API Service Interruption August 30, 2019 [RESOLVED]

TELUS is reporting an unexpected service interruption in the TELUS API affecting all TELUS EMRs since around 6:50 AM EST this morning. This means that Ocean cannot connect to TELUS EMRs via the API. TELUS is investigating and we will pass along any updates we receive from them.

While the API is offline, PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

More information to come.

---

Update 9:50 AM EST: TELUS has marked the issue resolved. We see successful API responses starting around 9:45. We will attach any root cause analysis that we receive from TELUS.


TELUS PS Suite Ocean Note Overwrite Issue

We are working with TELUS to investigate an issue that appears to occasionally cause an Ocean note to overwrite another Ocean note in the PS Suite patient chart when multiple forms are completed in the same Ocean session (e.g. a patient using a tablet to complete multiple forms). The issue can cause one of the new notes to disappear from the chart. It remains visible in the PS Suite "note change history" for the patient although it appears crossed out as if it had been replaced by a subsequent note. 

The issue has been reported by three independent clinics running PS Suite versions 5.13.406 and 5.13.407 in the past two days so we are treating it as a high severity issue even though it's not fully understood nor has it been reproduced in internal TELUS or CognisantMD test instances. The issue appears to be limited to notes added via the TELUS API (e.g. via Cloud Connect) as opposed to the legacy Ocean custom form. We do not expect any impact on eReferral functionality.

To work around the issue, log into Cloud Connect, click "Edit Settings" and enable "Combine Progress Notes". That will cause a single merged note to be added even if multiple forms were completed in the same session.

We are awaiting analysis from TELUS that will inform next steps. In the meantime, if you have encountered this issue, please apply the workaround described above, and let us know at support@cognisantmd.com.

2019-06-10 5PM EST: According to the latest info from TELUS, the issue is associated with the receipt of multiple notes within a single second for a single patient. Ocean notes are often added in rapid succession upon patient session completion when "combine progress notes" is disabled, often within the one-second threshold and therefore a subset of the notes may not render in the UI due to this bug, which has not yet been traced to a specific PSS version. The workaround "combine progress notes" documented above would only protect new Ocean notes from being affected, but it would not help with previous notes -- a fix from TELUS will be required to make such notes visible again. In order to mitigate the issue based on this information from TELUS, we are planning to patch the behaviour of Ocean Cloud Connect tonight (June 10) so that all new Ocean notes in a single patient session are combined for a given patient in PSS. In other words, we will deploy the workaround for all PSS clients. We will update this ticket when such action is taken.

2019-06-11 4PM EST: The "combine note" workaround patch was deployed yesterday evening and no issues have been reported so far today with the updated behaviour, which would have affected approximately two-thirds of our PSS client base. We plan to revert to previous behaviour (i.e. to respect the setting in the Cloud Connect configuration) once TELUS' fix is in place.

2019-12-9:

TELUS reports that the root cause is triggered by multiple progress notes on a chart with identical transaction ID's inserted via the API; these notes failed to appear on version 5.13.404 or higher.
It has a preliminary fix in 5.14.303 and later, which ensures that the notes display even when the transaction ID is duplicated. Existing progress notes are fixed via a repair which was released in 5.15.400.


TELUS PSS Layout Crash Issue

A number of Ocean sites have reported an issue in which very recent versions of PS Suite crash on some charts with a "java.lang.StackOverflowError" layout error.

Our theory is that dynamic text in a toolbar triggers a new bug in the PSS layout engine. When the Ocean Toolbar includes a display of pending forms for a patient, which gets longer as forms are added to the patient’s form queue, this can potentially cause this layout error to occur.

The issue is not specific to the Ocean Toolbar (and you may encounter it with other dynamic toolbars) but the Ocean Toolbar seems to trigger it in some instances. TELUS is aware of the issue and we have provided diagnostic information to facilitate a fix in an upcoming version of PS Suite.

In the interim, we have created a special version of the Ocean Toolbar that omits the "pending forms" field to attempt to workaround the bug.

If you are experiencing this issue, please email support@cognisantmd.com if you haven't already so that we can track affected sites, and then try upgrading your Toolbar.cfm to v5.22 or newer.

 


Telus API Issue Impacting OceanConnect Note Uploads for PS Suite [RESOLVED]

Our monitoring system has been receiving error reports this morning indicating that note uploads from OceanConnect are failing in the Telus API.  OceanConnect versions prior to 58 are affected.  After investigation, it appears that a recent change to the Telus API has broken older OceanConnect versions.

To correct the issue, tablets running OceanConnect versions prior to 58 can be updated to the latest version of OceanConnect through Google Play or by direct download. Alternatively, clients can upgrade to Ocean Cloud Connect and disable OceanConnect.  See Upgrading OceanConnect on Tablets for information on how to update to the latest version.  Or see Setting Up Ocean Cloud Connect for information on how to move to Cloud Connect. 

We are working with Telus support to determine the root cause and will update this ticket as new information comes available.

----

Dec 13, 2018: Telus has released a patch to their API server which has rectified the issue.

----

Jan 10, 2019, 4pm: We have seen a recurrence of this issue today. After discussion with Telus support, it appears that it has reappeared "because the API Services 2.37.01 deployed Jan 9th did not to have the emergency change that removed this error in the previous services release."  Telus is planning a patch release to rectify the issue.

----

Jan 11, 2019, 2pm: Telus patched the API servers last night at approximately 7:30pm ET.  The problem is resolved.


Telus API Issue Impacting Ocean Demographics Updates for PS Suite [RESOLVED in PSS 5.12.407]

We have been receiving error reports this morning from our monitoring system indicating that patient demographic field updates from Ocean forms are failing in the Telus API due to an internal system error from the TELUS API.

It's not year clear whether the issue is related to a new PSS version or a change to the API service itself. Note uploads are unaffected, but will include text indicating the demographics changes that were attempted but failed.

As an interim workaround, these values can be manually updated by copying and pasting into the appropriate fields. We're working with Telus support to determine the cause and will update this ticket as new information comes available.

Update 2018-12-19: We have confirmed with TELUS that the issue has been fixed in PSS version 5.12.407. If you are experiencing this issue, please request this patch from TELUS.


TELUS API outage impacting All TELUS Clients [RESOLVED]

The TELUS API server has been unresponsive since 8:25 EST this morning. We've seen connection timeouts (generally indicative of an offline system) returned to all TELUS/Ocean clients, including those on PS Suite and Med Access.

We are not aware of any scheduled or unscheduled downtime planned by TELUS and have escalated the issue to TELUS support for more information and requesting an ETA on the fix. We will update this ticket with more information when we get it.

While the API is offline, PS Suite clients may continue to use the Ocean Custom Form as a workaround to upload and download patient information manually; Med Access clinics will see failed appointments syncs, and will need to manually mark patients as arrived and copy/paste notes into charts.

More information to come.

--- 

10:00 PM: The TELUS API started responding again at around 9:30 PM. It appears to be responding normally since then. We will continue to monitor. As part of our investigation, we discovered that the TELUS EMR Mobile App was also affected. We'll post any root cause analysis that we receive from TELUS.


TELUS API issue impacting Cloud Connect and OceanConnect users [RESOLVED]

We have been notified that there is an issue with the TELUS National API, affecting Ocean clients using Cloud Connect (or OceanConnect). We are investigating the issue, and we will update this article as we receive more information from TELUS.

Update 12:30 PM EST: the issue appears to have been resolved (we are no longer seeing the same frequent errors from the TELUS API) although we are awaiting an explanation from TELUS. All EMR actions have been processed successfully now and the integration is operating normally again.


Telus API Issue impacting OceanConnect Users

We have been notified by Telus that there is an issue with the PSS API affecting Ocean clients using OceanConnect.  The impact seems to be limited to marking patients arrived, but we are still investigating.  We will update this article as we receive more information from Telus.

---

Update 1:30PM: TELUS has confirmed an issue with the stability of the API connections to PSS instances. They are investigating.

--- 

Update 1:40PM: TELUS says the API connections have stabilized and the issue should be resolved. We will continue to monitor.

---

Update May 30, 12:10 ET: The issue is recurring intermittently today.  Telus is aware of the issue.

---

Update June 1, 10:10 ET: The issue went away for a day or so but has returned and a number of clinics are experiencing the PSS arrival arrow issue again this morning. We are seeing the same error message from the TELUS API from those PSS instances and have escalated to TELUS again. TELUS confirmed that they are actively working on the issue. No root cause available from TELUS yet, other than general API instability.

---

Update June 1, 3:20 ET: TELUS says the API tunnel is stable again and things should be running normally. We will follow up with TELUS to try to understand the root cause of the API instability.

---

Update June 7, 10:30 AM ET: We are seeing the TELUS API failing across multiple sites again in the same manner as previously documented in this ticket. We are escalating to TELUS again. TELUS has acknowledge continuing issues with their API and are investigating. We will update this ticket as information becomes available.

--- 

Update June 8, 11:30 AM ET: It appears that the TELUS API has been stable since approximately 4:30 PM ET yesterday.

---

Update June 21, 10:50 AM ET: We are seeing the TELUS API failing across multiple sites again in the same manner as previously documented in this ticket. We have escalated to TELUS again.  We will update this ticket as information becomes available.

--- 

Update June 21, 10:00 PM ET: It appears that the TELUS API has stabilized.  TELUS is accelerating technical work to reduce the likelihood that the issue will recur.

--- 

Update June 22, 10:50 AM ET: TELUS says they successfully migrated their API endpoint to a new server last night to address the persistent instability.

 


TELUS API Connectivity Issues April 9, 2018

We are seeing errors returned from the TELUS API this morning. TELUS is investigating the issue. TELUS clients that have connected an OceanConnect device to the TELUS API may experience the following:

  • Patients not loaded for kiosk use
  • Patients not marked as arrived
  • Generated Ocean notes not being added to chart
  • Ocean Form Reminders delayed

As a temporary workaround, clinics can download clinical notes as required using the "download" link in the toolbar.

Clinics that are not yet connected to the TELUS API will not be affected.

We will update this ticket as more information becomes available from TELUS.

----

Update 2:55PM from TELUS: "Please note that we continue to experience an issue with the SSH Tunnel service for PS Suite clients. This issue is intermittent and is causing the SSH tunnel connection to drop. This results in a “nTELUS EMR Mobile cannot connect to your EMR Server. Please see the following link for more information.” error message from the API. TELUS has a team currently troubleshooting and have made it a high priority."

----

Update 5:30PM: TELUS believes they have isolated the issue to a change in the latest PSS version, and a ripple effect of a handful of clinics upgrading Sunday morning that resulted in Ocean connectivity disruptions for other clinics (that were not upgraded). They are going to disconnect the upgraded clinics from the API tonight, apply a PSS patch and monitor the API tomorrow. TELUS is contacting clinics directly that need to be disconnected, so if you haven't heard from them, there is no action required.

----

Update April 10, 2018 9:45AM: Although we do not yet have confirmation from TELUS, it appears that the maintenance work done by TELUS last night has helped at least some clinics that have reported success with kiosks this morning. Three clinics were disconnected from the API by TELUS; we assume that TELUS has informed those clinics but will be reaching out to confirm. If you are not one of the "disconnected" clinics, please try your kiosks this morning and contact us if you have any issues.

----

Update April 11, 2018 9:45AM: TELUS has confirmed that the issue was related to a PSS upgrade and patches were applied to affected clinics last night to address the issue. All clinics, including those disconnected from the API yesterday, have been reconnected by TELUS and your Ocean Kiosks, Ocean Online and other Ocean technology that interfaces with a TELUS EMR should be back to normal. Let us know if otherwise.

 

 


Ocean Compatibility for PS Suite 5.4

We are aware of a few sites that are running PS Suite 5.4 and having issues with PS Suite and Ocean, particularly with concurrent access to charts in PS Suite. We are working with TELUS on updating the custom form for 5.4 and have released a few custom form iterations in an attempt to address the issues.

If you are running Ocean and PS Suite without issue, we recommend that you do not upgrade the Ocean custom form until both 5.4 and the Ocean custom form are more stable.

On the other hand, if you are experiencing issues downloading notes in 5.4 (freezing, checkout errors, etc.) OR have already upgraded to Ocean custom form 2.84 or higher, please contact us at ocean.tips/support

Update 2015-12-08: We continue to work on the issue and have built a version 3.0 of the custom form for selective distribution that addresses some of the issues and adds some error handling logic. Please contact us if you would like it; it will be made publicly available at the standard download location later this week.

Update 2015-12-11: We have now released version 3.0.1 to the public download site. If you are running PSS 5.3 or 5.4, you should upgrade to this version as soon as possible. If you are running PSS 5.2, you should hold off upgrading the Ocean custom form for the time being.