Ocean/Accuro Integration Alert [RESOLVED]

We are investigating an issue with the Ocean/Accuro integration this morning. If you are running Accuro with Ocean, please see this article for more information:

https://support.cognisantmd.com/hc/en-us/articles/360030136051

Note that we will be transitioning EMR specific bulletins to dedicated categories in our support site. To this end, please take a moment to subscribe ("follow") this category for Accuro notifications:

https://support.cognisantmd.com/hc/en-us/sections/360004876851


Delay Adding Ocean Online Notes to EMR Chart (Cloud Connect/Ocean Connect Sites Only) [RESOLVED]

A bug introduced in last night's Ocean release broke the immediate download of notes generated by the Ocean Online module. The notes are saved in Ocean successfully, but the request to the EMR to add the note is deferred, and will only occur after a subsequent event (such as the patient confirming the Ocean Online message, for example). 

The bug has been identified and we intend to patch the issue at 9PM tonight. At that time, we will also apply a script to trigger the download of such notes that have not  yet been downloaded.

In the meantime, PS Suite users can download the notes by inserting the Ocean custom form and clicking "download". Users of other EMRs can achieve the same workaround through the Ocean Web Portal Patient's tab via the "Reset Downloaded Status for Notes" option for a patient under the "advanced" menu.

Update 2019-01-17 12:00PM: Ocean was patched last night to fix the underlying issue, and all delayed Ocean Online notes should be in the respective charts.


OceanConnect Tablet App End-of-Support Notice

CognisantMD will formally end support for the OceanConnect Tablet app on December 31st 2018, so please ensure you have migrated to Cloud Connect by then to ensure no disruption to your Ocean/EMR integration.

We are thrilled with the uptake of the new Cloud Connect integration platform for QHR Accuro, TELUS PS Suite and TELUS Med Access since it was made generally available in the summer of this year.  This new cloud-based platform replaces the tablet-based OceanConnect app and provides a faster, more reliable integration for your EMR.

We've now successfully migrated over 125 clinics and will be transitioning any remaining OceanConnect users over the coming weeks. Migrating is quick and easy, and our support team is here to assist with any questions or issues.

If you haven't heard of Cloud Connect, read the introduction announcement here: https://support.cognisantmd.com/hc/en-us/articles/360007319892

To migrate to the new Cloud Connect platform, go here and follow the simple steps: https://support.cognisantmd.com/hc/en-us/articles/360006687612

As always, please contact us if you have any questions or concerns.

For Accuro Users:

Please note that Cloud Connect does not support Accuro API v1, more formally known as the "appointment API". Accuro clients running against this older version of the Accuro API (the version without demographic updates and rich text notes) will need to contact Accuro customer support to enable v2 of the API for their Accuro instance. This can take up to two weeks, so we recommend starting this process as soon as possible.

 


Ontario MoH HCV Service Outage September 18 2018 [RESOLVED]

The MoH Health Card Validation service is not responding to requests this morning, starting approximately 9:30AM ET. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---

Update 11:20 AM ET: The MoH HCV server seems to be responding properly again.

Update 12:40 PM ET: The MoH HCV server is now returning empty invalid results, so sites will again be experiencing errors on check-ins or a flashing screen. We recommend that sites disable HCV as described above until further notice.

Update 5:25 PM ET: We are still seeing invalid responses from the MoH server.

Update 9:45 AM ET: The HCV server seems to be responding properly again.


Ocean Health Map Service Interruption July 5, 2018 [RESOLVED]

We experienced a service interruption with the Elastic Search engine that backs our health directory from 4:25PM ET until 4:35PM ET. Users running searches in the directory during this window would have encountered an error. The rest of Ocean was unaffected.

The system is performing normally again. We will update this ticket with additional information as it becomes available in the analysis.


Ontario MoH HCV Service Outage July 5, 2018 [RESOLVED]

The MoH Health Card Validation service is timing out this morning, starting approximately 10:45AM ET. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---

Update 4:10 PM ET: The MoH HCV server seems to be responding properly again. Issue resolved.


Ocean Service Interruptions June 25 [RESOLVED]

We have experienced intermittent technical issues in our data centre over the past several hours that have triggered service interruptions on the Ocean Server and health directory. We have stabilized the system and are investigating the impact and root cause.  We will post updates on this article shortly.

--- 

Update June 27, 10:45 AM ET: We have identified the problem and fixed the underlying issue. This issue is now resolved.


Ocean Service Interruptions May 30 [RESOLVED]

We have experienced a number of network issues in our data centre over the past hour that have triggered intermittent service interruptions on the Ocean Server for a minute or two at a time. We are investigating to determine the root cause and will post updates on this ticket shortly.

---

Update 12:30 ET: We've increased the concurrent connection pool capacity on our web tier and will continue to monitor carefully over the next couple hours.

---

Update May 30 11:30 ET: It appears that the capacity changes made yesterday have corrected the problem.  This issue is now closed.


Ontario MoH HCV Service Outage

The MoH Health Card Validation service has been intermittently unavailable this afternoon, starting at around noon ET, although some errors were seen as early as 10:30AM. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---

Update 3:45PM: It appears that the HCV service is back online. HCV-enabled kiosks should be back to normal.


Scheduled Maintenance for May 18, 2018

The Ocean Server will be offline briefly after 9pm, Eastern time (6pm Pacific) on May 18, 2018 for a software upgrade.  We expect the system to be offline for less that 30 minutes, during which a system maintenance page will appear in place of the Ocean Portal.


Ocean Performance Issues related to Friday's Server Migration [RESOLVED]

Tablets and Ocean Portal users experienced intermittent performance issues this morning as a result of an incorrect configuration setting on the new server.  The configuration error became apparent when server load returned to normal daily volume.  Ocean server was up throughout, but some requests would have failed due to timeouts.  The issue has been corrected as of 11:15am ET and performance has returned to normal.  Please let us know if you continue to experience issues.


Email Delivery Issues Relating to Friday's Server Migration [RESOLVED]

We are experiencing some issues with email delivery from our new data centre from Friday night. Emails may fail to be delivered and no error is presented.

In particular, clients using the "from" address in their email settings are affected (under Admin->Site Account), and batch sending may fail on large files due to an unexpected throttling limit from the email service provider. We are working to resolve these issues throughout today. In the meantime, as usual, clinics should rely on Ocean's patient confirmation status to determine whether a message was delivered successfully.

We will post updates to this article as they become available.

---

May 7 5:30PM ET: We've configured an internal mail server to which we will migrate and are now waiting for our data centre provider to configure some firewall configuration changes to match. We currently expect to migrate to the internal server tomorrow night.

---

May 8 11:30PM ET: We've updated the Ocean server to use the new internal mail server, akin to our previous configuration at the Toronto data centre. We've tested a number of email send scenarios and it appears to be working, and no error reports have been seen from the system. We believe the email delivery issues described above are resolved and we will close this issue accordingly.

--- 

May 9 4:00PM ET: Our logs indicated that our new server was been flagged by ProofPoint.com, which is a spam blacklist. We requested removal.

---

May 9 4:20PM ET: ProofPoint blacklist removal was completed.


Scheduled Maintenance for May 4, 2018

The Ocean Server will be offline briefly after 9pm, Eastern time (6pm-9pm Pacific) on May 4, 2018 for a system migration.  We expect the system to be offline for less that 30 minutes, during which a system maintenance page will appear in place of the Ocean Portal.

The production Ocean servers will be migrated from our existing production data centre in Toronto to our disaster recovery (DR) data centre in Montreal.  Once migration is complete, our Toronto data centre will be designated as our DR data centre and Montreal will become our production data centre.  This migration is being undertaken to increase server resources and provide additional future hosting configuration flexibility.  Connected devices will automatically switch over to the new data centre.  Please contact us at ocean.tips/support if any devices fail to move to the new location.

Update 2018-05-04 11:55PM ET: The migration to the Montreal data centre is complete. Service interruption was approximately 10 minutes shortly after 9:00 ET. Data replication has been reconfigured to allow for failover to our Toronto data centre if needed.


Scheduled Maintenance for April 9, 2018

The Ocean Server will be offline briefly after business hours on April 9, 2018 for a system upgrade.  We expect the system to be offline for less that 5 minutes sometime between 9pm and midnight, Eastern time (6pm-9pm Pacific).


System Interruption Jan 24 (4 minutes)

We experienced a system interruption between 10:46 and 10:50 AM EST on Jan 24 due to a networking problem.  We are working with our hosting provider to determine the root cause of the outage and will update this ticket when more information is available. We apologize for the inconvenience.




Performance Issue with eRequest Inbox Tab: June 27, 2017 10:01 [RESOLVED]

June 27, 2017 10AM ET: We are investigating a performance issue relating to the eRequest inbox tab relating to last night's release. Users may see delays of around 5s loading the page. We will update this post with more information and an ETA for a fix shortly.

Update 12:30pm: we found two independent sources of slowness (unnecessarily queries for some hidden inbox folders plus an inefficient database query path) that we will fix tonight in a patch. The system will be offline for approximately 5-10 minutes tonight at 9:30pm ET while we add the new indexes.


System Interruption Feb 28 (6 minutes)

We are currently experiencing a system interruption on the Ocean application due to excessive system load. We are investigating and will update this ticket shortly as more information is available. We apologize for the inconvenience.

UPDATE 5:24PM: the Ocean system is back online after a system restart. Total outage was six minutes. We will be investigating the root cause based on some captured diagnostic information and will update this ticket.

ROOT CAUSE ANALYSIS 2017-03-02 10PM: After analysis, it appears that the initial source of the excessive load was a series of directory searches that exposed a poorly indexed path. Compounding the problem was some system instrumentation designed to log "performance warnings"; due to the volume of requests that triggered this logging, it caused a knock on effect, forcing the hard restart by CognisantMD operations staff.

We are addressing the issue in three ways:

- We have put in place a measure through a patch release tonight to help protect against such events in the future

- We have some index changes planned for the next major release of the Ocean server

- We are adding protective code to reduce redundant performance warning logging in the event of a system wide slowdown


Critical patch release Feb 16 at 12PM EST

We will be releasing a critical patch release between 12:00 and 1PM EST today to address an issue that prevented EMR users without Ocean accounts from being able to send messages. We expect the Ocean system to be unavailable for approximately 30s. We apologize for any inconvenience this may cause.


Service Interruption January 25, 2017 (4 minutes)

The Ocean system was unavailable from 10:41 to 10:45 AM EST (4 minutes) on January 25th due to an unexpected lock related a database integrity check run manually by the Ocean system operations team. We apologize for any inconvenience it may have caused.


Critical patch release January 19 at 1PM EST

We will be releasing a critical patch release between 12:30 and 1PM EST today to address a couple of critical bugs from last night's release. We expect the Ocean system to be unavailable for approximately 30s. We apologize for any inconvenience this may cause.


Critical bug in tablet settings management [RESOLVED]

We are investigating a bug that affects tablet group setting management introduced as part of last night's release (Dec 13). It will cause favourites layout items and rules to be duplicated in the display. If the user clicks "save", the "doubled" favourites and rules will be saved to the database and may cause the forms to appear twice to the patient.

We are working on a fix and will deploy it tonight and fix any affected settings groups. In the meantime, please be very careful when updating tablet settings groups to ensure you don't save duplicate rules. If you can wait until tomorrow to update tablet settings rules, please do so.

Please let us know if you have any questions or concerns by contacting us at ocean.tips/support, and we apologize for the inconvenience.

Update (2016-12-14 10:15PM): The fix has been deployed to our production data centre and we ran a script to repair a number of duplicated rules and favourites (we contacted those sites directly to let them know, so if you didn't hear from us, you were not affected). Again, we apologize for the inconvenience.


System Interruption October 4 (2 minutes)

The Ocean system was offline for approximately 2 minutes this morning at 10:41 EST. During some routine maintenance (storage increase for backups) on our disaster recovery server, the production database server was accidentally rebooted instead of the disaster recovery database server. We apologize for any inconvenience this may have caused.


System Interruption Sept 14 3:43 EST (2 minutes)

We experienced a brief service interruption on the Ocean server between 3:43 and 3:45 PM EST today for a duration of just under 2 minutes. Some users may have experienced delays or errors during certain system operations.

We are looking into the root cause and will update as more is known.


Ocean on Android 6 Advisory

With Android 6, Google changed the way that a tablet app can identify its hardware by removing access to the wireless "mac address". Whereas Ocean previously could uniquely identify tablets in this manner, and use it for session management (like a username), this doesn't work in Android 6.

Fortunately, most tablets do not auto-update to Android 6, so the problem occurs only for certain tablets like newer Samsung and Lenovo tablets. In early July, we released a fix in version 125 of the Ocean tablet app that identifies tablets using a "hardware ID" instead. We highly recommend clients upgrade their tablets to v125 to "get ahead" of any potential Android 6 auto-upgrades.

If your tablets are running Android 6 and an Ocean app below v125, there are a few things that you might notice:

  • General connectivity issues based on the sessions getting mixed up (if you have 2 or more tablets on Android 6 with Ocean below v125)
  • Billing issues: if you notice redundant tablets on your month-end bill, it may be because your tablets changed identifiers and confused Ocean into thinking there were more tablets than in reality.

If you need help with either of the above, please contact ocean.tips/support and we'll help sort things out. You should also review your Tablets tab in the portal to ensure you don't have legacy registrations for tablets on your account.

Finally, in order to get ahead of the issue, next week we'll force upgrade all tablets below v125 to the latest version. If you have a large number of tablets running below v125, you may wish to upgrade them in advance. To learn how to do this, please refer to "Upgrading Your Tablet".


Ocean Studies Data Issue [RESOLVED]

We are investigating a bug that caused Ocean Studies submissions to be lost retroactively when using older versions of Ocean Tablets. The bug was introduced in Thursday night's Ocean server upgrade as a side-effect of the feature to eliminate redundant submissions to an Ocean Study in the same patient session.

We have fixed the bug and we deployed the fix at 10am this morning. We are in the process of recovering data from backups and replication transaction logs. 

At this point, we expect to have full data recovery completed by the end of the day. We apologize for the inconvenience. Please contact us at ocean.tips/support if you have any concerns.

2016-03-23 11:30PM: Issue resolved. We have restored all study submissions and verified against audit logs. Your Ocean Study repositories are back to normal and we do not believe that any data was lost (although some data was unavailable during the day today for export). We apologize for the inconvenience.


Critical system maintenance - Friday, February 19, 2016 at 10pm ET [COMPLETED]

A critical vulnerability has been identified in a library used by the operating system running Ocean servers. The vulnerability has been identified by security researchers at Google and Red Hat and a patch has been made generally available.  We have applied this patch to all Ocean servers however a reboot is required to complete the patch application.  As this reboot will impact Ocean services, we are scheduling it outside business hours.  We expect the reboot will take approximately 5 minutes, during which time you may encounter errors while using Ocean tablets or Ocean EMR integrations and the Ocean Portal will be unavailable.  If the outage lasts longer than 5 minutes, please contact CognisantMD support.

Technical details on the vulnerability can be found here.

Although this is a critical vulnerability, we have no reason to believe that any Ocean server has been compromised in any way.  Please contact support if you have any concerns.

10:40pm: reboot process complete, all servers responding


February 12 2016 System interruption (7 minutes)

The Ocean system is currently offline due to a misconfiguration relating to the final step of migrating to our new data centre (changing the DB replication settings). We are in the process of correcting the misconfiguration.

2016-02-12 5:04pm: The system is back online for a total downtime of 7 minutes. Root cause coming shortly.

Root cause analysis: 

Background: the Ocean database is replicated in real time from Toronto to Vancouver for redundancy. Since the February 2nd migration to the new Toronto datacentre, we've been running a third replicated database -- our old Toronto data centre -- as an insurance policy.

Cause: as part of the final stage of migration in our new data centre, our old Toronto data center and its replicated database was decommissioned at 5pm today. Unexpectedly, this disrupted the replication set in such a way as no database was elected "primary", leading to a situation in which the application had no active database connection. The replication set was repaired manually and the system was brought back online.

Actions: we will be investigating the underlying misconfiguration that would cause the unavailability of a secondary node to disrupt the primary.


Issue with Emailed Links After Datacentre Migration

Due to a misconfiguration related to the datacentre migration this past Friday night, links to ocean.cognisantmd.com in generated emails (e.g. web questionnaires, referral notifications, etc) sent since then were incorrectly pointing to "peer1.cognisantmd.com" instead.  When clicked these links would cause web browsers to display a warning message regarding an invalid SSL certificate. The links are safe to use but patients may be understandably reluctant to follow them.  The problem was corrected at 10:30am ET today (Feb 2 2016). We apologize for any inconvenience.