Email Delivery to cogeco.ca is being blocked - Nov 4, 2020 [RESOLVED]

On Nov 4, 2020 at approximately 12pm the Cogeco.ca mail server began blocking outbound email from the Ocean platform. Although we have not heard a formal explanation from Cogeco, it may be related to a spike in use of Ocean patient messaging which triggered a spam blocking rule.

As a result of the block, emails sent to @cogeco.ca addresses are not being delivered. We have opened a ticket with Cogeco support asking for the block to be lifted ASAP, but have paused email sends to that domain for the time being to avoid additional email loss. 

We will continue to update this ticket with information as it becomes available.

--

Nov 5, 2020 11:00am: Email deliverability to @cogeco.ca addresses from Ocean appears to no longer be blocked and we have resumed sending email to Cogeco.ca recipients.


System Maintenance Planned Friday October 2nd, 2020 at 9:30pm EST (Complete)

We will be applying necessary software updates to the Ocean Portal beginning at 9:30pm EST tomorrow. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

 

-------

Oct 2, 2020 10:05 PM: Maintenance completed successfully.

 


Ocean System Maintenance Planned for Sunday, August 2nd, 2020 at 5:00pm EST (Completed)

On Sunday, August 2 at 5:00pm EST, we will be performing system maintenance on the Ocean Portal that will require downtime. We expect this to take a maximum of 60 minutes, during which time the Portal may be intermittently unavailable.

We will update this article once maintenance is complete and Ocean is available again for use.

----

7:25 PM - Maintenance complete. Ocean was unavailable for approximately 25 minutes between 5:05 and 5:25 PM ET.


Email Delivery to Hotmail and Related Domains Blocked March 24, 2020 [RESOLVED]

On March 24, 2020 just after 5pm the Microsoft Hotmail public mail servers (including hotmail, outlook.com, msn.com, live.com and live.ca) started blocking outbound email from the Ocean platform. Although we have not heard a formal explanation from Microsoft, it may be related to a spike in use of Ocean to notify patients of COVID-related updates, which triggered a spam filtering rule.

As a result, emails sent to addresses within these domains are not being delivered. We have opened a ticket with Hotmail support asking for this block to be lifted. It has been marked high priority and is being review by Microsoft management. We made it clear to Microsoft that this block interferes with the delivery of critical information for patients in the COVID-19 crisis.

We will continue to update this ticket with information as it becomes available.

---

2020-03-27 8:54am: The Microsoft Hotmail Delivery Support Team has reviewed our case and determined that they should not be blocking our email. They've notified us the blocking policy that was put in place has been removed, and that the removal process could take 24 - 48hrs to take effect for all affected inboxes.

We'll be monitoring the deliverability of email from Ocean to Hotmail inboxes closely and will update this article as we see the block start to clear up.

After the block is cleared, any patient emails that were not delivered because of the action taken by Hotmail on March 24th will need to be resent.

 ---

2020-03-27 5:57pm: Delivery to Hotmail inboxes has returned to normal.

Note: Emails sent between 5pm March 24th, and 8:30pm March 26th will need to be sent again.


Cloud Connect Server Migration Planned for Saturday, Jan 18, 2020 at 10:30pm ET (COMPLETED)

On Saturday, Jan 18 at 10:30pm ET, we will be migrating our production Cloud Connect infrastructure to a new environment. We expect this to take a maximum of 30 minutes, during which time EMR integrations with Ocean via Cloud Connect will be unavailable.

DNS changes may cause browsers to temporarily return error messages. These should be resolved by refreshing your browser after a short wait.

-----

Jan 18 11:30pm: Migration complete and tested. All systems running normally.


Ontario MoH HCV Service Outage (Resolved)

The MoH Health Card Validation service has been intermittently unavailable today, starting at around 10am ET, and is now unresponsive. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---- 

5:25pm ET - The HCV service appears to be back online. We will continue to monitor.

---- 

8:50am ET - The HCV service continues to be intermittently unavailable to fully unresponsive this morning. We will continue to monitor the status and update this ticket when the Ministry service is back online.

----

11:45am ET - The HCV service appears to be back online. We'll continue to monitor throughout the day.


Ocean/Accuro Integration Alert [RESOLVED]

We are investigating an issue with the Ocean/Accuro integration this morning. If you are running Accuro with Ocean, please see this article for more information:

https://support.cognisantmd.com/hc/en-us/articles/360030136051

Note that we will be transitioning EMR specific bulletins to dedicated categories in our support site. To this end, please take a moment to subscribe ("follow") this category for Accuro notifications:

https://support.cognisantmd.com/hc/en-us/sections/360004876851


Delay Adding Ocean Online Notes to EMR Chart (Cloud Connect/Ocean Connect Sites Only) [RESOLVED]

A bug introduced in last night's Ocean release broke the immediate download of notes generated by the Ocean Online module. The notes are saved in Ocean successfully, but the request to the EMR to add the note is deferred, and will only occur after a subsequent event (such as the patient confirming the Ocean Online message, for example). 

The bug has been identified and we intend to patch the issue at 9PM tonight. At that time, we will also apply a script to trigger the download of such notes that have not  yet been downloaded.

In the meantime, PS Suite users can download the notes by inserting the Ocean custom form and clicking "download". Users of other EMRs can achieve the same workaround through the Ocean Web Portal Patient's tab via the "Reset Downloaded Status for Notes" option for a patient under the "advanced" menu.

Update 2019-01-17 12:00PM: Ocean was patched last night to fix the underlying issue, and all delayed Ocean Online notes should be in the respective charts.


OceanConnect Tablet App End-of-Support Notice

CognisantMD will formally end support for the OceanConnect Tablet app on December 31st 2018, so please ensure you have migrated to Cloud Connect by then to ensure no disruption to your Ocean/EMR integration.

We are thrilled with the uptake of the new Cloud Connect integration platform for QHR Accuro, TELUS PS Suite and TELUS Med Access since it was made generally available in the summer of this year.  This new cloud-based platform replaces the tablet-based OceanConnect app and provides a faster, more reliable integration for your EMR.

We've now successfully migrated over 125 clinics and will be transitioning any remaining OceanConnect users over the coming weeks. Migrating is quick and easy, and our support team is here to assist with any questions or issues.

If you haven't heard of Cloud Connect, read the introduction announcement here: https://support.cognisantmd.com/hc/en-us/articles/360007319892

To migrate to the new Cloud Connect platform, go here and follow the simple steps: https://support.cognisantmd.com/hc/en-us/articles/360006687612

As always, please contact us if you have any questions or concerns.

For Accuro Users:

Please note that Cloud Connect does not support Accuro API v1, more formally known as the "appointment API". Accuro clients running against this older version of the Accuro API (the version without demographic updates and rich text notes) will need to contact Accuro customer support to enable v2 of the API for their Accuro instance. This can take up to two weeks, so we recommend starting this process as soon as possible.

 


Ontario MoH HCV Service Outage September 18 2018 [RESOLVED]

The MoH Health Card Validation service is not responding to requests this morning, starting approximately 9:30AM ET. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---

Update 11:20 AM ET: The MoH HCV server seems to be responding properly again.

Update 12:40 PM ET: The MoH HCV server is now returning empty invalid results, so sites will again be experiencing errors on check-ins or a flashing screen. We recommend that sites disable HCV as described above until further notice.

Update 5:25 PM ET: We are still seeing invalid responses from the MoH server.

Update 9:45 AM ET: The HCV server seems to be responding properly again.


Ocean Health Map Service Interruption July 5, 2018 [RESOLVED]

We experienced a service interruption with the Elastic Search engine that backs our health directory from 4:25PM ET until 4:35PM ET. Users running searches in the directory during this window would have encountered an error. The rest of Ocean was unaffected.

The system is performing normally again. We will update this ticket with additional information as it becomes available in the analysis.


Ontario MoH HCV Service Outage July 5, 2018 [RESOLVED]

The MoH Health Card Validation service is timing out this morning, starting approximately 10:45AM ET. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---

Update 4:10 PM ET: The MoH HCV server seems to be responding properly again. Issue resolved.


Ocean Service Interruptions June 25, 2018 [RESOLVED]

We have experienced intermittent technical issues in our data centre over the past several hours that have triggered service interruptions on the Ocean Server and health directory. We have stabilized the system and are investigating the impact and root cause.  We will post updates on this article shortly.

--- 

Update June 27, 10:45 AM ET: We have identified the problem and fixed the underlying issue. This issue is now resolved.


Ocean Service Interruptions May 30, 2018 [RESOLVED]

We have experienced a number of network issues in our data centre over the past hour that have triggered intermittent service interruptions on the Ocean Server for a minute or two at a time. We are investigating to determine the root cause and will post updates on this ticket shortly.

---

Update 12:30 ET: We've increased the concurrent connection pool capacity on our web tier and will continue to monitor carefully over the next couple hours.

---

Update May 30 11:30 ET: It appears that the capacity changes made yesterday have corrected the problem.  This issue is now closed.


Ontario MoH HCV Service Outage

The MoH Health Card Validation service has been intermittently unavailable this afternoon, starting at around noon ET, although some errors were seen as early as 10:30AM. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---

Update 3:45PM: It appears that the HCV service is back online. HCV-enabled kiosks should be back to normal.


Scheduled Maintenance for May 18, 2018

The Ocean Server will be offline briefly after 9pm, Eastern time (6pm Pacific) on May 18, 2018 for a software upgrade.  We expect the system to be offline for less that 30 minutes, during which a system maintenance page will appear in place of the Ocean Portal.


Ocean Performance Issues related to Friday's Server Migration [RESOLVED]

Tablets and Ocean Portal users experienced intermittent performance issues this morning as a result of an incorrect configuration setting on the new server.  The configuration error became apparent when server load returned to normal daily volume.  Ocean server was up throughout, but some requests would have failed due to timeouts.  The issue has been corrected as of 11:15am ET and performance has returned to normal.  Please let us know if you continue to experience issues.


Email Delivery Issues Relating to Friday's Server Migration [RESOLVED]

We are experiencing some issues with email delivery from our new data centre from Friday night. Emails may fail to be delivered and no error is presented.

In particular, clients using the "from" address in their email settings are affected (under Admin->Site Account), and batch sending may fail on large files due to an unexpected throttling limit from the email service provider. We are working to resolve these issues throughout today. In the meantime, as usual, clinics should rely on Ocean's patient confirmation status to determine whether a message was delivered successfully.

We will post updates to this article as they become available.

---

May 7 5:30PM ET: We've configured an internal mail server to which we will migrate and are now waiting for our data centre provider to configure some firewall configuration changes to match. We currently expect to migrate to the internal server tomorrow night.

---

May 8 11:30PM ET: We've updated the Ocean server to use the new internal mail server, akin to our previous configuration at the Toronto data centre. We've tested a number of email send scenarios and it appears to be working, and no error reports have been seen from the system. We believe the email delivery issues described above are resolved and we will close this issue accordingly.

--- 

May 9 4:00PM ET: Our logs indicated that our new server was been flagged by ProofPoint.com, which is a spam blacklist. We requested removal.

---

May 9 4:20PM ET: ProofPoint blacklist removal was completed.


Scheduled Maintenance for May 4, 2018

The Ocean Server will be offline briefly after 9pm, Eastern time (6pm-9pm Pacific) on May 4, 2018 for a system migration.  We expect the system to be offline for less that 30 minutes, during which a system maintenance page will appear in place of the Ocean Portal.

The production Ocean servers will be migrated from our existing production data centre in Toronto to our disaster recovery (DR) data centre in Montreal.  Once migration is complete, our Toronto data centre will be designated as our DR data centre and Montreal will become our production data centre.  This migration is being undertaken to increase server resources and provide additional future hosting configuration flexibility.  Connected devices will automatically switch over to the new data centre.  Please contact us at ocean.tips/support if any devices fail to move to the new location.

Update 2018-05-04 11:55PM ET: The migration to the Montreal data centre is complete. Service interruption was approximately 10 minutes shortly after 9:00 ET. Data replication has been reconfigured to allow for failover to our Toronto data centre if needed.


Scheduled Maintenance for April 9, 2018

The Ocean Server will be offline briefly after business hours on April 9, 2018 for a system upgrade.  We expect the system to be offline for less that 5 minutes sometime between 9pm and midnight, Eastern time (6pm-9pm Pacific).


System Interruption Jan 24, 2018 (4 minutes) (RESOLVED)

We experienced a system interruption between 10:46 and 10:50 AM EST on Jan 24 due to a networking problem.  We are working with our hosting provider to determine the root cause of the outage and will update this ticket when more information is available. We apologize for the inconvenience.




Performance Issue with eRequest Inbox Tab: June 27, 2017 10:01 [RESOLVED]

June 27, 2017 10AM ET: We are investigating a performance issue relating to the eRequest inbox tab relating to last night's release. Users may see delays of around 5s loading the page. We will update this post with more information and an ETA for a fix shortly.

Update 12:30pm: we found two independent sources of slowness (unnecessarily queries for some hidden inbox folders plus an inefficient database query path) that we will fix tonight in a patch. The system will be offline for approximately 5-10 minutes tonight at 9:30pm ET while we add the new indexes.


System Interruption Feb 28, 2017 (6 minutes) (RESOLVED)

We are currently experiencing a system interruption on the Ocean application due to excessive system load. We are investigating and will update this ticket shortly as more information is available. We apologize for the inconvenience.

UPDATE 5:24PM: the Ocean system is back online after a system restart. Total outage was six minutes. We will be investigating the root cause based on some captured diagnostic information and will update this ticket.

ROOT CAUSE ANALYSIS 2017-03-02 10PM: After analysis, it appears that the initial source of the excessive load was a series of directory searches that exposed a poorly indexed path. Compounding the problem was some system instrumentation designed to log "performance warnings"; due to the volume of requests that triggered this logging, it caused a knock on effect, forcing the hard restart by CognisantMD operations staff.

We are addressing the issue in three ways:

- We have put in place a measure through a patch release tonight to help protect against such events in the future

- We have some index changes planned for the next major release of the Ocean server

- We are adding protective code to reduce redundant performance warning logging in the event of a system wide slowdown


Critical patch release Feb 16 at 12PM EST

We will be releasing a critical patch release between 12:00 and 1PM EST today to address an issue that prevented EMR users without Ocean accounts from being able to send messages. We expect the Ocean system to be unavailable for approximately 30s. We apologize for any inconvenience this may cause.


Service Interruption January 25, 2017 (4 minutes)

The Ocean system was unavailable from 10:41 to 10:45 AM EST (4 minutes) on January 25th due to an unexpected lock related a database integrity check run manually by the Ocean system operations team. We apologize for any inconvenience it may have caused.


Critical patch release January 19 at 1PM EST

We will be releasing a critical patch release between 12:30 and 1PM EST today to address a couple of critical bugs from last night's release. We expect the Ocean system to be unavailable for approximately 30s. We apologize for any inconvenience this may cause.


Critical bug in tablet settings management [RESOLVED]

We are investigating a bug that affects tablet group setting management introduced as part of last night's release (Dec 13). It will cause favourites layout items and rules to be duplicated in the display. If the user clicks "save", the "doubled" favourites and rules will be saved to the database and may cause the forms to appear twice to the patient.

We are working on a fix and will deploy it tonight and fix any affected settings groups. In the meantime, please be very careful when updating tablet settings groups to ensure you don't save duplicate rules. If you can wait until tomorrow to update tablet settings rules, please do so.

Please let us know if you have any questions or concerns by contacting us at ocean.tips/support, and we apologize for the inconvenience.

Update (2016-12-14 10:15PM): The fix has been deployed to our production data centre and we ran a script to repair a number of duplicated rules and favourites (we contacted those sites directly to let them know, so if you didn't hear from us, you were not affected). Again, we apologize for the inconvenience.


System Interruption October 4 (2 minutes)

The Ocean system was offline for approximately 2 minutes this morning at 10:41 EST. During some routine maintenance (storage increase for backups) on our disaster recovery server, the production database server was accidentally rebooted instead of the disaster recovery database server. We apologize for any inconvenience this may have caused.