Critical System Notifications

At 3:25pm ET, we became aware of users experiencing slow load times of Ocean services. We have traced this issue to an issue being experienced by CloudFlare, which the Ocean Platform uses for proxying and Web Application Firewall services, among other things. We are currently monitoring the issue and will update this article as additional information becomes available.

3:50pm: Cloudflare has implemented a fix for this issue and is currently monitoring the results.

More information is available at Cloudflare's status page.

4:06pm: Cloudflare has resolved the issue on their side. Latency to Ocean appears to have returned to normal. We're continuing to monitor the issue.

Early in the morning on Friday December 10th we became aware of a very serious security vulnerability found in a Java library we use in the Ocean platform called log4j2. (In this case severity is determined by how easy the vulnerability is to exploit as well as log4j2's widespread use.) Technical information on the vulnerability can be found here: https://www.lunasec.io/docs/blog/log4j-zero-day/

We wanted to update you as to how we are responding to this threat:

• First and foremost it was determined after initial investigation that Ocean and Cloud Connect were not vulnerable to the most serious attack vectors identified in the research - our patch processes keep our server software up to date and these patches prevented exploitation of the worst attack variant.
• On Friday night we released new patched versions of Ocean and Cloud Connect that included an upgrade to log4j 2.15, removing the remaining known attack vectors from the vulnerability in the Ocean platform.
• Working with 3rd party vendors, we identified some additional tools and libraries in our server monitoring infrastructure that could also be indirectly vulnerable. These 3rd party tools were patched Monday night.
• On Tuesday, further industry research led to the discovery of some remaining vulnerabilities in log4j version 2.15. Log4j 2.16 has been released to address these, which will be patched into Ocean platform Tuesday night. This eliminates the remaining known vulnerabilities.
• On Thursday December 17th a 3rd party vendor we rely on notified us a patch was available for their tool. We applied the patch last night. (Note: There is no added exposure to the Ocean platform with respect to the log4j vulnerability from this tool.)
• On Saturday December 18th we became aware of another update to the log4j library to mitigate an newly detected vulnerability. We upgraded to this version and applied the patch Sunday, Dec 19.

As of Dec 20 all vulnerable versions of log4j used in the Ocean platform (either directly or indirectly) have been fully patched. No action is required on your part.

After careful review of system logs, we find no evidence that the vulnerability was successfully used against the Ocean platform. While the known risks to the Ocean platform have been fully mitigated, we will continue to monitor the situation as our partners and software providers continue their investigation and mitigation efforts. This article will be updated if additional information becomes available.

Please contact support@cognisantmd.com if you have any questions or concerns.

2021-09-22 5:30pm ET: We are investigating an email delivery issue affecting some messages sent to @cogeco.ca and @cogeco.net addresses that began on Monday, September 20th. Delivery of secure messages and reminder emails to patients with Cogeco email addresses is impacted. We are contacting Cogeco Email Administrators to request that the block be lifted.

2021-09-24 The block has been lifted.

Please contact support@cognisantmd.com if you have any questions. We will update this article as new information comes available.

September 17th, 2021

8:45am EST: We've identified a bug that is preventing patients from uploading attachments to Ocean this morning after a new release of the Ocean Platform was deployed last night. We are investigating.

9:45am EST: The cause of this issue has been found and the fix will be deployed at 9:30pm EST.

Please email us at support@cognisantmd.com if you have any additional questions.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work to take no more than 15 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

CognisantMD is making infrastructure changes to increase the security, reliability, and performance of the Ocean platform. As a result the IP addresses used by Ocean and Cloud Connect will be changing.

This will not impact regular use of Ocean and/or Cloud Connect. However, if your site restricts outbound internet traffic via an allowlist, then you must make the changes below by June 30, 2021 to avoid any disruption to your use of the Ocean platform. This change is safe to make today.

Outbound Allowlist to be in place as of June 30, 2021:

• 35.182.46.13
• 15.222.173.33
• 52.233.47.110
• 13.71.164.223
• 103.21.244.0/22
• 103.22.200.0/22
• 103.31.4.0/22
• 104.16.0.0/13
• 104.24.0.0/14
• 108.162.192.0/18
• 131.0.72.0/22
• 141.101.64.0/18
• 162.158.0.0/15
• 172.64.0.0/13
• 173.245.48.0/20
• 188.114.96.0/20
• 190.93.240.0/20
• 197.234.240.0/22
• 198.41.128.0/17

If you have any questions, please contact support@cognisantmd.com

2021-06-29: We will be temporarily postponing the planned changes to Ocean IP addresses, and will provide an updated system notification including a new target date shortly.

2021-07-30: We will be applying this change next Friday August 6th, 2021.

1:54pm We are currently experiencing an issue affecting Ocean. We are investigating and will update this announcement as further information becomes available.

1:59pm The issue has been resolved and we are currently investigating the root cause. Note: Ocean will have experienced delays processing requests to EMRs via Cloud Connect. All pending updates will be synced to EMRs over the rest of the afternoon.

8:27pm Two issues impacted Ocean today. The first was a brief network interruption in our hosting facilities which caused the Ocean database to become unavailable for 2 minutes and 21 seconds at 1:11pm ET. No data was lost, but Ocean was unavailable during this period. The second was due to a failure between an EMR and the corresponding API server due to a very large volume of requests. The delay in processing the requests due to the failure caused a backup in Cloud Connect processing between 12:30pm and 1:55pm ET. During this period, Ocean features dependent on Cloud Connect EMR integration were delayed. 

Please email us at support@cognisantmd.com if you have any additional questions.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

At approximately 6am ET this morning, Ocean began experiencing irregular delays in processing requests to Cloud Connect. These delays impacted prompt processing of Ocean Reminders, patient note downloads to the EMR and patient appointment updates. No requests were lost. 

After investigation, it was determined that a defect in error handling in our SMS service was impacting Cloud Connect message processing. The SMS message that was triggering the bad error handling was removed and system processing returned to normal at 1:57pm ET.

We will be patching Ocean tonight to prevent recurrence of the issue and to insulate Cloud Connect processing from SMS.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work to take no more than 15 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work to take no more than 15 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

We are investigating an Ocean server service interruption from 08:59 - 09:05 AM EST this morning (approximately six minutes). The system is back up and responding normally. We will be investigating to understand the root cause of the issue and will post conclusions in this article.

--

1:55pm: The outage was caused by an out-of-memory condition related to an inefficient server function that was invoked repeatedly as part of an unusual user activity pattern. A remedial patch will be released tonight to guard against future occurrences.

On March 10th at approximately 1 pm, the  Bell.net and Sympatico.ca mail servers began blocking outbound email from the Ocean platform. Although we have not heard a formal explanation from them, it appears to be related to a spike in use of Ocean patient messaging for cohort emails which triggered a spam blocking rule.

As a result of the block, emails sent to @bell.net and @sympatico.ca addresses are not being delivered. We have paused email sends to these domains for the time being and will slowly increase sending to avoid triggering the spam blocking rule further.   

Please note that all emails to these domains sent since March 10, 1 pm may not yet have been received by their intended recipients. 

We will continue to update this ticket with information as it becomes available.

7:15pm: We have observed a small number of successful email deliveries to Bell / Sympatico and are slowly increasing the volume of email we sending to these email providers. We're starting to work through the backlog of email we've held back in the last 24 hours and will keep you up to date on our progress  

March 12: As of this morning, all backlogged emails have been processed properly and normal functionality has resumed.

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST on Friday, February 19. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

We want to make you aware that this Saturday (01/16) at 11AM there will be scheduled downtime in our support ticket system for approximately 3 hours while we migrate to a new ticketing system.

We are performing this migration during the weekend to ensure a smooth transition with minimal disruption. You will still be able to submit new support inquiries, but we kindly ask to refrain from replying to any open support tickets you may have during this time until we’re live again. All operations and activities around the Ocean platform will remain unaffected.

We appreciate your patience and understanding. As always, if you have questions or issues, please submit them at support@cognisantmd.com.

We'll update this article when migration is complete.

In order to maintain a high level of security and privacy for clinics and patients using the Ocean platform, we have upgraded the security protocols required by all clients (browsers and tablets) to connect to the Ocean server. Specifically, TLSv1 and TLSv1.1 protocols have known vulnerabilities, so they have been disabled as is considered best practice in the industry.

All modern web browsers support TLS 1.2, and most secure internet resources (including EMR systems) also require TLS 1.2 and thus it is unlikely that you will encounter issues on workstations or home computers.

However, clinics using tablets may experience a problem connecting to Ocean if they are running Android versions prior to 6.0 (which was released more than six years ago). Although Android 4.4 supports TLS 1.2, some manufacturer-specific versions of Android (including Samsung) may be missing this support.

If your tablet is unable to connect to Ocean after Dec 30, 2020, apply any outstanding updates or patches on your tablet. Given the vulnerabilities in TLS 1.0 and 1.1, most manufacturers have upgrade paths available. Please contact the tablet manufacturer if you have any issues.

If you have any questions or require assistance, please contact us by visiting ocean.tips/support.

As of this morning, the TELUS PSS Custom Form started to throw errors whenever a user attempts to upload patients to ocean.cognisantmd.com with an error like this:

This will be seen when the user initiates any actions from the PSS Toolbar, including sending secure messages ("email" button) and initiating an eReferral ("refer" button).

We are investigating now and will update this ticket with more information as it becomes known.

----

8:50am: The issue has been resolved and Ocean functionality is back to normal. At this point, it appears to be related to a web server tier configuration change made last night that unexpectedly affected the SSL certificate handling by Java application applications (but not web browsers, hence not noticed immediately). The change has been reverted.

At approximately 6:30pm Nov 30, 2020 ET, Ocean began receiving error responses from the Telus API for a number of API calls. The API failures prevented Cloud Connect synchronization as well as Ocean features that rely on Telus API integration including Patient Reminders and Online Booking. After informing Telus support, the incident was resolved at approximately 10:10pm ET.

Users may have experienced errors in Online Booking links and Patient Authenticated Website Forms during this interruption. In addition, Patient Reminders may have been delayed but should have completed successfully when service was restored.

On Nov 4, 2020 at approximately 12pm the Cogeco.ca mail server began blocking outbound email from the Ocean platform. Although we have not heard a formal explanation from Cogeco, it may be related to a spike in use of Ocean patient messaging which triggered a spam blocking rule.

As a result of the block, emails sent to @cogeco.ca addresses are not being delivered. We have opened a ticket with Cogeco support asking for the block to be lifted ASAP, but have paused email sends to that domain for the time being to avoid additional email loss. 

We will continue to update this ticket with information as it becomes available.

--

Nov 5, 2020 11:00am: Email deliverability to @cogeco.ca addresses from Ocean appears to no longer be blocked and we have resumed sending email to Cogeco.ca recipients.

We will be applying necessary software updates to the Ocean Portal beginning at 9:30pm EST tomorrow. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

-------

Oct 2, 2020 10:05 PM: Maintenance completed successfully.

On Sunday, August 2 at 5:00pm EST, we will be performing system maintenance on the Ocean Portal that will require downtime. We expect this to take a maximum of 60 minutes, during which time the Portal may be intermittently unavailable.

We will update this article once maintenance is complete and Ocean is available again for use.

----

7:25 PM - Maintenance complete. Ocean was unavailable for approximately 25 minutes between 5:05 and 5:25 PM ET.

On March 24, 2020 just after 5pm the Microsoft Hotmail public mail servers (including hotmail, outlook.com, msn.com, live.com and live.ca) started blocking outbound email from the Ocean platform. Although we have not heard a formal explanation from Microsoft, it may be related to a spike in use of Ocean to notify patients of COVID-related updates, which triggered a spam filtering rule.

As a result, emails sent to addresses within these domains are not being delivered. We have opened a ticket with Hotmail support asking for this block to be lifted. It has been marked high priority and is being review by Microsoft management. We made it clear to Microsoft that this block interferes with the delivery of critical information for patients in the COVID-19 crisis.

We will continue to update this ticket with information as it becomes available.

---

2020-03-27 8:54am: The Microsoft Hotmail Delivery Support Team has reviewed our case and determined that they should not be blocking our email. They've notified us the blocking policy that was put in place has been removed, and that the removal process could take 24 - 48hrs to take effect for all affected inboxes.

We'll be monitoring the deliverability of email from Ocean to Hotmail inboxes closely and will update this article as we see the block start to clear up.

After the block is cleared, any patient emails that were not delivered because of the action taken by Hotmail on March 24th will need to be resent.

 ---

2020-03-27 5:57pm: Delivery to Hotmail inboxes has returned to normal.

Note: Emails sent between 5pm March 24th, and 8:30pm March 26th will need to be sent again.

On Saturday, Jan 18 at 10:30pm ET, we will be migrating our production Cloud Connect infrastructure to a new environment. We expect this to take a maximum of 30 minutes, during which time EMR integrations with Ocean via Cloud Connect will be unavailable.

DNS changes may cause browsers to temporarily return error messages. These should be resolved by refreshing your browser after a short wait.

-----

Jan 18 11:30pm: Migration complete and tested. All systems running normally.

The MoH Health Card Validation service has been intermittently unavailable today, starting at around 10am ET, and is now unresponsive. Kiosks that are configured to check the HCV status of health cards may time-out during the check (20s) and report an error to the patient.

For clients using Ocean Kiosks with Ontario HCV, as a short-term measure, you may wish to disable the HCV checking in your tablet settings. To do so, log into your Ocean account, go to the Tablets tab, click on the relevant tablet group "edit" button, and choose "disable" under the HCV tab.

We will continue to monitor the status and update this ticket when the Ministry service is back online.

---- 

5:25pm ET - The HCV service appears to be back online. We will continue to monitor.

---- 

8:50am ET - The HCV service continues to be intermittently unavailable to fully unresponsive this morning. We will continue to monitor the status and update this ticket when the Ministry service is back online.

----

11:45am ET - The HCV service appears to be back online. We'll continue to monitor throughout the day.

We are investigating an issue with the Ocean/Accuro integration this morning. If you are running Accuro with Ocean, please see this article for more information:

https://support.cognisantmd.com/hc/en-us/articles/360030136051

Note that we will be transitioning EMR specific bulletins to dedicated categories in our support site. To this end, please take a moment to subscribe ("follow") this category for Accuro notifications:

https://support.cognisantmd.com/hc/en-us/sections/360004876851