Ocean Toolbar Issue: Update Required

We have identified a bug in a recent, limited release Ocean toolbar for TELUS PS Suite. This toolbar, which includes a link to the Ontario eConsult Listing (OTNHub), may prevent referral updates from appearing in the patient chart. 

Clinics that downloaded and installed this toolbar between August 15 - 17 should immediately update their toolbar. Instructions for updating the basic Ocean Toolbar can be found here (please note this version does not include the OTNHub link).

To learn more or receive a link to download the Ocean toolbar linking to the Ontario eConsult Listing in Ocean (OTNHub), please contact the Ontario eServices Program here.  

If you require any further assistance, please contact our support team at ocean.tips/support.




Article Updated May 4, 2022: URLs, and IPs That Need to be Allowlisted for Ocean

New IP addresses have been added to the Ocean platform. You may be impacted if:

  • You have enabled a CloudConnect integration with an OscarPro EMR, or
  • You have a custom integration with Ocean that receives webhook events for patient engagement and / or eReferral updates, AND
  • You have a firewall that prevents traffic from the public internet reaching either of the services mentioned above

The full article can be reviewed here. Check with your network administrator about whether this change affects you.

Please email us at support@cognisantmd.com if you have any additional questions.

 


Failures Uploading Attachments to Ocean April 13, 2022 [RESOLVED]

9:07am We're aware of an issue affecting some attachment uploads to Ocean and are investigating. We'll update this announcement as further information becomes available.

10:18am We are currently working with Cloudflare to investigate this issue. It looks like large file uploads are failing for some Cloudflare customers. We're trying to determine whether or not we are similarly impacted by a failure on Cloudflare's side.

Note that this issue also affected the eForm Editor; eForm 'save' actions are failing because they rely on the same functionality.

1:00pm Cloudflare has pushed a fix on their side that appears to have corrected the issue. Please resume regular usage of Ocean and let us know if you encounter any more trouble. We will be monitoring the platform for errors closely over the next few hours.

Please email us at support@cognisantmd.com if you have any additional questions.




Ocean Service Interruption January 10th, 2022 [RESOLVED]

9:19am We are currently experiencing an issue affecting Ocean patient reminder emails. We are investigating and will update this announcement as further information becomes available.

9:45am We applied a fix to our firewall configuration that should allow patients to follow Ocean links in reminder emails again.

3:29pm A new config was applied to our firewall service over the weekend that prevented patients from reaching Ocean by clicking urls contained in email sent by the Ocean platform. We have prepared a patch release to be deployed tonight that will prevent this issue from recurring.

9:50pm Patch release deployed.

Please email us at 
support@cognisantmd.com if you have any additional questions.


Ocean Portal Performance Degradation Dec 16, 2021 [RESOLVED]

At 3:25pm ET, we became aware of users experiencing slow load times of Ocean services. We have traced this issue to an issue being experienced by CloudFlare, which the Ocean Platform uses for proxying and Web Application Firewall services, among other things. We are currently monitoring the issue and will update this article as additional information becomes available.

3:50pm: Cloudflare has implemented a fix for this issue and is currently monitoring the results.

More information is available at Cloudflare's status page.

4:06pm: Cloudflare has resolved the issue on their side. Latency to Ocean appears to have returned to normal. We're continuing to monitor the issue.

 


Security Update: Log4j Vulnerability December 14th, 2021 (Updated Dec 20, 2021) [RESOLVED]

Early in the morning on Friday December 10th we became aware of a very serious security vulnerability found in a Java library we use in the Ocean platform called log4j2. (In this case severity is determined by how easy the vulnerability is to exploit as well as log4j2's widespread use.) Technical information on the vulnerability can be found here: https://www.lunasec.io/docs/blog/log4j-zero-day/

We wanted to update you as to how we are responding to this threat:

  • First and foremost it was determined after initial investigation that Ocean and Cloud Connect were not vulnerable to the most serious attack vectors identified in the research - our patch processes keep our server software up to date and these patches prevented exploitation of the worst attack variant.
  • On Friday night we released new patched versions of Ocean and Cloud Connect that included an upgrade to log4j 2.15, removing the remaining known attack vectors from the vulnerability in the Ocean platform.
  • Working with 3rd party vendors, we identified some additional tools and libraries in our server monitoring infrastructure that could also be indirectly vulnerable. These 3rd party tools were patched Monday night.
  • On Tuesday, further industry research led to the discovery of some remaining vulnerabilities in log4j version 2.15. Log4j 2.16 has been released to address these, which will be patched into Ocean platform Tuesday night. This eliminates the remaining known vulnerabilities.
  • On Thursday December 17th a 3rd party vendor we rely on notified us a patch was available for their tool. We applied the patch last night. (Note: There is no added exposure to the Ocean platform with respect to the log4j vulnerability from this tool.)
  • On Saturday December 18th we became aware of another update to the log4j library to mitigate an newly detected vulnerability. We upgraded to this version and applied the patch Sunday, Dec 19.

As of Dec 20 all vulnerable versions of log4j used in the Ocean platform (either directly or indirectly) have been fully patched. No action is required on your part.

After careful review of system logs, we find no evidence that the vulnerability was successfully used against the Ocean platform. While the known risks to the Ocean platform have been fully mitigated, we will continue to monitor the situation as our partners and software providers continue their investigation and mitigation efforts. This article will be updated if additional information becomes available.

Please contact support@cognisantmd.com if you have any questions or concerns.

 



Email Delivery to Cogeco Mailboxes Interrupted Sept 22nd, 2021 [RESOLVED]

2021-09-22 5:30pm ET: We are investigating an email delivery issue affecting some messages sent to @cogeco.ca and @cogeco.net addresses that began on Monday, September 20th. Delivery of secure messages and reminder emails to patients with Cogeco email addresses is impacted. We are contacting Cogeco Email Administrators to request that the block be lifted.

2021-09-24 The block has been lifted.

Please contact support@cognisantmd.com if you have any questions. We will update this article as new information comes available.


Patients Unable to Upload Attachments to Ocean [RESOLVED]

September 17th, 2021

8:45am EST: We've identified a bug that is preventing patients from uploading attachments to Ocean this morning after a new release of the Ocean Platform was deployed last night. We are investigating.

9:45am EST: The cause of this issue has been found and the fix will be deployed at 9:30pm EST.

Please email us at support@cognisantmd.com if you have any additional questions.


System Maintenance Planned Sept 3rd, 2021 Starting at 9:30pm EST [COMPLETED]

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 120 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

 

--

11:15pm: All maintenance completed successfully.



System Maintenance Planned July 30th at 9:30PM EST [COMPLETED]

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.



Changes to Ocean IP Addresses

CognisantMD is making infrastructure changes to increase the security, reliability, and performance of the Ocean platform. As a result the IP addresses used by Ocean and Cloud Connect will be changing.

This will not impact regular use of Ocean and/or Cloud Connect. However, if your site restricts outbound internet traffic via an allowlist, then you must make the changes below by June 30, 2021 to avoid any disruption to your use of the Ocean platform. This change is safe to make today.

Outbound Allowlist to be in place as of June 30, 2021:

  • 35.182.46.13
  • 15.222.173.33
  • 52.233.47.110
  • 13.71.164.223
  • 103.21.244.0/22
  • 103.22.200.0/22
  • 103.31.4.0/22
  • 104.16.0.0/13
  • 104.24.0.0/14
  • 108.162.192.0/18
  • 131.0.72.0/22
  • 141.101.64.0/18
  • 162.158.0.0/15
  • 172.64.0.0/13
  • 173.245.48.0/20
  • 188.114.96.0/20
  • 190.93.240.0/20
  • 197.234.240.0/22
  • 198.41.128.0/17

If you have any questions, please contact support@cognisantmd.com

 

2021-06-29: We will be temporarily postponing the planned changes to Ocean IP addresses, and will provide an updated system notification including a new target date shortly.

 

2021-07-30: We will be applying this change next Friday August 6th, 2021.


Ocean Service Interruption May 17, 2021 [RESOLVED]

1:54pm We are currently experiencing an issue affecting Ocean. We are investigating and will update this announcement as further information becomes available.

1:59pm The issue has been resolved and we are currently investigating the root cause. Note: Ocean will have experienced delays processing requests to EMRs via Cloud Connect. All pending updates will be synced to EMRs over the rest of the afternoon.

8:27pm Two issues impacted Ocean today. The first was a brief network interruption in our hosting facilities which caused the Ocean database to become unavailable for 2 minutes and 21 seconds at 1:11pm ET. No data was lost, but Ocean was unavailable during this period. The second was due to a failure between an EMR and the corresponding API server due to a very large volume of requests. The delay in processing the requests due to the failure caused a backup in Cloud Connect processing between 12:30pm and 1:55pm ET. During this period, Ocean features dependent on Cloud Connect EMR integration were delayed. 

Please email us at support@cognisantmd.com if you have any additional questions.



Ocean Processing Delays April 23, 2021 [Resolved]

At approximately 6am ET this morning, Ocean began experiencing irregular delays in processing requests to Cloud Connect. These delays impacted prompt processing of Ocean Reminders, patient note downloads to the EMR and patient appointment updates. No requests were lost. 

After investigation, it was determined that a defect in error handling in our SMS service was impacting Cloud Connect message processing. The SMS message that was triggering the bad error handling was removed and system processing returned to normal at 1:57pm ET.

We will be patching Ocean tonight to prevent recurrence of the issue and to insulate Cloud Connect processing from SMS.




Ocean Service Interruption March 23, 2021 [Resolved]

We are investigating an Ocean server service interruption from 08:59 - 09:05 AM EST this morning (approximately six minutes). The system is back up and responding normally. We will be investigating to understand the root cause of the issue and will post conclusions in this article.

--

1:55pm: The outage was caused by an out-of-memory condition related to an inefficient server function that was invoked repeatedly as part of an unusual user activity pattern. A remedial patch will be released tonight to guard against future occurrences.


Email Delivery to Bell / Sympatico Being Blocked - March 11, 2021 [RESOLVED]

On March 10th at approximately 1 pm, the  Bell.net and Sympatico.ca mail servers began blocking outbound email from the Ocean platform. Although we have not heard a formal explanation from them, it appears to be related to a spike in use of Ocean patient messaging for cohort emails which triggered a spam blocking rule.

As a result of the block, emails sent to @bell.net and @sympatico.ca addresses are not being delivered. We have paused email sends to these domains for the time being and will slowly increase sending to avoid triggering the spam blocking rule further.   

Please note that all emails to these domains sent since March 10, 1 pm may not yet have been received by their intended recipients. 

We will continue to update this ticket with information as it becomes available.

7:15pm: We have observed a small number of successful email deliveries to Bell / Sympatico and are slowly increasing the volume of email we sending to these email providers. We're starting to work through the backlog of email we've held back in the last 24 hours and will keep you up to date on our progress  

March 12: As of this morning, all backlogged emails have been processed properly and normal functionality has resumed.



Support Ticket System Maintenance Planned Saturday January 16th, 2020 at 11AM EST

 

We want to make you aware that this Saturday (01/16) at 11AM there will be scheduled downtime in our support ticket system for approximately 3 hours while we migrate to a new ticketing system.

 

We are performing this migration during the weekend to ensure a smooth transition with minimal disruption. You will still be able to submit new support inquiries, but we kindly ask to refrain from replying to any open support tickets you may have during this time until we’re live again. All operations and activities around the Ocean platform will remain unaffected.

 

We appreciate your patience and understanding. As always, if you have questions or issues, please submit them at support@cognisantmd.com.

 

We'll update this article when migration is complete.


System Maintenance Planned Friday January 15, 2021 at 9:30pm EST [Completed]

-- updated with correct date (tonight)

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST tonight. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.