Critical System Maintenance Planned for December 12th, 2022 at 10:00 PM EST [COMPLETED]

In response to the Ocean Portal Performance Degradation on December 9th 2022, we will be performing critical system maintenance between 10:00 PM and 12:00 AM EST tonight (December 12th, 2022) to adjust system resources and settings related to the Ocean database. During this time, Ocean will be intermittently offline and unavailable for use.

We'll update this article when maintenance is complete and Ocean is ready for normal use.


Ocean Portal Performance Degradation Dec 9, 2022 [RESOLVED]

Friday, December 9th 2022

9:08 AM EST: We have identified a performance degradation within the Ocean Portal resulting in slow loading times. We are currently investigating the issue and will update this article as additional information becomes available.

9:35 AM EST: A potential resource allocation fix will be applied this morning, during which time the Ocean Portal will be unavailable for 3-5 minutes. This article will be updated when the downtime begins and ends.

9:43 AM EST: Downtime will begin at 9:45 AM EST. This article will be updated when downtime ends.

9:55 AM EST: Downtime has ended and the Ocean Portal is now available.

10:11 AM EST: We are continuing to monitor the performance of the Ocean Portal.

Monday, December 12th 2022

9:25 AM EST: We are aware of the continued performance degradation within the Ocean Portal resulting in slow loading times. We are continuing to investigate the root cause of the issue and will update this article as additional information becomes available.

10:20 AM EST: We continue to see heavy loads and contention on certain database resources. We will perform a database restart in the next 15 minutes to attempt to work around the issue in the short term.

10:35 AM EST: The database restart helped and our load monitoring shows much better performance since the restart. This morning’s database pressure has provided additional clues as to the root cause, which we will continue to investigate throughout the day.

4:41 PM EST: We will be performing Critical System Maintenance at 10:00 PM EST tonight to adjust system resources and settings related to the Ocean database. Ocean will be intermittently offline and unavailable for use during this time. The linked bulletin will be updated when maintenance is complete and Ocean is ready for normal use.

Tuesday, December 13th 2022

11:44 AM EST: The critical maintenance performed last night to adjust system resources and settings has improved the performance and reduced the pressure on the Ocean database. We are continuing to monitor.

Wednesday, December 14th 2022

11:15 AM EST: After an extended monitoring period, we can confirm that performance of the Ocean Portal has stabilized at normal levels. This system notification bulletin will now be marked as resolved.

Please email us at support@cognisantmd.com if you have any additional questions.



Email Notification Interruption - November 29, 2022 [RESOLVED]

We are currently experiencing an issue affecting Ocean eReferrals and Website Forms.   Email notifications for newly submitted website forms and eReferrals are not being received by the designated recipients due to a bug introduced in the November 24th Ocean platform release.

[Nov 29, 9:20 PM]: We have identified the root cause and are releasing a fix tonight.  We will update this announcement as further information becomes available. Please email us at if you have any additional questions.

[Nov 30, 10:0 AM] We released the fix last night and have verified that email notifications are now being sent for newly submitted website forms and eReferrals.   Ocean is also in the process of sending any email notifications that were missed since November 24th.  

Please email us at support@cognisantmd.com if you have any additional questions.


Ocean FHIR / Open API Service Interruption November 4, 2022 [RESOLVED]

At 9am, we became aware of an issue for some sites using integrations on the FHRI APIs or Open APIs receiving referrals that in some cases the webhook notification may not be issued. This could mean that referrals have been received but downstream systems have not been notified. We are investigating as an urgent issue and will post more information as it becomes available.

In the interim, integrated sites should access the Ocean Portal's eReferral View to monitor for new referrals and process existing referrals.

10:30am We've identified an issue in Ocean related to referral webhook notifications and are working on a fix.

2:10pm: We have deployed a fix to our production environment to address the core issue. We will be investigating methods to re-trigger missed notifications and will update this article as we investigate.

5:00pm As of 2:15pm, Webhook notifications for new and updated referrals to FHIR integrated sites are being sent successfully. Outbound API messages that weren't sent since last night starting at 10pm will be re-triggered in 2 steps:

  • First we'll send message for newly created referrals
  • Following this we'll send messages for updated referrals for all referrals modified in the impacted range (both new and existing)

Note: We'll be spacing out the updates so they're not sent at the same time.

In partnership with the eServices Program, we will be providing FHIR integrated sites with a report of affected referrals that can be used to verify all updates have been downloaded to EMRs.

November 5, 2022 2:00pm: Missed webhook notifications to integrated sites have been re-sent. We have verified the successful delivery of these API messages and completed our planned data repair activity for this event.

Please email us at support@cognisantmd.com if you have any additional questions.



System Maintenance Planned for October 11th, 2022 at 10:00pm EST

We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.




Ocean Toolbar Issue: Update Required

We have identified a bug in a recent, limited release Ocean toolbar for TELUS PS Suite. This toolbar, which includes a link to the Ontario eConsult Listing (OTNHub), may prevent referral updates from appearing in the patient chart. 

Clinics that downloaded and installed this toolbar between August 15 - 17 should immediately update their toolbar. Instructions for updating the basic Ocean Toolbar can be found here (please note this version does not include the OTNHub link).

To learn more or receive a link to download the Ocean toolbar linking to the Ontario eConsult Listing in Ocean (OTNHub), please contact the Ontario eServices Program here.  

If you require any further assistance, please contact our support team at ocean.tips/support.




Article Updated May 4, 2022: URLs, and IPs That Need to be Allowlisted for Ocean

New IP addresses have been added to the Ocean platform. You may be impacted if:

  • You have enabled a CloudConnect integration with an OscarPro EMR, or
  • You have a custom integration with Ocean that receives webhook events for patient engagement and / or eReferral updates, AND
  • You have a firewall that prevents traffic from the public internet reaching either of the services mentioned above

The full article can be reviewed here. Check with your network administrator about whether this change affects you.

Please email us at support@cognisantmd.com if you have any additional questions.

 


Failures Uploading Attachments to Ocean April 13, 2022 [RESOLVED]

9:07am We're aware of an issue affecting some attachment uploads to Ocean and are investigating. We'll update this announcement as further information becomes available.

10:18am We are currently working with Cloudflare to investigate this issue. It looks like large file uploads are failing for some Cloudflare customers. We're trying to determine whether or not we are similarly impacted by a failure on Cloudflare's side.

Note that this issue also affected the eForm Editor; eForm 'save' actions are failing because they rely on the same functionality.

1:00pm Cloudflare has pushed a fix on their side that appears to have corrected the issue. Please resume regular usage of Ocean and let us know if you encounter any more trouble. We will be monitoring the platform for errors closely over the next few hours.

Please email us at support@cognisantmd.com if you have any additional questions.




Ocean Service Interruption January 10th, 2022 [RESOLVED]

9:19am We are currently experiencing an issue affecting Ocean patient reminder emails. We are investigating and will update this announcement as further information becomes available.

9:45am We applied a fix to our firewall configuration that should allow patients to follow Ocean links in reminder emails again.

3:29pm A new config was applied to our firewall service over the weekend that prevented patients from reaching Ocean by clicking urls contained in email sent by the Ocean platform. We have prepared a patch release to be deployed tonight that will prevent this issue from recurring.

9:50pm Patch release deployed.

Please email us at 
support@cognisantmd.com if you have any additional questions.


Ocean Portal Performance Degradation Dec 16, 2021 [RESOLVED]

At 3:25pm ET, we became aware of users experiencing slow load times of Ocean services. We have traced this issue to an issue being experienced by CloudFlare, which the Ocean Platform uses for proxying and Web Application Firewall services, among other things. We are currently monitoring the issue and will update this article as additional information becomes available.

3:50pm: Cloudflare has implemented a fix for this issue and is currently monitoring the results.

More information is available at Cloudflare's status page.

4:06pm: Cloudflare has resolved the issue on their side. Latency to Ocean appears to have returned to normal. We're continuing to monitor the issue.

 


Security Update: Log4j Vulnerability December 14th, 2021 (Updated Dec 20, 2021) [RESOLVED]

Early in the morning on Friday December 10th we became aware of a very serious security vulnerability found in a Java library we use in the Ocean platform called log4j2. (In this case severity is determined by how easy the vulnerability is to exploit as well as log4j2's widespread use.) Technical information on the vulnerability can be found here: https://www.lunasec.io/docs/blog/log4j-zero-day/

We wanted to update you as to how we are responding to this threat:

  • First and foremost it was determined after initial investigation that Ocean and Cloud Connect were not vulnerable to the most serious attack vectors identified in the research - our patch processes keep our server software up to date and these patches prevented exploitation of the worst attack variant.
  • On Friday night we released new patched versions of Ocean and Cloud Connect that included an upgrade to log4j 2.15, removing the remaining known attack vectors from the vulnerability in the Ocean platform.
  • Working with 3rd party vendors, we identified some additional tools and libraries in our server monitoring infrastructure that could also be indirectly vulnerable. These 3rd party tools were patched Monday night.
  • On Tuesday, further industry research led to the discovery of some remaining vulnerabilities in log4j version 2.15. Log4j 2.16 has been released to address these, which will be patched into Ocean platform Tuesday night. This eliminates the remaining known vulnerabilities.
  • On Thursday December 17th a 3rd party vendor we rely on notified us a patch was available for their tool. We applied the patch last night. (Note: There is no added exposure to the Ocean platform with respect to the log4j vulnerability from this tool.)
  • On Saturday December 18th we became aware of another update to the log4j library to mitigate an newly detected vulnerability. We upgraded to this version and applied the patch Sunday, Dec 19.

As of Dec 20 all vulnerable versions of log4j used in the Ocean platform (either directly or indirectly) have been fully patched. No action is required on your part.

After careful review of system logs, we find no evidence that the vulnerability was successfully used against the Ocean platform. While the known risks to the Ocean platform have been fully mitigated, we will continue to monitor the situation as our partners and software providers continue their investigation and mitigation efforts. This article will be updated if additional information becomes available.

Please contact support@cognisantmd.com if you have any questions or concerns.

 



Email Delivery to Cogeco Mailboxes Interrupted Sept 22nd, 2021 [RESOLVED]

2021-09-22 5:30pm ET: We are investigating an email delivery issue affecting some messages sent to @cogeco.ca and @cogeco.net addresses that began on Monday, September 20th. Delivery of secure messages and reminder emails to patients with Cogeco email addresses is impacted. We are contacting Cogeco Email Administrators to request that the block be lifted.

2021-09-24 The block has been lifted.

Please contact support@cognisantmd.com if you have any questions. We will update this article as new information comes available.


Patients Unable to Upload Attachments to Ocean [RESOLVED]

September 17th, 2021

8:45am EST: We've identified a bug that is preventing patients from uploading attachments to Ocean this morning after a new release of the Ocean Platform was deployed last night. We are investigating.

9:45am EST: The cause of this issue has been found and the fix will be deployed at 9:30pm EST.

Please email us at support@cognisantmd.com if you have any additional questions.


System Maintenance Planned Sept 3rd, 2021 Starting at 9:30pm EST [COMPLETED]

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 120 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.

 

--

11:15pm: All maintenance completed successfully.



System Maintenance Planned July 30th at 9:30PM EST [COMPLETED]

We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.

We'll update this article when maintenance is complete and Ocean is ready for use.



Changes to Ocean IP Addresses

CognisantMD is making infrastructure changes to increase the security, reliability, and performance of the Ocean platform. As a result the IP addresses used by Ocean and Cloud Connect will be changing.

This will not impact regular use of Ocean and/or Cloud Connect. However, if your site restricts outbound internet traffic via an allowlist, then you must make the changes below by June 30, 2021 to avoid any disruption to your use of the Ocean platform. This change is safe to make today.

Outbound Allowlist to be in place as of June 30, 2021:

  • 35.182.46.13
  • 15.222.173.33
  • 52.233.47.110
  • 13.71.164.223
  • 103.21.244.0/22
  • 103.22.200.0/22
  • 103.31.4.0/22
  • 104.16.0.0/13
  • 104.24.0.0/14
  • 108.162.192.0/18
  • 131.0.72.0/22
  • 141.101.64.0/18
  • 162.158.0.0/15
  • 172.64.0.0/13
  • 173.245.48.0/20
  • 188.114.96.0/20
  • 190.93.240.0/20
  • 197.234.240.0/22
  • 198.41.128.0/17

If you have any questions, please contact support@cognisantmd.com

 

2021-06-29: We will be temporarily postponing the planned changes to Ocean IP addresses, and will provide an updated system notification including a new target date shortly.

 

2021-07-30: We will be applying this change next Friday August 6th, 2021.


Ocean Service Interruption May 17, 2021 [RESOLVED]

1:54pm We are currently experiencing an issue affecting Ocean. We are investigating and will update this announcement as further information becomes available.

1:59pm The issue has been resolved and we are currently investigating the root cause. Note: Ocean will have experienced delays processing requests to EMRs via Cloud Connect. All pending updates will be synced to EMRs over the rest of the afternoon.

8:27pm Two issues impacted Ocean today. The first was a brief network interruption in our hosting facilities which caused the Ocean database to become unavailable for 2 minutes and 21 seconds at 1:11pm ET. No data was lost, but Ocean was unavailable during this period. The second was due to a failure between an EMR and the corresponding API server due to a very large volume of requests. The delay in processing the requests due to the failure caused a backup in Cloud Connect processing between 12:30pm and 1:55pm ET. During this period, Ocean features dependent on Cloud Connect EMR integration were delayed. 

Please email us at support@cognisantmd.com if you have any additional questions.



Ocean Processing Delays April 23, 2021 [Resolved]

At approximately 6am ET this morning, Ocean began experiencing irregular delays in processing requests to Cloud Connect. These delays impacted prompt processing of Ocean Reminders, patient note downloads to the EMR and patient appointment updates. No requests were lost. 

After investigation, it was determined that a defect in error handling in our SMS service was impacting Cloud Connect message processing. The SMS message that was triggering the bad error handling was removed and system processing returned to normal at 1:57pm ET.

We will be patching Ocean tonight to prevent recurrence of the issue and to insulate Cloud Connect processing from SMS.