This section contains critical updates for Ocean Site administrators.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
April 24, 2024, 10:20 AM EST: We are aware of the service interruption this morning that affected access to the Ocean Portal for users. The service interruption has since been resolved, and access to the Ocean Portal has been restored for all users.
Initial investigation of the interruption indicates a database connection issue. We have since made changes to resolve the issue, and will continue to test and investigate the root cause.
April 24, 2024, 4:35 PM EST: As part of our investigation, this was the sequence of events during this morning's service interruption:
Please email us at support@oceanmd.com if you have any additional questions.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
9:10am EST: We're investigating delayed webhook style integration event deliveries this morning, and will update this announcement as further information becomes available.
9:36am EST: We've identified the issue and are working on applying a patch to Ocean. There may be a 2 - 3 minute service interruption to Ocean to apply the fix in order to avoid delayed referral event deliveries.
10:08am EST: We're deploying a new Ocean release. We'll be monitoring closely to confirm the issue is fixed.
10:44am EST: An issue preventing delivery of referral integration notifications was identified this morning. The issue was resolved at 10:24am via an Ocean patch. Over the next hour all referral integration notifications will be delivered. The root cause was a regression introduced in a patch release deployed last night for an unrelated issue. Both issues are now corrected.
Please email us at support@oceanmd.com if you have any additional questions.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We became aware of an issue with Ocean eConsults, beginning on Saturday, April 15, 2023. Users submitting eConsults were encountering errors and, while the eConsult was successfully stored in Ocean, it was not transmitted to OTN immediately. The issue was due to an unintended configuration change on April 14th that blocked the eConsults from being sent. The issue was resolved at noon on Tuesday, April 18th and all eConsults submitted during the incident have since been transmitted to OTN successfully.
Please email us at support@oceanmd.com if you have any additional questions.
11:37am EST We are currently experiencing an issue affecting a subset of non-eReferral receiving Ocean Health Map directory listings. We are investigating and will update this announcement as further information becomes available.
Last Wednesday, Ontario Provincial Provider Registry sent a large directory listing update with bad or missing data that was imported by the Ocean platform corrupting these directory listings in the health map. We're investigating options to repair data.
2:24pm EST The issue has been traced to a corrupted Provincial Provider Registry data file that was processed by Ocean on March 30. We're working with Ontario Health to recover the lost data. They've provided new data which we are verifying and preparing to process.
We will provide an estimated time to recover full directory listing information to the health map shortly.
10:30pm EST We have successfully completed a data restore operation in production. Directory listings on the health map should now appear as they did before they were corrupted. We are resuming regular updates to the Ocean Health Map directory via the Provincial Provider Registry data source.
Please email us at support@oceanmd.com if you have any additional questions.
9:01am EST We are currently experiencing an issue affecting Ocean reminders that is impacting a small number of patient reminders this morning. We are investigating and will update this announcement as further information becomes available.
9:29am EST We have determined that all reminders (email and sms) have been sent to patients as normal. A bug has been identified where some patients will have received more than 1 reminder for an appointment. A fix is being prepared.
The bug was introduced last night as part of a regularly scheduled Ocean Platform upgrade.
9:45am EST A bugfix was deployed this morning to prevent duplicate reminders being sent. We're continuing to monitor the system closely.
4:49pm EST The issue with duplicate reminders being sent to patients is resolved.
Please email us at support@cognisantmd.com if you have any additional questions.
1:43 PM EST: We have identified an issue impacting the way that Ocean validates SPF records for custom email domains which allows Ocean Sites to email patients using their own email address. Impacted sites have received an email notification indicating that their address has become invalid.
This means that emails sent by Ocean since the validation failed will be sent by no-reply@cognisantmd.com instead of the custom email address to ensure that patients continue to receive emails. For clarity, the Patient Messaging and Patient Reminders products continue to function as expected, with the sender appearing as no-reply@cognisantmd.com.
We are actively working on a fix to restore the functionality to send from a custom email address and re-validate the affected Ocean Sites automatically.
2:25 PM EST: We have applied a fix for the validation issue, and the ability to send email using a custom 'From' email address has been restored for all Ocean Sites. The fix that was applied both automatically restores and re-validates any custom email addresses. No action is needed on behalf of Ocean Site Admins.
This bulletin will now be marked as resolved. Please email us at support@cognisantmd.com if you have any additional questions.
In response to the Ocean Portal Performance Degradation on December 9th 2022, we will be performing critical system maintenance between 10:00 PM and 12:00 AM EST tonight (December 12th, 2022) to adjust system resources and settings related to the Ocean database. During this time, Ocean will be intermittently offline and unavailable for use.
We'll update this article when maintenance is complete and Ocean is ready for normal use.
9:08 AM EST: We have identified a performance degradation within the Ocean Portal resulting in slow loading times. We are currently investigating the issue and will update this article as additional information becomes available.
9:35 AM EST: A potential resource allocation fix will be applied this morning, during which time the Ocean Portal will be unavailable for 3-5 minutes. This article will be updated when the downtime begins and ends.
9:43 AM EST: Downtime will begin at 9:45 AM EST. This article will be updated when downtime ends.
9:55 AM EST: Downtime has ended and the Ocean Portal is now available.
10:11 AM EST: We are continuing to monitor the performance of the Ocean Portal.
9:25 AM EST: We are aware of the continued performance degradation within the Ocean Portal resulting in slow loading times. We are continuing to investigate the root cause of the issue and will update this article as additional information becomes available.
10:20 AM EST: We continue to see heavy loads and contention on certain database resources. We will perform a database restart in the next 15 minutes to attempt to work around the issue in the short term.
10:35 AM EST: The database restart helped and our load monitoring shows much better performance since the restart. This morning’s database pressure has provided additional clues as to the root cause, which we will continue to investigate throughout the day.
4:41 PM EST: We will be performing Critical System Maintenance at 10:00 PM EST tonight to adjust system resources and settings related to the Ocean database. Ocean will be intermittently offline and unavailable for use during this time. The linked bulletin will be updated when maintenance is complete and Ocean is ready for normal use.
11:44 AM EST: The critical maintenance performed last night to adjust system resources and settings has improved the performance and reduced the pressure on the Ocean database. We are continuing to monitor.
11:15 AM EST: After an extended monitoring period, we can confirm that performance of the Ocean Portal has stabilized at normal levels. This system notification bulletin will now be marked as resolved.
Please email us at support@cognisantmd.com if you have any additional questions.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We are currently experiencing an issue affecting Ocean eReferrals and Website Forms. Email notifications for newly submitted website forms and eReferrals are not being received by the designated recipients due to a bug introduced in the November 24th Ocean platform release.
[Nov 29, 9:20 PM]: We have identified the root cause and are releasing a fix tonight. We will update this announcement as further information becomes available. Please email us at support@cognisantmd.com if you have any additional questions.
[Nov 30, 10:0 AM] We released the fix last night and have verified that email notifications are now being sent for newly submitted website forms and eReferrals. Ocean is also in the process of sending any email notifications that were missed since November 24th.
Please email us at support@cognisantmd.com if you have any additional questions.
At 9am, we became aware of an issue for some sites using integrations on the FHRI APIs or Open APIs receiving referrals that in some cases the webhook notification may not be issued. This could mean that referrals have been received but downstream systems have not been notified. We are investigating as an urgent issue and will post more information as it becomes available.
In the interim, integrated sites should access the Ocean Portal's eReferral View to monitor for new referrals and process existing referrals.
10:30am We've identified an issue in Ocean related to referral webhook notifications and are working on a fix.
2:10pm: We have deployed a fix to our production environment to address the core issue. We will be investigating methods to re-trigger missed notifications and will update this article as we investigate.
5:00pm As of 2:15pm, Webhook notifications for new and updated referrals to FHIR integrated sites are being sent successfully. Outbound API messages that weren't sent since last night starting at 10pm will be re-triggered in 2 steps:
Note: We'll be spacing out the updates so they're not sent at the same time.
In partnership with the eServices Program, we will be providing FHIR integrated sites with a report of affected referrals that can be used to verify all updates have been downloaded to EMRs.
November 5, 2022 2:00pm: Missed webhook notifications to integrated sites have been re-sent. We have verified the successful delivery of these API messages and completed our planned data repair activity for this event.
Please email us at support@cognisantmd.com if you have any additional questions.
At 10:40am EST, we became aware of users experiencing degraded performance with Cloud Connect. We traced the issue to a particular workflow within Ocean. A fix has been implemented, and normal Cloud Connect performance has been restored.
Please email us at support@cognisantmd.com if you have any additional questions.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 60 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We have identified a bug in a recent, limited release Ocean toolbar for TELUS PS Suite. This toolbar, which includes a link to the Ontario eConsult Listing (OTNHub), may prevent referral updates from appearing in the patient chart.
Clinics that downloaded and installed this toolbar between August 15 - 17 should immediately update their toolbar. Instructions for updating the basic Ocean Toolbar can be found here (please note this version does not include the OTNHub link).
To learn more or receive a link to download the Ocean toolbar linking to the Ontario eConsult Listing in Ocean (OTNHub), please contact the Ontario eServices Program here.
If you require any further assistance, please contact our support team at ocean.tips/support.
Old Ocean platform IP addresses retired. The full article and related change log can be reviewed here. Check with your network administrator about whether this change affects you.
Please email us at support@cognisantmd.com if you have any additional questions.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
New IP addresses have been added to the Ocean platform. You may be impacted if:
The full article can be reviewed here. Check with your network administrator about whether this change affects you.
Please email us at support@cognisantmd.com if you have any additional questions.
9:07am We're aware of an issue affecting some attachment uploads to Ocean and are investigating. We'll update this announcement as further information becomes available.
10:18am We are currently working with Cloudflare to investigate this issue. It looks like large file uploads are failing for some Cloudflare customers. We're trying to determine whether or not we are similarly impacted by a failure on Cloudflare's side.
Note that this issue also affected the eForm Editor; eForm 'save' actions are failing because they rely on the same functionality.
1:00pm Cloudflare has pushed a fix on their side that appears to have corrected the issue. Please resume regular usage of Ocean and let us know if you encounter any more trouble. We will be monitoring the platform for errors closely over the next few hours.
Please email us at support@cognisantmd.com if you have any additional questions.
We will be upgrading server components for the Ocean Portal beginning at 10:00pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
We will be upgrading server components for the Ocean Portal beginning at 9:30pm EST. We expect this work will take no more than 30 minutes during which time Ocean may be intermittently offline.
We'll update this article when maintenance is complete and Ocean is ready for use.
9:19am We are currently experiencing an issue affecting Ocean patient reminder emails. We are investigating and will update this announcement as further information becomes available.
9:45am We applied a fix to our firewall configuration that should allow patients to follow Ocean links in reminder emails again.
3:29pm A new config was applied to our firewall service over the weekend that prevented patients from reaching Ocean by clicking urls contained in email sent by the Ocean platform. We have prepared a patch release to be deployed tonight that will prevent this issue from recurring.
9:50pm Patch release deployed.
Please email us at support@cognisantmd.com if you have any additional questions.
At 3:25pm ET, we became aware of users experiencing slow load times of Ocean services. We have traced this issue to an issue being experienced by CloudFlare, which the Ocean Platform uses for proxying and Web Application Firewall services, among other things. We are currently monitoring the issue and will update this article as additional information becomes available.
3:50pm: Cloudflare has implemented a fix for this issue and is currently monitoring the results.
More information is available at Cloudflare's status page.
4:06pm: Cloudflare has resolved the issue on their side. Latency to Ocean appears to have returned to normal. We're continuing to monitor the issue.