This bulletin is to notify all users of any known issues real-time, it will detail when issues arise and when they have been resolved. Please ensure you "follow" this article for automated e-mail updates.
| Date | Description |
|
27th August 2024 |
Update 13:00hrs GMT: The issues with database performance have been resolved. |
|
27th August 2024 |
As of 08:45hrs GMT: The root cause of the issue has been found to be performance issues with areas of the database. A fix for this issue is currently being worked on. |
|
27th August 2024 |
Cloud Services and Evidence Locker have been experiencing issues with user verification, currently disrupting sign-ins as of 07:00hrs GMT. We are currently working on a fix for the issue. |
|
15th August 2024 |
There will be some slight disruption between 2100-2130hrs GMT on 15/08/24. This is for updates to resolve a few issues that we've identified after the C24.4 release, specifically in Cloud Services within the Telematics Page where it wasn't displaying historic queries for GPS data. This has now been resolved. |
|
10th August 2024 |
In anticipation of our C24.4 release, some of the services may be impacted between 0700-0900hrs GMT on 14/08/24. Release notes can be found here: Release C24.4 14/08/24 – Global Support Portal (synecticsglobal.com) |
|
9th August 2024 |
Off the back of the IoT function pipeline update, below there was no reported downtime. All services are running. |
|
8th August 2024 |
In anticipation of our upcoming UI front-end updates to our Cloud Platform we will be updating our IoT functions pipeline at 0800hrs GMT on 09/08/24. Every production tenancy could be impacted with momentary downtime of up to 10 minutes subject to any issues surrounding the deployment. |
|
30th July 2024 |
Microsoft declared the incident resolved at 20:48 UTC. All issues accessing the cloud platform should be resolved. |
|
30th July 2024 |
Microsoft has provided an update that explains the issue has become worldwide and they are working on solving the issue. Quote: "We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible. This message was last updated at 13:13 UTC on 30 July 2024" |
|
30th July 2024 |
Microsoft is experiencing network issues with Azure meaning all European users are being affected. Quote: "We are investigating reports of issues connecting to Microsoft services in Europe. More details will be provided as they become available." This issue could temporarily cause issues when attempting to connect to cloud services. |
|
23rd July 2024 |
We noticed a few issues with thumbnail generation within the Cloud Evidence Locker for incidents created between the 17th and 23rd July 24, whilst these videos will playback fine the thumbnails won't generate. This issue has been resolved and all future incidents will generate thumbnails in the Cloud Evidence Locker as expected. |
|
14th March 2024 |
Our Azure Kubernetes Service is retiring - specifically versions: v1.26.x on 31st March 2024. In anticipation of this we will be upgrading our production deployments to 1.27.9 between 2200-0000hrs GMT on 14/03/24. Every production tenancy will be impacted with momentary downtime of up to 10 minutes. |
|
20th November 2023 |
There is an upgrade planned across multiple Cloud Services and Cloud Evidence Locker sites today, Monday 20th November. The update should only take a few minutes. If you are trying to use Cloud Evidence Locker or other Cloud Services during this time you may see an error. If this happens please wait a few minutes and try again. |
|
16th November 2023 |
Redaction services are currently unavailable. We are working on a resolution and will update this bulletin with further news. |
|
16th November 2023 09:38am |
UPDATE - sites not upgraded today will be planned for upgrade from Monday 20th November.
|
|
16th November 2023 |
10am GMT, Thursday 16th November - There is an upgrade planned across multiple Cloud Services and Cloud Evidence Locker sites this morning, commencing at approximately 10:00am GMT. The update should only take a few minutes. If you are trying to use Cloud Evidence Locker or other Cloud Services during this time you may see an error. If this happens please wait a few minutes and try again. |
|
29th September 2023 |
The upgrade on 28th September completed successfully. |
|
27th September 2023 |
7pm BST, Thursday 28th September - there will be an upgrade to our back-end Kubernetes services to improve security. There are no user-facing changes. This upgrade will take approximately 30 minutes. There should be minimal disruption to services during this upgrade. In case of any errors, please log out and try again in a few minutes. |
|
7th September 2023 |
We will be upgrading our redaction services for Cloud Evidence Locker week commencing 11th and 18th September. During this time the redaction functionality may be unavailable. If you have persistent issues with redaction during this time please contact your local support team. |
|
29th June 2023 10:46 |
The planned maintenance upgrade is completed and all services are operational. |
|
27th June 2023 |
There is a maintenance upgrade planned for Thursday 29th June at approximately 10:00am BST. This is an update to the backend IoT device management; there are no user-facing changes. The update should only take a few minutes. If you are trying to use Cloud Services during this time you may see an error. If this happens please wait a few minutes and try again. |
|
15th June 2023 |
Planned maintenance upgrade completed successfully |
|
13th June 2023 11:35 |
There is a maintenance upgrade planned for Thursday 15th June at approx 11:30am BST. This is a database update only; there are no user-facing changes. The update should only take a few minutes. If you are trying to use Cloud Services during this time you may see an error. If this happens please wait a few minutes and try again. |
|
26th May 2023 15:37 |
Email notifications are being received OK. |
|
26th May 2023 |
We've received reports that some users are not receiving email notifications. We are investigating the root cause. |
|
15th May 2023 |
There are two maintenance upgrades planned this week: 10am BST, Tuesday 16th May - Synectics Cloud Media Services maintenance upgrade 7pm BST, Thursday 18th May - Kubernetes upgrade to improve security. This upgrade will take approximately 15 minutes. There should be minimal disruption to services for both these upgrades. |
|
21st March 2023 |
A maintenance update for T2000 health data will be carried out from 8am to 2pm (GMT) on 21st March 2023. Any downtime will be minimal and should not be notice-able to most users. |
|
20th March 2023 |
A maintenance update for T2000 health data will be carried out this afternoon (GMT). Any downtime will be minimal and should not be notice-able to most users. |
|
16th March 2023 |
All cloud services have been updated and are online. |
|
14th March 2023 |
Regular maintenance upgrades are scheduled for Synectics Cloud Services and Cloud Evidence Locker from 08:00 GMT on Wednesday 15th March and Thursday 16th March. Loss of access to Cloud Services during this time should be minimal and operations may be slower during this period. |
|
8th February 2023 |
Update 12:11 GMT: Maps are back online in Cloud Services (e.g. Telematics Mapping) |
|
8th February 2023 |
Some sites are seeing a message "This page can't load Google Maps correctly" when trying to view maps. We are actively investigating the root cause and will update this bulletin with further info. |
|
4th November 2022 |
Regular maintenance upgrades are scheduled for Synectics Cloud Services from 18:00 GMT on Tuesday 8th November. There will be 2 planned maintenance windows of 2 hours each; loss of access to Cloud Services during this time should be minimal and operations may be slower during this period. |
|
3rd November 2022 |
Regular maintenance upgrades are scheduled for Synectics Cloud Services from 18:00 GMT on Thursday 3rd November. There will be 2 planned maintenance windows of 2 hours each; loss of access to Cloud Services during this time should be minimal and operations may be slower during this period. |
|
27th October 2022 |
Update: As of 17:30, the issues have been resolved. Cloud Services are back to being operational. |
|
27th October 2022 |
A problem has been identified with a recent update for optimising data ingestion. As of 10:00, we have identified the issue and are in the process of rolling back the upgrade. |
|
22nd June 2022 |
Update: email services are fully operational |
|
21st June 2022 10:44 |
Update: The 'SendGrid' email component is now operational. We are assessing the impact to emailing from Cloud Evidence Locker. |
|
20th June 2022 |
An issue was reported with the 'SendGrid' component used to send emails within our Cloud Services including Cloud Evidence Locker. This meant emails generated by Cloud Evidence Locker from 6am this morning were not sent. We are tracking the issue with the provider of SendGrid. |
|
9th March 2022 (09:00 - 09:44) |
Update: Overnight back-end upgrades caused an issue with dependent front-end cloud services. We identified the dependency and rolled-back the upgrade. Mitigation: the inter-dependency has been identified and steps taken to ensure future updates will not impact front-end services. Cloud Services back online 09:44 |
| 23rd December 2021 (13:09) |
A recent Azure upgrade to the AKS for the Cloud Evidence Locker cluster reset our VM types back to a basic level. We have changed the VM types back to the higher specification level. We are implementing process improvements to ensure this never happens again. Cloud Services back online. |
| 23rd December 2021 (09:00) | We have identified that all of our Cloud tenancies are offline. We are working to resolve this as soon as possible. |
|
14th December 2021 (16:00) |
Update: Microsoft Azure have determined that a recent deployment task impacted a backend service configuration which caused devices to disconnect and fail to reconnect to IoT Hub. Mitigation: We rolled back the deployment task to mitigate this issue. Cloud Services back online. |
|
13th December 2021 (23:41) |
We have been made aware by Microsoft that there is an ongoing issue, starting 23:41GMT on the 13th December, specifically the UK West region IoT Hub. This is affecting our Cloud Services platform with no vehicles being able to connect to our platform. Current status from Microsoft: We are aware of the issue and are actively investigating. Next update will be provided in 60 minutes or as events warrant. |
| 6th October 2021 (11:18) |
The issue was due to the expiry of a legacy certificate, as a result access to services failed. There were complications with the existing certificate manager which have now been resolved; we are now acquiring certificates that validated automatically. Cloud Services back online. |
| 5th October 2021 (12:18) | We are aware of some instances of our Cloud Services including Cloud Evidence Locker being offline. We are working to resolve this as soon as possible. |
| 25th May 2021 (15:20) |
Our hosting provider has narrowed down possible causes for this system outage and is working as a priority on identifying the root cause. Cloud Services back online. |
| 24th May 2021 (10:00) | We are aware of some instances of our Cloud Services including Cloud Evidence Locker being offline. We are working to resolve this as soon as possible. |