Systems Incidents
Updated: September 16, 2024 12:30 pm
16.09.2024: Considerable slowing down of creating products in Cobuilder Supply
Dear customers,
On 16.09.2024, between 9:00 and 13:48 CET we experienced a significant delay in the creation of products in Cobuiler Supply that appeared after the migration to the Norwegian data dictionary. All other services worked fine.
The normal work of the system has been restored.
Event Start: discovered 16/09/24, from 09:00 CET
Event End: 16/09/24, from 13:48 CET
Duration: 4 hours and 54 minutes
Affected Platforms: Cobuilder Supply
Reproducibility: always
19.08.2024: Not possible to open Cobuilder Collaborate
Dear customers,
On 19.08.2024, between 8:00 and 8:54 CET, for a total downtime of 54 minutes (working hours), we experienced a critical incident with Collaborate – opening the service did not function. After clicking on Start Collaborate, the users were sent back to the login page. All other services worked fine.
The normal work of the system has been restored.
Event Start: discovered 19/08/24, from 08:00 CET
Event End: 19/08/24, from 08:54 CET
Duration: 54 minutes
Affected Platforms: Cobuilder Collaborate
Reproducibility: always
31.07.2024: Slow performance in Cobuilder Collaborate
Dear customers,
On 31/07/2024, from 07:22 CET to 08:52 CET we experienced an issue for a total downtime of 1h30min (52 mins within office hours). The issue was slow performance in Cobuilder Collaborate.
The normal work of the system has been restored.
Event Start: discovered 31/07/24, from 07:22 CET
Event End: 31/07/24, from 08:52 CET
Duration: 52 minutes
Affected Platforms: Cobuilder Collaborate
Reproducibility: always
30.07.2024: Cobuilder Platform doesn’t load
Dear customers,
On 30/07/2024, from 14:05 CET to 16:07 CET we experienced an issue with loading the platform for a total downtime of 2h02min. Logging in for all users was thus impossible.
The Root cause was issue with the Azure portal, more precisely – with Azure Front Door.
Affected solutions: Portal, Collaborate, Supply DM, Define, Link – all that require authentication.
None of the Public views in Supply DM were affected.
Event Start: discovered 30/07/24, from 14:05 CET
Event End: 30/07/24, from 16:07 CET
Duration: 2h and 2 minutes
Affected Platforms: All
Reproducibility: always
30.07.2024: Scanning QR code not working in Cobuilder Collaborate
Dear customers,
On 30/06/2024, from 07:18 CET to 10:20 CET there was an accident where scanning QR codes in Collaborate led to a not found public page, instead of the public list of chemicals. The incident lasted 2 hours and 20 minutes during our working hours.
A hotfix has been deployed and Cobuilder Collaborate is fully functioning.
Event Start: discovered 30/07/24, from 07:18 CET
Event End: 30/07/24, from 10:20 CET
Duration: 3h and 2 minutes
Affected Platforms: Collaborate
Reproducibility: always
29.05.2024: Cobuilder Platform doesn’t load
Dear customers,
On 29/05/2024, from 09:33 CET to 09:46 CET there was no access to Cobuilder Platform.
The Portal API failed to retrieve OpenID configuration metadata from the Azure Active Directory endpoint.
Access to Cobuilder Platform has now been restored.
Event Start: discovered 29/05/24, from 09:33 CET
Event End: 29/05/24, from 09:46 CET
Duration: 13 minutes
Affected Platforms: All
Reproducibility: always
23.04.2024: Cobuilder Collaborate not loading
Dear customers,
Today, from 08:00 CET to 08:08 CET there was no access to Collaborate. The incident was detected after the standard release process.
The main cause of the problem was the incorrect start of the application due to a newly introduced command in the new version. The bug was that the command only worked correctly for test and staging environments, but not for Prod. This is the reason why we did not recognize it in testing.
Access to Collaborate has now been restored.
Event Start: discovered 23/04/24, from 08:00 CET
Event End: 23/04/24, from 08:08 CET
Duration: 8 minutes
Affected Platforms: Collaborate
Reproducibility: always
10.04.2024: When creating a new version of a template with foreign classification, the code relations are dissapearing from the new version
Dear customers,
On 10.04.2024 16:20 CET a major bug was identified in Define. When updating Data templates, if they are classified with codes from another data dictionary, the classification codes disappear from the new version of the template.
On 12.04.2024, 8:15 CET a fix has been deployed and the problem has been resolved. No data loss.
Event Start: discovered 10/04/24, from 16:20 CET
Event End: 12/04/24, from 08:15 CET
Duration: 39h 55 minutes
Affected Platforms: Define
Reproducibility: always
10.04.2024: Codes in Cobuilder Link are mixed when two objects with the same classification are updated
Dear customers,
On 10/04/24, 16:00 CET a bug was found in Link that the new released functionality for bulk updating Define objects and properties to latest version is not working properly and assigning wrong classification to objects.
On 12/04/24, 10:53 CET The fix that was developed has been completely tested, no sequential issues were found.
Event Start: discovered 10/04/24, from 16:00 CET
Event End: 12/04/24, from 10:53 CET
Affected Platforms: Cobuilder Link
Reproducibility: always
15.06.2023: WCF service on Connect does not work
Dear customers,
Today, 15/06/23, from 16:13 CET to 16:32 CET we discovered a problem with the WCF service on Connect. The affected affected accounts received an error when trying to send orders through Connect.
The problem was caused by an infrastructural issue and has now been fixed. Тhe service is working normally again.
Event Start: discovered 15/06/23, from 16:13 CET
Event End: 15/06/23, from 16:32 CET
Duration: 19 minutes
Affected Platforms: Connect, Collaborate
Reproducibility: always
02.06.2023: Login to Platform
Dear customers,
On 02/06/23, from 10:13 CET to 10:26 CET, login to the platform was not possible for a total downtime of 13 minutes. The root cause was a network/connectivity issue.
This is now resolved, and the platform login is functioning normally.
Event Start: discovered 02/06/2023, 10:13 CET
Event End: 02/06/2023, 10:26 CET
Duration: 13 minutes
Affected Platforms: Platform
Reproducibility: always
17.05.2023: Report download access
Dear customers,
On 17/05/2023, 9:00 CET it was discovered that a shared folder of Cobuilder (Azure blob storage) theoretically could be accessed externally, given that one already had a correct GUID number to use it to revise the URL address.
The access to the folder was stopped and the configuration was reviewed for ensuring it was correctly protected. All the report files that could have been accessed are now moved to a secured storage and Cobuilder could provide details to concerned clients upon request.
The team is performing additional security measures to further secure the report download mechanism.
24.04.2023: Users get returned to welcome page
Dear customers,
On 24/04/2023, 9:26 CET it was discovered a problem where users get returned to the welcome page.
The problem has already been fixed.
This is now resolved, and the systems are functioning normally.
Event Start: discovered 16/03/2023, 9:26 CET
Event End: 16/03/2023, 9:28 CET
Duration: 2 minutes
Affected Platforms: all
Reproducibility: always
16.03.2023: Logging into the platform
Dear customers,
On 16/03/2023, 9:07 CET it was discovered a problem with logging into the platform.
The problem was caused by network connectivity issue most probably in Azure, that occurred between 8:56 and 9:11.
It is verified that the problem has already been fixed.
This is now resolved, and the systems are functioning normally.
Event Start: discovered 16/03/2023, 9:07 CET
Event End: 16/03/2023, 9:23 CET
Duration: 16 minutes
Affected Platforms: all
Reproducibility: always
14.03.2023: Data Dictionaries low speed
Dear customers,
On 14/03/2023, it was discovered that all of the Data Dictionaries (Public View Pages) were loading much slower than usual. The event began at 16:11 CET and lasted until 15/03/2023, 08:00 CET, for a total downtime of 15h 49min. The business hours downtime was 49min.
There was an issue with the configuration on production environment.
Engineering have successfully deployed a hotfix on production environment this morning which has solved the issue. The Data Dictionaries are working speedy and without problems!
Event Start: discovered 14/03/2023, 16:11 CET
Event End: 15/03/2023, 08:00 CET
Duration: 15 hour and 49 minutes
Business hours downtime: 00h 49min
Affected Platforms: Data Dictionaries (Public View Pages)
Reproducibility: always
10.03.2023: Synchronization from CRM to the platform fails
Dear customers,
On 10/03/2023, 14:20 CET it was discovered a problem with synchronization from CRM to the platform.
This is now resolved, and the systems are functioning normally.
Event Start: discovered 10/03/2023, 14:20 CET
Event End: 10/03/2023, 15:40 CET
Duration: 1 hour and 20 minutes
Affected Platforms: all
Reproducibility: always
20.12.2022: Users get returned to the welcome page
Dear customers,
Users get returned to the welcome page in an attempt to login. The access got restored by restarting the app service.
Event Start: discovered 20/12/2022, 08:11 CET
Event End: 20/12/2022, 08:24 CET
Duration: 13min
Affected Platforms: all
Reproducibility: always
03.11.2022: Issue with the mail server – emails don’t get sent on production
Dear customers,
On 03/11/2022 from around 07:00 until 10:36 CET, for a total down time of 03h 36min (business hours downtime: 02h 36min), emails did not get sent. These can be emails about products delivered in FREE, new user invitations, invitations on projects etc. There was another issue with the mail server. On 01/11 Engineering switched to another mail server to fix the issue we experienced then. Today they discovered that emails did not get sent through it. The issue was in the user, and after resetting it, the issue was fixed. Nevertheless, our team switched back to our old mail server for now, and emails are once again successfully sent.
Event Start: discovered 03/11/2022, 07:00 CET
Event End: 03/11/2022, 10:36 CET
Duration: 03h 36min
Business hours: downtime: 02h 36min
Affected Customers: all
Reproducibility: always
Root cause: issue with the mail server
01.11.2022: Issue with the mail server – emails don’t get sent on production
Dear customers,
On 01/11/2022 from 11:11 until 13:14 CET for a total of 2h13min it was not possible to send emails on production. The reason for the issue was in our mail server and is now temporarily fixed as Engineering changed the microservices to use another smtp server when sending emails.
Event Start: discovered 01/11/2022, 11:11 CET
Event End: 01/11/2022, 13:14 CET
Duration: 02h 13min
Affected Customers: all
Reproducibility: always
Root cause: issue with the mail server
25.10.2022: Some users in Define cannot open and execute specific actions
Dear customers,
On 25/10/2022, from 08:23 CET until 10:47 CET for a total downtime of 02h 24min, some users in Define could not open and execute specific actions. Some users were not able to login, but the QAs fixed that quickly. One user cannot open some concepts. Another one cannot approve some sharing requests.
Another, when trying to share concepts from Cobuilder context to other contexts, the list with context that they can share to is empty. All these issues appear to different users and different contexts.
The issue was due to the fact that the new endpoints which return info about subscription, role and permission, don’t return info about users without First Name in CRM. By adding the missing fields and syncing the user, the problem is resolved.
Event Start: discovered 25/10/2022, 08:23 CET
Event End: 25/10/2022, 10:47 CET
Duration: 02h 24min
Affected Customers: Define
Reproducibility: always
Root cause: new endpoints not returning information about user with missing information fields
11.10.2022: Еrror message when logging in to goBIM
Dear customers,
On 11/10/2022, from 09:33 CET until 10:07 CET, logging in to goBIM was not possible for a total downtime of 00h 34min. Users got an error message when opening goBIM.
The reason for the issue was a removed microservice that was released before the release of goBIM. This means that goBIM tried to use an endpoint that does not exist in microservices and for that reason threw exceptions to the users. The issue was fixed with a new release of goBIM that doesn’t use the removed endpoint.
Event Start: 11/10/2022
Event End: 11/10/2022
Duration: 34 min
Affected Customers: Cobuilder goBIM
Reproducibility: always
02.08.2022: Multiple emails are sent to Cobuilder FREE users
Dear customers,
Cobuilder Support has been receiving many inquiries from FREE users. They have been receiving emails regarding orders delivered in Cobuilder FREE. The email is legit, but looks like a spam/phishing email. Email subject: Delivery documentation has been received. The amount of emails increased after 10/08/2022. Engineering is working on a new email template.
Event Start: 02/08/2022
Event End: 11/08/2022
Affected Customers: Cobuilder FREE users
Reproducibility: always
27.07.2022: Login to goBIM not possible, documents in Collaborate did not load
Dear customers,
On 27/07/2022 from 10:16 CET until 11:05 CET login to the goBIM platform was not possible for a total downtime of 49 minutes. Documents connected to products in Collaborate did not load either. The issue was caused by updates in Azure security, a couple of internal legacy services (goBim and Collaborate still depend on) had to be reconfigured, so they can still connect to our Key Vaults in Azure and extract their sensitive configurations from there.
Event Start: discovered 27/07/2022, 10:16 CET
Event End: 27/07/2022, 11:05
Duration: 00h 49min
Affected Customers: goBIM, Collaborate
Reproducibility: always
Internal tracking:
21.07.2022: Multiple systems have problems loading
Dear customers,
On 21/07/2022 at 07:21 CET, it was discovered that multiple systems have problems loading for a total downtime of 1 hour and 2 minutes. The downtime during business hours was 23 minutes. Products on projects in Collaborate were not loading. When users log in to goBIM, they got returned to the main page. There was also an issue with Define. Engineering found that the cause was connectivity issue with Microsoft Azure. The platforms are now functioning as they should. Microsoft have not confirmed the issue yet.
Event Start: discovered 21/07/2022, 07:21 CET
Event End: 21/07/2022, 08:23 CET
Duration: 01h 02min
Affected Customers: goBIM, Collaborate, Define
Reproducibility: always
Internal tracking:
17.06.2022: Log in to goBIM
Dear customers,
On 17.06.2022, login to goBIM was not possible from 07:52 to 08:03 CET for a total downtime of 11 minutes. The user was stuck in an infinite login loop.
After restarting the CRM API Registration service, login started working again.
Event Start: discovered 07:52 CET
Event End: 08:03 CET
Duration: 11 minutes
Affected Customers: goBIM
Reproducibility: always
Internal tracking:
05.05.2022: Log in to goBIM
Dear customers,
On 05.05.2022, from 07:52 CET until 08:58 CET, goBIM didn’t load for a total downtime of 01h 06min. Login to goBIM was not possible, users were caught in a login loop.
The problem appeared in the connection to the internal Login service which keeps the customer credentials. Restoring the connection fixed the problem.
Event Start: discovered 07:52 CET
Event End: 08:58 CET
Duration: 1 hour 6 minutes
Affected Customers: goBIM
Reproducibility: always
Internal tracking: task 144848
01.03.2022: Log in to Cobuilder platform
Dear customers,
On 01.03.2022 between 14:21 and 14:41 CET we experienced an issue where the login page couldn’t load (20min). All users of the platform were affected. Login to the page is possible now.
Event Start: discovered 14:21 CET
Event End: 14:41 CET
Duration: 20 minutes
Affected Customers: all platform users
Reproducibility: always
Internal tracking: task 140280
Service Requests
In support of services outlined in the Service Level Agreement, Cobuilder will respond to service related incidents and/or requests submitted by the Customer within the following time frames:
- Two (2) hour (during business hours) for issues classified as Critical.
- Eight (8) hours (during business hours) for issues classified as High
- Three (3) business days (during business hours) for issues classified as Medium
- Three (3) business days (during business hours) for a general service Request.