Systems Incidents

Updated: July 31, 2023 1:10 pm

15.06.2023: WCF service on Connect does not work

Dear customers,

Today, 15/06/23, from 16:13 CET to 16:32 CET we discovered a problem with the WCF service on Connect. The affected affected accounts received an error when trying to send orders through Connect.

The problem was caused by an infrastructural issue and has now been fixed. Тhe service is working normally again.

Event Start: discovered 15/06/23, from 16:13 CET
Event End: 15/06/23, from 16:32 CET
Duration: 19 minutes
Affected Platforms: Connect, Collaborate
Reproducibility: always

02.06.2023: Login to Platform

Dear customers,

On 02/06/23, from 10:13 CET to 10:26 CET, login to the platform was not possible for a total downtime of 13 minutes. The root cause was a network/connectivity issue.

This is now resolved, and the platform login is functioning normally.

Event Start: discovered 02/06/2023, 10:13 CET
Event End: 02/06/2023, 10:26 CET
Duration: 13 minutes
Affected Platforms: Platform
Reproducibility: always

17.05.2023: Report download access

Dear customers,

On 17/05/2023, 9:00 CET it was discovered that a shared folder of Cobuilder (Azure blob storage) theoretically could be accessed externally, given that one already had a correct GUID number to use it to revise the URL address.

The access to the folder was stopped and the configuration was reviewed for ensuring it was correctly protected. All the report files that could have been accessed are now moved to a secured storage and Cobuilder could provide details to concerned clients upon request.

The team is performing additional security measures to further secure the report download mechanism.

24.04.2023: Users get returned to welcome page

Dear customers,

On 24/04/2023, 9:26 CET it was discovered a problem where users get returned to the welcome page.

The problem has already been fixed.

This is now resolved, and the systems are functioning normally.

Event Start: discovered 16/03/2023, 9:26 CET
Event End: 16/03/2023, 9:28 CET
Duration: 2 minutes
Affected Platforms: all
Reproducibility: always

16.03.2023: Logging into the platform

Dear customers,

On 16/03/2023, 9:07 CET it was discovered a problem with logging into the platform.

The problem was caused by network connectivity issue most probably in Azure, that occurred between 8:56 and 9:11.

It is verified that the problem has already been fixed.

This is now resolved, and the systems are functioning normally.

Event Start: discovered 16/03/2023, 9:07 CET
Event End: 16/03/2023, 9:23 CET
Duration: 16 minutes
Affected Platforms: all
Reproducibility: always

14.03.2023: Data Dictionaries low speed

Dear customers,

On 14/03/2023, it was discovered that all of the Data Dictionaries (Public View Pages) were loading much slower than usual. The event began at 16:11 CET and lasted until 15/03/2023, 08:00 CET, for a total downtime of 15h 49min. The business hours downtime was 49min.

There was an issue with the configuration on production environment.

Engineering have successfully deployed a hotfix on production environment this morning which has solved the issue. The Data Dictionaries are working speedy and without problems!

Event Start: discovered 14/03/2023, 16:11 CET
Event End: 15/03/2023, 08:00 CET
Duration: 15 hour and 49 minutes
Business hours downtime: 00h 49min
Affected Platforms: Data Dictionaries (Public View Pages)
Reproducibility: always

10.03.2023: Synchronization from CRM to the platform fails

Dear customers,

On 10/03/2023, 14:20 CET it was discovered a problem with synchronization from CRM to the platform.

This is now resolved, and the systems are functioning normally.

Event Start: discovered 10/03/2023, 14:20 CET
Event End: 10/03/2023, 15:40 CET
Duration: 1 hour and 20 minutes
Affected Platforms: all
Reproducibility: always

20.12.2022: Users get returned to the welcome page

Dear customers,

Users get returned to the welcome page in an attempt to login. The access got restored by restarting the app service.

Event Start: discovered 20/12/2022, 08:11 CET
Event End: 20/12/2022, 08:24 CET
Duration: 13min
Affected Platforms: all
Reproducibility: always

03.11.2022: Issue with the mail server – emails don’t get sent on production

Dear customers,

On 03/11/2022 from around 07:00 until 10:36 CET, for a total down time of 03h 36min (business hours downtime: 02h 36min), emails did not get sent. These can be emails about products delivered in FREE, new user invitations, invitations on projects etc. There was another issue with the mail server. On 01/11 Engineering switched to another mail server to fix the issue we experienced then. Today they discovered that emails did not get sent through it. The issue was in the user, and after resetting it, the issue was fixed. Nevertheless, our team switched back to our old mail server for now, and emails are once again successfully sent.

Event Start: discovered 03/11/2022, 07:00 CET
Event End: 03/11/2022, 10:36 CET
Duration: 03h 36min
Business hours: downtime: 02h 36min
Affected Customers: all
Reproducibility: always
Root cause: issue with the mail server

01.11.2022: Issue with the mail server – emails don’t get sent on production

Dear customers,

On 01/11/2022 from 11:11 until 13:14 CET for a total of 2h13min it was not possible to send emails on production. The reason for the issue was in our mail server and is now temporarily fixed as Engineering changed the microservices to use another smtp server when sending emails.

Event Start: discovered 01/11/2022, 11:11 CET
Event End: 01/11/2022, 13:14 CET
Duration: 02h 13min
Affected Customers: all
Reproducibility: always
Root cause: issue with the mail server

25.10.2022: Some users in Define cannot open and execute specific actions

Dear customers,

On 25/10/2022, from 08:23 CET until 10:47 CET for a total downtime of 02h 24min, some users in Define could not open and execute specific actions. Some users were not able to login, but the QAs fixed that quickly. One user cannot open some concepts. Another one cannot approve some sharing requests.

Another, when trying to share concepts from Cobuilder context to other contexts, the list with context that they can share to is empty. All these issues appear to different users and different contexts.

The issue was due to the fact that the new endpoints which return info about subscription, role and permission, don’t return info about users without First Name in CRM. By adding the missing fields and syncing the user, the problem is resolved.

Event Start: discovered 25/10/2022, 08:23 CET
Event End: 25/10/2022, 10:47 CET
Duration: 02h 24min
Affected Customers: Define
Reproducibility: always
Root cause: new endpoints not returning information about user with missing information fields

11.10.2022: Еrror message when logging in to goBIM

Dear customers,

On 11/10/2022, from 09:33 CET until 10:07 CET, logging in to goBIM was not possible for a total downtime of 00h 34min. Users got an error message when opening goBIM.

The reason for the issue was a removed microservice that was released before the release of goBIM. This means that goBIM tried to use an endpoint that does not exist in microservices and for that reason threw exceptions to the users. The issue was fixed with a new release of goBIM that doesn’t use the removed endpoint.

Event Start: 11/10/2022
Event End: 11/10/2022
Duration: 34 min
Affected Customers: Cobuilder goBIM
Reproducibility: always

02.08.2022: Multiple emails are sent to Cobuilder FREE users

Dear customers,

Cobuilder Support has been receiving many inquiries from FREE users. They have been receiving emails regarding orders delivered in Cobuilder FREE. The email is legit, but looks like a spam/phishing email. Email subject: Delivery documentation has been received. The amount of emails increased after 10/08/2022. Engineering is working on a new email template.

Event Start: 02/08/2022
Event End: 11/08/2022
Affected Customers: Cobuilder FREE users
Reproducibility: always

27.07.2022: Login to goBIM not possible, documents in Collaborate did not load

Dear customers,

On 27/07/2022 from 10:16 CET until 11:05 CET login to the goBIM platform was not possible for a total downtime of 49 minutes. Documents connected to products in Collaborate did not load either. The issue was caused by updates in Azure security, a couple of internal legacy services (goBim and Collaborate still depend on) had to be reconfigured, so they can still connect to our Key Vaults in Azure and extract their sensitive configurations from there.

Event Start: discovered 27/07/2022, 10:16 CET
Event End: 27/07/2022, 11:05
Duration: 00h 49min
Affected Customers: goBIM, Collaborate
Reproducibility: always
Internal tracking:

21.07.2022: Multiple systems have problems loading

Dear customers,

On 21/07/2022 at 07:21 CET, it was discovered that multiple systems have problems loading for a total downtime of 1 hour and 2 minutes. The downtime during business hours was 23 minutes. Products on projects in Collaborate were not loading. When users log in to goBIM, they got returned to the main page. There was also an issue with Define. Engineering found that the cause was connectivity issue with Microsoft Azure. The platforms are now functioning as they should. Microsoft have not confirmed the issue yet.

Event Start: discovered 21/07/2022, 07:21 CET
Event End: 21/07/2022, 08:23 CET
Duration: 01h 02min
Affected Customers: goBIM, Collaborate, Define
Reproducibility: always
Internal tracking:

17.06.2022: Log in to goBIM

Dear customers,

On 17.06.2022, login to goBIM was not possible from 07:52 to 08:03 CET for a total downtime of 11 minutes. The user was stuck in an infinite login loop.

After restarting the CRM API Registration service, login started working again.

Event Start: discovered 07:52 CET
Event End: 08:03 CET
Duration: 11 minutes
Affected Customers: goBIM
Reproducibility: always
Internal tracking:

05.05.2022: Log in to goBIM

Dear customers,

On 05.05.2022, from 07:52 CET until 08:58 CET, goBIM didn’t load for a total downtime of 01h 06min. Login to goBIM was not possible, users were caught in a login loop.

The problem appeared in the connection to the internal Login service which keeps the customer credentials. Restoring the connection fixed the problem.

Event Start: discovered 07:52 CET
Event End: 08:58 CET
Duration: 1 hour 6 minutes
Affected Customers: goBIM
Reproducibility: always
Internal tracking: task 144848

01.03.2022: Log in to Cobuilder platform

Dear customers,

On 01.03.2022 between 14:21 and 14:41 CET we experienced an issue where the login page couldn’t load (20min). All users of the platform were affected. Login to the page is possible now.

Event Start: discovered 14:21 CET
Event End: 14:41 CET
Duration: 20 minutes
Affected Customers: all platform users
Reproducibility: always
Internal tracking: task 140280

Service Requests

In support of services outlined in the Service Level Agreement, Cobuilder will respond to service related incidents and/or requests submitted by the Customer within the following time frames:

Two (2) hour (during business hours) for issues classified as Critical.
Eight (8) hours (during business hours) for issues classified as High
Three (3) business days (during business hours) for issues classified as Medium
Three (3) busines