Systems Incidents
Updated: December 20, 2022 9:50 am
20.12.2022: Users get returned to the welcome page
Dear customers,
Users get returned to the welcome page in an attempt to login. The access got restored by restarting the app service.
Event Start: discovered 20/12/2022, 08:11 CET
Event End: 20/12/2022, 08:24 CET
Duration: 13min
Affected Platforms: all
Reproducibility: always
03.11.2022: Issue with the mail server – emails don’t get sent on production
Dear customers,
On 03/11/2022 from around 07:00 until 10:36 CET, for a total down time of 03h 36min (business hours downtime: 02h 36min), emails did not get sent. These can be emails about products delivered in FREE, new user invitations, invitations on projects etc. There was another issue with the mail server. On 01/11 Engineering switched to another mail server to fix the issue we experienced then. Today they discovered that emails did not get sent through it. The issue was in the user, and after resetting it, the issue was fixed. Nevertheless, our team switched back to our old mail server for now, and emails are once again successfully sent.
Event Start: discovered 03/11/2022, 07:00 CET
Event End: 03/11/2022, 10:36 CET
Duration: 03h 36min
Business hours: downtime: 02h 36min
Affected Customers: all
Reproducibility: always
Root cause: issue with the mail server
01.11.2022: Issue with the mail server – emails don’t get sent on production
Dear customers,
On 01/11/2022 from 11:11 until 13:14 CET for a total of 2h13min it was not possible to send emails on production. The reason for the issue was in our mail server and is now temporarily fixed as Engineering changed the microservices to use another smtp server when sending emails.
Event Start: discovered 01/11/2022, 11:11 CET
Event End: 01/11/2022, 13:14 CET
Duration: 02h 13min
Affected Customers: all
Reproducibility: always
Root cause: issue with the mail server
25.10.2022: Some users in Define cannot open and execute specific actions
Dear customers,
On 25/10/2022, from 08:23 CET until 10:47 CET for a total downtime of 02h 24min, some users in Define could not open and execute specific actions. Some users were not able to login, but the QAs fixed that quickly. One user cannot open some concepts. Another one cannot approve some sharing requests.
Another, when trying to share concepts from Cobuilder context to other contexts, the list with context that they can share to is empty. All these issues appear to different users and different contexts.
The issue was due to the fact that the new endpoints which return info about subscription, role and permission, don’t return info about users without First Name in CRM. By adding the missing fields and syncing the user, the problem is resolved.
Event Start: discovered 25/10/2022, 08:23 CET
Event End: 25/10/2022, 10:47 CET
Duration: 02h 24min
Affected Customers: Define
Reproducibility: always
Root cause: new endpoints not returning information about user with missing information fields
11.10.2022: Еrror message when logging in to goBIM
Dear customers,
On 11/10/2022, from 09:33 CET until 10:07 CET, logging in to goBIM was not possible for a total downtime of 00h 34min. Users got an error message when opening goBIM.
The reason for the issue was a removed microservice that was released before the release of goBIM. This means that goBIM tried to use an endpoint that does not exist in microservices and for that reason threw exceptions to the users. The issue was fixed with a new release of goBIM that doesn’t use the removed endpoint.
Event Start: 11/10/2022
Event End: 11/10/2022
Duration: 34 min
Affected Customers: Cobuilder goBIM
Reproducibility: always
02.08.2022: Multiple emails are sent to Cobuilder FREE users
Dear customers,
Cobuilder Support has been receiving many inquiries from FREE users. They have been receiving emails regarding orders delivered in Cobuilder FREE. The email is legit, but looks like a spam/phishing email. Email subject: Delivery documentation has been received. The amount of emails increased after 10/08/2022. Engineering is working on a new email template.
Event Start: 02/08/2022
Event End: 11/08/2022
Affected Customers: Cobuilder FREE users
Reproducibility: always
27.07.2022: Login to goBIM not possible, documents in Collaborate did not load
Dear customers,
On 27/07/2022 from 10:16 CET until 11:05 CET login to the goBIM platform was not possible for a total downtime of 49 minutes. Documents connected to products in Collaborate did not load either. The issue was caused by updates in Azure security, a couple of internal legacy services (goBim and Collaborate still depend on) had to be reconfigured, so they can still connect to our Key Vaults in Azure and extract their sensitive configurations from there.
Event Start: discovered 27/07/2022, 10:16 CET
Event End: 27/07/2022, 11:05
Duration: 00h 49min
Affected Customers: goBIM, Collaborate
Reproducibility: always
Internal tracking:
21.07.2022: Multiple systems have problems loading
Dear customers,
On 21/07/2022 at 07:21 CET, it was discovered that multiple systems have problems loading for a total downtime of 1 hour and 2 minutes. The downtime during business hours was 23 minutes. Products on projects in Collaborate were not loading. When users log in to goBIM, they got returned to the main page. There was also an issue with Define. Engineering found that the cause was connectivity issue with Microsoft Azure. The platforms are now functioning as they should. Microsoft have not confirmed the issue yet.
Event Start: discovered 21/07/2022, 07:21 CET
Event End: 21/07/2022, 08:23 CET
Duration: 01h 02min
Affected Customers: goBIM, Collaborate, Define
Reproducibility: always
Internal tracking:
17.06.2022: Log in to goBIM
Dear customers,
On 17.06.2022, login to goBIM was not possible from 07:52 to 08:03 CET for a total downtime of 11 minutes. The user was stuck in an infinite login loop.
After restarting the CRM API Registration service, login started working again.
Event Start: discovered 07:52 CET
Event End: 08:03 CET
Duration: 11 minutes
Affected Customers: goBIM
Reproducibility: always
Internal tracking:
05.05.2022: Log in to goBIM
Dear customers,
On 05.05.2022, from 07:52 CET until 08:58 CET, goBIM didn’t load for a total downtime of 01h 06min. Login to goBIM was not possible, users were caught in a login loop.
The problem appeared in the connection to the internal Login service which keeps the customer credentials. Restoring the connection fixed the problem.
Event Start: discovered 07:52 CET
Event End: 08:58 CET
Duration: 1 hour 6 minutes
Affected Customers: goBIM
Reproducibility: always
Internal tracking: task 144848
01.03.2022: Log in to Cobuilder platform
Dear customers,
On 01.03.2022 between 14:21 and 14:41 CET we experienced an issue where the login page couldn’t load (20min). All users of the platform were affected. Login to the page is possible now.
Event Start: discovered 14:21 CET
Event End: 14:41 CET
Duration: 20 minutes
Affected Customers: all platform users
Reproducibility: always
Internal tracking: task 140280
Service Requests
In support of services outlined in the Service Level Agreement, Cobuilder will respond to service related incidents and/or requests submitted by the Customer within the following time frames:
- Two (2) hour (during business hours) for issues classified as Critical.
- Eight (8) hours (during business hours) for issues classified as High
- Three (3) business days (during business hours) for issues classified as Medium
- Three (3) business days (during business hours) for a general service Request.