MPexchange - Outlook requesting password
Incident Report for Manage Protect Pty Ltd
Postmortem

Manage Protect Incident Report

Overview

Issue: MPexchange login errors on Outlook for Mac, Outlook for PC and Mobile (Active-Sync) devices.

Incident Open Date: 23 Sept 2022

Reviewer: Andrew Johnson (andrew@manageprotect.com)

Some end-users were getting continuous password prompts when trying to open Outlook for Mac, Outlook for PC and on mobile devices (Active-Sync).

OWA, POP, IMAP and SMTP clients were working as expected.

Timeline

Date/Time Description
11:54am AEST, 23rd September 2022 Reports received from partners stating Outlook on PC’s intermittently connected. MP team verified issues, noted OWA & Mobile access was still functional and advised partners to use OWA & Mobile access while the issue was being investigated.
1:23pm AEST, 23th September 2022 Incident has been resolved by MPexchange operations team.

Root Cause

Multiple geo-distributed DNS appliances service the 2FA infrastructure DNS records of EXCH690.serverpod.net domain. Operations engineers were performing a scheduled maintenance with one of the appliances.

The 2FA DNS records become unavailable as the DNS forwarding rule contained only this appliance and was missing others. MPexchange servers were not able to perform DNS lookup of the 2FA records and check mailboxes 2FA configuration. Customers were getting continuous password prompts in Outlook for MAC or on mobile devices and were not able to open 2FA portal.

Summary

One of the DNS records for the 2FA infrastructure in EXCH690.serverpod.net become unavailable.

Corrective Action

Operations engineers reconfigured the DNS forwarding rule in EXCH690.serverpod.net domain to use all virtual DNS appliances and scheduled an audit for other Exchange domains.

We sincerely apologise for the inconvenience this incident has caused you.

Posted Oct 17, 2022 - 11:47 AEDT

Resolved
This incident has been resolved.
Posted Sep 23, 2022 - 17:33 AEST
Monitoring
We have been notified by the system admin team that the issue is resolved. Our testing seems to agree.
We will continue to monitor and test to see if the issue is fully resolved.
We apologise for any inconvenience.
Posted Sep 23, 2022 - 15:44 AEST
Update
We are continuing to investigate this issue.
Posted Sep 23, 2022 - 15:43 AEST
Update
We are still urgently investigating the root cause of this issue. The scope of the issue extends to Outlook apps on Mac and mobile devices as well as Outlook on Windows. We apologise for the inconvenience.
Posted Sep 23, 2022 - 14:43 AEST
Update
We are still urgently investigating the root cause of this issue. The scope of the issue extends to Outlook apps on Mac and mobile devices as well as Outlook on Windows. We apologise for the inconvenience.
Posted Sep 23, 2022 - 13:26 AEST
Investigating
MPexchange is having issues with via Outlook. Symptoms are the password request continually repeating, and not proceeding.
OWA from webmail.manageprotect.com/ is operating and can be used as a temporary workaround.

Mail flow is unaffected.

We apologise for the inconvenience and are working to resolve this as a matter of urgency.
Posted Sep 23, 2022 - 11:54 AEST
This incident affected: MPexchange.