Client: Ultra-Large, Global Food Corporation
Problem: Users were reporting sporadic instances of Office 365 Outlook calendar events doubling when entered or changed. The client had tried to identify the commonalities in the sporadic events but was unable to pinpoint the problem. With over 180,000 employees worldwide, even though this was an intermittent issue it was affecting thousands of users and was creating considerable noise within IT. As this was a new US Cloud client, there had also been a previous ticket escalated to Microsoft, but no resolution had been offered in the four weeks the ticket was open.
Resolution: A US Cloud ticket was opened and worked that same day. This was a problem that US Cloud engineers have seen multiple times, so the client was instructed to investigate the users and instances where the problem occurred, looking for a common thread of multi-device access.
When the client confirmed the hypothesis, US Cloud informed the end user that typically this happens when there are multiple devices synced with the mailbox – especially if there are other delegated users for the mailboxes. The client was given instructions on how to reconfigure the problematic mailboxes / user accounts and the ticket was resolved and closed within a day of submittal.
Client: Leading Online Retailer
Problem: This very large and famed 100-year-old US retailer turned e-tailer suddenly found that none of its employee user base could login and access their SharePoint site. The company used this hub as their primary internal document sharing and access hub and the lack of access was causing significant disruption to their business.
Resolution: A US Cloud ticket was opened. And after an initial quick response and triage, a Premier engineer was engaged that day. After digging into recent activity with the client it was discovered that the certificate for SharePoint servers had been updated very recently. US Cloud determined that this was most likely the cause of the problem and advised the client to import the “SharePoint Root Authority” certificate to the Trusted Root Cert store on all SharePoint servers. The client completed that step and the log-in ability for all users was restored.
Client: Top 10 International Apparel Brand
Problem: The client was attempting to set Policies for Devices Intune Managed for two Phases to force Windows Updates to process. Policies were applied, but corresponding devices were not taking action. The client opened a ticket with US Cloud to help review the policies and learn what they may be missing in order to get the devices to process outstanding updates.
Resolution: US Cloud researched the issue and then advised the client that the likely problem revolved around Microsoft Teams Rooms App (MTR). MTR has built in logic for a Windows 10 feature that only allows updates (for devices running Microsoft Teams Rooms) after six months from the time when Windows makes a release update. This is accomplished by putting a special block for Microsoft Teams Rooms devices on the Windows Update for Business channel (that is, Semi-Annual Channel) and through the app settings. During this blocked period Microsoft performs various tests both in-house and through device OEM partners to make sure that new Windows 10 feature release is working in harmony with the Microsoft Teams Rooms app and peripherals connected to it.
This is important to both ensure device security, consistent user experience and to make sure quality of experiences offered through Microsoft Teams Rooms app. The issue and cause were clarified in detail for the client and IT was able to reset expectations regarding those delayed updates.
Client: Large East Coast Health Plan / Insurer
Problem: Client had Windows Server 2012 running a repeating task scheduled to reboot a server weekly. Without any obvious cause, the server began kicking off that task earlier in week, and then again on the scheduled time.
Resolution: US Cloud first researched to ensure the system was not in hibernating mode. In addition, there were no events related to Kernel Power loss in the event logs, nor an event ID 42 in the logs as well. US Cloud engineers instructed the client to remove the AC condition from the task and ensure that there were no GPO settings or a SCCM Reboot Schedule setup. The client reset task scheduler to be configured for Windows Server 2012 r2, as it was set for 2008. After changes were made, the client let the task run with the change over the weekend and confirmed the issue was resolved.
Client: Top 3 International Cruise Line
Problem: The client entered a ticket with US Cloud after users with mailboxes in Office 365 suddenly could not see any calendar information, even free/busy times, for users still working in the cruise lines on-prem environment. The scheduling assistant appeared as a hatched-out bar through the users’ availability (when viewing in Outlook 2016), or as a solid gray bar when viewing in webmail. Productivity was limited by the inability to effectively coordinate time between users. The issue was escalated to the company C-suite prior to entering a ticket with US Cloud.
Resolution: US Cloud engineers first confirmed the Exchange version, current CU and discovered that it was limited to O365 on-prem users. They then performed a connectivity test to check for any issues with AutoDiscover. Engineers also checked the TargetSharingEpr value using the ” Get-Organization relationship | FL ” command through Powershell in the Exchange Online console. The report showed that this was set incorrectly prevented the O365 users from seeing the free/busy info. The client was given instructions on how to properly set the correct value and the issue was quickly solved.
Client: Major Business and Consumer Electronics Manufacturer
Problem: This US Cloud client put in a ticket when recovery key information was not being stored in the MBAM database for thousands of BitLocker encrypted systems within the enterprise. The issue had been ongoing for months and continued to create problems for the client’s Help Desk when systems were forced into recovery mode. The larger concern was that the client would encounter an incident in which hundreds of systems would be impacted because a BitLocker recovery key does not exist. Multiple troubleshooting steps were taken including: reinstallation of MBAM, ensuring group policy was applied and attempting to force a recovery key refresh before engaging US Cloud.
Resolution: US Cloud’s investigation showed that the MBAM Version was 2.5.1100.0 which was insufficient to properly manage current windows 10 devices. The client was advised to update Server infrastructure to 2.5.1143.0, and the users to at least that level. Following the prescribed fix, Bitlocker Keys began accurately populating in the Database again and were able to be queried by client support staff.
Client: Prominent International Not-for-Profit
Problem: The client had transitioned to using Teams recently and user errors were still common. It was discovered that some critical Teams Messages had been accidentally deleted and all efforts to recover them by the internal IT team had been unsuccessful.
Resolution: US Cloud first obtained background and details about the messages and confirmed that they / the Team had actually been deleted vs. archived, which it had. The US Cloud engineers then checked to see if the client’s instance of Microsoft Teams had possibly been configured with ‘Legal Hold’. Although not a feature commonly used by IT teams, Legal hold is often enabled at set up to ensure all information and interactions within Teams is kept indefinitely so someone can search and export the evidence for a court of law. Litigation Hold will maintain a copy of the data even after it has been deleted. A user can still delete the content as they’d expect, but Microsoft will continue to retain the data in a hidden location only administrators can access.
The US Cloud team found that the deleted Teams Room was indeed backed-up on Litigation and the messages were restored. The Audit logs were reviewed during a screen share and it was found that there were no levels of auditing for this particular type of event. US Cloud advised the client to review from the Exchange side and discovered the user who had deleted the messages. They were then guided through adjusting the relevant permissions to resolve issue and prevent the incident from occurring again.