Microsoft says it is aware of a code fault that caused some Exchange servers to stop processing emails on New Year’s Eve as the clock struck midnight. The bug has been called Y2K22 by system administrators who are spreading remedies on social media, in the style of the Y2K bug, which afflicted some systems at the same time 22 years ago.
Microsoft said its experts were “working around the clock on a fix” that wouldn’t require users to mess with their on-premise servers to get things back up and running, but warned that it “would take several days to create and deploy.” Instead, those developers are working on a new upgrade that “is under final test validation” and will demand client action while still providing “the quickest time to resolution.”
The problem appears to be related to the way Microsoft named updates for its malware-scanning engine, which included the year, month, and day (220101) in front of another four-digit number (0001). Microsoft appears to utilize this technique because if an update is labelled “2,201,010,001,” it’s easy to figure out which one is the most recent because it has the greater value.
The problem appears to be related to the way Microsoft named updates for its malware-scanning engine, which included the year, month, and day (220101) in front of another four-digit number (0001).Microsoft appears to utilize this technique because if an update is labelled “2,201,010,001,” it’s easy to figure out which one is the most recent because it has the greater value.
The issue appears to be that the field in which this number was kept had a 31-bit restriction, implying that the greatest number that could be represented was 2,147,483,648 or 2 to the power of 31.This naming scheme would exceed the maximum value that could be represented in 31 binary symbols as soon as the clock struck 2022.The company hasn’t validated the technical details, but their explanation appears to back up the theory: “Version checking against the signature file causes the malware engine to crash, resulting in messages being held in transport queues.”
The problematic software update is for Microsoft’s anti-malware scanning software, which means that messages that should be queued and examined are just being queued. On Microsoft’s website, one managed service provider cautioned that a client queued 10,000 messages in less than 24 hours. They said that doing so would risk filling up the server’s storage and causing it to fail, perhaps shutting down the firm.
“If you’re not sure your Exchange Server storage has the capacity to keep all queued messages without filling up discs and crashing,” they added, “don’t wait for the Microsoft patch.”
“Use the workaround right now to get the messages out sooner rather than later.”
The problem is that the fix includes turning off the malware filtering feature, which might leave businesses vulnerable to hackers. “You should only use one of these solutions if you have an existing malware scanner for email that is not the engine in Exchange Server,” Microsoft says. According to the Microsoft Exchange team, “We expect to send you this update, together with the actions you must take, as soon as possible. We apologize for any trouble this problem has caused.”
