Turning Things Off is Harder Than It Looks…Erasing Basic Authentication from Exchange Online

[MUSIC PLAYING] Hey, everyone. My name is Greg Taylor. I'm a principal product manager at Microsoft. And it's my pleasure today to talk about something that's been near and dear to my heart for the last three years, which is turning off basic auth in Exchange Online. And let me assure you that turning stuff off is way harder than it appears, and that's the goal of today. So I'm going to backtrack a little first and talk about, what is basic auth, and why is it such a big issue? And why are we even bothering to talk about it today? Well, the basic auth, or the use of nothing more than username and password, is one of the biggest problems that many of our customers face with Exchange Online today. Almost all the password spray attacks that we see happening against our tenants, their data, their user accounts happen using these legacy auth or basic auth protocols. Now, it's exacerbated by the fact that so many customers and users-- and we all do it-- reuse passwords. And so things like these kind of attacks are all intended to try and capture simple, easy-to-guess usernames and passwords. Let me just say, usernames-- bear in mind, usernames are typically people's email addresses. So we really don't-- we don't protect those anymore. Email addresses are freely given around. All we got to do is-- all the bad guy has to do is guess your password. And unfortunately, too many of us use the same password. We use simple passwords. And these Exchange Online protocols are used as a testing method to guess people's passwords all the time. Now, it's pretty clear that enabling things like multi-factor auth and disabling basic auth has a huge beneficial impact on the security of your tenant and your data. I mean, the statistics here on the slide give it away. There are huge amounts of password attacks going on every minute of every day. And throughout the course of the last few years, I've borne first witness to the fact that many of the biggest customers in the world don't really know this is going on. Their tenants are under attack constantly. Now, this is why we're turning off basic auth in Exchange Online. It has nothing to do with just trying to give everyone more work or force you to buy more stuff or whatever it might be. It's the fact that we just simply see too much abuse, too much compromise of customers' data, customers' accounts, every single day. Now, from the customer's perspective, what is it we're actually doing? And hopefully, this will-- I'll quickly go through this. By the time you've certainly seen this recording, this should be very obvious and nearly all done. But beginning October 1, 2022, we began to start to turn off basic auth for POP and IMAP, all the protocols that Outlook uses, ActiveSync-- used by mobile devices-- and Remote PowerShell. Now, to clarify, we're not turning off the protocols. We're turning off the ability to use basic auth with those protocols. All those protocols support OAuth-- modern auth-- and some even support things like certificate-based auth. It's just the basic auth-- username and password and nothing else-- that that's what we're turning off. Now, if I've said it once, I've said it a million times on our blog. SMTP, used for sending mail, we are not touching. If you are not using it, we'll turn it off, and we probably already have. But if you are using SMTP auth, I have no interest in turning it off at this moment in time. And the reality is, that's simply because there are so many kind of multi-function devices, scanners and all of these things, that send email-- scan-to-email kind of things-- that send emails via SMTP auth. And all of these are firmware-based hardware devices. No chance of updating them at all without replacing them. So while, eventually, we'll get to a place where SMTP auth, with just nothing but basic auth credentials, will go away, we are some time away from that. So September 1, we offered everyone an opportunity to get a little more time. So we basically said, hey, you've got the month of September to tell us that you need a bit more time before we turn off basic auth sometime early October. And that opportunity is closed. Certainly by the time you've seen this, we are done. And I'll tell you that probably about 20,000 customers told us, during the month of September, that they wanted a little more time. I'll tell you as well, that was not nearly enough, because the first few weeks of October have been really interesting. And right at the end of this presentation, if you stick around for that, I'm going to give you some up-to-date numbers on where we stand with the whole process. Now, given that we're in October, anything that was not opted out prior to October, we are turning off. And we have been doing so for the last few weeks, at the point at which we are recording this. But we are giving customers one last chance to re-enable. So if you missed the opportunity to say, hey, I need a bit more time, and we turned off something in your tenant, you should, by now, have known-- well, hopefully, by now, you've definitely known because we've turned it off quite some time ago-- you can go and re-enable that protocol for basic auth for the protocol. So if we turned off POP and you weren't quite ready or we caught you by surprise, then you can go and re-enable POP. Now, I'll give you some ideas of how many customers have done that, too, a little bit later on. But all of those expire December 31. So come January 1, it's going to be pretty-- we're going to turn everything back off again. And this time, no re-enablement. Now, the bulk of this presentation is really about running the project, how the whole thing kind of came about and how we do it, rather than what we're doing and why. We're all done with that for the moment. And the main reason-- I would say, the real big thought, the key to unlocking what we were doing this fall, is really at the start here. Which is realizing that turning off basic auth for Exchange Online doesn't really-- the primary goal is not just turning something off that you're all using. It's actually about protecting your data and your user accounts. And I mentioned this upfront as to why we're doing the whole thing in the first place. But the realization that we're not just turning off stuff people are using, but protecting tenants, allowed us to focus heavily on protecting the tenants that don't even use basic auth. So let's think about the fact that basic auth has been enabled in all of these tenants since their day of creation. And so many customers don't use basic auth, but it's still enabled. And they haven't blocked it. Which means they are, essentially, one password spray away from being compromised. So there was a big focus on, let's protect the tenants that don't even know it's still enabled while simultaneously working on reducing the usage in those that do. So I really had to think about who's using basic auth. Let's figure out who's using it. How are they using it? Where am I going to store all the data on this usage? Because it's clearly-- there's a huge number of tenants and a huge amount of usage. And then do some work on thinking about, well, what are the clients? What are the apps? How is it being used within all of these tenants? And then, are they based on scenario or solution or protocol? Or how are they based? In other words, try and make sense of what is a vast amount of data, a vast amount of customers, and then come up with plans. And we'll come back to that as well. Then, of course, there was the, how on Earth are we going to actually turn it off? We've got to figure out how we're going to block basic auth. And how are we going to avoid everybody calling us when we do? And more importantly-- and, honestly, most importantly for me-- is, how do we let customers know that we're going to do this? What the impact is going to be and what they can do about it. And so the project, from a customer's perspective, is, we're turning off some things, and here are some dates. From my perspective, these are the kind of questions that have been on my mind for the past couple of years. I did manage to find an internal Microsoft site that talked about turning stuff off for customers, and it was incredibly helpful. And I say that sarcastically because what it really said was, you should announce that you're going to turn something off. You should help customers stop using it, and then you can just turn it off. And obviously, if I read that site and realized that what I had to do were these three simple things, trust me, I wish it had been that easy. It wouldn't have taken three years, as well. So let's talk about how my team and I gathered the data and what it really told us. So we start with where we keep everything. We keep everything in a big Azure SQL database. And what we do is, we feed into that database a multitude of things. First thing we feed in is basic auth usage. Every 30 minutes, we have automation that queries our authentication logs in Exchange Online for the affected protocols. And we get a report back. We get data back that tells us which tenants, which protocols, how many logins. I do not capture user-level data. I am not interested in user-level data. Trust me, that's a whole privacy nightmare I don't want to get into. But I don't need it anyway I don't care. I deal at a tenant-by-tenant protocol basis. So I know, within a particular tenant, there were 10 logins to POP by 10 unique, different users in any period, for example. I feed that into my database. The other thing I feed into the database is tenant configuration information. So we have a multitude of automated jobs that go and query tenant configuration information. So for example, is modern auth for Outlook enabled in a tenant? Which bits are turned on or off in the tenant that affect authentication in a variety of different ways? Which forest is the tenant in? How many users has the tenant got? All that kind of stuff. I feed all of that in. Then another piece of data that I feed into this same database is information of tenants who opted out. So perhaps, during that September time frame, they said, please don't turn off IMAP or POP or whatever. And the tenants that, subsequent to us turning things off, have re-enabled. So I know all of that, too. And what that allows me to do is build a really decent picture of a tenant's usage, configuration, and need, if you like, through opt out and re-enable. And I can then query intelligently things like the tenant configuration and figure out who I can disable basic auth for. Now, one of the things that before-- just going back to one of the comments earlier, which is, it's not just about turning off everything in [INAUDIBLE] We've tried to be quite surgical about this. So in the months leading up-- in the year, I would say, leading up to October 2022, when we started turning off things people were using, we've been turning off things that people have not been using. So for example, I can say that a particular tenant never uses POP and IMAP. I can then query using my data. I can then figure out this tenant does use, let's say, Outlook. They use MAPI and EWS and Offline Address Book. But they don't use POP and IMAP. So I can then turn off POP and IMAP without them know-- well, noticing. Or being impacted, is another way to put it. Securing them. Not just turning them off, securing them. I run my queries. I figure out what the configuration is. I then pass this along to something called Symphony. Symphony is a workflow platform we've built that does a lot of mailbox moves. A lot of the mailbox moves between geographies are run by the Symphony platform. And I send Symphony a whole bunch of jobs. I tell Symphony to go and turn a whole bunch of stuff off. And then, obviously, the tenant configuration gets updated. I've turned things off, tenant configuration gets updated. And then it forms a full circle as that data then trickles its way back into my tenant configuration information, which essentially allows me to validate that the change I made in the tenant came back to me and was reported as being made in the tenant. And then, obviously, if a customer goes and re-enables something, I'll learn about that quickly, too. And then I'll update my view of the tenant's data. I know what's on, what's off, their usage, and so on. And that's how we track the data and how we turn things off at a high level. Now, the data that we've gathered is actually pretty fascinating. I think it's kind of interesting to see how customers use the service. And so three years ago-- I want to give you some-- I'll share some real big numbers-- there were about 70 million user accounts using basic auth in about 4 million tenants. Now, that number might be-- I don't know whether it surprises you or not. It's a big number. It's a lot of users and a lot of tenants. And by no means is basic auth used in the majority. So we have a lot of users and a lot of tenants. But the data also told me, interestingly, you can't just track users. You have to track requests. And so the reason for doing that is 10 million-- here's an example. 10 million Outlook users, MAPI users might make 60 million requests in a day. But 1 and 1/2 million POP users might make 80 million requests a day. Or if you think about it another way, let's just say a single user that logs in 10,000 times a day is obviously an application, typically. A script, some kind of application. But you could have one user-- so it's the difference between-- when you only look at user accounts, you could have one user in this tenant and one user in this. This user might be using ActiveSync in a mobile device, and this one could be an application doing some vital line of business function. And if you only look at users, not requests, you can't really differentiate the two. It's just a user. Other things we found-- obvious things. ActiveSync and EWS, biggest basic auth users, by far-- ActiveSync on mobile devices. And mainly that's because the mobile device manufacturers made migrating to new devices so easy that they transfer all the data and settings across. That is the reason why they keep using basic. And a lot of the EWS and MAPI usage came from Outlook, comes from Outlook because the OAuth2-- or modern auth for Outlook-- flag was disabled in [INAUDIBLE] I'm going to come back and talk about that a little bit later, as well. There was an awful lot of POP and IMAP usage still. That was pretty obvious. But when you-- particularly in EDUs, as well. It's sort of the EDU segment because a lot of POP and IMAP goes on in those industries, or market segments, if you like. But the reality is, when you look at the number of users or tenants, the number of users in them, the number of protocols-- eight or nine, including SMTP-- that could be potential breach points for basic auth, it is trillions. There are trillions of potential combinations of those that are at risk. So the whole project here is [INAUDIBLE] trying to shut that risk down. A few other charts-- the data is fascinating. It tells you a. Story. It tells you a story that, here's a typical week. Well, I've taken the numbers out, but ActiveSync's at the top, Outlook or MAPI and EWS in the middle. And then, down at the bottom, you've got IMAP, POP underneath. And way down at the bottom, there is PowerShell. Now, ActiveSync-- you can see where the weekends are. First thing you can see. Some people with mobile devices on ActiveSync don't sync their mail at the weekends. Which I think is crazy, but I've never-- they're better at email than I am, then, because I've-- it's not me. But some people don't sync their phones at the weekends. You see the slight dips. A whole lot of people don't use Outlook at the weekends, so you see a massive dip. More expected, right? POP and IMAP have stayed incredibly consistent throughout the entire duration. People don't-- a lot of it is apps for the POP and IMAP. But there's also a lot of variance between the number of users we see per day and the number of users we see over the course of a month. So that could easily be explained by things like shift workers or people who don't check mail every day. But because of the size of the numbers, they balance out. You also see things like, Tuesday's our busiest day. It just is. Not a lot of people work on Fridays, I'll tell you that. The data is pretty clear. Now, looking at our tenants as well, here's a chart with two sets of data on. So this is-- gives us an idea of where the basic auth usage in Exchange Online comes from. So the blue lines here represent the size of-- the number of users using basic auth. And then the orange line here is representing-- or the horizontal bar's here representing the size of the company as well. So in the 20,000-seat companies over on the right-hand side, you have a huge amount of basic auth usage. But then, you also have a huge amount of basic auth usage in the left-hand side, the companies under 100 seats. Essentially, a third of our usage comes from each of these two segments, with a third in the middle. So a third of our usage comes from the 10,000-seat and up customers. A third of it comes from the 1-to-100-seat customers. And a third of it comes from between 101 and 10,000 seats. Now, the orange line represents how many tenants there are in those particular same groupings. And so what it clearly shows is, there are-- 99% of the tenants have got less than 1,000 seats with basic auth usage. A very small amount of tenants with a very large number of seats at the right-hand end are also accountable for the usage. What this really tells you-- what does it tell you? Well, it tells you that we have customers of all different kinds of shapes and sizes. But it also tells you that the left-hand end of this is hard to contact. Because the left-hand end of this, where a third of the usage comes from, are small customers. And small customers are typically pretty hard to reach. They don't have account teams, they don't have a whole bunch of IT pros working for them. They are small businesses getting on, doing their day-to-day business. We'll come back and touch on this theme a little bit later. But the reality is, what it means is, they're hard to get hold of. Now, as part of this other-- part of the exercise, as well, I've had the unfortunate-- no, I'm going to change the word. It's not unfortunate. I've had the opportunity-- it sounds almost like I rehearsed that, and I didn't-- the opportunity to explain to a bunch of customers how they are already being breached through password spray attacks and helping them secure it. And I'm going to explain how I spot that and what you should do, or what can be done about it. So I mentioned password sprays earlier. And password sprays are just this technique used where the attacker tries to use a multitude of passwords-- commonly used passwords-- against a whole bunch of usernames. And they spray, seemingly randomly, across the entire collection because it's a lot harder for systems like ours to detect that than try the same single-- they also avoid account lockout, of course. If you keep trying the same account over and over again with different passwords, you will lock it out. But if you just try huge numbers of accounts with a reasonable number of passwords and then come back and start again, any time of a timeout is going to have gone away. So you can just keep trying. It's very successful. Often, the dead giveaway, when I look at a customer's data, is the number of failed requests being greater than the number of successful requests, particularly when it comes to something like IMAP. And so take a look at these two tables here in the example. So in the upper table, these are references to users and how many times they successfully logged in over a 30-day period. So there's six accounts here, and the top five, in 30 days, all logged in 500 plus times. And then one account logged in once. Once in 30 days. How many people do you know who check their email once in 30 days? OK, my mother might be one. Great. But she's not part of this tenant. But that's just kind of weird and unusual. Then, I look at the same tenant over just a three-day period. And I don't just look at logins. I look at failures. And that's in the lower table. And you can see that, in three days, there were 3,381 successful logins, but there were nearly 14,000 AccountNotProvisioneds, 1,300 incorrect passwords. Applications-- which is where a lot of the IMAP usage is done these days-- applications, once configured, do not generally get the password wrong. You type a password in, it keeps using it. People get the password wrong or attackers get the password wrong. If they can't guess the username correctly, the account will not be provisioned in the tenant. So these two pieces of data are a bit of a dead giveaway that there's an attempt to guess usernames and passwords, and then, there's a sign of success. And we actually put a blog out talking about how you can go about defending against this. And the way you defend against this is using something called authentication policies, which, hopefully, you're familiar with. And most people are familiar with the Set-CASMailbox way of blocking protocols. And they're familiar with authentication policies. And in fact, I'll go back slightly because I'm going to just see if I can-- let me see if I can get my pointer here to work. Let's just see if I can do a laser pointer. And I'll explain the difference between the two quickly. So in the case of a-- here we go, laser pointer. That's brilliant. OK. So in my basic auth client, my basic auth client sends its credentials to Exchange Online. A modern auth client goes and gets a token from Azure and comes back with it to Exchange. But basic auth sends its credentials to Exchange, and Exchange, on its behalf, goes off to Azure and gets tokens. But what happens is, credentials are sent to us. We validate them against the Identity provider-- ADFS, Azure, whatever it might be. Okta, whatever. We then go off, and we check conditional access for access rules. And then step number 5, finally, when we come back here, when we've passed everything else, we then evaluate Set-CASMailbox. And then we potentially block. Authentication policies are a step 1. So step 1, authentication policies. Step 5, CASMailbox. Now, the reason that's important-- if I try and jump forward here now, again-- is, the error the client gets back varies depending on how they're blocked. And Set-CASMailbox is blocking at the authorization layer, not the authentication layer. And what that really results-- here's an example with IMAP. When you block an attempt with IMAP with an authentication policy, or when we block it with this project, the message the client gets back is, IMAP server responded with, login failed. Basically, bad username or password. But when you block with Set-CASMailbox, we actually authenticated the user. They got through authentication, but you're denying them access at the mailbox level to use IMAP. The error, then, is, the IMAP server responded, user is authenticated, but not connected. Now, in both cases, the user can't get-- or whatever it is that was trying to access it cannot access the data. But if you're an attacker, you've just learned that you got the password right, but you can't get access to the data. And that's perfectly good. They'd be quite happy with that as a result because now they know that username and password is valid, and they'll go off and use it somewhere else. Set-CASMailbox is about blocking off access to an entire protocol at the mailbox level. It does not block basic auth. It blocks all kinds of auth. Or in fact, that's not true. It does not block any kind of auth. It blocks authorization and access to the data. Don't use Set-CASMailbox for any of this kind of behavior at all. If you want to block basic auth, auth policies or the way we're blocking. What about the strategy, then? So let's talk a bit about high-level strategy. So the data is fascinating. You learn a whole bunch of stuff about how people use the technology and how people are being breached by it. But for me, it was, OK, I've got to figure out, with the volume of tenants, the volume of users and data and everything else, how am I going to solve this? How are we going to try and get customers off basic auth? So clearly, we have to just break stuff down into different buckets. So we're going to break it down by scenario. So they're using basic auth or they're not using basic auth, these customers. And like I said, there was a lot of tenants that were not using basic auth. So there was an immediate opportunity there to think, well, how can we just-- can we just-- and, in fact, we did-- can we just turn off basic auth for the customers that don't use it? By the time we even reached October, I had turned off basic auth in millions of tenants completely. Millions of tenants. Then it was-- we broke it down, then, looking at the problem a bit more around technology-- so client apps or protocols. What opportunities are there for that? And so actually, on a per protocol basis, once we finished blocking basic auth for the tenants that didn't use a single thing, I started looking at, who doesn't use particular protocols, but does use others? I used the example earlier-- tenants use Outlook, but they don't use POP and IMAP. So can we go and turn it off? The answer was, in most cases, yeah. We also had to look at customer types. So these customers directly of Microsoft, or do they come through partners? There's a huge amount of customers that come through partners. And then there's, we have different environments. Most of us are familiar with the worldwide multitenant cloud-- the normal stuff, as you might think of it. But we have government cloud customers. We have department-- a whole bunch of secret clouds, a whole bunch of stuff. Gallatin, 21Vianet-- sorry, the cloud in China, for example. So there's all these different ways of looking at this problem and breaking these customers-- segmenting them, if you like. Then, I had to establish a realistic timeline. And you might be familiar with us having shifted the timeline once or twice, I know. First time was because we didn't get enough usage, really, to come down. And the second time was the pandemic, which was clearly a little unpredictable for all of us, or unforecast. But we understood everyone had different priorities. But trying to keep to a timeline with a project this big has been really hard. But we've done it this time. Then, communication plan. how are we going to tell everyone what's going on? Let's figure out how we're going to do that. How are we going to actually turn things off at scale? What's the technology we're going to use? And then, of course, we've got to figure out how we're going to track it as we go, report it so we actually can tell our own progress. So these are just the high-level thoughts on the strategy. Now, the actual turning stuff off. How do we do that? Excuse me. So we added a organization config-level parameter that operates in the authentication flow before anything. So even before auth policies-- authentication policies that we talked about earlier, and you see in the Admin Center UI. And you can use-- even before we even get that far in the authorization stack, we have a switch that says, basic auth for this protocol is either on or off. You can actually run Get-OrganizationConfig. BasicAuthBlockedApps is the parameter. It's an 8-bit binary value. If it's 255, we turned off eight things. If it's 1, we just turned off ActiveSync. There is a secret decoder ring that you can use to figure out what's on or off, but that's how we do it. So basically, it's a per-organization cmdlet that we need to run. And you can't run Set. You can only run Get. We get to run the Set. So that actually blocks up. Otherwise, everyone would just go and turn everything back on, right? So what we started doing-- we were gathering the data for tenants that I mentioned earlier. Who's using what, what kind of settings they have, and everything else. And then we built this picture of usage. We started to build a picture, probably over about three months of usage, and we began to get a pretty consistent view of who's using it and who's not. And we know which tenants are using security defaults. So we thought we'd start with them because, to be honest-- if you've enabled security defaults, basic auth is blocked anyway. So what we figured was, let's test our process by blocking basic auth at the organization level and a whole bunch of tenants that have already got security defaults enabled because they shouldn't notice. Unless they go and turn off security defaults, and then they'd notice. But that was why we built the diagnostic that we did. Anyway, we turned off a bunch of tenants. We gave them 30 days notice, and we said, you've got security defaults enabled. Well, we're going to block basic auth anyway. Let us know if that's a problem. Nobody said a word. Nobody re-enabled. It was all good. We watched all the telemetry. Everyone seemed fine. So then we started moving away from the security-defaults customers because you can't just keep turning those off. Because that's pretty uneventful. And we started picking tenants that use basic auth-- or don't use basic auth, rather. Sorry. We started picking tenants that don't use basic auth, had no sign of usage. And we started saying, we're going to proactively turn it off because you don't use it. And we went through millions of tenants. And then, as I said earlier, then we went back and started picking off individual protocols. We've run millions-- tens of millions of jobs like this with very few customers turning anything back on. Now, in fact, the chart here-- here's just an example of some of the key-- some of the dashboards and Power BI that I use. All the metrics are out of here, so you probably can't really tell too much from this. But I'll tell you that the chart in the top right here is a report I get on how many customers have re-enabled or opted out. Now, ignore the opt-out section. But for the re-enabled, these are the protocols people re-enabled after we started turning them off. And so there are some numbers here, but I'm not going to give you a time reference as to when this was because we had a few different points throughout the project. But we were tracking how many people were re-enabling stuff. And then, on the left here, I've got both requests and number of users. And the bottom right here is how many things we've turned off in each of these tenants. And you can see we've been steadily turning things off, with a few spikes and a few pauses as we run along. Now, some of the things that we've done along the way-- here are some of the things we're going to quickly-- I'm going to try and talk about as we go through. So that was the high-level strategy. Figure out who's using it, who's not. Keep going forward. Turn stuff off. Some of the things that we've done along the way are things like Message Center posts, Service Health Dashboard notifications. I learned some interesting things. I learned that-- as you probably all know, Message Center posts are the official way we communicate change to our customers. What I learned is, nobody reads Message Center posts. Or very few customers do, sadly. And you can plot on a graph how big a tenant is, and you can guess their probability for reading a message in a post. The bigger the customer, the more they read them. The smaller the customer, where's Message Center? What is Message Center? Is typically the response you get. Now, the challenge there, clearly, is, how do you communicate consistently to tell customers, hey, we're going to turn something off? So we started doing a whole bunch of stuff to try and make the Message Center post more actionable. So rather than just send a generic message, send a post saying, we're going to turn stuff off and we're not going to tell you what or when or how it will impact you, we started doing the mother of all mail merges. Merging in data to say, this is your usage for the past month. This will be impacted if we turn something off. And we found that to be pretty helpful. And those, they're more actionable. That was great. What we then started to do, as well, was posting to the Service Health Dashboard when we turned stuff off, not just posting to Message Center. And the statistics were clear that Service Health Dashboard posts? Read considerably more than Message Center posts. Message Center posts are required. We have to do that. SHD posts are notifiable, notifications of stuff going on. And they're read considerably more. So we found that was interesting. I just found that the whole exercise was pretty interesting to do. Now, the self-service diagnostic-- and I showed some data from that earlier, and here's it repeated here as well-- has been brilliant. There's a whole bunch of self-service diagnostic stuff in the Admin Center. And so rather than build something from scratch, I built upon that. Let's use an existing self-service tool that helps customers, and let's build our scenario into it. And our scenario has been used for re-enabling stuff, opting out of stuff, telling you what's disabled already in your tenant. It's brilliant. It has saved a ton of support calls. Clearly, a bunch of customers still call us, and we point them to the diagnostic. But for those that read the post, they go straight to the diagnostic. They can re-enable a protocol, and they save a call to support, which is great. And what was interesting, as well, was, the percentage of re-enable was really consistent throughout the project, at times. When we were doing some of the phases, you could predict and use the number of re-enablement as a sign for, are we going too fast? Are we going too slow? Because you can't go too-- how do I put this? If some people aren't screaming, you're probably not going fast enough. So it allowed us to find a happy medium where we could deal with the support issues, but we still made progress. Now, another very successful, I would say, part of the project has been the successful partnership with Apple that we had. So I've been working with Apple for three or four years on this. And there was a reason for doing that. So a ton of-- I mentioned earlier that, often, when you buy a new mobile device-- in fact, let me step back a second. There are lots of iOS devices connecting to our service using basic auth. Or there were. And the main reason for that was, when you buy a new iPhone, and it says to you, do you want me to transfer your data and settings from your old iPhone to the new one? You're like, yeah, that sounds awesome. I'll do that. And all your settings and your photos of your cat and everything, they all come across. Your settings come across, too. And that means it keeps using basic auth. So we have millions of devices connecting using basic auth that are brand new-- brand new iPhones-- still doing basic auth. So we hatched a plan with our partners at Apple to try and resolve that, rather than just break everyone. And that included taking advantage of something called the Resource Owner Password Credential grant flow, the ROPC flow. Which, if you read our documentation, will say, it's like Ghostbusters. Don't cross the streams. Don't use the ROPC flow, because it basically uses the user's credentials to get tokens. Yeah, it does. And in fact, we used it to our advantage in this case. So Apple wrote code for us that said, I'm going to try and use the user's credentials, which I already have on the phone, to go to Azure AD and get a token. And if I can, I'm going to try and transition the profile to use OAuth. And then, when I'm successful, I'll dump the creds from the device, thereby switching the device to use modern auth rather than basic auth. Now, that's different than an ROPC app that continually holds the creds and just keeps reusing them to get tokens. That's not what Apple do. They use it as a one-time operation, and when they're successful, they throw the creds away. They carry on using OAuth from that point forward. Now, we've had-- the change here went into iOS 15.6. The number at the bottom of the slide here is out of date. Millions of devices have successfully migrated, millions. So I'm very happy with how that's all gone. Now, another thing goes back to my point of, if you're not going-- if they're not screaming, you're not going fast enough with something we tried that we've never really tried before. And this was-- I liken this to when you're at the theater or somewhere, and it's at the interval. And they turn the lights on and off to let you know the show's about to start again. We flipped basic auth off and back on for a whole bunch of customers in early 2022. And the goal was just to get some attention, to be honest. Mainly, it was small customers. They don't read the Message Center. We've talked about that already. And so we picked 20,000 tenants in the first week, and we turned off POP or IMAP. In fact, we picked 20,000 for POP and IMAP and ActiveSync, and we turned it off for 36 hours. And then we turned it back on. And you know what? Nobody noticed, which was kind of interesting. Like, email got turned off. So we ramped up until we hit a point where we caused some customers a little discomfort. I'm sorry for those that-- if you were one of those, I'm sorry. But the message was, this is going to happen for real in October, and we're not going to let you turn it back on again after December. So we ran 1 and 1/2 million jobs turning these things off. Some customers saw it against multiple protocols. But the percentage that re-enabled was tiny by compared to the number of tenants that we actually affected. Which gave us confidence, as well, in a way, that when we come to turn things off for real in October, a good chunk of that usage might just disappear. I mean, bear in mind, I can't tell, when I see a successful log on, if it's a normal user doing their mail or it's an account that's already been compromised. So it's a good chunk of the usage that we probably turned off in October that was probably compromised usage already. One of the next things I quickly want to talk about is another big change we made in late summer, which was this thing called the OAuth2 flag in every tenant. And the OAuth2 flag, OAuth2ClientProfileEnabled flag, is a setting on every tenant that determines if Outlook for Windows can do modern auth or not. Now, you might not be familiar with this, but if you think, in the Admin Center, there's a checkbox for Enable Modern Auth for Outlook 2013 or Newer-- it's that. Well, that checkbox and this setting was only set to True by default in August 2017. And there are many, many, many tenants created before that time. And so they had new versions of Outlook, but they were all using basic auth. Now, we told customers about this many, many times and said, hey, you should go turn it on. And many of them did. And many of them did not. So we decided-- I decided-- that we should start doing it for them because the data showed me that 96% of the Outlook versions connecting to those tenants were modern versions of Outlook. Up to date, should be doing modern auth by default, have no known issues with modern auth. The only reason they're not is because that tenant-wide flag is set to False. So I picked 20,000 tenants, I gave them notice via Message Center, and we turned on basic auth. I'm sorry, we turned on the OAuth2 flag for them. And not a single person called us. So then, I picked a few more and a few more and a few more and a few more. And while there's no scale here, I'm going to tell you that the usage dropped-- and this is Outlook connectivity over a period of four weeks. 50% drop in usage or more. We saw an enormous amount of Outlook clients convert and very, very few support cases. I know a few customers were a bit upset, but honestly, I could count them on one hand. And if you had any idea how many tenants this was, you would consider that to be an acceptable number. It was huge difference. And that's testament to Win32 Outlook doing a great job transitioning from basic auth to modern auth. And what also made it a lot easier was because everyone was expecting-- whenever I mention this idea to people, they're like, well, what about the auth cred pop up that the user's going to get? And the question is always, do they use Teams? Do they use OneDrive? Yes. Then they'll use the same auth stack, which shares tokens, and Outlook will just do it silently. And it did. So I'm going to wrap this up pretty quickly by showing where we're at now. And this is-- we're recording this mid-late October. And we are about-- I'm about 3/4 of the way through the tenants that we're going to block during the month of October. And usage is down by well over 50% at this stage. The charts on the right here are a giveaway. Can you see? So the big-- in the upper chart here, the big drop of the purple and orange line here started with the OAuth2 flag that I was just talking about, turning that on late August. And then you can see the big drop in ActiveSync users. That's about the beginning of October. You can see where that drop comes from. There's some pretty enormous drops over the last three weeks. And I'm actually really happy currently-- and I hope I'm not proven wrong when this recording comes out-- but very happy with progress so far. There's about 30,000 tenants who've gone and re-enabled something right now, which, as a percentage of those we've turned off, is tiny. Which is great. And then the bottom chart, I'm showing you, as well. The red line here-- the very faint blue line you can maybe see towards the bottom is successful logins. And the red line, the red chart, the red stack are blocked logins per hour. And I'll tell you that we're blocking an average of about 30 million requests per hour with the basic auth right now. And we're OK with that peaking to 50, 60 million. As we block basic auth for tenants, these are applications that are still trying to use basic auth. And you can tell they go up, and then they come down as people figure out what's going on. And perhaps they re-enable. But we're OK with this right now. But I'll tell you, too, that basic auth attacks are increasing all the time. So we're blocking and protecting tenants every single day right now, and I'm pretty happy with that-- very happy with that. And the final warning, as well, I'll give you, is come December, you will not-- there will not be another blog saying, hey, if you need a bit more time, you can have a bit more time. There is no more time. Time is up. We're going to turn everyone off pretty much by the end of October. And then we're going to turn everybody who re-enabled or opted out off beginning of January. So be aware. Big thanks to all the customers who've done all the work here to get off basic auth by this time. Clearly, there's still a bit of work to do. Customers have a few more months. So with that note, I'm going to say thanks for watching through. I hope this is interesting. I've certainly found it a really interesting project. And I'm also going to be available online now to answer any questions that you might have following the recording. So thank you very much. [MUSIC PLAYING]