Securing the enterprise by securing APIs (Google Cloud Next '17)

[MUSIC PLAYING] CHRIS LATIMER: Good afternoon, everybody My name is Chris Latimer Thanks for joining us here I know that we’re the only thing standing between you and drinks at the cocktail reception, so we definitely appreciate your attendance for our session here, today, on securing the enterprise with secure APIs I just now realized that rhymed I never actually said it out loud before My name is Chris Latimer I’m one of the field engineers from Apigee, now part of Google, and I’ll let my cohorts here introduce themselves later on, when they get to their part of the presentation as well Really excited to be here and talk to you guys today about this topic Being Apigee, we absolutely love APIs And so what do we mean when we say APIs, when we talk about APIs and API security today? Are we just talking about the technical aspects of API, what protocols we’re using, system to system integration– that type of thing? The answer is no We really think about APIs more broadly We think about them from a technical perspective, but we also think about them from a business perspective And later on, we’ll get to hear from Ticketmaster and some of the interesting things that they’re doing from an API perspective and how they’re securing their APIs So if you look at what all we’re doing with APIs today, it’s hard to go even more than five minutes without using an API of some form If you check social media, you’re using an API When you checked into your flight, checked the flight status, that’s another API call When you make a purchase online, that’s also going to be another API call, so of course, we need to secure these APIs Very, very important data and functionality are running through APIs, so we often want to make sure that we’re going to secure those, and our first reaction is to think about something like this We’re thinking about, well, that’s a lot of valuable data that somebody could get access to, that maybe they shouldn’t get access to, and so what we’re going to do is we’re going to lock this down as tightly as possible Put everything in the vault Don’t let anybody get access to this stuff Put in place all the monitoring, all the tools that we can to really lock this important capability down And the problem with that is that, if all of your API security starts to look like this, you’re missing out on a lot of the value that APIs can bring So the hard part of API security isn’t just locking everything down If we just wanted to have a secure API, we could just not build any APIs That’s the most secure API of all But putting together a secure API while still being able to foster innovation inside of your organization, avoid putting in place unnecessary delays, red tape, bureaucracy– doing all that while keeping your API secure is a much, much harder problem to solve And so how do we do that? And in order to think about this problem, I like to put it into the context of imagine that you are just creating a new account online– you see a new site; you want to register for an account– and the way that we would think about this process If the site wants your email, sure, why not? I give this out all the time to any site Its pretty usual that they want that They want my address Not a big deal to give that information out It’s usually pretty easy to find You start to get to something like a phone number Well, now, I’m a little skeptical Am I going to get robocalls? Are you going to text me? Is this for two-factor auth? Is there a good reason for it? And if they say something like, oh, what’s your social? Well, now, you’re going to slam the brakes on Almost nothing is going to get access to that information You might think about it a little bit Is there a legitimate need for me to give this information out? There might be in certain cases You’ll look and see, is this actually the site that it claims to be, or is this a broken SSL cert? And if you’re comfortable with the risk reward, you’re comfortable with handing out that information An API security requires a similar approach We’re going to expose different types of APIs, and they’re going to have different types of security requirements that we’re going to want to impose on those APIs So let’s look through a few examples here Let’s imagine that you’re working for a health care company, maybe a hospital, and you want to create an API that’s going to expose hospital locations and hours, public information There’s nothing sensitive about this, so we can just open it up and just let everybody go use this thing like crazy Well, in a sense, yes, but there are still things we need to be concerned about For example, this may be part of a mission critical application for us This may be what– you know when you call up the hospital, and you ask where the nearest location is, somebody might be using the same API When you go to the website, you may be using the same API So if all of a sudden, you open this up and somebody starts sending a flood of traffic your way, that can still cause pretty major problems for you So we need to think about that The traditional approach to this is let’s put in place heavyweight governance Let’s lock this down Let’s not give anybody access unless we

go through a rigorous process to understand how you’re going to be using this API, how many calls are you going to send to this But a better approach to that is traffic management, so we can put in place things like quotas and spike arrest and so forth to limit our exposure while still maintaining that culture of fostering innovation Another thing that we’ll usually do is try to put into place a service registry Let me figure out exactly who’s using which API and make sure that we’re tracking this information very, very closely A better approach of that is to use something like an API key, where we can do this automatically So rather than have a registry that’s probably going to get out of date and out of sync and give us unreliable information, what we can do is we can use an API key to automatically make sure that each application that’s making an API call is authorized to do so We also have to worry about content based attacks Even though this API may not expose sensitive information on purpose, if somebody starts sending in SQL injections or cross-site scripting or those type of things, we have to still be cognizant of the payload and make sure we’re validating that payload And a common way of approaching this problem is with an API gateway, and because I came from Apigee, I’m going to happen to mention, we just have a really good API gateway that solves a lot of these problems that you have that we’re talking about today Now, I promise this is not going to be a commercial for Apigee, but I did just want to mention this So if you want to talk more about Apigee and its capabilities, we are happy to do so So let’s think about another API Let’s think about one that, all of a sudden, is opening up scheduling or doctor availability So now, we’ve got a few more things that we have to think about We have to start thinking about we don’t want anybody just registering for access to this API and starting to create appointments for doctors We want to have a process in place that lets us vet the partners who are going to be using this API, and the traditional approach to this is to have a manual process, the same type of thing We’re going to have a form that somebody’s going to fill out We’re going to take this information We’re going to decide whether or not you get access to this API A better approach is self-service, so rather than you answer a bunch of questions about how your API works, instead of you looking through and having conversations and meetings and slowing things down and coordinating schedules, we can have a developer portal that’s going to give you the ability to give your partner self-service aspects– self-service capabilities to register for access to an API, see the documentation, and start to use it right away We are also going to need things authentications, so we’re going to need identity We’re going to need to make sure that the person that is registering for a– or that is scheduling an appointment is the person that’s the patient in our system So we need some way of identifying folks, and the traditional approach is going to be direct user authentication If I have partners who are going to be authenticating my patients, I don’t necessarily want them to have access to that patient’s credentials Instead, a better approach is to use delegated identity and user authorization, so we would have things in here like OpenID Connect, social logins, other logins where our partners can redirect that user back to our identity management solutions so that they can sign in, and then we know that it is actually them without the partner ever actually seeing those credentials Now, what if we start to get into some really tricky areas? What if we’re going to expose patient data? We’re going to give information about their medical history What do we have to think about then? Well, now, we have to think about making sure we’re not end up in jail because we’re breaking a bunch of laws So there is a time and a place that we need to have very, very locked down, tight security in our APIs The point of this presentation isn’t to say that we absolutely don’t need to think about tight API security We do, and there are times when that’s absolutely appropriate So we want to make sure that the person who’s looking at this data is authorized Is it your primary care physician that should have access to this information, or is it somebody from marketing purposes that wants to sell you a drug? They probably shouldn’t have access to that information We want to be able to have an audit trail We want to be able to look back and see who has looked at this over time At one of the hospitals that I’ve worked with before have had issues where celebrities have come in, and people have wanted to kind of see what’s going on there Obviously not something that you want to have happen in a HIPAA compliant environment And then there are all the basic things, too Of course, we need to still think about TLS We need to use HTTPS We’re going to want to look at payload encryption in certain cases to make sure that our data is very, very tightly locked down And so from a general perspective, these are some of the things that we need to think about from an API security– as we’re designing our API security And with that, I’m going to hand it over to my colleague Greg Brail, who is going to talk about a few specific use cases and case studies that we’ve seen in the wild GREG BRAIL: Excellent Thanks Chris So I’m going to talk now a little bit– Chris has talked about the context

Why do we want to select different types of API security? When is it important to have everything locked down to the max versus making things flexible developers? But what I really want to also talk about is get into a little more detail about API security and ways to secure APIs And the reason why I think it’s important to talk about this– if you’re a web developer or have even come close to web development, and you work for any sort of moderately large company, you probably had to go through some sort of mandatory training where they taught you about all the things that can go wrong with web app development And you learned about CSS attacks and SQL injection and all kinds of things like that We don’t have that for APIs, and APIs are a little different than web apps There are some things that can happen to web apps that don’t happen to APIs There are some things that happen to APIs that you don’t think about as much when you’re building a web app So I wanted to make sure we talk a little bit about API security, and one of the ways I like to do it is by talking a little bit about some of the things that can go badly when you don’t think about all the aspects of API security that are important So for instance, if you remember nothing else, everything that has a URL– or if I’m really fancy, I’ll say URI– has an API So Chris said earlier, one way to make an API secure is not to have one That’s true, but if you built a mobile app, and the mobile app talks to some service on the internet, you have an API You might not have advertised it, developers You might not have a Swagger spec for it You might not have GitHub repo with sample code But you have an API, and a reasonably smart person can, at home, jiggy up their Wi-Fi router to see what’s going on between your app and your API and figure out what’s happening And a bunch of smart people have done this to embarrassing effect For instance, the highly critical enterprise API run by Kylie Jenner was broken to by a 19-year-old who looked at it and discovered that his Kylie Jenner mobile app, or actually her website, had no API security, and he could get some interesting information Similarly, here’s a great article I found on Programmable Web that came out about a year ago titled “How to Use Undocumented APIs to Hack Your Tesla.” That’s something that, if you own a Tesla, it’s kind of cool Hey, I can hack my Tesla But the truth is that these are all cases of when people actually were able to look in and see what was happening between the client and the server and figure out the API So when we ignore the fundamentals of API security because we don’t have an API, you have to really ask yourself, well, do we have an API? Maybe if everything is inside one room or one LAN, OK, we don’t have an API But otherwise, we do, and as enterprise networks get larger, as companies get larger, as things like– as companies have global networks, everything is effectively an open network nowadays, so you have to assume that when you design your apps So why are APIs important, and why is API security important? Well, a really great example I can talk about here is the world of banking Especially in the US, a lot of people use services like Mint or Yodlee, which will do aggregation of their bank account information I use Quicken a lot It has a similar feature And if you use any of those services, the first thing they do was say great, give me your bank account username and password I should have used one of Chris’s disturbed person images for that one because, once someone says give me your bank account username and password, it means what they’re going to do is they’re going to store your bank account password Yeah, they’re going to encrypt it, but they’ve got to decrypt it in order to use it, and then they’re going to screen scrape your bank’s website They’re going to make HTTP requests as if they were a web browser They’re going to parse the HTML, and they’re going to figure out what your bank is telling them So basically, some of the banks have started to realize that this is not great for security because when their end users are giving their passwords to these third parties, you don’t know what the third parties are doing with them You don’t know if the third parties are storing that data in a safe way Even if they’re storing that data in the safest way possible, they still have access to their end users’ unencrypted passwords, and those passwords give you access to do everything you can do from your bank’s website, including transferring large sums of money, making big stock trades, whatever you can do If we had took an API approach, on the other hand, where banks were to offer APIs that used a security mechanism such as OAuth, that would mean that, as an end user, you would not give these third parties your password You would instead go through an authentication process with the bank that gives the third party token, and that token can now be set up by the bank to only offer access to these specific things that the bank wants them to have access to So for instance, JP Morgan Chase realized this and wrote something in their annual report, and they got into a little bit of trouble with the “New York Times” that said [MOCKING] Jamie Dimon wants to protect you from innovation [NORMAL TONE] And he has a point, and they have a point, too But the idea here is by offering open APIs with well-known API

security techniques such as OAuth, we can have a much, much better handle on security because security is not about eliminating all risk Security is about managing risk And by giving third parties a way to have access to important information like financial information in a controlled way, where you’re only allowed to do what the bank allows you to do, we’re in a much better position than we are today, where you give someone your password, and they can do anything So I’m now going to talk about five or six specific API security things and kind of use some examples So one big API security mistake we can make is to not use TLS, or SSL as it used to be called, and now the protocol we’re supposed to use today is actually called TLS So for instance, a security researcher did a study last year, and he discovered that, around the world, especially in South Asia, at the time, something like 30% used non encrypted links Two years ago, that was 90% Now, maybe that’s not your password, but we’re to the point now on the internet where everything that is even slightly important, and in fact, even everything that isn’t, should be encrypted with TLS That’s why google.com, when you do a Google search, is encrypted with TLS The cost is just not there But it’s more than just, hey remember to use TLS, don’t put an unencrypted API on the internet It’s how you use TLS So for instance, in some frameworks for the client, it’s very easy to turn off TLS verification because you’re doing development, and it’s kind of a pain to get the right certificate, and the name on the DNS record doesn’t match the certificate, and we’re in a hurry So we’ll just check that box in Node.JS or PHP or Java or whatever that says don’t verify the TLS certificate That opens it up for your client app to be able to have its supposedly secure web traffic be intercepted by anyone in Starbucks who has control over the Wi-Fi or over the DNS router TLS verification is tremendously important Not only that, TLS has a bazillion options What version are you using– 1.0, 1.1, 1.2? What cipher suite are you using? Are you using Elliptic Curve? What encryption algorithm are you using? And these things change all the time So as anybody who’s offering an API or any web app and is using TLS, you have to remember to keep up with what’s happening in the security world, follow the best practices from your security people and your company and elsewhere, and make sure that you’re continually evaluating how you’re using this technology and keeping up with the latest attacks For instance, I still see developers sometimes occasionally want to write code, and they want to hash something, and they use MD5 MD5 has been a no-no in the security world for years In fact, at Google, it’s not even in the code base They took it out But people still remember this from a few years ago So this is why it’s very important, anytime encryption is used, to really think about this So now, we move on to some of the other basics of APIs Chris talked about APIs that require an end user to be authenticated There are all kinds of things that can go wrong with this So for instance, the Nissan LEAF a little while ago came with a mobile app, and the mobile app let you do things to your car like turn on and off the climate control or start your car remotely And to make it easy for people to use the app, the authentication mechanism for the app required you to put in the car’s VIN number, which is a unique ID that’s on every car It’s actually on the dashboard of your car, so it’s not a secret, and it is structured in such a way that every type of car, most of the digits are the same, and only the last five digits vary So you can imagine that if you knew that, hey, I can turn on the climate control on one Nissan LEAF using a VIN number, what if I were to cycle through 10 or 15,000 VIN numbers and see if I can do that other Nissan LEAFs in the world? And a security researcher named Troy Hunt figured out that the answer was yes Now, I’m not saying this to make Nissan look bad, because they fixed this I’m saying this to give you an example of what happens when you don’t think through API authentication There are a lot of best practices out there for basic things, like signing users in, checking passwords, resetting passwords It’s actually pretty hard to get right Companies like Google do a good job of it Smaller companies that are specialized in the security world do a good job of it Large companies that don’t, don’t always do a great job of it I’ve seen some horrible password reset practices from various websites I use in my life These kinds of things can show up as massive API security vulnerabilities if you’re not careful But it’s not just about authenticating the end user In the world of APIs, one thing we do that we did not do in the world of web apps is we also authenticate the application If you were to ask me what does API management do– because I am an API management guy That’s what we do at Apigee API management is about managing the relationship between the developers who build applications and the APIs

that they use, and a big part of that is managing those security credentials OAuth, which is the de facto standard for API security now– and in fact, from a lot of security now– has built into it the concept of application credentials In order to build an application that gets an OAuth token– say they use a Google Cloud API– you not only have to supply some user credentials, but you’ve got a supply application credentials, which the application developer gets as part of the development process Now, the application of credentials are not 100% proof against attacks I mean, anybody who has access to the application can use those application credentials, But what they do is they give the security team a lot of extra layers of defense, and that’s what security is really about So for instance, we have had cases of Apigee Edge customers who had– we had one who accidentally pushed a version of an app to the App Store that had a bug in it And it was actually hitting an authentication API in an infinite loop, hundreds of times a second, and it was soon overwhelming their whole technology stack Now, if they had a different set of application credentials for every version and platform of that app, they actually would have been able to cut that app off instantly in their API management tier, or redirect traffic from that app to some sort of other server, which is able to mediate it And as a matter of fact, they didn’t quite did that, but they did have the application authenticated, so they were able to take some of those steps But that’s an example of why it’s important to not only authenticate the end user but the application This also gives you an opportunity to do some things like analytics, and we’ll talk a little bit later about bot detection But knowing which application is making an API call is one of the pieces of information you can use in that world Now, I mentioned OAuth OAuth has been around for a couple of years It’s become the de facto standard for API security And basically, one reason why it works is because what it allows you to do is it allows a client who makes an API call to exchange some credentials for a token, and that token gives you access to the API And with OAuth, it’s possible for the API provider, using software like Apigee or many other bits of software, to control what a token is allowed to do Furthermore, unlike a password, a token uniquely identifies a single application on a single device It’s not just a generic bit of security information, and OAuth tokens can be refreshed But OAuth is a fairly complex family of specs They are IETF standards There are a lot of smart people, including some of our security folks at Google, working on it, but there are lots of ways to use OAuth So for instance, one way to use OAuth is to make an API call where you send in your password and get back the token Another way to do it is to have the user’s device open up a web browser window where they can get a token by being redirected to a website, and that website can do additional things like check cookies, ask for a second factor, do all kinds of things like that Now, it turns out that the OAuth folks have actually been working on a spec that describes, hey, these are the best practices for how to use OAuth in native app We recommend using this web browser redirect method Here’s how to do it safely on different mobile platforms Furthermore, there have been a couple of things that have been found as vulnerabilities in the OAuth specs that have been fixed in some of the newer IETF specs So it’s really important, if you’re going to use OAuth, to pay attention to some of the requirements that you see in the community and some of the discussion that goes on, especially in the IETF, because there are a lot of smart people who have written a lot of really good stuff about how to use OAuth correctly I should also mention authorization, and in this form of authorization, I mean once you’ve authenticated a user, and you know who they are, what are they allowed to use It’s easy to make some dumb mistakes This one security researcher discovered that his bank actually– I don’t think it was a very big bank, but they actually were not verifying the account numbers on the back end So he figured out the API that his mobile app was using, and he was able to transfer money from any account to any other account in the bank Now, that’s a huge mistake That’s like people getting fired mistake, but these kinds of things can happen Developers don’t always think of these things One of the things I like about APIs is that an API is a contract, and it’s exactly clear what is supposed to be sent between the client and the server So unlike a web app, where you’re downloading JavaScript and executing who knows what, a provider of an API can specify exactly what’s sent, exactly what’s received, what the valid responses are That means you can write test plans and regression tests and penetration tests, and you can verify that what you think your API is, is what you actually implemented I think that’s actually an advantage of APIs, and it’s an advantage of API security And by having that kind of spec, and by thoroughly testing that spec, you should be able to prevent

these kinds of dumb authorization mistakes that– they’re easy to laugh at now, but there but for the grace of God go all of us Right? Another one, this is really important, is rate limiting So the best example of API rate limiting I could find came from a researcher that did some work a couple of years ago on Snapchat when they were a much smaller and less mature company He discovered that, given a phone number, it was actually possible to get information about someone’s Snapchat profile Now, that doesn’t sound like a big problem, but it turned out that there was no rate limit on this API, and he was able to send in huge batches of phone numbers Now, if you could imagine say Beverly Hills, California– I guess people use cell phones now, but even then, it doesn’t have that many phone numbers in it It wouldn’t take very long to get a lot of interesting information that way So this was, to me, the best example of, in the real world, an API that should have had some sort of rate limit to help with security And I mention this because pretty much any API that could be subject to a brute force attack is something that will be subject to a brute force attack And it may be subject to a brute force attack by someone who is just hammering it for fun, but it could be somebody with legitimate credentials Lots of people can sign up for these services, so things like product IDs, user names– if you can put in someone’s phone number and get information Another great example is that OAuth password grant type, where you send in your password in an API call, and you get back to a security token Well, if there’s no rate limit on that, now you have a distributed password cracking API that people can now use to crack your password database So it’s great to monitor, and this stuff is important But by having basic rate limits on APIs– like no one should call this API more than 500 times a second, or a given application is limited to a certain number of APIs calls per day or hour or month, or even possibly introducing artificial delays to some APIs that may be called very frequently Those make a very, very big difference in the kind of security posture you have and prevent these kinds of attacks And also we’re going to finally mention probably the biggest security list of things that developers need to know about, comes from OWASP It’s called the Top 10 I couldn’t find anything since 2013, but basically, they list the top threats to applications on the internet Interestingly enough, they don’t all apply to native apps that use APIs If you’re building APIs, some of the JavaScript things and cross-site scripting things don’t really apply to you, but the rest of it does Injection attacks, misconfiguration of security, bad authorization decisions– a lot of the things that I’ve talked about are in here, but there are some things I didn’t talk about as well For instance, the number one attack is an injection attack, so classic SQL injection stuff, like can I put semi-colon delete star from customers in my user name It applies to APIs just as much as it applies to web apps Yes, there are things you can do with regular expressions and stuff to protect from this, but it’s really, really important that, as application developers, we remain just as cognizant of this in the world of APIs as we do in the world of web apps So to sum up this part, and then I’m going to talk for two or three more minutes and turn it over to Ismail I think everyone needs to know some basic things about API security, and this is not necessarily the same things you need to know for web app security And a big part of it, in my opinion, is understanding the importance of things like a rate limiting, traffic management, content attacks, and basically, the idea that anything that can be called with a URI, someone is going to figure out what your app is doing and try to break it And as app developers, we have to be very cognizant of that At the same time, by designing APIs that follow the right security principles, by making smart choices about how much security we need on an API And by making sure that our API definitions are clear and well understood, we’ve now designed something that we can test that has a much smaller surface area than a web app And I would argue, if you follow all the rules, and you do everything right, its possible to make things with APIs that have fewer security risks than what we see today with web apps Now, I’m going to talk for just a short time about Apigee Edge This is the product that has been acquired by Google as part of Google Cloud, and basically, what we do in Apigee is we provide a product that lets us insert ourselves into the world of APIs So many companies, when they’re looking to offer an API, they’re asking themselves how do I implement a lot of these things? How do I implement OAuth and connect it to my users– to my existing user authentication system, my existing single signon system? How do I implement a rate limit?

How do I do checks, basic threat protection checks like malformed JSON and XML These are the kinds of things that we’ve always been built into the Apigee Edge product And the idea of Edge is that, as an API provider, you can put the API management product– and there are a lot of products in this category– in front of your API, and it can handle a lot of these aspects, so that your back end, the stuff your developers are building– you don’t have to build this into every service The larger the company you are, the more it makes sense to have one place where you apply many of these security policies as opposed to building them into every single application Many large companies– Google has an API management for its own APIs All the big companies that have APIs have API management as part of their software stack at the edge of the network, somewhere between the load balancers and firewalls and the stuff that actually runs the APIs What Apigee Edge is– it’s a product that allows you to configure it to be that API management layer for the API and configure it however you need it to work Now, I’m going to talk for two minutes here about Apigee Sense, and this is the last thing I’m going to talk about, which is a thing that we’ve used to extend Apigee And this is going to segue nicely to what Ismail’s going to talk about, is we’re going to talk a little bit about bots So there are all kinds of bad things that can happen to APIs People who are not legitimate users may try to hammer your API, to crack it, to brute force it just to mess with you But what can also happen, which is a little bit more subtle and harder to deal with, is when legitimate users who have got legitimate application and end user credentials use your APIs in an illegitimate way This happens a ton, for instance, in the travel industry, where, gee, every hotel has the same prices Well, it’s because somebody somewhere is screen scraping or botting the APIs of all the other hotel websites and trying to book a room in every single price category to try and find out what the prices are and what the availability is There’s a lot of bots that happen in retail Ismail comes from Ticketmaster Some of you may have used Ticketmaster They have some experience with bots, as much as they’re able to tell us So Apigee Sense is a product that uses the fact that we collect a lot of data in the Apigee platform about the APIs that we handle, and basically, we can use that information– it’s information about IP addresses and things– to figure out and to score incoming API calls on whether or not we think they might be a bot Some of this is heuristic, like for instance, American retail companies don’t have a lot of customers in Eastern Europe Or some of it is heuristic like iPads tend not to be located in Amazon data centers Some of it is machine learning, and some of it is standard big data stuff Was one OAuth token used to make more APIs calls than a human user could have made using an app? Was one OAuth tokin used to scan through APIs, like to scan through all the phone numbers in Beverly Hills, from 0000 to 99999? These are the sorts of things that you can see, and by looking for patterns like this, we’re able to then score incoming API calls as definitely a bot, like we know this IP address is a bot because everybody is getting bot traffic from it, to probably not a bot, to something in between, and then take action Sometimes, take action is block the API call, but that lets the bot people know that you’re blocking them So sometimes a better action is to let the back end know so it can provide a different result, a different price, a different availability, even slow down the API traffic So that’s kind of what we’re looking at doing So with that, that’s really all that I have to say about Apigee I’m now going to turn it on to Ismail, who is an SVP at Ticketmaster and is going to talk to us about his experiences with APIs and API security Thank you very much ISMAIL ELSHAREEF: Thank you, guys Oh, whoa, it’s loud Hi, everyone Thank you, Apigee and Google, for having me here This is really exciting to be here to talk to you guys about our experience with APIs from a business perspective I’m going to talk a little bit about the tech involved as well So at Ticketmaster, we’re all about delighting global fans in a distributed world A little bit about myself before I jump into that I’ve been in this battlefield of tech and product for a while now I started as a software engineer, back end engineer and front end engineer I’ve been doing APIs for the last seven years or eight years Joined Ticketmaster 18 months ago to open up the platform and open up their APIs, and six months ago, I started running the entire product organization– consumer product organization, so very, very exciting So show of hands– who’s been on Ticketmaster the last six months? Tried to buy tickets?

I can’t see Not a lot, actually, which is surprising So keep your hands up Keep it up if you were not able to buy tickets when you visited Ticketmaster I don’t need this I don’t need to talk I don’t need to talk It’s all good It’s happening Oh, we have one person OK, I’m here for you, man I’m here for you So we have problems Not every fan that comes to Ticketmaster.com could buy tickets So that’s why, when we started, what really got me excited about the opportunity at Ticketmaster is the leadership there– Jody Mulkey, the CTO, and the entire team– is that we’re really reinventing Ticketmaster We’re changing the way we’re selling tickets, and that’s by having a fan first thinking From our infrastructure, and completely changing and upgrading our infrastructure– we have an incredible partnership with AWS and Amazon where we’re moving our private cloud into AWS, and it’s a process that we’re working on right now– into opening our APIs, and that’s a partnership that we have with Apigee that’s been fantastic So infrastructure opening up the APIs, and then our products had to be built with fans in mind first, which is not traditionally the way it’s been done in the past And what that means is that really looking at all our fans, different personas, and understanding why we’re building the products that we’re building And our promise to our fans is that we are unlocking moments of joy for global fans through unforgettable product experiences So if you think about that, and thinking about the fan and fan first, you’ll see that– and I don’t know if you guys follow Ticketmaster– we have been changing the way we offer value to fans So for example, this happened today It’s a pre-sale for Ed Sheeran, who’s coming to North America It’s going to be one of the biggest tours this year His new album dropped a week ago, and it’s killing it But this way is for us to say, hey fans, preregister, and we will text you a link before the presale where you can buy Ed Sheeran tickets We did it with 1975, John Mayer, Depeche Mode– you name it Because what we find is that, when fans come to Ticketmaster for Depeche Mode, for example, who– they haven’t toured in seven years, and if you can get tickets, you’re going to freak out because I will freak out, too, because I’ve been wanting to go And I signed up, and I got– yes, we have a fan in the house I love it But it’s real No one’s going to drop $4,000 to go see Beyonce at the Rose Bowl if they’re thinking logically So it’s an emotional business that we’re in We just want to make sure that fans are not disappointed, are not turned down, and are actually getting the tickets that they want We see 72 million unique visitors a month Not all of them are fans So we have a real, real need here, and a real problem to solve with some of the challenges we run into So tickets are the only commodity I can think of that go up in price– up in value once you buy them I don’t know if you guys bought tickets to Hamilton recently I have not seen it I can’t afford it But we deal with a lot of issues and a lot of problems Bots are one of them Limit violations– people are trying to circumvent the limits we put on an event So as a fan you cannot buy– for certain events, you cannot buy more than eight tickets For some events, you cannot buy more than four tickets, just to make it fair and give access to a lot of other fans that want to go see the act And some people and some bots and some bad actors trying to circumvent that by they want to get, I don’t know, 100 tickets I’m talking about concerts now, and theater, not sporting events per se So there’s that level of complication and challenge that we have, and then the biggest challenge we have is reserve and do not buy So during an on sale, when people bad actors come and select a pair of tickets or a bunch of tickets, and they don’t actually end up buying them, what happens is that when they actually select them, those tickets are put into a hold, and no other fans can get to them, which is bad We have ways of going around that, obviously We’ve been doing this for a while now, so we have our own algorithms and our own systems that can kind of determine or try to glean whether this traffic is coming from a good actor, a real fan, a bad fan, a robot, a non robot, and all of that good stuff But that’s just ticketmaster.com, Ticketmaster apps That’s just our closed system But in order for us to deliver on our promise to our fans, our global fans, we determined that we

have to open up the platform We have to make our APIs available to other developers, to other partners, clients, where they can build applications or can build experiences for their own fans, so everyone gets access to those tickets So when I started 18 months ago, what we did was we consolidated all of these different platforms that we have into one simplified, very easy to use platform partnered with Apigee to expose those APIs And they would start using those APIs across the board in all different channels starting with our own apps and their own properties, but also for pre-sales on fan clubs, for example, last minute channels, you name it So we have different outlets for those APIs But the reason we wanted to go this route, obviously to open up the platform, but also for any measurement of security that we apply to one channel now, we apply it to the APIs, and that’s it We don’t have to think about it anymore Because when you apply it to the APIs, it doesn’t matter who’s using the API The same level of security is going to be applied In our world, what Greg said, that security is not eliminating risk It’s mitigating risk, and for us, it’s all about mitigating risk It’s all about mitigating risk We’ll never know, in real time, who’s a true fan and who’s a bot So if you’re like me, if you go to a show once a week– and I pay out of pocket, by the way I don’t get free tickets If you know how to get free tickets, let me know My behavior and my pattern of activity can look like a bot to our system because I’m coming from my home wifi trying to buy tickets to John Mayer, to 1975, this and that And it’s like, oh, my god, this guy or that entity is buying a lot of tickets, so maybe it’s suspect We’re trying to make it smarter and better, but true fans and super fans can come across as bots, and that’s a problem that we’re trying to solve But we opened up the platform We have a portal, developer.ticketmaster.com We created an ecosystem where you can go to the portal and understand what APIs we have, who’s using them, and how you can get access to them We really followed all the best practices and everything that Apigee advocates and puts out there And from my previous experience as well, all the best practices that I learned the hard way sometimes, that we should be doing in order for us to have a successful API program So we’ve done all of this stuff We integrated with some incredible partners to get access and to reach other fans that might not necessarily come to ticketmaster.com And as a result of that opening up the platform and really thinking broadly, we’ve seen, just last year, that 10% of all tickets sold by us came off of platform, came through partners, came through our APIs, outside of our products Fascinating, right? But guess what– the challenges still remain The challenges that we’ve seen in our own products still remained true in the open platform world They just got amplified Actually, almost all the errors we see in the Apigee portal coming through our APIs are of these two kinds of errors Too many requests, rate limiting– so people trying to make too many API requests, trying to get access to those tickets, reserve them And because we have rate limiting, it’s preventing our system from going down and overwhelming our system, but also really controlling the API access for that particular user or application But also look at the unauthorized calls A lot of people are trying to make calls without being authorized, without the right OAuth token It’s a real problem And because building an API– a lot of people think, oh, it’s a simple interface, and I have the SDK and documentation It’s pretty simple Right? It actually takes a lot of effort and planning, especially when it comes to security, identity management, who has access to what, error messaging, all of that– it takes a village for the API to actually be useful and to be simple That’s the challenge we run into, so when we focus on the fan and opening up everything and trying to think how do we make that access of this API that we just put out there favorable to fans, and for developers to build applications that actually let fans through So the way we did it was really follow the basic best

practices I cannot emphasize that A lot of times, the biggest problems happen because you don’t follow the guidelines and the best practices, like authorization, authentication, OAuth, rate limiting, using bot blocking that Apigee provide if you don’t do this for a living like we do, SSL, just the basic stuff And if you do the basic stuff, you’re actually 50% there, like Greg and Chris were talking about In our case, we really had to think about the reserve API call That’s where actually people put real tickets, real value, on hold, so we need to rethink how that API call actually works So the way we did it was we’re introducing friction We’re trying to introduce friction It’s like you cannot do that anonymously You authenticated your application We have the OAuth token Great But now, we want to know who’s the actual user making that call, because then I’ll have extra information about that user that will help me determine is that really a legitimate fan, or is that a bot that’s trying to– that hacked the app, and now using it in a bad way So introducing friction by forcing registration or signing in, using whatever you want to sign in for We use Ticketmaster Accounts for that And using the fan score, which is our algorithm to determine whether you’re a good actor or a bad actor– basically, a fan or a bot And the last thing, obviously, which is really, really important– and we just did that with Apigee– is identity management, entitlements So when somebody’s making an API call with their OAuth token– so if that checks out, that means Apigee says, OK, this OAuth token is good Now, taking their API key and their identity and pinging our system, saying what role does this player have in our system? Are they a supervisor? Are they a fan with a bunch of tickets that they can transfer, and all of that stuff? And once you get that back, you add a bunch of headers to the call, and then when we get it in the back end, we understand if this action is permitted by this user or not So if you want to transfer some tickets– now just because you’re a fan doesn’t mean anything You have to be a fan and also have the rights to transfer those tickets or to sell those tickets or to cancel an order So these are the three learnings that we found Just opening up an API ecosystem and a platform actually works really well Is it perfect? It’s not It’s never going to be perfect, and we’re always learning every single day But the fact that we sold 10% of our tickets off of platform using our APIs that we opened up just 18 months ago is a huge testament to the power of APIs to business, but also the fact that you have to stay vigilant in order for you to really expand I mean, we haven’t scratched the surface with APIs just yet And with that, I thank you for being here, and we’re hiring We want people to help us with this transformation that we’re on And I guess I’ll open up to questions [MUSIC PLAYING]