Modern Web Testing and Automation with Puppeteer (Google I/O ’19)

[MUSIC PLAYING] ANDREY LUSHNIKOV: Hi, everybody My name is Andrey This is Joel JOEL EINBINDER: Hi ANDREY LUSHNIKOV: We are Chromium engineers working on DevTools and Puppeteer So, we know this is the third day of Google I/O, and it might have been overwhelming So thank you for staying awake, and keep coming to the talks Today, we will talk about modern web testing and automation with Puppeteer We will split our talk into a few parts First, we will introduce you to Puppeteer Then, there will be a surprise announcement Then we’ll talk about testing the modern web with Puppeteer And then we’ll wrap up with a [? community. ?] So, what is Puppeteer? Puppeteer is a browser automation solution It brings the power of web technologies with a simple to use and easy APIs With Puppeteer, you can do many things You can open pages, navigate the websites, evaluate JavaScript inside these websites, and many, many more stuff In fact, we encoded this in our logo You see over there is a browser frame This is Chrome Up there– the green thing, with bars and strings? This is nodegs that controls Chrome through the APIs Seeing once is better than hearing So I want to actually show you the Puppeteer And for this, let me introduce you to Joel’s personal merchandise website On this website, Joel sells T-shirts, coffee mugs, stickers, all with his name on it My personal favorite is this T-shirt, but unfortunately, it is a little bit pricey for me It is $50 So every once in a while, I keep coming to Joel’s website and hope for a discount But unfortunately, it doesn’t work like this And being a lazy developer, I thought that this is a perfect test for web automation Let’s see how I did this with Puppeteer The first thing I do, I install Puppeteer with NPM And when Puppeteer is getting installed, it downloads the browser And this already eliminates the biggest pain point of browser automation So with Puppeteer, there is no need to configure, link, or otherwise manage the browser installation I just do NPM install, and I am ready to go coding So let’s code This is not just script The first thing I do is require Puppeteer Now all my code will go in between these parentheses I use the async function so that I can use the async await syntax of the JavaScript The first two lines– I will launch a browser with one line, and I’ll open a new page in this new browser In the top right corner, you can see the browser frame This is actually what it will look like Next, I will navigate the page to Joel’s website And then I know that Joel has his price in this specific CSS selector So I will use this Puppeteer API to fix the price and output it to nodegs console And, yes, I will not forget to close the browser This is as easy as it gets This is just six lines, one NPM install, and I have [INAUDIBLE] automation for Joel’s website that pulls the price It’s really simple Now let’s see how it worked I had it running for a few days before I/O And this is a chart of Joel’s T-shirt price As you see, the price unfortunately didn’t go down, but, in fact, increased, which is fair given the I/O demand JOEL EINBINDER: I want to make some money at I/O ANDREY LUSHNIKOV: However, I have one last hope I have heard that certain websites change prices depending on the browser I use But Puppeteer is all about Chromium But we are happy to say that there is another browser that is joining the Puppeteer family With this, let me introduce you to Puppeteer Firefox [APPLAUSE] Puppeteer for Firefox is currently an experimental project It is available on PM, as Puppeteer Firefox, and it is a work in progress But it already supports 90% of Puppeteer API We are working with Mozilla to bring the product out of experiment and support more Puppeteer APIs Let’s see what it takes to convert my script from Puppeteer to actually Puppeteer Firefox Again, the first thing I do, I do NPM install Puppeteer Firefox And similar to Puppeteer, it downloads a browser, but this time it is not a Chromium It’s Firefox After that, it’s just one line I change require from Puppeteer to Puppetter Firefox

My script works with Firefox So let’s see how it actually works So this is the Firefox window It does resemble Chromium But actually something is going on over there Let me zoom in for you Gosh OK, so it looks like Joel’s website doesn’t work in Firefox So clearly my automation adventure didn’t succeed, but at least we found the problem So Joel is a great web developer, and Firefox is a great web browser But clearly, certain things require testing, and require cross-browser testing for this kind of thing So Joel, can you please tell us, why didn’t you test your website? JOEL EINBINDER: You’re calling me out at I/O for not testing my website So I was scared to test my website because I thought that it would be really slow to test and complicated And I was worried that I’d spend more time maintaining my tests than actually writing my website But, of course, this being the Puppeteer testing talk, we hope that, with Puppeteer, we can solve these problems for testing on the web So with Puppeteer, there’s no complicated setup It’s really simple You just do NPM install And it works the same as any other NPM library, like ESLint or TypeScript, that you’re probably already using It being a Node.js library, it supports any JavaScript test runner, like Jest, AVA, Mocha, Karma, JUnit There’s a lot of them But people are using all of them with Puppeteer, and it’s working great So getting Puppeteer working on your desktop environment is really simple But it also works on your CI We personally test Puppeteer on Travis and AppVeyor on our GitHub, but it works on others, too And it works on all the cloud providers– GCP, AWS And if you need to, it works in Docker containers So you can run it pretty much anywhere So web tests have a reputation for being kind of slow But why actually is that? Because I’ve spent a lot of time making my website really fast Hopefully your websites are fast So then why are your tests slow? One reason tests can be slow is because you’re launching a new browser for every single test This is actually a really good idea because it keeps your tests isolated from each other Otherwise, tests might interfere– cookies could be set, network caching So running a new browser is great for isolation But it has a downside because it makes your tests really slow With Puppeteer, we have a special API that we call Browser Context that we hope fixes this for you It’s based on incognito mode in Chrome, and it isolates your pages from each other But you don’t have to launch an entire new browser So this gives you all the benefits of launching a new browser, but way faster It’s actually 100 times faster So we really encourage you to use Browser Context And let’s take a look at some code and see how you do that So here I have a test, that I just verify that a Pay button exists on my merch site It’s a very important button At the beginning of the test, I’m launching a new browser, and at the end of the test, I’m closing that browser Let’s look at how we make it [INAUDIBLE] browser contexts So first, I move the browser launch call outside of my test because I’m going to share the same browser between all of my tests Then, in my test, I create a new incognito browser context, and close it when I’m done It’s that simple, and it gives you a huge performance win So web testing has another bad reputation, which is for being flaky Sometimes your tests pass, and sometimes it’s fail Personally, for me, I feel like a flaky test is actually worse than having no test because then I have to deal with the flakiness and the problem with my code A common fix is retrying tests, but there are some other better solutions So here’s a test for my website that checks that I can click the Pay button and then the purchase is successful But this test is actually flaky My Pay button appears asynchronously after the page load So sometimes it doesn’t appear yet when I try to click it, and sometimes my purchase hasn’t completed when I check for the Successful Purchase button So this test right now fails about 70% of the time A common solution for fixing this, if you’ve done web testing, is to sleep for a bit in between asynchronous tasks So with Puppeteer, we’re waiting for a second after the page load for the button to appear, and then we’re waiting for another second for the payment to complete This makes the test less flaky– it’s still a little flaky– and it slows it down a lot But with Puppeteer, we have an even better solution So we Puppeteer, we have a set of reactive APIs that we call our waitForX APIs You can wait for Dom elements to appear, you can wait for network requests to be sent and finish, and many more So let’s look at the right way to fix this test So instead of waiting for a second after our load for the Pay button to appear, we’re just going to wait directly for the Pay button Then, instead of waiting for a second for our payment

to finish, we’re going to wait for the payment network response to come into the website Then we know we can check for our successful payment element ANDREY LUSHNIKOV: OK, so let me recap So Joel didn’t test his website because he was scared But it turns out that if he were to use Puppeteer, his fears would not apply So with Puppeteer, tests are fast because of Headless Chrome and browser contexts So tests in Puppeteer is reliable because of all the reactive APIs we have, and testing in Puppeteer is actually simple because it’s just one NPM install away from you So now, with Puppeteer and Puppeteer Firefox, you can actually test for different browsers, for Chrome and Firefox Let’s see how we can use Puppeteer to test the modern web And modern web is not what it used to be like 10 years ago Every other website today is in a web app We spend a lot of time to make sure the performance is good and that the website is accessible So how do we actually test for this? Let’s go and see certain features that Puppeteer lets you do So first up, mobile So it is 2019, and we as web developers spend a lot of effort to make sure that our websites work on both desktops, mobiles, and tablets And we do it different ways, right? We can do fluid layouts, responsive designs, or we can even build separate websites for different devices and [? sniff ?] for the user agent So how do we test for this? Well, manually, you can pull up your phone and load your website over there That’s easy Or you can unload your website on a desktop, open DevTools, and enable device simulation to see how it works over there But how do you make sure that your website doesn’t regress over time? How do you automatically test this? Well, it turns out it’s really easy with Puppeteer So let’s write a test together that makes sure that Joe’s website works on iPhone, and that there is a Pay button there So first I will open the page And it’s a very important– in the bottom right corner, there is a browser window It looks like Chrome Now, with one comment, I will emulate an iPhone 8 And now the page becomes an iPhone So now whenever I navigate the page towards Joel’s website, his website is absolutely sure that it’s been loaded in an iPhone Now I can easily assert that there’s a Pay button Now when I say that the website is sure this is a real iPhone, I really mean it because our emulation is really extensive We emulate not only the viewport sites, but also the user agent, DPI, touch support, landscape mode, all the things you should care about when the devices are real And what is even better, we have more than hundreds of these devices available there [INAUDIBLE] So with Puppeteer and mobile emulation, you can make sure that your important checkout [INAUDIBLE] works in mobile This is really great Next up, offline support So offline is really critical to certain types of web applications, for example, [INAUDIBLE] maps, or subway maps But it turns out that regular websites, every other website, can actually take advantage of support in offline mode So let’s see what Joel did on his website So when the user goes offline, his website shows a big red banner that [INAUDIBLE] me that I cannot actually purchase a T-shirt So how do we test the offline mode? Well, manually, I can load it on my phone and enable airplane mode Or, for example, I can open DevTools and trigger an offline mode in DevTools Or I can even turn off my Wi-Fi on laptop So automatically, I can do the same thing with Puppeteer So let’s write a test that makes sure that this website shows this red box whenever it goes offline So as usual, I’m going to open a new page I will navigate to Joel’s website And now, with this next [INAUDIBLE] this page setOfflineMode, I will transition the page from online state to offline state This will trigger all the necessary APIs inside the page And if you were to do any request, it will fail It gives No Network Connection error So now it’s really easy for me to assert that the offline alert banner is over there That’s it So for me, this test is not only easy, but actually a pleasure to write Now if your deal with offline, chances are high that you actually have to do it with service worker, and you know what it is So service worker, this is a big, very important part of web platform today And they help us to do caching on the web Now, how do I actually make sure that service worker registers? To help me with my demo, I want to introduce you to the other website So this is This is a website with [INAUDIBLE] [? documentation ?] available online– for Puppeteer documentation It is actually [INAUDIBLE] searchable,

it works offline, supports devices and everything It’s a real modern application And it does support service workers, which is important for us So since service workers are so complicated, we used Workbox to help us configure the service worker But how do we actually know that services workers will work? Well, the only way today is to go open DevTools and navigate to Applications panel, and find service worker over there If they’re there, it’s good But automatically, you can do the same thing with Puppeteer, and it’s actually even easier So let’s write a test that verifies that service worker registers for my page Again, I open a page, and now I navigate it to the Now something really important is actually happening here So not only is the website being loaded in the page, but also service worker is getting registered in the browser context of the page So with this, I can actually wait for it to appear in the context with these three lines of code And this is already a good test It makes sure service worker gets registered But there is actually more And to show you more, let me bring you to another slide So now when we have the service worker target, [INAUDIBLE] we can actually attach to it And once I attach, I can evaluate JavaScript inside the service worker global scope Now with this, I can make sure that, for example, my logo is properly cached But there is more, and this actually opens limitless possibilities for service worker testing OK, let’s talk about geolocation So the web is global, and your application is global as well So there are many different cool applications for geolocation, for example, if you have a map and you want to show where user is, or if you have a news feed on your website and you want to serve the relevant content So how do I actually test these things today? Turns out, I do two things First, I navigate to my website and I grant it the permission to query for my geolocation Well, next, I’ll open DevTools, and it will emulate some location For example, pretend I’m in London Now with Puppeteer, I can do exactly the same thing So let’s write a test that makes sure that Joel’s shows pounds when we are in London So first thing I do, I set up my browsing context, and I make sure there is a permission for Joel’s website to query geolocation Now I can navigate the page, and with this one call, page.setGeolocation, I can convince the page that it is actually in London So now when Joel’s website is getting loaded inside the page, you can query for geolocation, and it actually thinks it’s in London And you can query the price, and it is actually 40 pounds So good job, Joel OK, off to the really powerful stuff we have in Puppeteer, and this is network monitoring So network is usually big point in different browser automation things And the golden standard for network monitoring is DevTools network panel So DevTools network panel gives you a waterfall of all the requests and all the network activity that is actually happening inside the page So you can see requests, responses, you can see status codes You can even get the content of the responses Now, what if I tell you that you can get the same Network Monitor but with Puppeteer? So Puppeteer gives you two events– request and response And with these, you can recreate the same DevTools monitor, but programmatically So there are many different APIs available over there You can [INAUDIBLE] for headers, for post data, and you can even check if the response is coming from service worker or from cache This is really powerful But there is more Puppeteer can not only do network monitoring, but also network modifications So let me introduce you to the request interception And to demonstrate this feature, let’s substitute every image on the web with a cat So if you were to go to the developer’s Google Chrome, you will have cats all around Let’s understand how it works So say I have a browser, the page on the left, and servers on the right And there is an image on the website that actually issues a request Now, if the request is getting intercepted by the Puppeteer, we can inspect it and see that it is actually coming from the image so we can fulfill it with a random cat image Now if there is a style sheet on the page, we can again intercept the request and see that– we don’t have anything to do with it, right? We care about only images So we can continue and let it [? keep ?] the service Now in code, it looks really simple So first thing we do, we enable the request interception Then, we subscribe to request [? event, ?]

and [INAUDIBLE] request actually gives me the type of the resource it is attached to So I’m going to connect to image, and if it is an image, I will respond with a random cat image Otherwise, I will just continue to the server So we really like this [INAUDIBLE] So this is the video of what it looks like if you [INAUDIBLE] of Twitter with this functionality It looks much better So what does it all have to do with testing? So Joel’s website has a piece of functionality that handles the case when payments don’t go through So in this case, Joel shows this red banner, Payment Failed So it turns out request interception is a really powerful thing to do server [INAUDIBLE] So the same thing– we’re going to subscribe to request event And for only the request of his payment endpoint, we will abort it Otherwise, we will just continue And with this, it is really easy to mock a server and APIs So let me hand it off to Joel to tell us about more input detection and other stuff JOEL EINBINDER: Thanks So I’m talking about keyboards, and mice, and user input, which you might think is simpler than the network stuff Andrey was talking about, but it’s still pretty complicated If you’ve done web testing before, you’re probably familiar with the Dispatch Event API This lets you send fake JavaScript events It calls all your event handlers, your onmousedown callbacks This works great for making sure that your site works for events It works all in JavaScript But this actually misses some of the story of real user events on the web When real mouse and keyboard come into Chrome, they get sent to Blink, where they affect your CSS– like your colon hover, pseudo selectors– they go into DOM and they change your text selection and your focus So with Puppeteer, we send events directly into Chrome in the same place that user events come in, and they have all the wonderful side effects of user events They go through Shadow DOM correctly, they go through iframes, you can resize a text area, and our keyboard actually sends the right text in with the keys So let’s look at a test that actually types something in with Puppeteer If you remember our documentation site before, it has an awesome feature where you can search So we’re going to open a new page We’re going to go to our documentation site, And then we’re going to find the search element, and we’re going to type JavaScript into it Then all we have to do is hit Enter, and we should see the results of the search Let’s look at it in action So we actually slowed this down so that you can see it happening Puppeteer is much faster But you see, it types into the page and it periodically performs the search So it’s not enough just to make sure that your site works correctly We also want to make sure that it works as best as possible, so performance testing is really important And Puppeteer has a lot of great APIs for you to automate performance testing on your CS My favorite API is Page Metrics It’s really simple It just gives you a nice list of all sorts of metrics on your site, the same as the Performance Monitor in DevTools My favorite number down there is the JS heap size used Some of you might complain about how much memory your website uses Maybe you blame a certain web browser But with Puppeteer, you can monitor your own memory usage and make sure that you don’t regress If Page Metrics that isn’t enough for you, you probably want to look into Chrome Tracing Tracing is a feature in Chrome where we instrument the entire browser– like CSS, animation frames, network– and you’re pretty familiar with it in the Performance panel in DevTools When you want to take a performance trace in DevTools, you hit Record and Stop, and get the trace And with Puppeteer, we have the same API So let’s make a performance trace for this test that selects the mug item on my site Before the test, I’m just going to call page.tracing.start, and I’m going to specify a path where I want the trace JSON to go Then, when my test is done, I hit Stop, and then I can drag that JSON into DevTools and see the trace, just like I took it in DevTools So Code Coverage is another really powerful tool in DevTools It tells you exactly which CSS and JavaScript were used on your site, so you can make sure that you only ship what was needed or that the right code ran And just like we have tracing in Puppeteer and DevTools, we also have Code Coverage in Puppeteer You can start and stop CSS Code Coverage and JS Code Coverage, the same as tracing So your site working fast is not the only aspect of it working correctly We also want to make sure our sites are accessible

You can test manually for accessibility, with screen readers and other assistive technologies, but it’s very hard to automate accessibility testing So that’s why we have this awesome API in Puppeteer, called the Page Accessibility Snapshot This gives you the accessibility tree out of Chrome The accessibility tree is what’s sent to assistive technologies, and they tell them what content is on your website So it very closely matches what is actually read to users or displayed This is the accessibility tree of my merch site As you can see, it has the title of the site, it has an image of my shirt, and of course it has the Buy button, the most important part of the site Let’s look at how we can use the accessibility tree to write a test to make sure that my site is accessible So I’m going to query for the button on my site, the Buy button, just regular DOM element Then I’ll take a snapshot of the accessibility tree for just that button, and I’ll verify that it has the correct Google Pay name I had a lot of fun with accessibility trees in Puppeteer, so I actually built a tool I called it Accessibility Render So this takes the accessibility tree that was generated from your page, and it generates another web page from just the tree So you can see if they match up and if all your content is displayed correctly in the accessibility tree of your website So here’s my merch site And it’s actually its own web browser You can navigate to any site and see its accessibility tree rendered as its own web page So here’s Google, and you can try your own sites and make sure that they look how you’d expect them to look You can find the accessibility render on GitHub, and of course you can get your accessibility trees through Puppeteer ANDREY LUSHNIKOV: Thank you, Joel So this was it for the features So let’s talk about community So Puppeteer has been around since 2017, and is available on GitHub And many different companies actually use it So Google uses Puppeteer internally But I guess no surprise here Netflix uses Puppeteer internally, and also Facebook uses Puppeteer to drive their performance analysis But not only companies– there are many different people who use Puppeteer So there is a Slack channel where thousands of people communicate with each other and help each other to solve questions related to Puppeteer But community not only talks about Puppeteer, but actually embraces and contributes to it So more than 100 people have stepped up to help us maintain our API docs So our API docs are really good quality We have code snippets for all our API methods Go check them out It’s really good stuff– all thanks to the community But really important, and we’re really proud of it that 20% of all contributions to the core library are actually community contributions This is a big number for an open-source project So last but not least, Puppeteer is an NPM-native project There is a huge ecosystem of packages that depend directly on Puppeteer– more than 1,000 of them– and we believe that Puppeteer has spawned a new wave of next-generation web developer tooling So I want to give you a few examples of what is possible with Puppeteer and what people do with it The first is the project called Size Limit And Size Limit uses Puppeteer to measure how much time it takes for a website to load and run [INAUDIBLE] So with Size Limit, web developers stay aware of the size of their website And it makes them ship less code, and websites load faster for all of us This is good So same thing, but for CSS, is called Penthouse Penthouse uses Puppeteer to extract the critical CSS, or only the CSS which is absolutely necessary for the page to load So with this tool, you can again ship less CSS and make your website load faster and your users happier So this is the end of our talk Thanks for sticking around So, summary– Puppeteer is the official Chrome API We are very excited about experimental Firefox support and our collaboration with Mozilla And yes, the modern web is testable with Puppeteer in both Chrome and Firefox So please test the websites Thank you This is Andrey, Joel [APPLAUSE] [TITLE MUSIC PLAYING]