Twisted tutorial part 1

see but somebody’s going to remind me to go my hotel rooms is that should we get started we is it 9:00 so no one hi I’m plugging things is in is hard for Lauren’s here but let’s be tricky and double-click on that ah you do yes just miss the errors never show that again so just to be sure we are doing the twister training and it was what with the with the other testing training that was here scheduled here so if you are here for the testing training you need to go upstairs in the pits and up your room alright alright welcome welcome so this is an introductory talk on how to write code with twisted we’re going to be covering how to write clients servers how to do flow control with deferreds a few tricks and tips different ways of doing things if you if you did anybody go to the talk last year the same training session last year that’s fantastic great does this clicker work well yes but you have to yes but you have you plug it in

post it oh well electronic says that so twisted is an asynchronous i/o framework meaning principally single threaded event loop driven callbacks the idea of the idea of this training is you’ll be able to comfortably write some code dive into the sauce find out the the right things to use and hopefully ask the right questions apparently even the spacebar doesn’t work anymore all right so my name is Steven thorne I’m a I’m an SRE with Google and I’ve done quite a bit of networking in my in my career and this is so I’m Lawrence and I’m I’ve been at risk or dev since 2009 currently doing a startup that uses twisted extensively and my last startup was also one that used twisted extensively and ended up being a good exit so yeah it’s been good to be somehow I’ve been a dev since sort of 2002 but it’s not the sort of thing okay so raise your hand if you don’t know work programming before at all right event loop programming you select pole all right and in other languages Java okay great now we’ve got on the files EP website some some files that would be useful for you I also have them on these USB keys if you want if you’re if anybody needs some files the DP should be quite quick passes around and all of them no no I haven’t loaded those yet yeah sure so they’re on these keys that they so um that’s a quick link to that but it you can actually just navigate to file so DP go down the tutorial and then the twisted training is the link you want a later version of twisted in fact I put 12.1 up there because that’s the current release so the twisted versions they go on here so that’s 2012 the second release of 2012 the I didn’t put the API Doc’s on there that was silly of me there on the net we go down on them uh yeah if you know yeah do that so there’s there’s some example files there which will be useful because it will save you a bit of typing during it during the presentation yeah yeah it’ll do that if you have twister installed we basically want to get to about that point and if you have a Mac and you need GCC tools there on the USB keys as well as well as a real version of Python oh is that the best way getting is that the best way really it’s Gnostic Oh Abed the best place for the camera around anyway the brainy yeah shop please feel free to ask questions at any time especially if I’m being unclear or just any time yeah sorry you

don’t have to take the whole thing you can just get a file sleepy that short link should work provided the internet does that if the interns are working it’s all on the USB keys yep the Mac port version is yet the one that’s installed on Snow Leopard is version 11 it’s kind of alright it’s just a bit painful if you want to install extra tools yep most annoyingly virtually doesn’t work for it no no don’t don’t trust too much but it version 11 should work yeah right right think I’ll just continue on that’s very strange what if it turns out that when you step on the cable it unplugs all right so I’m just going to go through some introductory network concepts I mean it’s it’s quite self-explanatory what a network program is sending sending a stream between two sites server client we’re going to be starting off writing clients move onto servers but of course servers end up being clients basic tcp/ip server listens on a port client connects that port with twisted fortunately now finally has rudimentary ipv6 support so but I haven’t really investigated that lately but you know this is once you establish a stream they’re all the same reading and writing data in a in a in a blocking not a sink well a system where you have one throw to one process per connection you can simply just call write and that will return when it’s finished writing you call read it will return what it when it has received some data the idea of see I didn’t write these slides originally so I don’t know what’s coming sometimes so ETA in on a single threaded situation where you have no concurrency whatsoever you end up in a situation where when multiple clients connect at simultaneously you end up with a second client getting rejected or in a queue or depending on your your queue length um but they’ll get timeouts and whatnot and it’s just you know not positive so there are different paradigms are doing things you want to be able to have multiple clients connect and there are various different techniques for P a new process having worker threads and asynchronous i/o now twisted is asynchronous IO I’m sure many of you have done various different ways of doing this so I mean if you’ve ever run a web server they often use a combination of techniques a patch uses multi process and can use multi thread and all sorts of things I think this is a description from the web site I mean you can read that so the idea is okay so this is from the API Docs this is some of the technologies which are available these are the sub packages of twisted dot so just looking down this list Twista has implementations for ssh v2 and telnet that’s conch ah authentication databases please ignore the documentation generation system the Sphinx is much much better mail manhole is actually very interesting it allows you to connect into a running process

and get a Python prompt and you can interact with the running process I might be a bit of inserting it here but if if you are running twisted and you want to be able to divide what the heck is going on well traditionally I would always open up a Python prompt and start typing at it you know if it’s a local program and just import my modules it’s very hard to do that when you go to the running event loop man Hall allows you to do that actually just tell it that telnet N or SSH in and find out blah blah blah this is a this is actually a bit of a dated graph but it shows that the components are twisted so this have a laser on it yes little button somewhere there’s a middle button supposed to well I wanted was freaking sharks with frickin laser beams alright so all the way on the left Twitter’s application that’s that’s what’s what the box is labeled the reactor is our event loop that can do GUI integration so you can run a twisted program with a event loop and that can I interact with their user interface event loop that’s there so if you can run GDK applications and it and it cooperates correctly with the GDK main loop the windows main loop the well WX which doesn’t really have a main loop but it’s still tries to integrate with it pretty much anything by Objective C there’s a way of if you’re writing Mac OS X applications you can have it neatly cooperate with all of us X’s native event loop stuff so pretty much any anything that you can imagine tk2 if you’re really into that anything you can imagine you can write an event loop and you can have it cooperate nicely with twisted all right over there we just below that we have twisty there that’s the if you if you are writing a application you want to sort of set it up as a daemon on your system well you know in etcetera init.d blah blah blah twisty is the best way to do entry point to that because it gives you all the things you need for demonization system logging all of that that that setup process is is encapsulated with into a sneak command line arguments trial as a unit testing framework I said ignore law really do up here we have twisted right at the top we have twisted Internet all right do have a laser so that’s that supports TCP UDP SSL SSL’s very important security is very important you need be able to support it it’s actually quite a quite easy provided you have your certificate set up protocols so this is all the high-level protocols there’s a lot that are pre implemented or or whatnot DNS SSH SMTP HTTP pop3 and some more esoteric ones so if you’re on a postfix server for instance you can have a postfix look up daemon widget which just ends up being you know subclass one thing override a method and then suddenly you can interact with so I said Postgres yes I meant postfix I was I was going to incorrect you it yeah but there’s post Chester’s native Postgres implementation in there too but I don’t think it’s interested protocols yet there’s a ticket for it listen because that’s in a txt Alaska so on launchpad if you go to launchpad net / TX there’s a you know it’s very cumbersome to have everything embedded within the one package within the twister package so we have the the TX project which is a third party code that people have just written and put on launchpad a lot of the Tris developers use be said are and therefore end up on launchpad but not all do there’s a whole lot on github yeah you use a lot of github and misc stuff so for instance everybody wants to interact with databases there’s really good support for that when you’re running an event loop program it’s very hard to interact with databases because databases are blocking interfaces so that’s a nice neat little way of encapsulating the the thread pool behavior required in order to talk to you know SQL a MySQL Postgres we don’t non-blocking now so we’re primarily a networking framework but general purpose software you need to have those capabilities oh now what is this this is twisted Internet this is the support for sources and destinations of events this is the list of packages within the

twisted internet sorry modules within twisted into their package and and we we discussed earlier the different main loops so this is where they are CF reactor there is sorry reactor is the name of the main loop but that’s more or less all it is CF reactor is what I call it call foundation that’s the back that’s basically oil soaks yeah end points you’ll be seeing a lot of examples of end points that’s just a neat way of saying this is the abstracting away something which represents a server port and the and the protocol it attaches to so you can create one protocol and attach that your SSL listener your TCP listener you UNIX socket when it’s listener glab GDK i notify a standard i/o is very important when 32 event reactor what we were pointing to no wheantr the win32 and WX the ones that we were talking about earlier yeah so um so it really does actually work on windows 32 and is it here as iocp here on this list you don’t actually want to use one 32 event reactor it turns out that that iocp is a much better way of doing things that’s also supported iocp czar edge driven iocp is kind of like a Paul but for is the II Paul equivalent or the KQ equivalent for Windows it’s a lot uglier but it’s the only thing to do really fascinate on with these be if you have you select on Windows with Python you find you have a 64 file to script a limit on your select that’s a little low we like it in the thousands and so therefore iocp is a better solution now I don’t know why it’s not on that list okay so this is a short list of things which are useful um tasks tasks is where you’ll find you know I want to call a function every five minutes defer this doesn’t actually did it belong in here because it is pure Python it doesn’t rely on the main loop the reactor is the place you import the the main loop from that abstracts to the correct one that using on your system protocol is the basics you need in order to set up a client-server protocol this is where all the exceptions are and endpoints of course is is where you do a listener or later so so this is a little representation you know if you’re doing multi-threaded you do some like this and each of each of the connections that’s been accepted will be waiting at different times so twister does it all in one loop and multiplexers the reactor calls callbacks now I should stress that if you has anybody heard of a deferred before there’s one in the book quite a few in the room okay so the deferred is the attraction we use for organizing callbacks that’s callbacks for questions that you’ve asked and there are promise of a result to come now that’s not the same callback as what we’re talking in the context of the reactor the callbacks what we’re talking about in the context the reactor or things like a connection has been made you’ve received more data you’ve just lost the connection somebody’s closed their terminal the whole thing shutting down now the difference is that I deferred as a the phone is only called once whereas these kind of callback functions get called over and over and over so a good example of a deferred would be for example let’s say you want to download a web page um then you call press add a web client get page and that gives you a deferred and whenever the web page is completely downloaded the deferred fire is with the body of the web page if you try to fire it twice it raises an exception as it should so this is the most basic person program go type run import from twisted on internet import reactor which is something you will see on a lot of slides most twisted programs so does everyone have a working twisted now can they open a ripple import twisted and not get an exception and preferably get a recent version if anyone has any problems with that I’ll be more than happy to come and help but so this is starting in main loop and this will run forever we’ll have to control see out of it now the reactor only runs and stops once so there are three methods which are very important react or unreacted stop and the one that you should never use which is reacted or crash the difference between stop and crash is that stop will shut down all your TCP connections and tear everything down and prepare for system exit crash won’t crash will leave from that so if you’re doing really

hacky stuff you won’t crash what you never do so this is not the slide that I thought would appear so I thought I’d gone through and edited this so that it would use end points maybe the next slide shows an end point note what happens but I don’t know what’s happening there all right so okay so this method listen TCP okay laser sight listen TCP is the simplest way absolute simplest way of science listen on port this listens on listens on the blank interface meaning all interfaces port 8000 and some question mark so those question marks are what we call the factory please go ahead type this out either do it you know text files probably easiest because you’ll be editing symbol files you’ll have to import protocol if you want to save typing you can actually just type factory the difference between a server factory and a client factory is semantic generally it’s a sick if you look at the inherited I heritage hierarchy to have a look at what something’s supposed to do but it’s it’s actually exactly the same interface currently with endpoints the there used to be a slightly larger difference but currently with endpoints it’s been completely done away with so yeah you can pretty much do either one and this is a bit hackish basically you set what protocol is but you know this is this is a well accepted patterns and fine you’re setting the protocol to be protocol or protocol that’s the superclass of all protocols and that just doesn’t implement anything so it doesn’t implement its events for connection may receive connection lost will do anything if you run that you’ll end up with a server that listens on port 8000 except starter and then throws it away it will never send you anything it won’t call you in the morning it sorry that was rude it’s it’s something simply a Alyssa now this will less on the default installation of twisted for 12.1 on a on a Mac system on a Linux system this could handle a couple of thousands connections fine so this is an event loop and can handle a high throughput of nothing and starting with the next version it will handle like tens of thousands of connections yeah I was doing absolutely nothing the difference is that right now the default for most of you I think will be select reactor no no don’t want no so actually 12.1 is the first release where it will magically figure out like oh look I’m running on FreeBSD so I probably want KQ or I’m looking on Linux I’m running on Linux so I probably want people or I’m running on windows etc etc so select as a hot limit of 1024 file descriptors which is actually just a hash defined in a source in a Python source file but I Paul Paul KQ is CP there better reactors and that they will now be the defaults in 12.1 and onwards in in the past you would have to do some fancy importance in order to get them working properly select being the one that works everywhere it works on windows it works on X so what’s actually happening here the reactor is listening on the port for each connection so that the the event loop knows what factory is associated with what file descriptor that it’s listening on it’s accepting connections every time a connection comes in it asks that factory an event that sense of the factory is is said you could any come in connection can you give me a protocol please so there’s one factory and you can actually have the factory or multiple ports because you know the reactor knows that one factory is on multiple ports will just work the and then one protocol object is created per connection so that’s why on this slide we’re not passing and AH so we’re not passing an instance of protocol we’re passing the class itself so that’s the protocol Factory you can actually make that a function which returns the protocol if you want it can be anything there’s also a method you have right called build protocol if you want to actually do run some code within the context of the factory you might see a demo of that later so build protocols pass the address the factory has the opportunity at this point to reject the connection by not returning a protocol you know that’s that’s quite useful depending on the kind of application you’re writing so so you can say you can do sort of your file balling in your

actual application choose to reject the connection now twister has these things called interfaces it uses oak dot interface has anyone used up to the interface yeah that’s not surprising so if you actually look at twisted on internet on interfaces there’s a number of interfaces there but you want to say a few words about that yeah so basically an interface essentially is it looks like a class definition it has no implementation it’s only function is to document what a certain thing in your project is or what it does or what it’s used for so for example an IEEE protocol factory isn’t a protocol factory but it explains if you ever come across a thing that’s called a protocol Factory here’s what it is here’s what it does here’s what you can do with it so these are purely documentation and sort of a statement of intent it is a little confusing but it does allow you to do things without inheritance hierarchies and twisted it’s very subclass Orange so not composition but but inheritance instead so having these these interfaces allows you to clearly document exactly what you mean and exactly what methods you you should be able to override if it’s listed in the interface you can override it we are going to now write a server you can write a server first that the returns all daughter receives uppercase this is this is one of the most trivial when you when you write it when you start learning how to write code with networking it’s just after the echo server echo server being the one that writes back not a back to you now a protocol has three methods I mentioned them earlier connection made connection lost data received the most important thing here is I before E except after C the number of times I’ve made that mistake data received you can find this in the bundle of code that’s on the website called replies in 2012 or on that USB key it’s called up to serve it up high or you can type it out I recommend you type it out because I always find you you learn better when you type run it go now and then tell that into it you’ll notice that this example contrary to the earlier one where we use loosen TCP will actually use a real endpoint from right here it’s not exactly evident why endpoints are fantastic the real reason is that you can write your code exactly once and then if you want to make it work on SSL that involves zero lines of code but we’re going to anything it involves zero line to make this work on SSL into the TCP you change TCP to SSL in that string and the idea is that you can come on line arguments say listen on a UNIX socket which is by the way you type UNIX colon slash path to UNIX socket and you and later you’ll see examples of clients where you create one endpoint use that endpoint multiple to connect multiple times so reactor protocol and endpoints protocols the is the superclass of sorry the place that all protocols come from ladle using a few helper protocols that they’re from a different location will point that out at the time you don’t have to implement connection made in connection lost or data received but those those are the three callbacks which the which you will receive when you do this then you quickly throw together a factory listen and then the last thing you have to have any program is directed run which actually runs the event loop the program won’t exit from reactor don’t run until you control see if you have any questions of like Lauren’s down I’m not moving so you’ll notice that connection lost is sort of doesn’t really do anything you don’t really have to implement it it’s just here so you know that that method exists if you wanted to look that up in the documentation it would be on a protocol following the explanation about interfaces I gave earlier reason sometimes you might print that out and sometimes it will say that actually often it will say error the connection was closed cleanly that’s actually perfectly all right it’s always called it’s always called with the reason and and the most common reason is that the connection was closed cleanly you’ll also see connection lost and that in terms of Education closed by PR and those kind of errors basic Network errors a reason what’s what’s a reason an instance of an instance of failure isn’t it yes so it’s going to exception object to know yes and twisted on errors the twisted internet taught errors is a list of the

exceptions you can see who’s got that running no you’re still typing you got a few people got it running excellent you copied it from the tar ball didn’t you no no excellent cool so you got the data coming back uppercase ah yes yes so the question was how do you change the host that listens on : interface equals local host I think interface meaning is yeah that’s a terminology from yeah but unfortunately it’s that the IP address not the actual Ethernet interface but you know there’s no way to do that same way when you’re doing connection you actually do TCP call in the hostname call in the port so it’s a bit different for clients but you call a different method for that that’s quite from string and sort of server from string if you if you’re doing SSL you also do the SSL keys by doing SSL : port name : key equals sorry question sure right some deep raisin do it into it with anything well no you should it should send a LAN connection you go have a look at this program well only if you Mexican how is you making a connection the is doing it we’re going to talk to so at this point you should be able to find everybody else’s laptop by connecting to they let two random IPS on point it out and that this this will support multiple connections of course as well so so there’s nothing here which is stopping the event loop from running self transport right that that is not a blocking operation what that does is it inserts the data into a queue if you did something you’ll advise you could write a 20 gigabyte file into there and it would portion that up you select to find out when the socket is writable write the data fill up the kernel buffer wait to local buffer is a little bit drained write more data to it so so that’s non-blocking lose connection there of course is optional you can just make it continue to send daughter um how’d it go perfect it was connection mode instead of connection made ah yes sorry connection made that’s another typo now now this is one one reason why there’s a feature in Java actually the wouldn’t use juice and and that sort of thing where you can say at overrides mm-hm which will raise an exception if you’re not actually overriding a method I actually really like that there was a talk at Euro Python this year about copying things like add override and Java yeah yeah I should like add override is the one thing I want final here but at override I want that so that you get an exception when you accidentally do connection receive it this is the slightly simpler example this is the older style with listen TCP so you know that’s that’s just a you’ll see this another in all the code and older it older examples I much much prefer endpoints and I use them consistently everywhere on all the examples these endpoints but just just to say that’s that’s the older style this is an this is an example of it running this is what you should see telnet I strongly advise use Tom it here is a client it’s called multi client high if you actually run this this will use your arguments to go you do Senate arguments localhost colon 8080 acumen C will actually make multiple connections so this is actually demonstration of a client which connects multiple times so if you run multi client you should see output like that I don’t recommend you write out multi client because that’s nasty code interestingly I inherited some of this code and very very common bug is to use send never you send always you send all questions coming up questions any questions ah yes so Autocode reloading um no I

don’t have a good good way of doing Auto code reloading so the question was is there a way to have Auto code reloading substitute to two E’s quick development know that there isn’t anything built-in I I did however experiment with this a while back on my on my bitbucket page I’ve got a little thing called harpoon which is a thing I wrote in order to make it easier to to quit quickly restart a server with a new protocol instance it’s just a command-line argument they come online thing where you pass it a protocol and a string endpoint and it automatically synthesizes a service so you can test there’s also a Python project called watchdog which is on Google code somewhere and it it’s essentially a generic inotify wrapper that works on pretty much any platform and I use it it comes with a tool called watch me do and I use that to restart my server when I’m doing development no it’s fine any other questions great so now we’re going to build on that original server that you wrote the the upper server try and do these things count the number of connected clients and announce the number of connected clients when at the point where you connect go for now all protocols have their factory as an attribute so it’s a soft top transport and soft factory are two very important things to have so transports the stream you’re connected to and factories the the factory that created you I’ll go ahead do this now and the important thing to remember is that a protocol is one single connection and a factory is sort of the father of all of those connections so if you have something that’s that’s shared amongst all connects all of your connections you do it on the factory if it’s specific to one connection you don’t know the protocol when we move on to doing clients later you’ll see there’s a there’s quite tight coupling there any questions about this exercise of niklas would be nice show of hands who’s got it running okay but sorry you

you don’t actually have to call super clot super on protocol if you want to give it it in it I think they’re also old-style classes so super doesn’t work very well so the the reason there’s a lot of old cell classes and twisted is because it originally worked on Python one point five and one point four and changing semantics is bad not sure which would be Zara protocol on a related note to twisted works on are used to work at least on one point four and one point five one of the most frequently asked questions is when will it work on Python three and to answer that we actually have a google Summer of Code student currently doing amazing work on getting it to work on twenty two which is one of the super ten core devs is also working on it they are both making great strides but well as Steven said it used to work on 1.4 it’s that old so it’s not exactly easy to pour it over that code so work in progress but much like Django we’re not quite there yet some stuff works the fortunate thing is that was it has extremely extensive unit tests so we do know what doesn’t Oh hey buddy

right okay so I’m going to this is this is a way of doing that that are that exercise so that here in this particular case were actually sub classing factory just so that we have the attribute set there that we need otherwise this is a bit ugly as I did see some people using this technique works fine accessing the factory increase the counter on the connection loss decrease the counter and pass the instance of counting factory through anyone having trouble with that oh yeah so multi clients not so good for this so so use a telnet or something shredded client is an interesting one if there’s a test program thread a client does allocations simultaneously with threads also it’s up to me why do you have to hook the count on the factory and because there is only one protocol per connection so if you want to have access to data that the other protocols have access to you have to do it on the factory not on the connection so so if you think about that the factories attach the listening port and the the protocol is only attached to each individual connection so they’re they’re all isolated they all have their own attributes so so the product so say you were writing a protocol you would do something like is user authenticated would be a method on or attribute on the protocol and it is the server actually running or a number of connections or that kind of thing down the factory okay thank you all right we’re going to move on from here any other questions okay perhaps to pick into that technically if you did it as a class attribute on the protocol for this example it would work it would just be semantically incorrect okay or a global value the but you know Global’s are bad I guess so it’s better things off so recapping reactor runs until told to close all stockett sockets and stop so if you call reactor stop by the way you actually so say for instance you actually want to send us send a signal to shut down or after some number of seconds shut down you call reacted stop ratchet run react about stop the reactor uses factory to act in the connections factories create a protocol an instance for each client and you subclass and implement so so the callbacks the callbacks are called multiple times in general and twisted are are not pass strata that as arguments to functions they are their class sub sub class and override inherited methods whereas callbacks for individual actions are generally passed around as attributes we’ll see a number of examples in a few slides with four more complex for like standard behaviors things like if you want to receive net strings you also do that by just sub classing a different protocol and implementing it specific methods to two more tips these these are just tips telnet is generally better than netcat or NC telnet much more aggressively flashes if you’re testing stuff I recommend helmet it’s it flushes more aggressively and it sends carriage return new line most Network protocols HTTP pop3 FTP standardized on carriage return in line not new line and so so you might actually find that if you’re trying to test you find your Pro doesn’t work when you use netcat but it does with telnet that’s why also this this is really neat I actually I introduced Laurence this yesterday but I but it’s really neat if you run this command that

runs the the main loop and an interactive prompt simultaneously so you can actually run that you get an interactive prompt it’s not really the pose an interactive prompt is very implementation of it but it understands deferred and it’s got the main loop running so you can import whatever code and bang away at it if you want to do things interactively yes hmm not really this the the issue is that in order to run the main loop and any tracks at a high level with and give you good results yeah we can’t even use lib read line we’re not using lip read line for this it’s actually really implemented so some things like I think control our might work but a few of the other more esoteric liberate line features your fingers might be used to don’t work so for ipython apparently someone had that working in the past back when they had twisted e things as their core in a deep dark recess of ipython and then they started doing a rewrite which I’m not sure if if it’s completed already or not but based on that new base which doesn’t really use twisted at all and has no way of crop of cooperating with other reactors they broke through two things one is you can’t use it with a with a GUI anymore and two well except their own and two it doesn’t work with twisted anyway so yes I someone to get it working with B Python alright so the next thing we’re going to do is a proxy server so a H well a rudimentary HTTP proxy server so the client just sends a URL use helmet the server returns the contents of the URL encloses connection so it’s like the upper server except we’re actually going and doing a connection ah now if you wanted to do new lines this is how you would have to do it you have to have a buffer you have to when you get data received you append to that buffer and then you check if the if there is a new line and then you this actually has a bug in it as well and then you you split off from that new line and then you that’s another bug there check that out it’s /n and shapes to slash are slash n so bla bla bla bla bla and then we’ll print each line this is much easier first up protocols is a collection of twisted protocols these are just implementations not actual servers so I set say for instance you see SMTP in here that’s the implementation the SMTP protocol with callbacks that you can subclass not an actual web email server it basic protocols in basic is line receiver I before E except enough to see that instead of having data received this implements data received and then has a new method that you can subclass called line received does a little buffering for you works really well you can customize the delimiter it works very well so that’s what we’re going to use this example and for most of the line based examples from now on I recommend you use line receiver not protocol protocols contains amp basic dict finger ftp GPS that’s actually talking to GPS devices – loopback is good for testing memcache mice that’s actually an implementation for a serial mouse protocol port forward postfix I mention POSIX early as shekhar sip there is a sip server written with twisted socks safe will tell that Tila some wire TLS meaning being able to upgrade a normal connection to an SSL connection midstream that’s question nope all right protocols are basic contains these things net string lie don’t only receive oh sorry line receiver actually has the ability to go between line based and stream based so in line only is slightly more efficient but it’s easier to type line receiver I’m sorry what yeah if you want to send and receive integers then there are these integer receivers there sorry I think these are lengths prefixed strings so you know they’re much more efficient for in terms of buffering tomorrow and files and I’d like if you want to be able to connect and send a file that’s a protocol it doesn’t it’s all in basic and you can you can leverage these things in your own programs yes so check twisted up protocols before you invent your own protocol having said that yep so this is

a line receiver so this this code is in that bundle of code that I have there it’s called proxy one top you I run it run it if you want run it and connects port 8000 and give it a URL go down do it you can even use multi client to send it multiple URLs so so this has some rudimentary checking a little sexy TP it does a bit of timing it uses URL Lib to fetch the URL closes connection doesn’t implement connection make initial loss but doesn’t need to a important thing don’t implement data received and line receives trying to be tricky because you’ll you’ll be overriding that the line receivers data received and brake line received I’ve done that there’s no working there is no working internet connection teach them about twisty well uh yeah ok yes so because this is a timing client you’ll find it out exactly how slow the internet is here if you want local stuff which is easier files EP is a izl is on the intranet so it should be milliseconds not seconds this is a client which does some timing as well who cares it’s like like the other clients it’s threaded like the previous client it’s not something that you really want to write yourself it’s something that’s just on the stick if you want to run it here’s an example of all these things so so this is running timing client on on a machine clearly not on the network but somewhere in Europe of course because it is multiple seconds and so this is the output from the client of the server and you can see the server took like one second each for these things part three seconds in total for the client just because there were so many things to retrieve now those sums are very very close the individual requests and the thready client the reason for this is because we’re using URL Lib dot URL open which is not not cooperative with the async event loop we’re blocking the event loop which means nothing else can be processed so no two webpage is being downloaded at the same time it’s generally quite negative I don’t know good what happens if they think there’s a yeah so we can’t use your own two URL open so let’s weave we’re going to be cooperative and we and when accessing the network we have to return controls the event loop so we can’t just you know in one method go and do everything you’ve got to return control while you’re waiting on the network so recap I yes this is the panini example translating a so the idea is I didn’t know that we had these animations did you know about these animations yes so so the idea is that you can actually cook multiple things at the same time it’s it’s better to not wait for the result for every connection at the time this is a bad example because we end up saying wait till my turn wait till your it was ready idling the cpu so we’ll use callback so you’ve seen callbacks before who’s used to a query that’s jquery who’s using jquery you’ve used callbacks in jquery all the time hey that that’s a callback and this is the you know this is a better api it’s like when ready call that method deferreds add to the whole callback paradigm error handling passing the result around cancelling all of that so that’s what deferred is deferred is so that you can very very easily say I want a result and I want to be able to handle the success but also handle the failure and you want to be able to chain these things together in a way that you can sort of say do multiple things and just tell me when it’s done people who do JavaScript development sometimes probably know that these days actually a lot of things have implemented their own including jQuery jQuery now comes with some deferred implementation did the deferred implementations more like an event yeah it’s source so it can be called

multiple times comment no it can’t be called multiple times but you can do chaining oh you got together all the callbacks get called with the same value so Tristan’s deferreds are a little bit more powerful than jquery’s but the essential idea is still the same you give it you get an object right now that represents a promise that something will happen in the future so has anybody use Maki kit it’s a JavaScript thingy it’s got very good deferred implementation but it gets the order of arguments are all going around which is a bit annoying so it’s promised the result a result will appear in the future you can pass the that promise around so it’s not you know you add a callback to be called you can actually return it without any callbacks attached and let your calling code attach the callbacks you can attach more callbacks to to you it is technically not standalone as in the twisted internet package because it depends on a tiny tiny thing and twisted into that to do with the reactor but in general it’s actually pure Python you I’ve done till the break the air okay so the idea is in the pinning example you’re the panini you get back a deferred immediately so this is the promise of the result and you and you can attach a method to to be called and you can add call backs and what we call Erb acts which is what what to do in case of an error they’re everywhere in twisted mostly so that you can ask questions and get answers yeah so we’ve got a twisted variant of URL Lib we call it twisted web client actually if you use that Python – M twisted concert on Saturday oh that’s actually a really neat way of doing that duh I recommend doing that now type Python – M I’ll demonstrate oh not objected properly Loren’s all right ah twisted dot con dog is bigger hmm lab work yeah except it breaks I turn – yeah all right really so show you some it’s um the twisted dot webclient import get page get page HTTP Oh big not found apparently I’m not on the internet are let me just quickly start a web server no I must end that’s because they’ve pulled you’re all right uh you don’t have any that’s alright I’m running a web server all right look and so if you do this you actually see your deferred instance D equals on the score so that’s your deferred and I can def foo page print page DD callback foo and it prints it so so that’s that’s actually a really good example of using this to to quickly prototype things and check out how api’s work you’ll see that when you what actually called this method it founded of creditor deferred and because it knew how to deferred on the previous line in it it noticed before I finish typing that have been called back and so actually showed the result immediately it’s really neat like that and this is an example of creating a callback adding the callback and it just prints of that right and that’s the return value of do I had call back so you can do so if I do that again this will print none because the previous callback didn’t return anything it returned none so any chain things together you can mutate the value as you go through if your return not it ends up being non so this should be none yep none so you can chain them together alright ah how do I get back into the slideshow does anyone know how to use this play was amazing play yay play alright I actually use keynote for the first time for this so that this is the previous example and this is how to do it asynchronously so what we do is is we call get page instead we get add

deferred object we add a callback when we’ve actually just implemented a method right there which you know prints the the data so this is those four lines moved up a bit and put it in a callback it’s a bit messy so just reminder this is a this is a closure so self is actually that self it’s not passed through teacup data line is that line start is that start but everyone’s use closures like this before maybe who hasn’t it has never seen this before Wow okay you can the a function definition is just an assignment you can do it in any scope and and it carries carries through the locals of its parent the only the only issue is you can’t bind a name and the enclosing scope you can do anything to do with modifying them or accessing the B you can’t remind them this is a bit of a tidier way so actually defining a method on the object itself and you’ll see so this method is not defining the object itself so has a self and that’s implicit that’s that’s the result of the defer that’s fired and you see these two parameters here URL and start time when you add a callback you can add more arguments it’s like a star arts when you add more arguments they’re passed through and so this is how you can transfer state between but between the call the call site and and the callback without using closures go for it whether this is proxy to this is the same example but using get page you run this now proxy to from the from the table the limiter at the end Lorenz has just pointed something out there’s about here we’ve got lined up starts with that because line is line we receive and then we get page in the line rashly passing through the line delimiter there we should strip it off I remember this magically working so I think that page will just take it and run with it but it probably you’re all encodes it yeah that’s fancy also a little silly all right so basically to repeat for the people who didn’t hear me the first time when you so when line received gets called the line still has the delimiter at the end of the line it’s not stripped off so the URL isn’t like HTTP slash Google com it’s HTTP slash here’sh Google com our /n you don’t have to type this out it’s on on the key now um this is another way of doing it ah I’ve got some bad indentation there I’m a bad person um sorry that print should not be indented sorry sorry sorry sorry okay so this is this is a thing to make this easier to write so it’s called inline callbacks it’s a decorator and it does and it turns whatever method it is into a into a deferred and because of that it actually stops up line received so so we call forget page method from front line received so what what this does is this turn this is a decorator which goes and calls this method and and returns a deferred and this is a generator is it uses a yield what this does is it if you yield the deferred this will this generator will not continue exiting until that deferred has fired it that essentially adds a callback of of calling generator send so in this particular case you can write this function as if it were like this so you see that URL Lib URL open that becomes this line here page equals yield get patch and so this is asynchronously returning control to the event loop and then return control back here now this is a nice neat way of avoiding messy callbacks callbacks are sometimes quite hard to read generally I use this technique when you’ve got a pipeline of things to do whereas I use defers explicitly when I’m writing say a server which which attaches a few callbacks before returning a value when I’m especially like when you’re writing one-off scripts to do something on a server somewhere then what you really want to express is do something then do something else then do something else which is inherently synchronous but because you’re already

right using twisted so you probably want to reuse your protocol you want to reuse all of your tools so this is this makes it really easy to express synchronous things because you’re expressing them synchronously it makes it really hard to express asynchronous things so depending on what you want the tool is there to make it look easy so this is a proxy 3py if you want to run it there one of one of the problems here is that because it’s doing fancy things with with generators and generators and and mucking around with the event loop and wrapping everything trace backs look awful so so be prepared for if using these to have trace backs that don’t seem to make any sense because you’re you’re in the bowels of some of another process for getting this running this is an example of using the timing client and you can see that the the cost is now the cost of the slowest page in this particular example being the slowest this is clearly Erastus was the person who wrote these slides originally clearly he was very close to his own server and further away from Apple and Google because the agency and you see you know the sum of the requests was four seconds and spread I did 1.5 because it’s running asynchronously multiple requests at the same time so in summary callbacks SP cooperative you return control back to the loop and look at a loop or call your code when the network is ready avoid calling blocking functions now there is a caller e to this if your if you want to do database stuff and your database is very fast like you don’t ask you a light you’re doing index queries or you’ve got my SQL on a local server and it’s very very fast and you don’t really care so much it’s not so bad but you just have to be aware that you have to balance out the number of clients you have this is the latency of your database queries but if you’re definitely doing things that could even timeout you always want to make sure you’re doing Mason currency or a thread so exercise to a take proxy to or proxy 3 whichever one you’re most confident with using and implement a caching proxy server which will not request the same website twice go run it now if it’s the same URL then return the case data so store each response and and return them a dictionary for four save the response order in a dictionary and then just return it this is the last exercise before the coffee break always sorry I there’s the person behind you has a USB key you saw that you’re speaking no whiskey anyone come on No oh yeah I or files EP is the is the website and you can navigate through the tutorials and if you’re feeling really advanced and you do that quickly making it so you don’t request the same web web page twice if you’ve got a query in flight that’s actually a really nasty one especially if you’ve got used to third before after next year we should actually get you to the test suite yeah to eight make tests to a pass yeah so the suggestion was we should have unit tests that you can run it run against this unit testing twisted is quite easy you don’t actually set up any network connections you just create an instance of the protocol and call a medical connection made and daughter received and see what happens we have another USB key let me just make a new one because I don’t know what the elements are within the trista community like testing is taken extremely seriously if you find some random TX project there is a good chance that it’s going to have like literally 100% branch coverage it and to twist it itself like for example does not accept new code unless it is completely unit tested we don’t take bug fixes unless there’s a unit test to prove that there was ever a bug sometimes this makes it really annoying like you know there’s weird-ass SSL edge

cases that just don’t make sense and are impossible to unit tests but it overall I’d say it makes it code quality a lot better my computer is just refused to believe that USB keys exist I you yes what is it because it follow ah the back of this I might hmm just get the same products that’s why I put a zip file on there as well the zip file should be on the website there’s two things next to each other once as it follows if anyone wants any help slide me down sorry he’s fiddling with his laptop so one of those is a zip file no no that’s the not ignore that one it’s the training ones just that your acquaintance yeah first ones a zip second ones that IDs at Arby’s up to that one so the other one is just a requirements file a pip requirements are it’s just that if I upload chat text files it it mocks up quite badly the requirements file isn’t required but it but I’ve got a demonstration of how you can do it did anyone go to the tornado talk the other day oh okay there’s a torque on tornado which is another asynchronous networking framework which integrates with twisted quite neatly so I just a demonstration in big demo dot py of using tornado and twisted and courage wine which is a

green ‘lets thing as well as inline callbacks I know I didn’t do something no cyclones are not the same cyclone is a reimplementation a tornado yeah but the the event loop that is implemented that is implemented by tornado can be used as a reactor and they cooperate yeah twisted you know tornado platforms or platform not twisted import reactor favor do this again unit tests we didn’t explain what a primed i’m deferred i know what the forensics egos but i can’t tell from that explanation um yes sir maybe i should mention that um i deferred there are two we didn’t actually demonstrate that i’ll i’ll quickly demonstrate that with with the with the console get a ripple going on oh yeah that’s going on so basically uh as you said before i deferred is something that is something that you get now that represents some data that will happen in the future right but sometimes you get things you have things already for example in the case of your if your caching proxy let’s say you’ve already made that request you already have the response somewhere in a dictionary right so you still want to return a deferred because your api is deferred but you want to return a deferred that has the data already right now so for that you have deferred so defer the module defer dot succeed which basically creates a deferred fires it immediately and just returns you to the returns a deferred it’s only like three lines but it’s something that’s common enough that you want to make it one line similarly there’s the FIR dot fail which is the same thing except for errors instead of successes so that’s a little example of creating deferred in our normal one and it calling callback as a way of firing it so you create it then then call it you can only call callback once or you can just call deferred succeed and that’s the same as creating one then immediately calling the callback function deferred fail is the same as creating one and immediately calling the back function I’m just quickly adding a callback and adding a note back there to demonstrate because that’s my foo from earlier which prints I’ve whatever I do examples I always use foo in them it’s very bad of me especially when I mark it up by actually using food my string and put my function and then different prints foo and it’s very food are from future import print function this is awesome if you’re writing code does this even work well because you can’t do future input oh this is it does work that’s amazing I thought you had to do future imports of the beginning ear module if you’re writing like if you’re writing a module yes in a ripple no ah Lorenz has just taught me something new so so one problem with print of course is that is not a function and so you can’t add it as a callback turns out if you do from future import print function you can edit add as a callback so you have a nice callback which prints stuff out that has been my biggest annoyance and now it is no longer Thank You Lawrence alright so it so in this case um I’m just going to show that deferred succeed my page I downloaded earlier dot add callback print EA and it prints out my page I downloaded it earlier so the idea is that if it if you’ve got it in your cache than you call then you return at the first to see that otherwise you are you don’t have to how much time do we have left okay so it’s time for the coffee break I’m going to leave the answer’s up to do you want to leave an

answer up over the coffee break all right I’ll do that that’s a answer which is a is not using inline callbacks but using straight-up deferreds all right well we’ll be back here at what time all right well I will be back here and I’m going to start again in 11:15 so I’ll see you in about half an hour I’ll probably be back here a little bit sooner if anyone wants like more once to ask more in depth questions so this solution does not do the fancy version where if there’s a request in flight that hasn’t been answered yet use that one and don’t make double requests this is the simpler version someone remain in the room yeah all right so somebody will remain in the room so