From Synchronous to Asynchronous Postgres with Tornado

microblog from synchronous to asynchronous Postgres with tornado alright hello everybody my name is Mike Rowe blurred and we’re gonna be talking about writing asynchronous code in tornado mainly talk about Postgres database stuff related to that these slides are available online there’s a URL for that and the code that I’m gonna be talking about it’s also available on a public github see that so a little bit about me right now I am the director of Technology which at a start-up which basically means I’m one of the people founded to start up it really means but we make a product called EHR tutor that nursing instructors use for teaching them how to use not EHRs oh my god I don’t believe it no no we don’t teach nurses about object relationship mapping that’s what we’re going to be talking about here I can’t believe I didn’t catch that before yes um no no teaching about electronic health records not object relationship mapping oh but we use the object relationship mapping in the code so that’s all good I am the former senior software architect and on shift and there’s a bunch of people from on shift here so before that I worked at American green jersey from American Greens people here it’s like that just follows me around I can’t leave my previous companies ok so that’s enough about me because it’s not that interesting okay so for this talk I’m assuming that you at least know what asynchronous is hopefully you went to David’s talk this morning because it was really good and who taught you about asynchronous if you didn’t know what it was I’m also assuming that you know what tornado is and that you understand how to talk to a database software so this is meant to be kind of an advanced talk and that there are some assumptions I’m making so I won’t be going into a lot of details about the basics of this stuff but we are gonna go over is I am going to review a little bit of RM some I gave a talk two years ago about object relationship mapping and we’re going to briefly go over a little bit of that because it does play into the code that we’re gonna see a little bit later and then we’re basically gonna do is I’ve written four versions of the exact same code it does the exact same thing but it does it four different ways first it does it the old-fashioned synchronous way and then I wrote the same code that uses a library called Momoko which is the basic users straight tornado asynchronous stuff to do asynchronous i/o then I wrote the same code using the async i/o library and using a library called ail PG to do the same database stuff exact same stuff but using new async i/o in three four and then for the final one I wrote this exact same code using G event and all of these are running on tornado for the web application so we’re gonna talk I’m gonna kind of show you each one of those the difference is if anybody has any you know anybody see something that they’re like why would you do that just blurt it out that’s fine and then at the ending to wrap things up a little bit and that’s basically we’re gonna talk about so object relationship mapping if anybody was at my talk two years ago I’m very opinionated about object relationship mapping code especially when it comes to things like Django’s ORM and sequel object and some of the other ones i really don’t like them but you know and seek welcome means good it’s really super complex but the reason a lot of that complexity comes up is because sequel alchemy is built to talk to every single database in the world and it also provides stuff for handing handling the DDL handling you know building queries doing the pooling and it does all kinds of stuff like that you know that’s all great and a lot of people like that and if you like that there’s nothing along with that especially if you’re trying to build something quickly but when you start to try and build advanced stuff and you’re working on an ORM the ORM tends to really start to get in your way and it doesn’t need to because all Norum really is is about mapping database tables to objects and mapping the relationships between all the know RM is supposed to be and all the other stuff is really fluff on top of that so you can do this a lot simpler than using

these big libraries and the code that we’re gonna be looking at today which makes this a lot easier to talk about the asynchronous stuff because we don’t have all that all that overhead of using an ORM is going to make it a lot easier to see the asynchronous part of it so real quickly a little bit more about that so cyclo fiji to the library that we all use when talking to Postgres provides this wonderful ability to have adapters which will basically take a type in from coming out of Postgres and turn it into a class for you and the records coming out of wrote rows in the table are a type that can be adapted to a class so conveniently you can just go and say take this row and spit it out and it’ll as a single element and it will turn it into a class for you and that is the main thing that we’re using to accomplish the object mapping and then which one I actually didn’t use this today so this isn’t really all that important other than to know that I’ve written it and it works but if you’re doing relationship mapping which this is such a simple application that we’re going to be using to show these examples there’s no relationships and a it’s one table so but in order to do that we basically have a recursive function that goes through a bunch of row results and says hey let’s put these together based on how the relationship should map and that’s what that’s how we did row that’s how I do the relationship mapping and that’s I used this method in production code and it seems to be working fine so that’s that that’s not really what we’re here to talk about we’re here to talk about tornado and asynchronous stuff but before we do that we have to look at how you would do this in a synchronous world so we go over here and you’ll see we’ve got the world’s simplest web application ever written it’s got a single link on here it says new game when we click on it it says give me a name for this game so we’re gonna call this oh hi oh and we’re gonna hit submit it goes back to the original one and now PI Ohio is in there in the database and came from the database so like I said it is the simplest thing you can possibly implement to actually show this so let’s take a look at the code everybody can read the code right the back alright so the important things here we do a bunch of imports imports are only important as long as you make sure you have them all but what’s really important here is our model class this is our model class and it uses my very simple RM a little bit but it’s like really simple so all this is saying is here the fields that exist this is the query that you would run if you want to insert data here’s the query that you want to run if you want to update the database update the table men the actual interesting things are if you want to do get the data out or get all the data out we actually write and provide a method for that and unlike if you’re using no RM you’ll see that we are actually writing sequel which is good if you need to write really complex before you can just take really complex sequel in it returns it but this is synchronous code it is straight up here we are getting a psycho PG cursor that’s the normal getting a cyclop√¶dia cursor we’re executing we’re fetching all this is the build relationship that actually doesn’t matter in this case and we’re returning the results same thing and to get all let me real quick show you so this is the or M most of stuff is not relevant so my admit just does some basic set up stuff to handle the fact that you can pass it and keep some keyword arguments and that they map them into the into underscore underscore dick this handles some of the relationship stuff this just returns a dictionary out so that you can get nice simple Jason encoding and then obviously this is a fairly important one if you want to use the get adder for accessing your things but the stuff that we’re really interested in here is the update which as you can see all it’s going to do is call that update all it does it calls the update query that we set in the class below save says hey if we already have an ID call update otherwise insert it so this is some really simple stuff and that’s literally that’s the entirety

of the ORM it’s you know the rest this is just some stuff for the relationship so anyways this is tornado so oh wait one more important thing here so this game composite right here this is the adapter this is the adapter for psycho PG so the existence of this is what allows us to say hey we’ve got a thing that came out of psycho PG we’re going to turn it into a dictionary we’re going to pass that dictionary into the constructor for a game class and now we have a game object coming out so for and so you have to provide one of these for every single model class that you create you can probably come up with some cool way to make that actually be part of the game class and stuff I just haven’t bothered because this is easy enough and I like to keep things simple now we’re into our controller stuff I wrote it on a single file because well it’s an example in seas so it’s transactional so when we finished the request we want to make sure he commits to the database otherwise nothing’s actually gonna happen to the database if there’s an exception we want to rollback and then here is our actual things we’re going to do so here’s our get which calls the game get all from above and then renders the HTML that you saw that you saw the output earlier here is the there’s the one for the edit you know same thing if no idea is passed in then we’re just assuming that we need to create an empty one to use otherwise we go and get it out of the database when we post we do the same thing if the ID is passed in we get it out of the database and modify it otherwise we go create a new one and modify it we call save save as you saw in the earlier code automatically knows whether or not it is a new one or an old one what based on whether or not the ID exists and then we redirect back to the main HTML page once we’re done so this is literally the entire app it’s very simple for this example for our purpose what is important is you can see we’re just doing a regular cyclo this is cyclo PG we’re starting a tornado IO loop the regular tornado I loop here and we just run it and it runs the normal way the tornado would run if you’re doing something synchronous it’s actually very boring stuff which is good because we’re gonna make it not boring here now so now so what from what we saw it is you know it’s Trent normal transactional code the one thing is is that we’re going to lose some of the advantages of using tornado because that was synchronous stuff so what’s gonna happen is you’re gonna make a request and you can have lots of requests come in but you still have the synchronous blocking on the on the database access code which would be nice to kind of get rid of that and that’s what we’re going to talk about doing next so one way to do that is to use a library called Momoko Momoko is a library written specifically for tornado to do asynchronous i/o with Postgres and i’m going to show you guys the code to do the exact same thing we just did synchronously but this time we’re going to use to do it so the first thing is we had to change in order to do this first thing we have to do is we have to change our ORM a little bit so I basically have subclass the previous one that we had this same or in that we had before but the only the difference here is now are updating our save methods our co-routines and what that allows us to do is by doing that now we’re going to instead of just calling our execute method now we have to yield it and the nice thing about Momoko is we can just say here’s the connection we’re gonna rather than normally before we got the cursor now we’re actually just saying yield off the database execute this and we’re passing the cursor factory here as a parameter instead of getting the cursor and doing the stuff Superboy it’s basically Momoko has a nice way of kind of shortcutting those two things otherwise you have to get the cursor yield that and then yield the results this is just a weighted shortcut that process of having to do

two things and and that is the change that we have to make it there isn’t me there aren’t many changes that we had to make at this level to do that you’ll see everything else is the same the queries have not changed from there from the original version what has changed though is our get and our get all just like our save and update into ORM levels it getting to get all have also changed in the exact same way with the yields here here now notice we only in this one we only have to put a yield here because this yield is saying hey this is a future and then when you call fetch all on this future inside Momoko it says oh okay go and handle that and return an actual value as the results and so the rest of that is handled under the hood so you only have to do the yield in one place here the other change is we had to make these all po routines on our controllers we needed to yield the results of our calls to the model now instead of making a straight Club the model now they have to be yielded instead and then down here we have to do a little bit of magic to get the database set up properly we have to create a pool because since we’re doing any synchronous stuff we’re going to potentially be using more than one connection at a time at the database now we have a pool of connections instead of a single connection this is just some stuff that you have to do to tell Momoko to load the get to to actually create the pool and stuff and then once you’ve done that this is a little hack I had to do in order to get the composites to register you could there’s you’d have to do some work in Momoko to actually allow it to call the register composite under the hood because it’s assuming a synchronous connection here and we have an asynchronous connection so I just make a second connection and once cyclo PG know what’s happening here is the in order to register the composite has to query the database and get the OID out of the out of Postgres for the game table in order to know to map those properly so in order to do that it means a database connection that’s the only thing it’s actually using us for the rest of that if that’s storing it in the Global’s of the cyclope g2 library so you don’t you can use psycho PG to with multiple different connections and all you have to register the composite once when you do this little equals true so that’s kind of a hack I had to do to get that to work you couldn’t fix that hack I just didn’t have time you’d have to go and you would have to go and work in the Momoko library too to actually kind of do register composite differently so I have to actually talk to the author and we’ve talked about how to do it we just haven’t written the code well no you what you would do is you would write a you would write a Momoko version of register composite that would go and do it so you know there isn’t as you can see there’s not a lot of differences here one thing that you may may have noticed was that all the transactional stuff was gone and that’s because by default when you go into asynchronous stuff the default issue is you’re now in Auto commit instead of transactional that doesn’t mean you can’t do transactional you can do transactions you just have to manually do them yourself which is kind of a pain in the butt but it can be done I didn’t bother to do it for this one because it would have made the code Messier and I was running out of time to do it but it is hard it makes it a lot harder to do the other thing that you probably saw is you have to yield and use Genco became pretty much everywhere any place you want to make something asynchronous you have to do the yield and Genco routine and the problem is once you’ve done that you cannot really unyte asynchronous things so you once you’ve made it a a synchronous thing you’ve got to use it asynchronously it won’t work the old synchronous way so you can’t mix match very well you have to go one way or the other with your stuff the other thing that is kind of sucky about this is let’s say that you were one using your models that you’ve written and you want to use them in a offline module as well as in your web application well now if you’re using multiple and you got

running for NATO i/o loop in your non web code and that’s kind of not the world’s best solution so there is a solution to that problem and that’s when we’re going to talk about next which is we could use async i/o and I hope you do and so instead of relying on Momoko and relying on tornado specific stuff now we can rely on stuff straight out of the standard library from Python 3 for AI OPG is a library for doing asynchronous i/o and Postgres No it uses psycho PG as well and you’ll see it my ide is not set up for three-four and it thinks there’s errors all over the place when there’s a yield from so so once again we have okay so once again I’ve taken the regular by regular or M that I created made a subclass of it so I could override update and save again this time you’ll notice that instead of doing a gender routine out of tornado I’m now using the standard library async ioco routine so there is nothing in here is no longer core Nadeau specified anywhere in here so you can use this outside of tornado which is a good thing the other thing that is different from the Momoko one there’s two other differences here you use yield from everywhere instead of you there are some differences between what yield does and what yield from does and they are like super technical detailing things that half the time I even forget what they are but you need to use guild from here yield will not work if you try to use yo you’re going to get all kinds of nasty exceptions the other thing is is I have if you’ll recall that was on one yield and the other one here there’s two yields and that’s simply because first you have to get the cursor and then you call execute on the cursor just like you would a normal psycho PG so this one actually looks in that way it looks more like the original code with the exception of yield from x’ instead of just calling the functions and then the same thing down here except also when you get the results we have to yield from the results as well here so the results are still a future and it’s no longer a future until we get down here where we’re actually using the results so when you’re doing so in asynchronous stuff when you’re doing a when you’re doing anything asynchronous all those yields are returning all those yields are yielding you know generative means we’re using generators here what they’re returning each time you call the yield is they’re returning a what’s called a future object and the future object basically keeps getting Pat gets passed around and stuff and that’s what actually allows the asynchronous part to happen but when you get to a place where you’re using the value as opposed to just kind of passing around this the first time you go to use it it goes and says hey evaluate evaluate it if if the asynchronous call has not completed yet right because it’s a call to the database it’s a call to the database it may not have been completed yet then we’re gonna sit here and wait and give up execution to some other asynchronous piece of code this is where the asynchronous part comes in so your code will run straight through it’ll go and tell it to run the database until it gets to a place where you’re using the results from your yields in something useful and then it’s gonna sit there and wait until the results actually come back on the database but it won’t block it what I’ll do is we’ll wait and then it’ll go and wait for the results from another it’ll go and look at the other things that you have asynchronously and try them as well so that is so this is act that’s where the important thing happens is when you actually go to use the results that’s what makes these things that’s what makes the all the asynchronous packages work we’ve talked about both Momoko and regard so once again as you can there’s very few changes we have to make except from here’s our yield from –za ghin the exact same thing that we saw just a minute ago you have to yield from the cursor yield from the execute yield from on your fetch all these are literally they a OPG was very they’ve tried to make it look as much like regular cycle PG as possible so you call the cursor function to get a car cursor so you call execute

execute a query you call a fetch all together to get the results and then you just yield from all of them and we’re doing the same thing I’m to get oh then the only other thing nothing else has changed here from the previous one except now we’re doing yield from sand study yields and one thing you’ll notice these are still Genco routines because we’re inside of tornado and for NATO is expecting the tornado stuff and tornado has some adapting that it does automatically under the hood that says hey this is a this is an async IO future but since we’re trying to have the to mix you’re allowed to take the async IO future and return it into a tornado co-routine and it works it works so the okay so here’s where the other big changes in order to get the set up to work in order to actually get the connection to work the connection is also a sinker the connection setup stuff is also asynchronous so what I had to do to make that is make a KO routine that does the database set up it does a yield from the pool in order to create the the database pool object which is basically it’s a connection object it works while I’m at you can see I used it as a substitution for the connection object in psycho PG HS and it’s actually a pool under the hood and or do that so we have to call this install which what this is saying is take the async IO loop and install it on the tornado loop so that the two loops are basically merged together now one can get the event loop we can run until complete this DB set up on that event loop which basically goes and says okay now we can go run this thing to get the pool all set up then we’re back to everything normal except for we’re now running loop run forever instead of the normal normal start app that we would do for tornado you do have a different thing there starting and that is what we’re going to do to make it work on async Iowa all right so one thing the transactional is even a little bit harder and async IO moco has some a couple of methods that kind of make the transactional harder and AI OPG they say run I call select you know to basically do a begin transaction and transaction as an actual query and that’s how you would do transactional there so transactional is a lot harder here once again exactly because and it makes sense if you think about it the rug because you can’t actually do that that way because you have two different content you you could do that and you just if you just blindly do that try to do that the asynchronous library by default is gonna go and try and get two different connections because the pool every time you call the pool saying get me a free connection and transactions run inside of a connection so if you have two connections you’re gonna have two different becoming them in two different places so that’s going to that’s gonna create problems for you there are like I said there is there are some ways around it for instance the Momoko library has a function call and basically what it does it says take all your take all your calls that go that are going into this transaction put them into a room an array wrap and it wraps basically wraps around it begin and end transaction and dumps the whole thing to the database once and then waits for the results from each of the queries as how Momoko handles it problem is is that won’t work particularly well if you want to have transactional stuff happening in three or four different functions and our those functions going to know that they’re communicating with each other so that creates so the transactional stuff when you’re getting the asynchronous stuff it makes up it’s hard once again you have to do the yield and use the async i/o co-routines just like we saw the floor you have to do the event a little bit of a vent loop switching but the nice thing about that is now because you have because your model code is written with async i/o you can run a you can run it in any place that you’re running an AC that you’re running an async IO event loop so you can run your async IO event loop in your code that is going offline and doing offline processing stuff and still using your model code without having a dependency on tornado um the

other thing that is nice is if you are a sequel alchemy user hey OPG the library that does this provide some monkey patching to allow it to work with sequel alchemy I have not tried this but I wanted to bring it up because a lot of people like seek welcoming there’s nothing wrong with it I have no issues with sequel alchemy and you can use it with this um the other thing is and I think I mentioned this when we were looking at the code the yield from is really important when I started trying to I spent two hours realizing that I was had yield where I need yield from and made it really hard so yeah you need to use yield from when you’re using async I know and use regular yield when you’re using Momoko those are very important ok so now the last one that I wanted to show you guys so last one I’m going to show you is using G event so if nobody knew you can use G event with with ug event with tornado and get asynchronous stuff out of it one thing you’re going to notice is there is no special or I’m code required for this so I have an async I own and Momoko and the regular synchronous stuff but I don’t need one for G event and that is because the way G of n is written G event for those that don’t know uses the whole green thread greenlit concept of things so you’re basically creating not real threads but little mini threads that are doing the work and the nice thing about that is G event basically does a little bit of monkey patching you have to do this at the very top in order of your application in order to actually get it to work right but basically what that does that goes and says hey any time I’m going to go and do something that will be normally that we would want to have asynchronous stuff go ahead and release the it’s not a guilt but it’s basically release the lock and viewer the normal green lid key event kind of pseudo thread and stuff that it works similar to threads but the thing is is you’re gonna see nice thing about this is the code is the exact same code from the original version get all the get everything is exactly the same we can do transactions normally commits rollback all that stuff the main places where we have a change the only changes we have its here we are in our get we’re using the regular tornado furnace it’s not a cobra key but an asynchronous request and then what we’re going to do is when we get the web request we’re going to say all right go spawn a green lamp to do this work and we go off and we do the work and we’re done and we can do the same thing here and the same thing here and the work is done in these very simple functions that are basically so it is my mind this is a far I is this seems like a much easier way to understand things it’s not true asynchronous in the way that the other two methods are but it’s a lot cleaner to read I think so the disadvantage here of course is that inside of inside of this async task everything is happening synchronously so if you went unlike our other ones we wanted to do multiple calls to the database each call to the database is going to be synchronous inside this asynchronous task whereas if we were using the synchronous code that I showed you earlier you could do every single database call as a separate asynchronous query and then you could wait we have one wait for the other one and B and have the asynchronously down to a lower level so the thing it’s asynchronous with this is what will happen though now is when this goes off to the database it’s going to give up it’s it’s going to basically give up its lock on the the Python interpreter and let other stuff running so it’s a slightly different model to be honest I will tell you right now I didn’t like performance tests any of these so I have no idea which things

work the best and what the performance trade-off is of doing it this way as opposed to doing it the traditional way but I was basically the one to present you guys with the options of how you can do it the other thing is you would probably want to actually put this in some sort of pool if you were doing this for real because otherwise you’re still actually one running off of a single database I’m a single database connection so this is not done properly for using the G event the right way but the point I was trying to make with the G of n example is it is much it’s a different way of doing it than the other ones I showed you and it’s simpler in a lot of regards because all you have to do is do this as opposed to populate you know for co-routines everywhere and it’s a very big paradigm shift to do a normal asynchronous stuff as opposed to the and basically all things I just said are repeated here I think you only have to modify your controllers the models were exactly the same as your saw which is nice the nice thing about this is if you want to mix synchronous and asynchronous code you can do that without a problem because you have a modified your models and flowing have to do your controllers to make the asynchronous stuff work you don’t have to have the co-routines everywhere and as I mentioned you can’t it’s not you can’t have easily lots of asynchronous things going on inside of a thing unless you spawn you have to spawn separate gremlins for everything you want to do asynchronously in order to get that to work so if you otherwise inside of the actual Greenland everything is synchronous so yeah the other thing is is since we’re not using the co-routines you can run the model code without the requirement of having an IO loop at all I think I really thing inside that asynchronous task once again it’s synchronous so you can use transactions just like you normally would because it’s all synchronous inside of there and the other thing is is you can use seek welcome you with this whoops okay so that blog link at the bottom is I will shamelessly tell you that I figure out how to do the G event stuff by reading that guy’s blog and copying and pasting it into my code and seeing if it work and it work so so that person’s blog is actually the go-to thing for figuring out how this works so I figured I should plug the person to actually go figure out how to do this for me I did not figure this out on my own okay so so it kind of in summary here who routines are hard it’s they’re easy to mess up they are it’s a paradigm shift you have to really wrap your head around it it’s not easy to do but the advantages you get the asynchronous stuff out of it now mind you I’ve only been talking about asynchronous from the point of your web request talking to the database as you know earlier today if we went to David’s talk he showed some asynchronous stuff from the front end side of things where where he was doing WebSocket and ancient Christy was there and that is a totally different it’s you know it’s the same kinds of things but it is a slightly different thing once you map put all those things together and you start to build a really complex that you get a lot of things but this this what we’ve really only talked about here is the backend side of doing asynchronous stuff not the front-end side and the methods that we talked about here even though they’re using we were specifically using psycho PG if you are switching it out with Redis or god forbid or whatever other thing whatever other thing whatever other things that you like to use the concepts are basically the same yes I’m opinionated I admit it so you know there are three choices here that I showed you and each one of them is good in their own way the Momoko stuff works on Python to seven as well as Python three four so if you’re stuck in Python to seven land you can use that async i/o gives you the ability to use stuff without having dependencies on tornado you know and then there’s the chi event model which makes things like there’s a lot of different ways you can accomplish

the same thing and I wouldn’t say that there’s a right or a wrong way I haven’t I have not used this in production yet I have only been using this to build some tests and samples and try and working on some projects some side projects with it so I haven’t really decided which which one I even like because I see advantages and disadvantages to all of them you know the ACE the async i/o method you can ng event you can both get to work with other ORM s particularly sequel for me which is a good thing you know I am NOT a huge RM person so that wasn’t a big thing for me but a lot of people are and there’s nothing wrong with that with caveats we happy to discuss those and the other thing is that you know you don’t have to necessarily be tied to tornado here you can use the async i/o stuff with whatever your favorite other framework is and G event will also work outside of tornado as well so anybody has any questions I’d be happy to take questions there’s lots of contact information for me feel free to contact me and whatever way you see fit and like the most any questions yeah dad can call him by name so going to auto commit just seems like a huge step backwards and I can’t wrap my mind around the implications for that but have you thought about that at all so in the one sample one of the things I was doing when I was actually trying to build something for real with a real project with this what I ended up doing is I created two connections one was a synchronous connect and this is before this was a my first start playing with this so I haven’t tried using some of the transactional things that you can do but the way I first decided to address that problem was I created two connections one was a synchronous connection and one was an asynchronous connection and when I wanted to read from the database I use the asynchronous connection and when I was doing a post I used the synchronous connection on the post and so everything was transactional inside there any more questions about anything what universe whatever and forfour was back there the yield from fetch all in AI OPG mm-hmm you had to yield from the cursor creation the execute and the fetch all yes but I don’t quite understand why I figured once once you got to the point of fetching all I thought the data was on your client machine in the psycho PG world psycho pin just simple psycho PG – so I don’t understand why the fetch all would be asynchronous so in Kop G they have decided because if you saw on the moko that was the case in the example code in mocha that was exactly what you what you said it was when you did the fetch all it actually went got it and a OPG case they’re actually waiting until you use the results fast so the fetch all is not as returning once again another future does be so it’s delaying it until you’re actually using the code inside of there so just delayed there’s basically they’re chosen to delay it to the absolute last possible minute and you can the one thing and async I owe that you can do with that fetch all that you fetch all normally in psycho Fiji returns a list in the case of a OPG it is actually returned because it’s a future and it’s a generator you can actually act on it it’s a generator so you can take that you put that fetch all in a for loop and it’s going to be treating it as a generator not as an iterator so it’s not you’re not iterating on all this trader aiding on generated now it’s good it’s still using the it’s just like fetch Hogs doing that but it’s doing it because even if you run fetch all fetch all doesn’t actually go get everything either thing it goes and gets them in batches of whatever the batch size is set to encyclopaedia any other questions okay thank you very much