Storlets: Making Swift More Software Defined Than Ever

morning everyone thanks for coming this is our subject for this morning and we are with ok I’m polish from from Intel i’m a software from intel i have not been contributing to starlets I’m one of the Swift core developers but I think store that’s our way freakin cool so I’m up here to give you a couple of slides on Swift to make sure we’re all level set there that’s makes it work hey I’m handy or Manny from IBM I want to really thank you all for coming out i know it’s kind of early today after the HP event yesterday but i’m also from IBM and I’ve been working on starlets for maybe three or four months and doing Swift for a little bit longer and like Paul said I think they’re really neat thing so I hope you guys enjoy and I mehran i’m also from IBM I’m the IBM technical lead for the stuff and I really like it so I hope you’ll like it too so let’s get started so this is like the usual agenda will talk about the concept the motivating use cases then Paul would give the Swift overview and then hum they would give the store let’s overview and they stole it um OpenStack project yes we have we have it as an open steppe project now and now I’ll end up with vision all right so the concept imagine you have a storage system that can hold petabytes of storage many petabytes of storage and the system is being used as a service so what would people put their their photos the right put their IOT data archive coyote data in the format of CSV files they might put their 3d designs this is a bit futuristic I’ll think about the 3d printing era or talk about it later and they might put their like digital media files huge files usually now what happens if one wishes to do some computation over the data without moving it around from various reasons that I’ll show next so the concept here is to co-locate compute in the storage system and the reason that I’m joined here a docker container as the compute engine is because we’re not talking about any computation we’re talking about computations that are uploaded by users so we want them to be well isolated so I could have drawn your like kvm or anything else but since we were walking with docker there’s the docker logo there so why would we want that let’s assume that user put his photos arm in the storage system now we know that photos have embedded metadata called EXIF metadata and JPEGs and that exists metadata is quite reach you can get there all the camera settings where it was photo and so on and so forth this means that the user would probably want to ask something like how many pictures were taking in Tokyo between those dates where the easel that was used was 400 right valid question it would probably want to use his favorites analytics engine spark here’s the problem the problem is that sparks know how to how to process semi-structured data whereas the metadata is saved is embedded in binary files which are the JPEGs right so there’s a problem here so I call this use case data preparation where we can use compute on the storage to extract the EXIF metadata from the pictures before we um before we do the instead of actually downloading the pictures to the Swift cluster sorry to the spark cluster so that the spark only gets the EXIF metadata so we save on bandwidth we save on memory in spark in the spark cluster and of course we actually make this use case of work otherwise we couldn’t do that so these years kids was explored by was Michael factor and gilt vernick from IBM gave a talk in the Prairie Swift xamot about this here is link the data preparation use case I’ll move to the next use case which i call it predicate push down so suppose that we had all this information about the picture taken in a CSV file rather than in the arm JPEGs so we have a huge table there and we want to make the same query right so let’s zoom in into the CSV files its huge table containing lots of rows one for each picture and for each picture picture we have all the EXIF metadata like the location the left of the ESO focal lens and so on so fourth for our query we’re not really interested in how in all the data right we’re interested only in the location column the ether column and the date according to the query moreover we only want the line coming from Paris so um sorry that’s the only I see ? interfaces

so the Paris line is the only one that matches the the query here okay so why not push this filtering to the storage right so we don’t need to download all the CSV to the spark cluster again we can just use toilets or to use the compute I’d in systolic yet to use the computer near the storage to actually do the filtering inside the storage system so this is predicate posh town we have seen significant reduction in the overall time taken to process the query I should I mean from from from entwined light from the spark point of view although i should say these are very initial results but it’s promising i do want to mention one more thing about his use case so on the tuesday talk we were talking about another kind of push down which was used by metadata search so what have you done with metadata search with metadata search we’ve shown the capability to narrow down the list of objects that were interested in using metadata search we have a bunch of objects in the object store that we want to process right however our query is such that only small part of them are we offer are of interest so we can do use metadata search to narrow down to that small group of objects and on them do this stuff right so these are like complementary solutions right let’s talk about data security arm and this brings me to the 3d printing era so it is said that in the 3d printing era the ability to manufacture becomes commodity so people are not longer going to pay for the ability of one to manufacture something but rather more on the design by the way the right hand side on the right hand side there is an illustration of the country concrete based 3d printer um they have it working there like nice new YouTube showing that anyway so assuming people would still use object stores in this era people would probably put their 3d designs in the object store um that would probably never want them sorry that was too early they would probably never won those design to leave the object store however they would be very happy to sell a printed version of the design what do I mean by that usually when you want to print something some 3d model you need to do some lossy transformation that allows it to be printed and that lossy transformation is dependent on the actual printer you’re using right so again using store let’s one can download the transformed version of the 3d design that fit is desktop or her desktop on printer but we don’t really need to wait for the 3d printing era think about this one or you have like medical records inside your storage system and you want to make it available to researchers but since this is medical record you probably want to D identify them before you give them to anyone else right so the personal information is being erased there um so this use case was further explored by the forget IT EU project they’ve done their face blurring so in the storage system there was the actual picture and once downloaded the faces were blurred I hope you can see this in the picture so there is a link to the YouTube that shows this demo it’s quite nice um I’ll skip about this one really fast so this one we’ve talked about in in the OpenStack Perry summit it has to do with digital films so the RAI the radio television italy national broadcaster was using stole ‘it’s to actually employ algorithms once the object stood already in the stores to think about it this way these are large files very large files so suppose you’ve already put them into storage and after putting them you came up with a new algorithm to extract feature for them for example the loudness of the film so instead of downloading downloading the object calculating it and then uploading it again or whatever you can just do this in place with compete on storage last use case inspired by a comment that was gagged by pole at the time so i call it the superuser use case so in recent years were seeing more and more storage systems there are not imply insist rather their software commodity hardware and there’s an operator there that does the installation of the software on the hardware operates the hardware and sell it as a service right so if he could use computer on storage he could add more

value of generic functionality to the storage system using this compute right it doesn’t need to wait for the arm for the software developers to do that for him you can just edit so long so examples are like antivirus compression encryption and so on and so forth like generic storage stuff that you can just add to the system perhaps poll which I will next how easy it can be done with Swift oh this idea was taken further by a project called aisle stack that’s a European project that by do I Finding Nemo Trippier what they do is that they have added the policy layer on top that says something like all right every time there is a request for a certain for from a user belonging to a certain account or sir to tenant to a compressional or encrypted so it’s kind of an automating layer to use those computations so with that I’m oh sure that’s most importantly so all this stuff is now available as an OpenStack project in github um-hum d we’ll talk about it and of course any help is mostly welcome so I’ll hand the mic to UM Paul the other way yeah me the clicker maybe I’m just using excellent alright so as awesome awesomer I’d love to see these use cases they really do a great job of identifying the motivation for the work these guys have done it’s so much more effective I think to see real use cases especially a variety of them as opposed to saying what if and painting some hypothetical stuff this is all really cool stuff it’s got a lot of us really excited about it so before how many gets into the details of starlets I wanted to make sure we were all sort of level set on some very basics of Swift I know there’s a lot of Swift people in the audience but if you don’t have the basics it’s hard to understand how all the stuff bolts together and really where things are making connections ok so a couple things about the Swift community as probably most folks know we’re one of the first two projects an OpenStack we’re up to somewhere around 40,000 lines of functional code and another 80 thousand lines of test code so we’re somewhere generally between two to one and three two one on functional versus test code which is pretty good we’ve got just a fantastic community I say there’s a lot of core folks in here we all love to come to each other’s talks and support us so so thanks for coming but John our PTL and one of our sessions yesterday gave some really cool graph showing the increase in the number of contributors in the Swift community and how often they come back and and how long they’ve stuck with the program and it really tells just a fantastic story of if you’re a developer what a great community this is to work in so besides the long list of company names here I did collect a few staff from the last three summits and you can see the number of presentations in the main conference not the design summit just was swift in their title right and I didn’t count it twice when they used swept twice in the title in Paris we had 14 and then Vancouver 19 and then up over 20 here in Tokyo so we just continue to see more and more interest and more people voting to hear more about Swift so it’s just a really exciting program to work on a fantastic community to work in won’t go through all the timeline below just to be clear these are things of sort of focus areas for these releases they’re not actual features in the release for so for example are not releasing encryption right now but this is a big focus area for us here at this design summit lots of discussions on the encryption work that’s going on okay so a couple of high-level bullets on Swift’s so you get a picture of what it is obviously it’s an object storage system we use a container model very much like s3 buckets to group things together for like characteristics easiest one to graphs of course is ackles right securities also storage policies is a big one if you’ve been following Swift that’s a year before last big feature that we added everything is through a restful interface right stateless restful interface so very easy to use very strong API lots of other storage systems have layers built on top so they can actually support Swift so we’ve sort of become a de facto standard in that area and then of course built on standard hardware and highly scalable and efficient so you don’t have to go with any specific vendor lock-in to go out and buy something and build yourself a swift cluster pretty much you can build it with what anything you’ve got right stitch together and make it work and eventually consistent if you’ve been in any Swift talks you understand the value of eventual consistency and it’s funny when you talk to people for the first time and they haven’t heard of this they think maybe it’s a bug it’s not a bug

it’s designed to be that way and there’s a really good reasons for it if anybody wants to talk more about that afterwards there’s plenty of us here that can explain that to you so before before how many gets into store let’s I wanted to do a high level architectural overview of sort of the main modules within Swift based on how their tiered this is important so you can see where store let’s bolts in okay so on the top the top box here this is the proxy tier so if you’re familiar with Swift it’s a two tier architecture the proxy tier is where we scale for concurrency so we can scale in two directions if you need additional concurrency or add more proxy services if you need additional capacity you add more storage services you don’t have to scale them at the same time the main blocks on the top is our whiskey server our proxy application and then the one that’s highlighted there which is important for this talk is middleware right so extremely cool extensibility feature within Swift there’s been lots of talks over the last couple of summits Kristen had a great one in Paris was it Hong Kong on the world was yes but fantastic talk we wrote middleware from that from the ground up like right during the session and and showed it so really shows the power of middleware and what store list does for middleware if you haven’t already seen one of these talks it’ll blow your mind it’s so friggin cool but but that middleware capability exists in the proxy tier and and if you look down on the bottom it also exists in our storage tier in our capacity tier so the the high level software architecture is the same we’ve got a whiskey server down there we’ve got the middle work the middleware a framework and then we’ve got multiple different proxy applications or multiple different applications I should say that run on the storage node to handle all the various Swift stuff and we’ve got more detailed slides on this stuff too and if anybody wants to get into you know more of the guts of Swift I don’t to take them anymore the store it’s time to do this but we can certainly do that because it’s it’s some really cool stuff but that’s the key point here is the the middleware capability is where store that’s bolts in and then just to give you a visual of you know sort of what it looks like to put an object into Swift and what it looks like to put an object at a swift and then after you see the store of stuff you’ll see how that intercepts things and does it’s magic when we put an object into Swift it comes in through our load balancer in our access tier where load balancer auth services and proxy is at so that’s where that little object is flying in it’s going to get routed in this case we’re using replication actually should have put the EC one in here thank you and shameless plug there’s a talk on AC like four rows down right after this so come down and hear about the progress on that stuff but there we aren’t shuffling this thing into three locations within the stores node and then on the get side we actually only get it from one place there’s actually configurable how we do the get but but you can see it comes in hits its three nodes and on its way back out it’s a one noon so that’s the that’s the general Swift overview and what I’m be taking on so much knowledge I want to start by saying I feel bad we didn’t include Swift in the type of his presentation because I would’ve been a plus one for you I think in terms of your account so I apologize go change the slide make it 25 I’ll go edit the title after so just to start I mean Aaron gave really good overview and so did Paul about what’s or letarte and what Swift is and I’m going to give it kind of a more in-depth detailed explanation of exactly how store lates fit into the Swift architecture so what exactly is a store late well it’s pretty simple store lights are compiled Java code now I want to start by saying Java code is currently what we support but it’s something that we’re looking to actually extend in the future to other programming language is maybe Python but I thought I would just throw that out there so store let’s reside as middleware and the really great thing about that is it essentially implies it required zero changes at all to core Swift we needed no changes we didn’t touch any of the core swift code whatsoever I really want to point that out because it’s a lot of complex algorithms but it all resides through middleware which implies it’s completely transparent to your Swift object storage and you can plug it in extremely easily so more specifically how do you store let’s work so store let’s utilize docker and talker may potentially reside on the same server a different server where your proxy or object server are and that’s what’s going to drive the compute engine so as a really kind of high-level overview what could happen is you’ll start by uploading the store late and all the store lidded is is code and it’s going to simply be an actual object sitting in Swift directly then you can actually inform Swift through as you’ll see later a header that you want to execute the store lit on a specific object we then kind of view all the magic of taking the input stream and output stream wiring them together and then letting that Java code executes on the Swift object so it’s actually really nifty um the reason why we use docker is it really does i mean like Aaron mentioned provide a lot of good security and multi-tenancy as well so we

essentially have a docker engine / Swift account and that kind of provides the isolation and multi-tenancy that we have across the system so Paul went into you really good overview of what middleware is and what this slide is really pointing out is that’s really all stirlitz are they essentially reside as middleware and the interesting thing about it is that they can actually live either on the proxy side or the object side so what that means is you can intercept data coming in at the proxy layer or even potentially at the object server layer which is all the next slide actually shows so how can you actually upload this compute to Swift well it’s it’s pretty straightforward I was trying to give an example Aaron had a bunch of good ones but I’m trying to thick of a silly one on the way in today and I was thinking okay let’s say you Swift as your code repository system maybe that’s silly it probably is and let’s say you store Python code in there and let’s say you really love Pepe I mean a lot of folks don’t so you want to Peppa Phi stuff on the way in so what would you do so you found kind of a nifty pep eight library lying around somewhere in the ecosystem first step is you write a simple Java program I know it’s silly you’re using Java to Peppa fight Python code but I can think of anything else and you take this library you write the Java code hopefully you test it you compile it you get it reviewed and then the very next step is essentially you create a package a jar file for for the Java code and then you can take this dependency is kind of a separate thing and you upload both to the Swift to the Swift object storage system and there’s a container you can put that end as a default one called the store let’s container but you can create your own and modify all that we actually have a lot of nifty tools that will actually allow you to to do all this automate the entire process all you have to supply is a jar file and we’ll upload it automatically to the swiss system another really interesting thing is that as we mentioned earlier store let’s actually execute in a docker container and an image and you can actually adjust that docker container to have any dependencies that you need so in this example it’s just one simple java package java library that does a peb eight code but for example say you wanted to pull in a lot of other package your better bet is actually to modify that docker image that will use to run your store lik code so here we just kind of go through the exact steps of the put data flow in terms of executing store let’s and the really key thing to note is really all you really need to use is this new header X run store lit an X runs stole it essentially says for this object I want to execute this particular store late the store late will reside in a different container and a different account and this will just say this is the exact code that compute per se you want to run on the specific object it’s exactly like uploading any other Swift object release so this kind of just goes through all the steps and really the key thing I want to point out is really all we do in the put flow is we take the input stream which is what the user is supplying as data why are that to the input of the actual store let the Java code in this case and the output stream is actually to go to the object server and it’s going to end up being written to disk so just kind of a silly example if your store late did nothing at all it just may be printed some logs and actually didn’t do any compete whatsoever the result of running that story it will be as if you had no stirlitz at all it’s there’s absolutely going to be and no changes down to the actual object you uploaded to store storage so this kind of goes over the get data flow and the one thing you should notice is it’s essentially exactly the same the only difference here is the input stream is good obviously be the incoming object from the object server whereas the output stream is going to go back to the client so in this slide over here we kind of go over the more detailed store late architecture rocket spend too much time on it but if anyone’s interested feel free to talk to me after and we can get into more details but the one thing I want to point out is essentially we have a docker container per Swift account and that’s how we do the multi tendency aspect in the security aspect and as you can imagine when you need to have some kind of communication mechanism between the two the two being the Swift system and then docker residing on potentially different system and this is kind of the way we actually enable that so we have the store let’s middleware itself and then we have these buses and then just domain sockets that we use to be able to send commands and messages and the data itself ask the input stream the file descriptors which is what we end up using between the Swift potentially proxy or object server and wherever your doctor container resides so how do you actually write a store late so I just wanted to show a really simple example to show it’s actually very straightforward if you look at the actual interface to this function this java function it’s really simple and what you can imagine is there’s an input stream and an output stream and then parameters and parameters in this case or simply the query string that you that you supplied for example if you were doing a put or it guess they’ll be actually supplied to your store late as

well so you can communicate data as well to the store late and the actual code sorry a little bit messy but the only thing I’m trying to show here is that really all you have to do is kind of what you would do is if you were doing anything else in Swift which is the very first step is you get all the metadata and then you set that into the output stream so you’re essentially flowing back by HTTP headers after that you’re free to play with the input and output stream as you wish and this is where all your compute resides so really all kind of the fun and magic of store let’s lives in this space here regardless of what you were doing so really writing store let’s couldn’t really be any more simple than that ya know this is it was made extensible for things we could change in the future for example if you had multiple input streams but the way it works now is simply on one object but it’s a really good question because that’s an idea that’s came to us in the past which is is there any way we can actually work on multiple objects as opposed to just one that’s a little bit more difficult to do biz as you can imagine when you’re working with Swift itself generally speaking there’s a specific object you’re always dealing with in the system whether it be a get or a put so but very good point and if you’d like to work on that we’d love you to you yeah okay so for example say you actually want to start helping with the store lit project what exactly can you do it’s actually really really simple so the very first step is you can get a u-bahn two of you bun to image oh we’re gonna probably expand this in the future to be compatible with other operating systems and it turns it yes and in terms of actually the install process really all you need is a pseudo a password list account that you can end up using clone the code from the github account which is yes an OpenStack project now and we have a nice script sqa i/o which it’ll actually set up everything for you this is lift and the soil its ecosystem so it’s kind of an ansible script that will put everything together and install store let’s but I can imagine if you already have switch systems out there and in that case and you still want to play on the store lates it’s actually still just as easy and there’s essentially an ansible script you can run there is a configuration file that point to where your proxy resides where your object server resides and a bunch of other details like that and you can use that to essentially allow the script to configure store list and install it on your system so this is actually my favorite slide I know we’ve mentioned this many times but yes store let’s is now an official OpenStack project for us that’s actually really great because it makes development a lot easier because it gives us all the magic of OpenStack and by that I mean we have the garret review system Jenkins for testing and we actually have tests they’re out now they will actually execute and on any changes that you submit better yet documentation documentation is a lot more clearer now we have an IRC channel as well and you can actually submit bugs and feature a request to but the real key thing I want to say in the slide is we’re very much looking for help and any contributions or even operator so if someone wants to try it out reach it reach out to us on IRC there’s some emails you can use there as well I think that’s it Aaron do you want to take over for the vision slides thank you very much honey I’m good alright so here’s my vision for stole it so clearly we’re currently focused on integrating into Swift using docker and running stores in Java but there is no reason why we couldn’t add more languages there I think that Scala is in particular interesting cause many spark developers spark has a nice feature called user-defined functions that can run over our spark data during analysis and I think they’re mostly using Scala so if we could write stories in Scala and push them down automatically that could be like a huge ecosystem thing next I mean why just use docker why not you know go 0 vm kvm or even integrate with OpenStack Nova and get whatever opens technova works with right and I should also mention Magnum in this case and then perhaps more futuristic is to look at other storage systems such as the safe object the gateway or databases that have armed blobs attached to them so this is the vision and I’ll end up with this thought here and before any guy just want to really really thank durin Han from our team Haifa who did the tremendous work of making this an OpenStack project all the integrations with Jenkins and Garrett it’s not reveal and then Gil Vernick and yourself and what you working very hard on the push down on the spark push down of solids to Swift and that is thank you very much and will be very happy to answers or before the questions ok thanks I’ll go for it oh yeah so on the Intel side if anybody hears participating in the marketing thing called the Intel passport program please see my hottie maybe you could stand up raise your hand please see her and she’ll take care of you and make sure you get stamped or whatever they’re

doing thanks and really quickly from IBM and again i’m forced to say this but we’re actually hiring so if anyone’s interested at all feel free contact me and imma leave it at that so thank you all so much thanks so any questions thanks just recording so thank you for a good presentation so just my question is for the developer said whether so how it’s working for the learning the container on the copy case so how do you explain the the the current case on put and get and the sometimes we want to make a new object from the old pan origin raw data into the some temporary metadata case yeah so absolutely good point so in Paris we actually gave we had a post verb which is um which I would say is not the right verb to do that copy is the right verb to do that and this is on top of my feature list we actually have used case in IBM that needs that so I’m going up anybody wishes to work on that it’s very welcome in other words that was really good question I thought about that sounds good thank you anybody else sang skoda so my question is about a resource isolation so for example if a user execute something that takes a long time what happens in case for example for timeouts what about if the load on one of the storage nodes gets too high because sometimes for example after rebalance you have to shuffle a lot of data to another node and that might interact with the loads that you see from the docker containers do you work on late or yes well yes we now know why Intel we now know why Paul is very supported on the future so um so basically um this is something that is not yet implemented but we’re trusting that docker would have the chainable it has some tuna balls they’re not as reaches what other container technologies gives you or what see groups gives you but at the end of the day we’re going to have to add the man those two Nobles as managed by the operator so at the end of the day the user can control the docker image but the operator would control the tunable of the isolation just answer the question or I must something yes basically what about if you want to isolate different accounts and give different accounts a different level of resources that they can use right so yeah yeah so so the this falls into the same category something that we haven’t done it again help is most so just just to quickly add though that’s a really good point and the way it works today’s you definitely need that store late’ to respond within a certain period of time otherwise you hit timeouts on the proxy object server side so it’s actually really excellent point we do clearly document that though in the code for now but it’s something we’re actually looking to in terms of resources another really good question actually because it’s very important you have spare CPU resources so just an interesting point is if you’re using something like spark spark actually ends up hammering the Swift object server with hundreds of range gets literally hundreds even for a simple job and the key is the way stirlitz work I didn’t actually mention it in detail but for example if you have a single store late what we’ll do is we’ll create a thread pool and while she run these all concurrently but the important part about that there is you need to CPU cycles to accomplish that so it’s a very good point and an interesting part about that is it’s another trade off to whether do you run store lets on the proxy side on the object side and it all comes down to if you have spare capacity okay thank you very much in the proxy pipeline obviously you insert the the middle where do i because store let’s run one per account do I have to the actual middleware can it handle multiple accounts and multiple store that so do I have to insert it for every account absolutely it can handle actually multiple accounts so you only need to insert that on your one what for example proxy pipeline and there’s actually a lot of configuration you can do that’ll actually determine the details in terms of what permissions are for which accounts which accounts can even run store let’s as well so it can a single account use multiple stirlitz as well yes oh absolutely so single count can upload hundreds of store ‘let’s and the ole that will execute in that single container for that account so and obviously though it’s multi-threaded so we have different threads for each one of these so thank you thank you okay thank you all so much that was really great thank you you