Enabling Research Using Cloud

Thanks a lot David As David said I lead the research part in the public sector Canada is really very dear to me. What one of the things especially in the research area is simply because you know I I grew up here in Montreal I did my doctorate from McGill I spent several years in Canada later on I went till to Germany. I studied there at Daisy and then followed by at CERN. It’s an international collaboration there. So today I’ll talk about some of the areas which impact scientific computing or research that’s related and how technology is enabling it. So let’s get started. I’m really thankful Paul is joining me here. The couple of areas as you can imagine which scientists tend to use for research and the first thing is the access to access to the infrastructure in minutes second is is really it enables global collaborations I’ll give you a couple of examples today related to biomedical field as well as as well as the the part related to CERN. It’s elastic that means you can automatically expand and shrink it. It is globally accessible secure and scalable. As you know AWS is is the global infrastructure so it is more available in 20 regions if you see that that particular left side you will realize that it’s in US in Europe Canada as well as various other countries and you need that for research right there interconnected networks such that you can share your data can share you algorithm and then can work together with various folks. We have 20 recent 61 availability zones so those small circles you see they are the vulnerabilities zones so every reason has one or more availability zones and there are numerous edge locations. In terms of compute for a long time the tradition of compute is what we what people call homogeneity in architectures right if you look at supercomputers they’re a huge homogeneous set of computers and you try to schedule a job around it but the time we realized that process need to be changed and based on the customer feedback the way we look at the compute is really in three major areas. One is what we call virtual servers so those are like that systems where you can schedule jobs but also you need resource isolations right For example if Paul is running a job in the same CPU you still have certain resources available right like memories are or network. Can we schedule them together if you do that then you have to define some rules on security and that’s called resource isolatin and that’s done via what we call container mechanism and we have a numerous set of services to do that container scheduling containers. And the third part is what we call serverless computing. Imagine your boss tells you to get something done today right that triggers a given mechanism to do some studies right so that’s a mechanism one can do that using step functions or lambda functions to do service computing so that once the job is done it shrinks back to the normal scenarios Even in CPU I talked about the super computers it started with homogeneity in processing but even in CPU you will realize that we based on customer feedback people need different sets. For example there are general purposes that are computer optimized there’s so much storage optimized and with the advancement of analytics like machine learning and AI we started seeing more and more usage of what we call accelerated computing I’ll come back to that. So in AWS we provide almost all sets even we provide something called bare metal instances where we can give it to you bare metal based on a given processing system. We all know what CPU is right but I just briefly want to mention CPUs are typically 10200 processing cores right. They use what we call predefined instruction as the data set path but in Assen you that is evolution accelerated computing people started using GPUs and they have roughly about thousand processing cores and they also use predefined instruction set and there is a path weights and then if you want to even go bigger than people started

using FPGAs. They have roughly about a million of programmable digital logics So the bottom line is based on what you want to do you can use a given set instead of running a large set on a CPU you can run either and GPU or FPGAs for accelerated computing That evolution is already started coming in this world. Even in for example here a couple of case studies for example if in amazon P3 instances we have something after order of 8 NVIDIA Voltas they typically people use it for machine learning or HPC or financial services and so on so forth we also have something called GPU graphics instance that is people use it For 3d rendering suppose you want to do a rendering and collaborating of various folks across the across the different places you can use that. we also have as I mentioned FPGAs f1 instances so you’ll ask what shall I do with FBG’s how can I use for scientific computing I’ll give you a couple of studies very soon on that. So here’s one class I came from as I said certain CERN is the international collaboration where many countries participate this is the Large Hadron Collider at CERN and this particular picture is one of the detector is called silicon detector and it’s me and my colleagues there. CERN has roughly about more than six thousand researchers around 40 countries and it produces approximately 25 petabyte of data that two major experiments they Atlas and CMS collaborations and they use all kind of processing ways to study various various parts of physics and several countries are involved as I said one is Canada Germany Spain and and you a UK us and so on so forth the way it is defined is after the data is after the collision happens at CERN it happens at 95 nanoseconds and once once you have the collisions we store the data at at the tier zero and then get distributed to various Tier one center one of them is Canada and as well as the US so one other thing which if you look at the detector this is one of the reactor I worked there for several years if I can use the laser pointer this is the size of a human it has 80 million electronic channels if you multiply by four bytes at a 40 megahertz teh data taking rate you’re talking about 10 petabytes of information per second so how do you analyze 10 petabytes of information per second so what we did there is to divide into two parts what we call online processing people use FPGAs for that so that you can analyze it that data stream at that rate there you select a given car particle kind and then you analyze the rest of the events which in a data at a batch level which becomes a little bit lower into a lower in size so they have roughly about 2,000 scientists from 180 countries with 180 institutions from 40 countries and then comes also at 25 nanoseconds you can imagine this is a collision happening here which one is the laws of nature it’s very difficult because there are overlapping events also there right so you need advanced techniques like machine learning to really detect the property of nature so one of the prawn face therefore for for simulations and modeling is imagine you have a data taking happening here right and imagine you have a set of resources called X number of CPUs doesn’t matter right as soon as you want to process this particular pink you know you cannot go beyond this line so you process it by delays in timing so elasticity started becoming important so what we try to do is work with formulas it’s one of the Tier one center like like a Canadian take Tier one center we change the schedule in the behind such that when the simulation jobs comes in it automatically expands to Amazon and shrinks back when you don’t need it so look at it with that simple make so the environment remains the same the processing remains the same with that pattern we created roughly about sixty thousand slots using AWS port instances Spartan instances are unused capacity in a given areas one can use it using a Attic at a price you request so this way you created factor of five formula amassing if you are two by five chapter five Fermilab how long will it take and then you shrink it back so what is not what is important is is that yes it

it it went up to sixty thousand CPUs but even more interesting is ups and downs right that means when there is no job in the scheduler the instance terminates you don’t pay for that and when there is more job in the scheduler it automatically expands and that’s and that’s what we need one needs insane in computing that it automatically elastically expands or shrinks based on requests and this is all done in a in a in a using unused capacity we run for a few weeks there details are there in this particular particular article you so it’s not only the formula can do that anyone can do that we made those software public for free it’s HD Condor mechanism this is the way to do it this is the link here there are some tutorials you can do it yourself as well the same the same thing we try to expand using what we call machine learning so we were trying to study what we call topic modeling modeling with Clemson researchers there it’s the same kind of situations where they they had a given set of resources in their campus but for machine learning they wanted to expand using using Amazon so the read so the entire environment remains the same the way it was in the Clemson University this automatically expanded to roughly about 1.1 million course it hasn’t somebody planned to buy 1.1 million course so this was extremely cost effective and it’s all used using spark instances we also had our partner helped us in this process they are the cloudy class sure so one can you still use those those processes even today by any of you so use cloudy class sure which has a bad system building which is slurm which ought to expands to a double s keeping exactly your environment the way it is and it will shrink it back when it’s not needed so it’s not only about you know how big can we go but it’s also what what what else can we do given the advancement in the processing so here is another study I like to show you as you know to sequence genomes it takes months two years with large set of computing right see magazine you’re running thousands of CPUs running for the whole year so instead of doing that one of the way these folks did at Children’s Hospital of Philadelphia with encode genome is they started using those FPGAs which I as I mentioned has roughly about million processing units right they analyzed about thousand human genomes in two and half hours using FPGA Zoar Amazon F one instance so they not only pay a few dollars to Amazon but also got the Guinness Book of World Record for fasted analysis in the in the world so within two and half hours you can do genomic sequencing using the kind of processor you’re using so they use Amazon f1 instances and and they completed in two hours 25 minutes or something like that so the bottom line is based on the kind of study you don’t have to use all the time the cpus and gpus a given resource would be super useful for the given kind of workload we also worked with Hubble telescope so we provided almost 28 years of the Hubble telescope data in cloud for anyone to analyze it we also work with neutrinos this is experiment at at at Nova it’s its international many many folks from all over the world use it so what I’m talking about neutrinos do you know neutrinos ok neutrinos are the particles they pass all over our body right so what’s the special about neutrinos one of the thing we learned about neutrinos is they change identity so imagine you shoot a John at a given place it passes through the Earth’s Rock and you detect a second one and you are detecting a Lisa so changing identity becoming a part of nature is really crucial this is one of the study they did that where a given particle change the identity some some of the time so here they used a AWS to do the processing but also various the analytics part so there you need machine learning are and other analytics to detect it so coming back to machine learning we look at machine learning in

various ways so we start with what we call a SS services where there is a preload dynamics idea for example related to in what we call recognition or poly or comprehend based and voice Alexa uses lex for example we also provide services associated with how you can use it in a given framework for example Amazon sage maker how you can label it automatically using ground truth but also we give freedom of infrastructure such that people can use their own algorithm you can use tensorflow if you like tensorflow if you like MX net so based on what users like they use the aid given kind of algorithm so this is this is a fantastic framework most of our researchers use within the machine learning as you know that are categorized into like supervised and supervised and reinforcement learning we provide all those algorithms building to the infrastructure so in this particular example like as you can see you all you need to do is start with a given notebook for your research and then start using the algorithms like came in class sharing a principal component analysis or on that but also you can use deep learning if exactly the same way it’s it’s very simple step three ways you just start building your framework using a simple Python based notebook then you train it using a couple of a couple of clicks you define your loss function and get your happy parameter so here is one of the studies people did in in Stanford so this is in the field of convolutional neural and in Intel volution with all that you have a set of input layer and hidden layer as well as the output so in this particular study as you know many people get diabetics right so what it happens it affects you in their vessels you go to the doctor and the doctor says you know it affected your eyes but sometimes you can train those images on a standardized you call it background only hypothesis any deviation in those nerve vessels you can detect it and thus people started doing it using a double yes so you can do it today in your studies you can also for example this is one of the study we did with in collaboration with National Science Foundation as well as other other folks where you can train based on the social information you have and start predicting the path of a given given hurricane in this particular example this but left one is the predicted path and this is the real part amazing you can predict hurricane 20 or 30 minutes before it is giving it given area that’ll help hugely the disaster response team right so we plan to use some of this make it public give it to Red Cross or something like that to try pilot out so this is one of the studies we did in research areas you can also use automatic detection of damages for example if you look at the top layer it’s very hard to know which damage was the most right but if you have a given mission mechanism you can detect it for example this has a div of 0.42 3 but this one corresponds to the highest damage so you can train the model put them in a camera or a drone and can Auto detect given damages the details of the study is there in this particular archive we also collaborated in in something called thousand Jima genome project that started roughly in 2008 where many collaborators including Canada was there we started this collaboration lashes to health but in collaboration of so many folks and we hosted this data even today that data is available for anyone to analyze thousand people donated it in order to study genomics for different different kind different race and and kind of characters Australia for example successfully studied mapping the coil coil our genomes it’s really interesting study it was in CNN and various other news media this particular species is very picky on what they eat they sometimes chew on the collab tips leaves and that can be poisonous to many of many of the many of the other species so this particular study maps out various areas of how to help this particular species this fantastic study it came out in in reefs in nature please have a look at details here that was that used AWS CFM class to add to analyze the the genome using using various various processing units I mentioned before and a chest is

also building what we call data leaks or data services in in UK and they are working with us to do the card and store in our platform with that I’ll give back to the poll then we’ll come back and ask for questions thank you thank you very much Sanjay good morning everybody I’d like this morning I’d like to tell you a story which we call democratization of high performance computing democratization meaning available to everybody in this context I’ll come back to that point later so the this work came out of a collaboration well my clicker isn’t working big green button no there we go between the Communications Research Center and the National microbiology laboratory the CRC is the government center of excellence on wireless research as you know the world is increasingly powered by a wireless devices whether it’s cell phone’s GPS emergency service dispatch air traffic control radar all these things require radio frequencies to operate and we’re now looking at new services like 5g Internet of Things coming along they all require frequencies to operate what we cause while a spectrum but spectrum is a finite resource how do we make sure that all these things work reliably in Canada how do we make sure they work better in Canada than anywhere else in the world and that is the job of the regulator we provide the scientific advice to the regulator which is our parent organization the nml is part of P HACC public health agency they have a the highest containment level microbial containment facility level four in Canada they keep all the nasties like Ebola there nml has dual role they are involved in scientific research looking at techniques for treating and dealing with disease they invented the vaccine for Ebola in fact right here in Canada actually done in Winnipeg they also have an outbreak detection and response capability where they’re looking to find out what the these are outbreaks which we hear about and infectious disease outbreaks like when you hear about an e.coli breakout they use genomic sequencing to figure out what the specific strains of E coli for example are and correlate that to try and figure out where’s the size the outbreak where’s the origin the outbreak and then coordinate at response to that and any type of outbreak response they will be the guys involved doing it and they have developed a number of world-class techniques and software applications to help that which are used internationally now it may seem at first blush that we’re very odd bedfellows to be collaborating why would the the wireless guys want to collaborate with the genomic guys but it turns out that in my discussions with a number of science departments and agencies we all have a common set of problems modern science requires ever increasing amounts of compute we all need to ingest data store data process data visualized data apply machine learning AI we want to collaborate with each other and those who are constrained on on premises Hardware just cannot do that and there’s no easy way particularly in a government context to scale that dramatically cloud is a good solution the CRC has been in cloud for more nearly four years we have built in that time a secure scientific computing environment which we’re going to boast and say is the most advanced in Canada in the government to Canada for certainly the nml has a on-prem data center they have expertise in high performance compute but they are finding during outbreak detection that they are running out of CPU so the question was could they use cloud to burst out of their data center in times of significant activity to improve the time to outbreak detection response and the other question was could you get cost effectively and then from what we learn from that could we use those techniques to I say real science real science being the discovery science which is part of animals mandate and CRC’s Monday as well as the operational science that they’re using so those are three questions so to answer that we created a six week challenge and the idea of six weeks is cloud enables you to move fast learn fast fail fast what can we do in six weeks there’s a common technique we use that the the CRC to try and move the yardstick on what’s possible and what works so can we take the outbreak part

of their nmls HPC infrastructure my grade into our virtual research domain our scientific cloud infrastructure on Amazon Web Services build this proof of concept and then do some benchmarking with real-world use cases from from sample data from situations of actual Public Health significance so we took a three phase approach first one very very simple lift and shift don’t think too hard just get it going take a week or two then we’re going to optimize or really in the time we had available let’s just call it clarifying and then the last we would do a measurement do a benchmarking how does it compare with on-prem so we put together very small very focused interdisciplinary team from the CRC and the nml of it as you can see very different backgrounds supported loosely by a conglomeration of other other folks and we used a very agile approach who weekly sprints to see how we were progressing so phase one lift and shift what we were trying to lift and shift was this and essentially in the bottom you see the slurm controller which is the nhpc Orchestrator and galaxy anaridis and their software that has been developed by the nml and is widely used in genomics sequencing worldwide they have a data store and on-prem and they have a database and then in their data center they have all these virtual machines which they can use for to burst out into their foot for various analysis as well as response analysis and they’re limited to at seven thousand CPU cores and about 40 terabytes of RAM so the lift and shift approach was to simply take that architecture move it into AWS and just use like for like so from the data store we put it into what’s called EFS storage from the galaxy database we put it into RDS just and everything else was just VMs and the cluster at the top there we put into an AWS auto scaling group his auto scaling will give us infinite resources so we did that good news was it didn’t take a week or two it took three days and we had it working so we were that was very very satisfying however it didn’t work very well so the simple dumb approach didn’t work I’ll show you some numbers to justify what I mean so then we started to immediately get into the clarifying the optimization and we had two major problems that we discovered when we did our lifters shift that we need to address one is scaling we couldn’t scale big enough so and then just to explain what that means this is the error we were getting in sufficient instance capacity within the availability zone and I’ll briefly explain sanjay mentioned regions and availability zone but within an AWS region for example canada central montreal or US east which is north virginia you will have a number of availability zones to us an availability zone as a data center in reality it’s one or more data centers and in each region you have in fact multiple data centers the way we had architected it was simple lift and shift you couldn’t burst across multiple data centers so we fixed that so we fixed the the size of scaling problem the other scaling problem we had is we couldn’t scale fast enough so in a high performance compute world what you’d really like to do is what the blue graph here is showing at the beginning of your job you scale to the full number of resources you know you’re going to need for that job you run them flat out during your your job and then you turn them off at the end and reduce your cost to zero but if you just do the basic default use the AWS auto scaling algorithm it’s not designed for hyper let’s compute so it was really the wrong tool to be using and we got this really very inefficient our jobs are taking far longer than they needed to so we very so for my perspective very inefficient so what we did we built our own out of that slim controller that slim orchestrated that I mentioned took us about a week and it’s obviously not perfect given more time I’m sure we could improve but with that we got far better scaling far better efficiency and then that things started to run much more quickly so that was one problem we were solving that or the scaling problem the second problem we were having is pumping enough data quickly enough through our compute cluster so we analyzed four different types of data storage I won’t go into too much detail we started on the Left which is clearly the wrong choice lift and shift again don’t think too hard just get it going and we tried a few others there was one of great interest to us at the time it was a new feature from AWS the fsx lustre design part for high performance compute and it seemed on the paper that would actually meet a lot of our needs

and we stride started to use it very very heavily and in fact we got a call from the the AWS guys what are you doing with our lustre you’ve been chily breaking it and the AWS development team worked together with us to improve it and it was it was a very clever a very good collaboration turned out for our application it still wasn’t ideal and we ended up using the s3 API I won’t go into details but it turns out to be a cost-effective and high-performance storage solution for a multi availability so an application like we were using so ok so we solve scaling we’ve solved the big problems with data so what does that look like essentially it looks very very similar to what we started with at the bottom you see our VPN firewall again this is a secure environment and in fact we built this is worth saying we built this in a secure Enclave that for collaborators the way we’ve architected our virtual research domain we can create an enclave for each collaborator they can work securely and privately in that they don’t even see data or work of any other collaborator or the CRC unless we choose to share so they had their own environment that could securely access it by a VPN from Winnipeg even though we were not running a Winnipeg obviously we had the same controllers the slurm controller now to control over the scaling as opposed to relying upon the AWS auto scaling we changed the storage type from EFS for data storage to s3 and then we had the the the ability to scale our cluster across multiple availability zones so that was the final architecture and now we’re about four to five weeks in so what does that mean how well did it actually work let’s do some some benchmarking so to do our benchmarking we use two real use cases so from 2017 there was this ecoli outbreak and what that meant there was in fact it was across six provinces they were trying to analyze the genomes to figure out where what kind of e.coli what would be our response to it where was the source and deal with it appropriately and this is kind of thing they have to do routinely at any given point this is all going on in the background we don’t get to see or worry about it so that the guys are taking care of it for us the other thing they were doing is answering looking for antimicrobial resistant gene detection so I think as we all know the the the antibiotic resistance is a major problem these days and what they were asked to do was to look through their entire database of genomic sequences that they had and say have we seen this in Canada that was a much larger use case not something that they do very frequently but obviously a very important activity when it comes up and it’s worth saying at this point that all the data we were using is publicly available and if that was not unlike the example that Sanjay mentioned about a human genome this is not human genome this is e coli genomes things like that so there was no privacy or ethical concerns about using this data so we pulled these these datasets into our cloud and what we did we created two benchmarks so the 10,000 genomic sequences came to about ten terabytes that more on the e.coli and then the hundred-thousand sample simulation was the the antimicrobial resistance but a hundred terabytes of data there and we ran each of these benchmarks on the on-premises system we did the cloud lift and shift only the ten thousand because we knew it was it was poor it wouldn’t be worth doing a hundred thousand and then our cloud optimized and it looked a bit like this so with the optimized version of the cloud we were approximately four times faster than on Prem and for the ten thousand samples and for the hundred thousand samples it was about seven times faster it’s worth saying that in this particular case this is outbreak detection so simply scaling up means now we have the capability to respond more quickly to two outbreak detection so we solved the problem with with scaling you can see clouds left and shift I’m not even gonna mention it it was just bad now we can’t talk about scaling into a cloud without talking about cost to understand this slide there’s two elements to it this is the base cost and the burst cost the base cost is all the elements which have to be running all the time to be ready to run a workload so obviously storage is a key one Wheatland the benchmark here is 100 terabytes of storage online we had the base CPU for the slurm control of the year a Ritter in Galaxy software so initially lifting shift thousand dollars a day with optimized we got it down to a hundred and thirty dollars a day the burst cost is when you actually want to run one of these workloads so for a ten thousand sample genomics equals farm plus 62 bucks or 100 thousand two hundred and twenty bucks so these are

great numbers but what does that mean well let’s say that you ran this the so called optimized data center running ten thousand samples every day 365 days a year a data center doing that workload will be something of the order of seventy thousand dollars now if anybody has experience with how much it cost to run on-premises data centers you realize that’s that’s just nothing it’s a drop in the ocean so certainly from the perspective of would this be a useful technique and a cost effective technique to to burst out from your data center we think yes so conclusions we have we think successfully demonstrated that the elements of the end of the National microbial biology laboratories HPC system the elements related to outbreak detection can be successfully migrated into the cloud and we think optimized for cloud uses and we also think we’ve demonstrated cost-effectiveness now the last point cloud HPC can be used for real science and what I mean by that is we were looking to find techniques that aren’t just useful for operational science can it be used for other types of science too and I know this is true because before we even finished our work with nml researchers that the CRC became aware of what we were doing and they came to us and said hey guys we see you’re doing this like performance to keep you I have a problem I’m trying to analysis across Canada and it’s taking so long I’m out having to restrict myself to one province can I use what you’re doing and we said sure of course you can so we gave it to them and helped them get going and now they’re running across Canada full analyses for this wireless spectrum for in just hours whereas they couldn’t do it for it for days before so it does immediately scale and apply to other science so what do I take aways from this so I’m going to say that HP is C is available to all in the cloud I’m going to take that back to the original title democratization of HPC when I say it’s available to all if you have a cloud account the code to do this is now in GC code you can download it you can use it you can you can run it we didn’t use any fancy additional services it’s just available what you can do and when you think about it making it available to all changes the paradigm of how we do science in Canada we have had in the past limited resources and yet we’re dealing with global problems like all the other science fits around the world up to now what we’ve had to do is to scale our problem down to the size of the resources we had avail to address those problems now with cloud we can scale our resources up to the size of the problem so Canadian science is now has the potential to be even more relevant than ever has been in the past we can compete with any other nation because now we have the compute resources we’ve shown that cloud is scalable on demand and and we think cost-effective we’ve also shown that collaborations can quite can be very valuable even between apparently very different science organizations because we have actually a common set of interests and we found that early winds are possible if you embrace cloud and the pace that cloud actually enables you to talk to move that you can if you have the boldness just to go ahead and do it you can learn a lot and early winds are indeed possible so thank you very much thanks a lot Paul it’s all about democratizing research so with that and research is all about everything right Paul talked about genomics HPC I talked about machine learning AI I really like to open it to the audience and I’d love to get your questions anyone from the audience actually yes please oh there’s a microphone coming to it so you can be heard so it’s actually a very small question the size of your team for their six weeks I guess before concept how big was your team so the people focusing for the six weeks was a team of three they were supported by large a team of a couple more at the CRC and a few more at the nml and bear in mind that they were working with an existing HPC system that existed a 10 ml an existing virtual research domain at the CRC so one starting from scratch and do they have help from AWS experts or suppliers expert so how do they figure it out how to use the cloud in such a very small amount of time well we’ve been in the cloud for 4 years so we knew how to use the and that was what we brought to the collaboration we knew cloud animal new

hpc together we could learn from each other and I will say though actually the AWS high-performance computer guys did consult with us to give us some guidance Just to add to that we do have what we call immersion days where we we provide bring our solution architect and others we provide some trainings to how to get on board or how to use it and so and so forth so if you can crested please feel free to reach to your your local account manager or local contact and we’ll arrange it one day of training as the immersion days from at least the research side and it’s for free yes you just have to ask exactly hi I think you said with the nml they had on premise of 7,000 CPUs so when you went to the cloud how many did you actually scale up to to get that performance increase was it a lot more? Actually what happened very good question how do we scale so 7000 CPU is their physical limit the because we were benchmarking and it wasn’t a real outbreak we were having to share see the the data center with the other researchers so in practice they were bursting out to 2,000 course for the benchmarks, we were able to burst up to eight and a half thousand in our simulations next question. Hi could you elaborate on the background work that was done prior to getting an m/l on board for example you know I couldn’t go on with my credit card go and decide and okay I’ll go on Amazon and I’m gonna do burst at 7,000 V CPUs so what kind of framework does the CRC have in order to accommodate that kind of workload. So that is a very long answer to that question we did we and I’ll be very happy to talk with you afterwards and some of my guys are here today will give you much more technical answers. We have built a full environment all the scientific computing from the CRC has been in the cloud now for three years so we’ve got a lot of experience on how to do scientific computing in the cloud and we have a full secure infrastructure to do that. Maybe just a brief comment for Government of Canada people in the room whom you might be with the governor of Canada CRC has their own situation at this point but any department in the government is able to work with the major cloud providers through contracts issued by shared services Canada and start small and work up to where Paul is at so that’s available to you if you wish to take advantage of it you do not need to use your credit card sir. Perhaps one more question and we need to wrap up Okay let’s wrap up. Thank you so much Paul and Sanjay it was fantastic