So this is pretty cool I’m not sure if all of you have the chance to play with it What we have here is the Maps Engine layer in Maps API Just loading some different layers I loaded parcels from San Francisco and trees from San Francisco And you can go ahead and scroll around and click on them And the thing I want to mention, just upfront– we’ll talk about some more of the stuff later– is that all of you are welcome, now to go ahead and get an account on Maps Engine You can head over to the Sandbox, the Google Maps Sandbox right after this. talk Give us your email and you’ll within a few minutes be signed up to a test account, where you can play around with this So throughout this session we are referring to some of the things that, hopefully, you guys will be playing with in a few hours SEAN WOHLTMAN: And if you want to know more about it, there’s also a session tomorrow called “Our Maps Your Business.” And so if you want to get a more in-depth look into Google Maps Engine than we’re going to do today, I highly suggest attending that session JOSH LIVNI: Yeah OK, so as we mentioned, there’s a lot of common patterns You look at all different kinds of applications– and we just showed a small subset of the kind you might be building on the previous animation– and we see very similar patterns So we’re going to go ahead and start out with some crowdsourcing example to show you what we mean SEAN WOHLTMAN: So also during Hurricane Sandy, we had another very popular layer that made its way onto the Crisis Map Because of the extent of damage to trees and power lines that were down, a lot of the residents had no power for days and weeks on end So a lot of folks actually did prepare, did have generators And there was, all of a sudden, a 1970s gas shortage inside of a New Jersey and New York So, some enterprising young students decided, hey, why don’t we get people, feet on the ground, to try to make a map of this So, let’s define a problem here All right, we need to map which gas stations have gas That’s a fairly simple problem statement How do you go about approaching that? So you might think, well, let’s go ahead and create a forum We’ll get a forum out there What’s the name of the gas station? What’s the address? Is there gas available? And you’re probably going to host that forum somewhere on the internet And then the idea is that you’re going to make this as viral as possible Hopefully get some press behind it And people are going to come in and submit that type of information But you can’t really trust everybody There’s most of the times people try to do good during crisis But there’s always going to somebody that’s going to come in there and troll So, you’re going to have some sort of moderation capability in there as well Somebody that can be able to approve things Somebody you trust, that’s been working on the project for a while and you trust their opinion And then you have to be able to make this spatial So you’re going to want to export that to some sort of format, whether it’s KML, or geoJSON, or whatever the geo-flavor du jour is And then, like we saw with Schuyler, you really want to make sure that if you have a database or if you have some service that’s generating these feeds, that you’re caching out the results, right? You don’t want people hitting your live service And then, and only then, once you’re up and set up and running, that’s when you want to put this data on a map JOSH LIVNI: Right So, we’ll make a brief segue here into something we see happen a lot, when people start collecting information that’s spatially related You come up with this type of forum we mentioned earlier You’ve got all the information you think you need; the address; even the latitude, longitude I’ve seen forms out there, just like this in the wild And I just want to say that this is not a very good form So part of best practices for your architecture, is taking advantage of other people’s architecture So this took me two or three minutes to modify this form, that I created in Google Spreadsheets under the form, and added an interactive map and a little drop down So if I’m looking for a gas station, say Exxon and it’ll pop it right up And this is just cut and paste code, directly from our example, super easy And if I happen to know the gas station’s there, I can kind of drag it over and update the latitude, longitude Those fields should be hidden Users should never know about that Anyway, you go ahead You say there’s gas there You submit the form And what you should see, there it is, is the crowdsourced information coming into your spreadsheet And we’ve got an extra column for confirmed, where a smaller select group of people can modify that Now, once you’re at this stage you have a whole bunch of data We want to get it out onto the map This is pretty easy The form took just a few minutes to create I could go here to tools and use Apps Script Now, if you’re interested in creating really amazing forms with really powerful Apps Script integration, you just missed a talk about that But you can catch it on video I think they’re all going live in the next day or two So the talk for that is called Forms, Use Apps Script to Create Dynamic Forms So, pretty cool talk where they go over the power of Apps Script, and all the kinds of things you could do For example, if something new comes in, you could send that data to a different API or service that you have to run some computations in it And all kinds of dynamic interaction you can do with Apps Script The easiest way to go from here is just a File, Publish to Web And then you get the power of the spreadsheets API The data comes out A RESTful request that you can make Make it public People can hit that feed It scales reasonably well There’s a little extra work though involved in sort of transforming that data feed in XML or JSON to something more map friendly Because this will not necessarily have that in mind So keep in mind those tradeoffs, if you’re just going this really simple way And also keep in mind the tradeoff of, this is not designed for tens of thousands of points Of course, spreadsheets can handle that It’s too much data to send over the wire to the client,
on all these dynamic requests So, a super simple way to get up and running Just think about the way you’re bring the data in and how you want to publish it out Oh yeah, and then a lot of people, when we mention scalability, they think immediately QPS What is the queries per second I don’t know how many of you have noticed the little sensors we have around Google I/0 Hopefully, you guys have seen that We have hundreds of those There’s one right on stage, tracking me as I talk And tracking you, if you walk by it And tracking five or six other different things, including temperature and all kinds of readings There’s a great talk on that, the Data Sensing Labs, tomorrow evening Check out the architecture there My point is here, we have a whole bunch of data This is also crowdsourcing The issue here isn’t tons of people hitting the site, but rather tons of data Every few seconds they are sending a variety of different signals back to a server With upwards of the thousands of these sensors, you’re talking tens, hundreds of millions of data points, over the course of days or weeks So that’s a lot of data and it’s more than most people have It kind of gets in the realm of big data And so, think about that If you’re going to have a big data set, that maybe the standard tools that you’re using for crowdsourcing, or even maps, aren’t going to apply Now, luckily, it turns out almost nobody has big data Everyone wishes they did and big data conferences are fun, but when it comes to mapping and GIS, most people have medium-sized data SEAN WOHLTMAN: So what do we mean by medium-sized data? This is a class of data, like Josh said, that most people actually have This is the data that’s too big to send the browser, but not quite big data We’re talking greater than 5 megs, probably up to a couple million rows at max So what are the strategies and what are the patterns are you want to employ when you’re dealing with this type of data? This is important since it is what most of you have So, if you have a small area in San Francisco, you could probably get away with pushing all the geometries in the Google Maps API and actually drawing them as an overlay here But what if you wanted to take that layer and push it out to all of San Francisco? We’re now talking about a couple hundred megabyte file that you have to be able to try to push into the client And that’s not going to be a happy day for anybody, right? Because you want to keep that density You want to make that map interactive and have all the data there So what’s the easiest strategy? Well, it’s nothing really novel What you really want to employ is take that data and then you want to break it up into a server-side rendering of that data You want to be able to just make a simple overlay of 256 by 256 pixels But it’s not just a rasterization process Because people are going to expect that these parcels are interactive So you have to have a strategy for being able to do some sort of interaction Whether that means you’re using some sort of an UTF grid or using some other technology, that you’re making that click actually respond and get data back So if we take a look at the diagram from the architecture standpoint, it’s not too different from the crowdsource You’ve got one or maybe a couple people collaborating on uploading a big data set You probably still have some moderation piece If you’re a county and you’re doing a tax assessment, and you’re going out and you’re changing polygons, you need to have some sort of moderation piece But then you need to have the server-side rendering piece that’s going to take that vector data, rasterize it and also keep track of the [? action ?] data for you Again, you’re going to want to have a cache The nice thing about having static tiles is there’s kind of cache built into that, in the sense that the tiles are the cache But you also want to make sure that it’s load balanced, of course If this becomes a very popular site, where your map goes viral, you’re going to want to make sure that you could handle the traffic load for it And then you publish it out to the map JOSH LIVNI: Yeah, so I saw this really innovative site and it’s almost sad that it’s innovative It’s a GIS portal for the City of Palo Alto I think they launched about a month or two ago And it kind of stood out to me because I go to a lot of GIS data portals And it’s usually, after going through some click-agreement, just a list of downloads Hopefully, in a format that your standard GIS software can read Often not And so you have no idea what the data is until you download it and put it into some sort of GIS system What I liked about the site, in particular, is you can kind of scroll down, play with the map Click on things actually directly See that you can have some fairly complex data Here’s some land use data, for example It looks pretty good You have an immediate idea of what it is, before you jump in And then potentially choose to download or otherwise play with the data Pretty cool So, let’s go ahead and see how you might go about building that type of application, and just a simple way to do it So, just as we walked you through it in the crowdsource example using Google Spreadsheets We’ll talk about our Maps Engine platform here So again, you guys can get some test accounts by going the Sandbox, if you haven’t signed up already Let’s go ahead and dive in So the first thing you’re going to do is you’re going to have some data, hopefully It could be a KML It could be a shape file, a variety of other formats are supported You can upload those files individually Or what I like to do is click the Upload button and choose a folder And that folder could just be full of all different kinds of GIS data And then you just batch upload it all with one click So you’ve uploaded what we call data sources Now you want to make a map And we do that by creating what we call a layer And that is, basically, some styles associated with your data So we have some pretty nice styling widgets here on the left You can go ahead and set filters, based on zoom level or attributes So if I want to style all the parcels that are on a certain type of street, [INAUDIBLE] will drop down auto-populated based on your data
medium-sized data This is where Google Maps got its start, like Brian mentioned, is actually doing the tiling and creating a static tile cache So the same principle applies You need to have some server in place that’s going to take that image, chop it up into 256 by 256 pixel images, and then serve them out Now, it’s really simple once you have those tiles, to overlay them with the Google Maps API So if I have a Maps API site, I can just make sure that the tiles are exported into the web Mercator, which is a standard for all slippy maps This got a EPSG code of 3785 And then if I want to play with the 3D, I want to make sure that I also want to put out tiles in the plaque array or 4356 In this case, this is really nice because I’ve got the bathymetry from Google Earth And I’m overlaying this nautical chart with all these soundings And the soundings are lining up under water So this is an option If you’re going to serve a 3D application, you want to make sure that you have tiles in that format as well So, just to review you’ve got your image You’ve got some sort of server running a tile rendering Once again, your cache is kind of already built for you because it is static tiles But you do have to think about if you’re going to put it out to two different projections There are a couple of categories, I would say, and different patterns that evolve when you’re taking a look at what the use case of the imagery is And the first one that I want to talk about is what I would call fast response imagery This could be– if you followed Public Laboratory and funded this Kickstarter– a great project where you take a cellphone or camera, a point and shoot camera, and tie it from a weather balloon Or you go and follow DIY Drones and build your own drone and put and Android phone there And a little Arduino to pilot it Or it could be actually formal imagery, such as NOAA NOAA goes those out and collects imagery after hurricanes And they actually fly with really nice commercial cameras in real planes But the key here is if it’s fast response imagery, it needs to be fast, right? Time is of the essence Here’s an example from Hurricane Sandy This is Mantoloking, New Jersey This is the first eyes on that we had of the area And it’s pretty dramatic if you take a look at what Mantoloking looked like before See the bridge coming into these beautiful homes, beautiful little beach town And then after Sandy rolled through, the dramatic change there We have all of a sudden, a new natural-made inlet and houses have been wiped out Josh is going to tell us a little bit about one strategy that we could use to get that imagery out as fast as possible JOSH LIVNI: Yeah, again, so we’ll take this super simple case first You have a little bit of imagery You want to get it out to people as quickly as possible There’s tons are proprietary and open source tools and workflows available for this I’m going to show you one example using the GDAL Library and Google Compute Engine Now, I’m not sure if you noticed any of the talks from the cloud track this morning, but Google Compute Engine, I believe, is available to everyone tonight And also they’ve announced a bunch of other great stuff, like [? sub-hour ?] billing, and you can boot up lost instances, and turn it down as needed, and so forth So the way to get access to this is you go to the Developer console, which is the same place you might, for example, get an API key for Google Maps You go to your Services tab and just enable Google Compute Engine So once you’ve done that, you’ll see a Google Compute Engine logo You can go ahead and click on that Go to the Instances tab And then there’s a variety of options I like to just choose a standard Ubuntu instance Don’t worry too much about the other parameters, booted up We’re going to show you how easy this is So you boot up the server You SSH into this this virtual server that runs in our data centers And then you just run two command lines to get up going with GIS in our server Install Apache2, so that I can have web requests coming in We’re going to serve out some tiles And then go ahead and install UbuntuGIS, which installs a whole bunch of software, including this GDAL Library After we’ve run those we’re going to go ahead and download some imagery data And then we’re going to use that GDAL tiles library Which is a little Python script, basically, that will mosaic that data together And create all of the appropriate tiles in all the appropriate projections for you So, it took about two minutes, I think, for the data we downloaded And rather than waiting two minutes, we made a video I think it just finished So that was it Download the data, created some tiles, done Now we have the tiles in a folder, and let’s take a look at the results SEAN WOHLTMAN: Great, so the results of that We have a nice little embedded Google Maps V3 API site here And I can zoom-in to that area that looked really damaged And I have a nice little custom overlay here, so I can do a little transparency slider And you can just really start to see the total destruction Now getting this out to people, especially people whose homes those were, first responders and rescue folks coming into the area, is critical So this is a perfectly fine way to address a small area, get that data out there as quick as possible with minimum friction As long as you understand a little bit about the GIS tools behind the scenes But, what if it’s not a small area? What if it’s a really big area America, the United States is a pretty big area It’s about eight million square kilometers and what if you wanted to do imagery on that scale?
And what if it was fairly high resolution? I’m not talking high resolution like we showed earlier, 15 centimeters But the USDA NAIP program, every three years basically, recovers the United States in one meter imagery So you can take a look at the Moscone Center here Obviously, it’s not going to be totally exciting in urban areas But it’s not really designed for urban areas It’s really designed to go out and survey agriculture They fly it in four bands, so they can also do some photogrammetery telling the different crops that are being grown So, this imagery, if you had it for the entire United States, which we do, for the past three years would be about 80 terabytes of imagery So my first problem is now I need to upload 80 terabytes of imagery somewhere And then, it’s not just that imagery Now, I actually we need to make tiles with it So take that same three years of data, I’m going to tile it out It comes to about 1.3 billion tiles that I need to write to disk And that in itself is about another 30 terabytes So, now I’ve got to worry about how many disks I need, how much CPU and RAM I need to spin up, how long this is going to take And there are a lot of really smart people working on how to do this with big map reduce processes My recommendation being not try to reinvent the wheel So we took all this and put it into Google Maps Engine And we blended and served redundantly over thousands of cores in a few hours And when I say we served it redundantly, we’re writing this to multiple data centers at once, for you with Google Maps Engine So we’re actually taking all that data We only tile it out to not just one data center, but across multiple data centers And it’s ready in a few hours So take a look at a quick preview of this Here we have a 2D map And this is the default map that you would get with Google Maps Engine This is built for you This is the Google Maps view And you can come in and see it’s just as performing as the underlying tiles that are there The one meter can go back and forth, on and off And the nice thing about Google Maps Engine, if you remember I brought up before that I wanted to serve both 2D and 3D Google Maps Engine doesn’t charging the overhead for also doing that in 3d as well So you get a lot of bang for your buck, running on the same servers that are running the rest of Google properties And there’s another thing that we should talk about as well There’s a security piece here I could share that map I can make it private Maybe it’s licensed data Maybe I can’t put it out to the entire public So I have the security things that we talked about with Josh But we’ve only talked about, so far, about putting this data out onto Google properties, Google Maps and Google Earth And if you work with anybody the GIS community, they loved it It’s nice, but they also want to be able to use it in their analytic applications So if you were going to do this yourself you now have to think about how am I going to implement things like OGC standards How am I going to put out a WMS feed? Am I going to have to have the overhead of returning WMS requests? Google Maps Engine does take care of that for you And so you can put it out into OGC compliant feeds that can work with other applications, outside of the Google framework JOSH LIVNI: Thanks, John Yes, so hopefully you can see that even from a little bit of data– maybe it’s vector data you’re crowdsourcing, maybe it’s a little bit of imagery, many of the same things apply– if you just have to serve it up a few people, it’s static, you’re not worried about a huge influx of data or dynamic requests taking you down, then there’s a ton of tools and options out there As people start adding complexity– and this is inevitable for most popular applications– and they need to think about things like the moderation piece, the simple sharing piece, the permissions, and so forth, the different projections, the OGC standards, make sure that you’re using a framework and thinking about that in advance So when you do get popular one day and someone starts slamming you, you have a cache ready When someone says, we need to take care some moderation issues, you have a method in place to do that So, hopefully, we showed you through some different stories that many of the same pattern arise We tried really hard to put out some software that makes this as easy as possible for you guys Hopefully, you go out and try some of that software, including Google Maps Engine Let us know you think And also let us know if you have any questions I think we have about 10 minutes Yeah, thank you very much If you do have questions, please come up to the mics And speak into the mics and we’ll do our best AUDIENCE: Hello, my name’s Pat I work for a high tech agriculture firm based in Des Moines, Iowa And I know that was interesting SEAN WOHLTMAN: Not a plant AUDIENCE: We have a problem And that is that a lot of people in the agricultural industry live and work in areas where they don’t have good access to the internet And so, I’m curious if you have any suggestions for patterns and practices that would help people when they’re wanting to look at map data, and they do not have an internet connection? SEAN WOHLTMAN: Yes, in fact we have a session tomorrow [? Sean Maday ?] and [? Harsh Bhatnagar ?] are doing, basically, what happens when you have to go off the grid I’d highly recommend taking a look at that There’s commercial applications, but there’s also grants that we do that, basically, allow cutting from the Google Maps Engine platforms And you can actually take portables of the maps that you create And take them with you in a portable cache that you could run on a PC, Mac, or Linux server
So what about performance? SEAN WOHLTMAN: Yes, so right now doing Dijkstra or A-Star or anything like that over a vector network is not something that Maps Engine does But when we talked about Compute Engine, you should think about that as a possibility To be able to run the traditional PostGIS or pgRouting types of applications in the database in appropriate form, until we can offer that as a direct offering in Google Maps Engine AUDIENCE: But if people create some vertices of raw graph by themselves then we have a [INAUDIBLE] or an other site Can we use your storage as a local cache, as front end server for our backends, in order to send it backend? JOSH LIVNI: Yeah, so the general pattern there is, if you would like us to render it out, we have the API So I showed a really simple example of editing some geometries, but, of course, you can batch post a whole bunch of features into an existing table, modify, or delete them So, if you need to do some type of synchronization or adding new things to data that’s in Google Maps Engine, you can do that using the API AUDIENCE: Sorry, do we need an Engine server for balancing between my backend and your storage? JOSH LIVNI: I’m not sure I understood SEAN WOHLTMAN: You’re asking how to connect between your backend database, to what it would be in ours AUDIENCE: I don’t know about average number of requests to your storage? What about it? JOSH LIVNI: You mean if you’re doing like a whole bunch of requests, very frequently, what are our limits? So, we have a few different limits I think right now, if you purchase Google Maps Engine– I’m not sure exactly offhand what the queries per second is that you can post and get It’s pretty high and I think you can, as many of our offerings, purchase more quota as you need I’m not sure what it is offhand, by default But it should be plenty I mean it’s many a second And you can batch post a large number of features that once So I think it should suffice SEAN WOHLTMAN: If there is a use case you really want to dig down into, please come by the Sandbox JOSH LIVNI: Yeah, we can talk a little more detail, something specific AUDIENCE: Quick question, in your imagery example, how was that 30 terabytes of data delivered? SEAN WOHLTMAN: How is it delivered? As tiles AUDIENCE: I’m sorry, delivered to the Google App Engine SEAN WOHLTMAN: Oh, how do we upload the initial? AUDIENCE: How do the– Yeah SEAN WOHLTMAN: OK, so there’s a technology that Google has called Xenon Which, like Josh said, there’s one way in which you could sit there and try to bulk upload of all that For the 80 terabyte source, you probably don’t want to do that Xenon is a technology that Google uses to monitor remote web servers, where we have a lot of partners, such as VEVO that puts out a lot of music videos And we monitor either a http or a ftp site, where we will actually pull that data across data center to data center And we can ingest that for you into a particular account So, it needs to be on a server somewhere Or, if it’s a really big delivery like that, we can actually have customer agreements through some of our partners that will help get that data online for you Because, yes, getting a large amount of data into the cloud is a barrier to entry JOSH LIVNI: OK, well thank you guys very much for coming Sean and I will be in the Google Maps Sandbox for the next hour or so Look forward to chatting with you guys SEAN WOHLTMAN: Thank you