CVFX Lecture 23: LiDAR and time-of-flight sensing

okay so the topic for today is well so the the topic of this chapter is methods for three-dimensional data acquisition okay and so this is becoming definitely a big issue for visual effects not only are you dealing with you know 2d images that come off of your cameras but it’s also very important to acquire three-dimensional information about the sets and the actors and the props as you’re filming movies that those things help to create the visual effects app in fact and so what I want to talk about this morning are basically different ways of acquiring 3d data with basically laser ranging okay so there are several ways you can get free data and we’re going to talk about those in the first three lectures of this chapter the first one like I tell you what today is called light on lidar and also we’ll talk about time-of-flight sensing the second one is called structured light scanning and the third one is called multi-view stereo and so those are gonna be the topics of the next three lectures and then the fourth lecture is going to be basically about you know computer vision algorithms for manipulating and processing this data once you have okay so for the most part we’re going to be mostly talking about just acquisition in the first couple chapters or the first couple lectures and so here’s an example what I’m talking about this is what’s called a lidar sensor lidar stands for well actually there’s some debate about whether it actually is accurate or not but you know it is it’s an analogy to radar lidar stands for light detection and ranging and so the idea is that this device which we have on campus you can see this is in Troy New York since I have tripod this slowly spins around and what’s happening is that laser pulses are going to come out of this window and hit objects in the scene and usually the way that these sensors work is that they measure the time it took for the laser pulse to bounce off an object in the scene and come back and that time is used to calculate the distance to Optimus I only had networks in just a second and so this is a pulse pace lidar scanner I can already tell it my computer’s a little bit laggy so let’s see whether it’s gonna keep up with me writing here how’s that been okay so today’s topic is basically lidar and time-of-flight scanning okay and so when we think about lidar is it’s kind of like the laser measuring tape that you get at Home Depot right so you have this little device you pointed at the wall you know a laser spot goes out and you aim where you’re looking to measure the distance to and it comes back and tells you immediately how far away that surface is right and so the advantage of such a system instead of using like cameras to infer the distance between objects and that’s time we’re going to talk about next week is that you are using a kind of a physical real-world measurement to directly acquire the distance of something like you’re directly probing the distance and so in some sense this is kind of like the gold standard for measuring how far away stuff is you can do it with cameras alone using things like structure for motion and multi-view stereo we’ll talk about that next week but you know when you’re talking about really doing accurate 3d scanning you’re going to use some type of laser scanner or structured light scanner so we’ll talk about those technologies right now and so what you get at the end of the day is kind of like what you’d call a death or a distance image and so basically you have a sensor I’m going to draw this kind of poorly I’m sure so this is a sensor you know it probes the scene you’ve got some object in the scene and then that pulse comes back to the sensor and what you get basically is depending on the angle of this pulse you kind of get this distance map as a function of you know theta and fie where these are the two angles that the sensor can be not it’s laser rays and so this can be interpreted as what you’d call a depth image or a range image and then if you do the conversion from spherical coordinates to Cartesian coordinates basically here you have a theta of fee and then you have the distance itself which is like the radius with respect to the scanner then you can convert that back into absolute values of XYZ to get you know a 3d position of something okay so like I showed you this is an example of one such laser scanner and then this is another kind of later laser scanner so I’m going to talk about in a second the distinction some of these scanners where you Paul pulse based and some of them are called face based looks like what that means on paper in just a second you’re also probably familiar with you know like the DARPA Grand Challenge self-driving cars also like the Google cars also have laser scanners on and operate on the same principles this is a filadyne scanner basically here there’s this you know laser scanner on the top of this car that is spinning around very rapidly and acquiring

3d points okay and so in the interests of trying to not have my computer slowed so much let me just kind of show you the type of data that comes off of the scanner and so this is a scan of the building the VCC on campus right and so this was acquired by one of my students you know many years ago and so what you’re seeing here if I can remember how to manipulate it is you know you can see that the quality of the scan is you know pretty good you can kind of make out the individual textures of patterns of bricks and stuff like that this is kind of a parameter you can decide how finely you want to scan the surface of the building of course if they take you a long long time to scan the building if you say I want to scan the surface every you know five building yourself like that and you can see if I turn this from surface two points that really the surface is made up of basically these columns of points each of which has been probed independently by the scanner and so you know one time I heard lidar point clouds being described as you know swarm of angry bees right because if you really start to look at the data that you get back from the scanner it’s not this beautiful surface mesh although you can connect up triangles into a nice-looking mesh later on but really all you have to work with are these 3d points and so sometimes it’s tricky to understand how do I know how to connect these points up into a you know reliable mesh because if I go back to my mesh you can see that well actually maybe like even go the wireframe that may be a little clearer right so here you can see the points have been kind of connected up into meshes and of course there’s a big question about you know how do you decide which point should be connected into a triangle and which points are too far away to be connected into a triangle right so we’ll draw some sketches in just a second of course you could say okay well the length of a triangle edge is too long well then obviously I shouldn’t connect it up into something but that’s kind of hard to distinguish sometimes because of the ways in which the laser rays kind of graze the surface so you could probably argue it like these right these triangles here probably shouldn’t be here but you know this is a scan that is pretty high resolution and you can make out all sorts of cool details one thing that’s definitely a characteristic of laser scanning which you can see in this view is that you’re always scanning the surface from a fixed perspective right and so if you fire a laser ray in one direction and you hit something that you don’t see anything that was behind that laser ray from that perspective the camera and so you can see for example that this tree here because the laser is way over here you know we held these things in the tree and then that creates this kind of tree shadow on the surface of the building and we never actually directly probed those intensities to get those we would have to move the scanner over to some other location and then we’d have to try and register or align these two 3d point clouds together to fill in the gaps it can also see that there are some surfaces that we did see head-on but for some reason didn’t actually get a very good image of and so for example you can see that these windows right we get the stone arch around the windows but we didn’t do a very good job being the window itself and so I’ll talk about that in just a second – another thing we can do is we can texture this 3d model with the points that come from a kind of co-located color camera and so here if I add some color onto this texture you can see that you know now I have something it really resembles the actual building and you know in some sense you can get away with a lot more I mean if you zoom in on you can guys see there are some issues but like the scan somehow becomes more kind of understandable more relatable when you’ve got some sort of 3d texture – on top of it and so later in this class we’ll talk about how do we you know take the camera image and align it to the 3d scan if they weren’t already kind of co-located in the scanner so anyway this is the kind of data that you’ll see coming off of 3d scanners and you know it’s actually pretty high quality I mean if you look at you know you can see here that kind of unlike the results that we got from like the structure promotion example are we – the reconstruction of the BCC using just the bundle adjustment alone in that case you could definitely make out the right angles of the building that we looked at from the top but you know if you were to zoom in on those it wouldn’t look super great I mean that’s not like super highly accurate whereas here the angles and so on that you can get from a lidar scan are really very accurate so I mean here you get the exact right angles and stuff that you know really exists in the scene okay so I’m gonna move off of this because I think it’s making my computer chug along so kind of what are some of the issues with lidar scanning so here’s the picture I showed you earlier of fundamentally we’re doing these we’re pinging the scene with a bunch of theta comma feed positions and the scare does that by basically rotating a mirror inside the top of the scanner and also rotating kind of you know around its base like this and so you can kind of say okay I want to get this solid angle

seen and I want to get it within this you know within every you know sending me your spacing on the surface and they just let it spin around that doesn’t stick and so like I said here’s a still example again here you can really see the shadows right so there’s a shadow from this tree there’s a shadow from the solar array that used to be in front of the BCC and then even here there’s there’s a kind of a vestibule that casts a shadow on the building from this side and so we’ll talk a little about how could we fill in some of those shadows using like a 3d version of in painting basically and here’s an example of what the points look like and what the mesh looks like after you connect them up and this is the picture if you take the registered color image you can turn into a color image like we saw earlier so here are some of the problems that you can get with lidar scanning so one thing is that obviously you can’t you know you get get 3d points on any surface that you didn’t see with the naked eye right from the perspective of the scanner and so for example if I’ve got a column in front of between the lidar scanner in the building well then I’m only going to get the front surface of this column I’m not gonna see the back of the column obviously I’m not going to see anything that the column occluded from the perspective of the sky and unfortunately these shadows kind of spread the further away the object is right if you guys since are far away other if you get these really big shadows on the scene one thing that you see a lot so this happened when we were scanning the MTAC building many years ago was that you know in some sense what you’d like to get is you know the physical surface of the building that you’re scanning right the problem is that when you have like bless the laser goes right through the glass and hits the object inside the building and comes back through the glass without getting you know refracted and so the problem is that you know when we scan the impact building we basically got the big wooden ball inside the building we didn’t get the front surface of the building Harvey at all right so if you’re really trying to scan something that’s glass you’re gonna have a problem and finally you know what happens sometimes is that if you have a object that is just if this is the laser ray and this is the surface of the object you know what you’re doing is you’re relying on the physical surface to return some laser intensity back to you right and so that works the best when the surface is oriented directly perpendicular to the laser ray and when also the way that when also that surfaces like for example you know you can imagine it like a matte white surface as almost the best-case scenario when that surface just returns a lot of that laser intensity right back to you if you’ve got a surface that is almost parallel to the laser ray it’s very unlikely that most of the laser intensity is going to get reflected off of that surface somewhere out into the world that’s never going to come back to you from the you know back to the scanner location so sometimes you may miss you know surfaces that are really parallel to laser rays do the so-called grazing angle effect and so you know it’s not a perfect science they’re also a bunch of other little issues like for example you know you could argue that the further away the object is the more that laser spot will expand and so there are some errors you have to do with distance although for the most part when you’re scanning I don’t think it to worry about the distance too much because what you get back I think has it’s fairly uniform hairs and also you know there are some minor interactions between the color of the laser and the color of the object right so for example you know if I have a red laser and then I hit a green object well then that green object may not reflect law that red light back to my camera whereas a white object might do better right so you know you could also analyze the kind of intensity of the return of the return you can to see other returns light gives you kind of a clue about how well do you scan a topic and if – how far away it is you can kind of tell how much light came back to me to make the decision and so the best-case scenario is you’re scanning objects that are roughly perpendicular to your you know view angle and those objects are kind of like you know light colored map objects is the best-case scenario so scanning stuff like a you know a light brick building is really not so bad but that being said if you want to scan like and I’ll show you some pictures in a second you had to scan like a you know beautiful new car right black shiny car just came off a lot that would really be problematic because again that surface has been a number one absorb light number two act like a mirror and so maybe you’ll get responses from the surfaces that are really perpendicular to the laser ray but other surfaces are just going to bounce right off and never come back to the scanner and so typically what they do in a visual effects scenario is that they will try to powder up for example these shiny objects and so I or someone tell me a story of one of the cars I think for like transformers right they have all these you know super you know those cars never dirty right they’re like these beautiful shiny concept cars right and so the first thing that the laser scan guys want to do is to scan this car and what they may want to do is they may want to like kick some dirt up on it or even worse like spray it with some sort of like white corn starchy stuff to give it this in that way think so I could imagine that the you know the people who deliver this $80,000 kind of concept car are not enthusiastic about you spray it with some sort of white you know mat

cornstarch thing to get a scam so sometimes there’s like a tension between what you can’t do on the set and you know then having to clean it up later on okay so let’s talk a little about the theory behind how these systems work okay so there’s kind of two fundamental ways of doing lidar scanning one way is what’s called pulse base okay so what you do is you send a small pulse of light kind of turn the laser on very briefly that pulse goes out hits the world and comes back to my scanner and so what you obtain is something like that so there’s basically a very simple equation that says pulse baseline R so you know I have my scanner I have my object I sent my laser pulse out it bounces off this object and comes back to the scanner right and so the thing I want to figure out is this distance D right and so it’s very simple equation right because I’m shooting a pulse of light out what I can say is okay this light and I measure the time that it took this light to come back to the scanner so it’s like say okay I have a pulse that traveled a distance of two D it took me T units of time and this is what we call the time of flight of the pulse and what do we know the velocity velocity it took me key time units to go to D distance units and that velocity is the speed of light right because I know that I’m shooting at pulse of light so I know this I measured that and I want to infer G and so that means that I can take D is equal to one-half Ct is like the key idea behind time-of-flight scanner and so really what’s that my question yeah no yes fetters right yes it’s got to be the timing is extremely precise right so if you if you go back and figure out okay so if I want to measure surfaces to within like five millimeter accuracy if for example if you do the computation that corresponds to I think it’s in the book here yes 33 Pico seconds so this is like sensing you know timing accuracy right so the receiver electronics on these systems have to be extremely accurate right for sure and that’s why these systems these pulse bae systems are generally pretty expensive so like we have a lidar scanner a pulse based lidar scanner on campus that we bought boy it was a long time ago maybe like eight years ago and i think at the time with the educational discount it was like close to two hundred thousand dollars and i think that probably the professional version of that would have been you know maybe double that right now things may be cheaper now but fundamentally that’s what we’re talking about and so the same thing you know like those filadyne scanners are also pulse based and so those are not cheap scanners either it’s like you buy one for your you know hobby you know you have to actually shell out some serious cash to put that together and so yeah it’s definitely true that you need to have a very high accuracy receiver and the other thing is that you know not only does it have to be good in terms of distinguishing the time difference between when the pulse come is shot out when the pulse is returned but you know the returned light intensity may be much much smaller than the sent out light density and so consequently the sensor has to be extremely sensitive in terms of you know photons as well as in terms of timers these are really high precision systems so the other things say though in some sense you know you do get to improve things a little bit you can mitigate any noise in the system by sending out multiple pulses and averaging the distances that you get so for example if the scene is stationary and the scanner is stationary you know you could send up say ten pulses to the same location and then measure those ten distances and take the average and we know that when you take the average of these measurements the variance of the measurements goes down right and so in some times you can kind of if you’re willing to wait a little bit longer to scan you can get more precise scanning by paying the same scene multiple times and so I’m not sure that when you you know I mean I don’t remember in our scanning whether we have the ability to trade off the number of pulses I mean some very advanced systems may let you make those a little very under the hood decisions but but all that being said you know the the resolution of these sensors in terms of the difference between the distance of the service you get and the distance it actually is really is like millimeter

accuracy even if you are tens of meters away from a surface so it’s really impressive to scan size I don’t get like this very nice high quality you know reading but that being said you know not everybody can afford to buy a pulsing system and not everyone can afford to wait for the pulse base system to give you you know the results of the kind of chug through the scene so there is a faster method called phase based liner okay so the idea behind phase based lidar isn’t falling so face based lidar let me show you a picture first the idea behind face based light are is a bit different so instead of sending out a kind of a you know pulse we have a continuously on laser beam okay so it’s basically continuously being modulated server those of you they’ve taken signals and systems hopefully some of you have taken single systems we talked about amplitude modulation so what you’re doing here is you’re basically saying okay I have a sine wave so that the sine wave has a very high frequency right this is the frequency of the light but what you do is you kind of shape the envelope of the sine wave with a much lower frequency wave so this is called the carrier frequency and the modulation because it basically turns that into something like you say okay here’s my you know slowly varying envelope instead I modulate this high frequency thing inside this envelope so it’s still wiggling very fast but the amplitude of the wave is changing much more slowly compared to the frequency of the of the carrier wave and so how does this help us well the idea is okay so what I do is I imagine that I again have my sensor and I have my object and I’m sending out in some sense you imagine that you’re kind of extending out some portion of a slowly moving sine wave right that hits the surface and it comes back to me and what I do is I measure the difference in the phase of the of the thing that I received right so that guy tells me at what point in the cycle am i when the laser came back to my Assessor or anyone that when the measurement came back to my sensor and so if the you know surface is not too far away then the phase delay is small because there hasn’t been that much time for this for this slow frequency sine wave to change that much when it comes back to me whereas if the surface is far away then maybe I’m all the way almost back to my original you know zero strategy and so here what I look at is what we call psy which is called the phase shift the phase shift of the modulated signal and so what I have is I have the you know if Megha is the frequency of the modulated signal you can show that the time of flight is related to these two things by the pulp where I have the phase shift the frequency and the time okay and so then if I put that together with my previous equation I can obtain that the distance your surface is again I have C like this if I kind of combine my previous two equations this is saying I know the frequency of the slow sinusoid I know the phase shift of that science side I know the speed of light and so this turns out to be you know quite a bit faster than the pulse pace but because the laser is always on it’s not sending out pulses and waiting for them to come back the laser is oay is just modulating this weight of life and so fundamentally you know I can get a whole bunch of continuous measurements coming back from a given point C and I can just kind of stare at that thing watched by a sinusoid as it comes back for a few seconds seconds a few fractions of a second then I move to the next point and I see the same thing right so instead of getting a bunch of discrete measurements in some sense you’re getting a whole bunch of continuous measurements of this office of phase and that can be you know a lot faster and so when you’re on a busy movie set you may want to use a phase based scanner instead of a pulse based guy okay the downside is that you know there is an ambiguity here which says that you know suppose that I’ve got an object that happens to be kind of exactly one period away or maybe like half a period away from the from the scanner so when I come back and I come back to the scanner I’ve gotten one full period right but now if I have another object that is one full period away I’ve gotten one full period here and I’ve got one full period back so the phase shift or both of those things will be exactly the same and so basically you have this kind of problem where you can’t distinguish things that are you know

well so there’s an ambiguity if you restrict yourself to say there’s a here there’s kind of a maximum unambiguous sensing range I mean you could imagine some sort of like you know computer vision techniques or a computer signal processing techniques that could kind of get around this issue so for example if you’ve ever done any sort of like microscopy imaging you may have heard of like phase unwrapping that’s got a similar problem from another biomedical ending problem where basically you can only see the phase but he really done wrap it into you kind of like to know okay well this is not actually you know three PI over two things shift it was actually 3 PI over 2 plus 2 pi or plus 4 PI so you kind of want to know how many 2 PI’s are you off from the return but that being said I think that in most you know kind of physical sensing situations in practice you basically say ok I’m gonna say I know that my sensor is going to be unambiguous within a certain range and that range could be something like you know you know 60 or 80 meters I mean it’s not like it’s you know 5 feet away so typically in most real-world scenarios you’re not up against this ambiguity with the kinds of environments that you’re scanning with things based lighter so in general you really have usually a problem with this so yeah so I think that you know again if we were to do it all over again we would try we would probably buy a phase based scanner not only for price but also for you know timing because right now our old scanner it takes a long time to scan stuff and so like we talked about no one on the movie set has any interest in the visual effects guy sitting around while something chugs around the scene and there are no actors present doing acting right so I mean you want to be able to get in and get out with your scanner as quickly as possible as I’ll talk about this a little more detail next time but I mean I had a great visit to a place called gentle giant studios in LA that basically does all sorts of types of scanner mix game movie sets they scan actors they scan props I’ll show some examples of that in just a second and so you know they have some good stories about you know the way it used to be that a pair the laser scanner used to be like the size of the washing machine it would take forever and now it’s like this little thing just it’s not tripod you can set a whole bunch of them up let them all kind of go at the same time to get these quick scans and maybe you have a master pulse basic scan that’s going slower that you know it’s not in anyone’s way and so need to see here of what I’ve got in my presentation right so this is a kind of example that you’d have in a visual effects scenario so what you do is you take one of these wide our scanners you pop it down on the scene and then you would acquire distances to points and so here we use what here what you’ve got is basically the scanning company has already helpfully classified these 3d points in two different classes right so gray is building Purple’s tree green is Bush yellow is sidewalk and so basically this helps the visual effects people later on the pipeline understand okay well I really only need to look at these points for for my algorithms now you know that being said you know some of this may be automatic and some of this may be extremely manual to kind of automatically segment down because again if all you’ve got is the literal 3d return it’s very difficult we’ll talk a little about 3d feature extraction right so in 2d you know things like sift features and so on are very easy to extract you can use those as the basis for telling what different objects are but in 3d you looked at that cloud of angry beings right be very difficult to say okay this clump of points is a building and this couple points is a bush right that’s kind of tricky to do sorry this is the set from one of those that’s from x-men first class the one where the mutants are getting broken out of this building you know young mutants getting broken out this is a example of a scan from battle Los Angeles quality movie and so here you can kind of see I think that what you’re seeing here these these circles on the ground are probably the places where the laser scanner was actually put and it rotates you know a few sources or rotation 360 degrees and so you don’t get data like right in the middle of where that scanner is sitting here’s another view of that of that environment so you can see this is a very large freeway set where they’ve got you know maybe 25 cars there’s a helicopter here that’s crashed you get a sense of the scale of this is very large they’ve done this by sticking together multiple stands from different perspectives and so I’ll talk about how that works next week so here this is a scan of the set from Thor and so when they when they did Thor you know so so most of the you know these days you know there’s lots of green screen on sets and so on but for Thor they actually built this kind of two blocks square town in the middle of the New Mexico desert I think and filmed on this real set and so they scan that set very thoroughly right and so again you can see that these circles are basically the Centers of where they place their scanner and this has been a registration on all those points into one coordinate frame and so here you can see just want

this diner from Thor and this is the kind of level of quality that you get and so you know you have us you know get a whole bunch of points and so again if you now wanted to take the if you want to destroy this building right when the guardian thing comes out and loans buildings away so now you’ve got very accurate 3d structure of things that you can use for example to blow up the building and say okay now what what if this 3d surface was made out of concrete and I apply a physical simulation to blowing something up great you can also do laser scanning for objects and so here are some cars from Fast Five where basically again to get these car scans probably required some sort of like corn starching of the cars to get really nice you know shapes because you can probably see that these are these are shiny objects also that being said you know because these are clearly been kind of post processed again into you know like they’ve segmented out this red thing they’ve segmented out the wheels and so it’s definitely highly possible that some artists have gone in and touched up the points that work goes oh by the scanner maybe the connected some surfaces you know here’s another example of a car again you can see this is not just a model because if for example there’s this big dent on the doorframe right and so maybe for visual effects realism you know they actually smashed this car up on set but now they want to further smash it up in the visually in the digital world and so they need to make sure that there’s a smooth addition between the actual car in the physical scene and the 3d version of them another car this is the there’s a train in fast delivery when they drove the cars off the train this is the safe that they were dragging around on the street right so so basically you know if you’ve got like a kind of a large object right like probably something that is car sized or building-sized then you want to use a light our kind of system I mean that’s the right kind of system to use we’ll talk next time about structured light scanning which is more like if you’re trying to scan a person sized object or a prop sized object and so again going back to the self-driving cars this is the kind of thing that you see from a valid ein scanner where you’ve got a car mounted lidar scanner that is on the top of the car and it’s basically swinging around in a circle and acquiring rings of data and you can imagine that the Google cars are you know automatically say ok this is the car this is a pedestrian this is a road sign and so on and of course those cars also have all sorts of cameras and Urschel sensors and stuff like that now we haven’t talked about we’ll talk about that maybe a little bit next week you know the car is moving obviously while the thing is scanning and so somehow you have to know how to put all these points together into the right coordinate frame so you have to have actually number one you have to have fairly accurate inertial measurements of how the car is going you probably also have to have cameras to help you register where the car is now to where it was then that helps you kind of align the 3d points also ok so that’s kind of it for for lidar any questions about that process I would have brought my lidar scanner in but it’s heavy and cranky and you know it’s you just kind of look at it but right now I think it’s thing over an impact they’ve been using it to scan and concert halls and stuff like that so again this there’s an example of there are lots of reasons to apply lidar other than visual effects right so they use lidar a lot in archaeological preservation right so for example they’ll go to pompeii and scan the ruins and of course that was an issue right because I think that just recently there was some sort of uh there’s some sort of an accident or not accident landslide or crumbling at Pompeii where they lost thing that had been there for a long time and they had scanned it hopefully previously that so they kind of preserve the cultural heritage with 3d scanning rights this is a very common thing to do pretty much any architectural or archaeological site you can imagine has probably been pretty well scanned by 3d scanners by now um and I was saying something else before this what was I saying about where else to use it I lost my train of thought machine anyway so yes all right so let’s talk a little bit about a different alternative to light okay and so that is what’s called a time-of-flight camera which is a little bit you know weird because we just talked about time-of-flight is the principle for light our scanning so it’s kind of like a slangy turn right so this is this here is what you’d call a time of flight or a flashlight are sometimes you hear it called flash lighter and so this is a $8,000 cube or at least it was at the time and so what this does is it acquired so one thing about lidar that is you know this advantageous is that you’d have to basically let this scanner percolate through the scene to get the 3d model whereas really what you’d like to get is like Y of 3d scanning I mean some sense you know the Villa dine scanner is doing live scanning but again that’s not cheap things like to be able to do is get like live real-time 3d and of course that’s the guy thing that is like what the first version of the Microsoft Kinect was doing we’ll talk about how that worked next time this actually we’re going to show in just a

few minutes a demo of the second version of the Microsoft Kinect which does use what’s called a flash liner or a tof camera and so the situation with this is that what you see around what we have around the border of this silver disc is a set of infrared lights and so they’re basically bathing the scene in front of the camera with basically this is usually phase based modulated infrared white and then there’s a whole bunch of sensors in here but not a lot I mean like compared to a digital camera these things are generally I guess prior to the Microsoft Kinect version to these generally three lowers illusion so I think this thing is really like a it sound like 176 hi something image you know so much lower resolution then even the crappiest webcam you might find on the market right but the advantages that it can do the scanning at basically 30 Hertz and so you’re trading off the amount of time it takes for you to get a scan with you know the resolution of the scan and so definitely there are a lot of cases where what you want to do is get kind of real-time framerate depth sensing and so when they say the third kind of modalities sometimes called flashlights are and sometimes you just call the tof camera and so again these are usually fast I would say about thirty Hertz frame rates they’re usually I would say low resolution and so this was you know this was a SR 4007 engine and that was a 176 by 144 image and then also you know I would say it’s noisy right by which I mean you know the measurements are not as accurate and you also get a lot of kind of you can see some visible noise I’ll show you example that just a second but they’re very easy to set up I mean so basically you usually just kind of pop the tripod turn it on and suddenly you’re getting real-time three depth that’s pretty cool and so let me show you actually we just see if there’s anything else I want to show you in this thing right so this is the kind of thing that you get so this is a picture of my students in the lab this is a real camera image from above the desk and this is a frame of a time of flight based rqf camera based image and so here the wider something is the further away the object distance that the floor is farthest away and you can kind of make out that you know the person’s back of the chair and their shoulders kind of gray and yeah the top of their head gets darker but then there’s a whole bunch of missing stuff for example this person has like shiny black hair and so the time-of-flight camera didn’t respond very well to that and just didn’t give me any response at all on that region of the image and so and also again if you look at the periphery of the image the camera response is much noisier as you get out to the edges of the image and so fundamentally you don’t really get much response in kind of like this outer circle and so actually we were using time flight cameras and one of my research projects for detecting occupancy in a smart room so you guys maybe know about the smart lighting Engineering Research Center on campus so basically we’ve tried to instrument these kind of smart rooms on campus to automatically be able to tell you know where are the people in the room and kind of hoarsely what is their pose and it’s standing up for this same down and then the idea would be that the room would respond by changing the lighting in the room to kind of match up with what the person was doing right so there’s connectedness so here’s an example this is a new room that we just built up in the CII where these things in the ceiling we’ve got 18 kind of very low resolution time-of-flight sensors and so that’s what these look like these are called iris sensors iris matrix sensors and again these are even lower so these are like literally 25 by 20 you know pixels right so extremely crude but you know one one advantage of them is that Tenafly is a good choice for this application because you want to feel like I mean you could do this very easily with cameras right why don’t we just put cameras in the room right but the thing is you want to feel like your room is monitoring you with cameras all the time right or even if you trusted that no one was looking at the video you don’t want to feel like someone could like use some sort of like heartbleed vulnerability to hack into the cameras and get real-time video of you in your home right that would be bad so Gideon behind time-of-flight here is that things are really really chunky pixelated but they’re good enough to be able to tell who’s in the room and are they standing up or sitting down that’s really all we wanted to do for this application and so and then you can set guys here we have these color tunable lights so these lights may you know dim in areas of the room where there are no people or where there’s already sunlight coming through to provide lighting in the room but they

may light up over the desks of where people are working right and so let me show you an example of that weapon alright so you’re going to see at the top is a mosaic image of all the time of flight sensors put together and that’s the raw output of these pilot flight sensors at the bottom you’re going to see a computer vision algorithm that is basically tracking people as they move around at 15 times normal speed so it’s not like okay so kind of what you’re seeing here is on the top you see the raw up where the type of light sensors you can see there’s actually a bit of noise in the sensors right so for example here on the table you can see there’s like this kind of wavy pixelated noise saying here this is a TV screen or testing extra light into the scene so the sensors themselves are pretty noisy but if you look at the tracking results you know they’re good enough I mean they’re good enough to track the various people as they move around the room and so here you know you got up to seven people that are standing sitting around you know and so our next phase of research is going to need to use these you know real-time person locations to develop control algorithms for the lights in the room that’s kind of a fun project and if you’re justed we can go over or you can stop by the you know the smart conference room or SDI to actually see it working okay so I guess while I’m on this page what I would say is that the most notable current use of this is for the Xbox one you know Kinect version 2 right so this version this can disconnect that just came out Christmas time I guess uses a real-time time-of-flight camera to extract three distances seen and also to do the real-time 3d skeleton that we were talking about kind of a devotion capture chapter and so since this just came out and there isn’t a lot of detail on it I thought we would take our beta kit here that my student Dan is working on kind of putting together will show us some results to it so you can see what it looks like so why don’t we try and do that demo but I need I may need your help Tim so let me close this up and okay so we don’t want to close this up either okay so I think this a little bigger look I think this is just a little slow to respond to my commands here there you go so what you’re seeing here is a live camera feed and then this is kind of like a crude depth image but the most interesting thing is this image right so let’s see what we can see here so I guess we can see that you know it’s picking up the the core these these are the ceiling lights right and then who is directly that there’s a chair in front of Matt right so Matt wave your arms little bit well baby this is you right so you guys can come in further connect let’s see our Altos there you go and so actually you know this is probably even faster than it looks just because my laptop is kind of pokey and so if I rotate this to look down from above right here you guys see again so actually sweet spot is really so it looks like you know we’re getting I don’t think we’re getting all the way to the back of the room or at least it’s hard to tell whether yeah that’s that’s pretty good there but you guys in the back of the room like I don’t think we see you guys at all so sorry oh yeah there we go there’s there’s somebody right and again this is probably partially because of the occlusion Matt guest so like if you think about what we’re seeing from the point of view scanner it may be something like this right so I mean if you think about it from the from the desktop view we do see about as much as you as we could so if it wants to move down to the aisle or if you guys are still so again I think that it’s actually probably a lot better than this in terms of frame rate but yeah actually gets so one thing is surprising about this is that this is actually a much denser time-of-flight image than I was expecting do you know damn what the resolution of this is well oh I guess actually you could figure it out from looking at the I guess we don’t know what the resolution of this image is so this is

basically the real-time yes I can make this bigger so we can’t make this Naxos bigger I guess or we could also do they connect we call the connect status right but we have a second source alright well we’ll come back to that in just a second so with the other demo is the skeleton right so if I do display body data oh yeah so that that’s a better image up there it’s like this is basically the real-time depth image kind of colored where I guess that blue is closer and red is further away and then I guess there’s no one in the field of view of the Kinect so we have to my lovely volunteer stand in front of this and see so when you moving to this you think is going to pick up your body yeah now go you think it’s maybe the board is something yeah maybe 32.1 different direction much little bus clutter in the background Scofield well I also just very much yeah you think that if we restore this oh we better do like just restart the service okay so should I restart MATLAB also all right okay so then I need to stop the stop all the services right and this one we also have to stop oh if we if we just stop this and look at the connect yeah so let’s just stop with this fail fail so connect status so here again you can kind of see like this is the the real yes actually you can re see that what Dan was showing was kind of cleaned up so again you can see this kind of characteristic noise around the edges of the time-of-flight image right so this is you know again definitely not as high quality of depth images you get from from uh you know lidar sensor but you know for the purposes of consumer you know real-time game control this is perfectly good right even around I mean it’s gonna hurt so you probably can’t see it just because of the way that it scored there’s probably you know a lot of noise that you can’t see because of this color map happening to and so alright so if we cancel out of this and then we restart the connect service which I have to do from here right the service right okay and then MATLAB – run your servers again yep okay so that I should probably back to that one I’m feeling that now I’m sorry yeah like it’s now I can unplug the hub account okay so what yeah I think maybe these things you is just to go yeah you’re probably right okay so we’re on a this is just you two split apologize for the people at home who are watching a slog through this demo

so I mean this is a beta SDK we’re fundamentally there’s not a kind of academic connect err face that’s been kind of generally approved yet just run this area hey here we go okay so let’s zoom in on this a little bit it’s gonna study head there you go that’s pretty good right so Dan put this together and so basically so when the frame drops out is that when it’s lost track and so you can see unlike the old connect this one has I know whether we can see the the grasping of the fingers like is it you know what yeah one thing that you know o as se is this is this ball your thumb is that the idea right right so he used to be that the Kinect didn’t do very well so again I don’t know whether or not the underlying I mean I’m sure when they made the version 2 or the Kinect they totally changed the word they totally revised the skeleton estimation – I think this is again the result of some crazy you know weeks-long trained decision tree classifier it’s very fast to apply but I don’t know whether there’s a temporal coherence here you can still see there’s a little bit of a you know tug roll wiggle I mean that the body seems pretty firmly planted I guess that the the hands seem like they’re kind of jiggly a little bit yeah but again you know you think about what you could really reasonably infer from you know looking at a depth image of a person you know I think it’s already asking a lot to be able to even estimate what the risks are doing and what their fingers are doing so it’s pretty impressive I think so we’ll work with somebody else the framer is only one person okay do you yeah try some roses so yellow means bad estimate okay so what if you like cross your hands in print your body well that’s okay what if you put one hand behind your back oh I see yeah yeah so what does it work what are you saying down it looks like it’s kind of see now it’s now it’s a lot okay sorry I’m trying to wriggle this around here how do you do oh the hand right yeah right so this is the side view of you sitting down so you know again it’s probably trained more or less on expected poses as opposed to oops sorry I’m flying around here I thought as yeah that’s pretty near it so okay so that’s the you know kind of latest and greatest tof camera based technology I think that probably there goes our crazy outs all right so I think that’s about it for today any questions about either lidar or POF cameras okay so let me stop my recording goodbye little sticks again