Paco Nathan: Jupyter for Education: Beyond Gutenberg and Erasmus

already thanks for the opportunity really appreciate being able to speak here and even more so really appreciate a lot of the talks that I’ve seen and the hallway conversations i’m at o’reilly it’s actually fairly recent that I joined O’Reilly just as far as background I’ve spent you know about 10 years leading data teams doing mostly machine learning at scale across a few different verticals and I’ve spent the past after that the past 34 years evangelizing open source projects working on cascading mesos and more recently spark and in particular out of those I’ve had a lot of opportunities to be teaching tutorials and working in universities teaching in various programs regarding data science and big data we spark in particular we had this huge adoption curve or sorry this huge adoption rate and we’ve been doing very large tutorials all around the world and I was leading a team that included a couple people a couple professors from you see I think Joseph end I’m a tall Walker who just did the two spark MOOCs on EDX so you know we we definitely saw a number of different kinds of use cases of both in class and remote learning and in a lot of cases trying to leverage ipython notebooks or something close so I wanted to present about where we’re going at o’reilly now with the new or la learning team which is very much using Jupiter but also looking at how do we bring in other types of media channels other kinds of learning modes and how do we integrate this in our context as publisher so who we are O’Reilly learning is a new business unit and we’re not these people as far as O’Reilly on we are these people so we have a lot of books a lot of videos a lot of conferences a lot of involvement with developer communities over the years and you know O’Reilly is closely associated with with open source we started to ask on um I just got out of Oz Khan coming up here the objective that I was given from the higher-ups as I walked my way into the door was as a verbatim examined make sense of and organize our various training products and learning channels for ourselves and our customers and speaking as an author I’ve been a writer for a Riley for I guess the last 11 or 12 years it is kind of daunting because there’s five different video channels and there’s a I don’t know how many different training channels all these business units that have been acquired over the years trying to understand that from an author’s perspective is daunting our editors I think are probably even more confused but we’re trying to make this something that’s a very cohesive whole how do we bring these different types of learning experiences together into something that’s more effective so one of the first things I’ve done is to try to figure out you know just with O’Reilly as a as a case study what do we have and how does it work and if I were to try to put together an overly simplified map of where the flows are I’m really interested in charting from an audience perspective always audience first but then closely behind that from the author’s perspective the authors coming in what is their workflow like what what’s the experience that they have and we’re really where are the routes the different touch points for different types of media for learning products as you can see there’s several here I’ve kind of collapsed a few but it really spans a range of at the top between in-person events which often have a very high transaction cost people will spend thousands of dollars or their employers will spend thousands of dollars and maybe a week out of work to go to some kind of conference that ranges from you know a high transaction costs to a relatively low transaction costs for on-demand behind a paywall with something like Safari Books Online and this process that we have here is I feel overly complex it’s still regarded as being relatively agile I don’t know if that’s really the right word I kind of I don’t like use that word too much I used to be an engineering director and I anyway I’ve written for a variety of different publishers and the workflow that we have here we can put out a book in like six months generally some of their six to nine months whereas in a lot of organizations you know that workflow is more like two years in some cases in our case typically you’re working with one

editor whereas in other places you might be working with 12 different editors and those handoffs all the way across so the publishing process is really complex and that’s if you’re doing a book if you’re doing a book and you’re doing a kindle and you’re doing some video to go along with it and you’re going out and doing appearances it gets oh so complex and the crux of what I think is a lot of the problem right now is that when we look to publish things we generally reduce them down to epubs of one form or another and there’s probably some video content that goes in there or can be linked there but it’s a combination of these two that are basically our source repository our content source and the standards for epubs are they don’t accommodate exactly what I’d like to do in teaching people programming for instance how to use code so we’re looking at how to react attack this here’s a article that came out last year and in some conversation with person but it’s called learning architecture it’s actually behind a paywall from Deloitte but there are some pretty good reviews of this online publicly available the idea of learning architecture that was I think something that’s being formulated in the words of michael pollan you are where you eat eats and I always try to keep that in mind in terms of how how do we provide things for the audience we have customers we have audience people who buy books and videos we also have customers in terms of companies that are sponsoring conferences and they’ll in some cases sponsored books but they in turn have customers that they’re accountable to that they’re trying to accomplish something with so I like to think of that hole chain of learning so we live within a community of makers and innovators learners implementers the objective that my team has is to come up with a learning architecture that’s very effective internally but then to externalize this for our customers and then apply that as a kind of design pattern for them to use for their customers so as far as background I’d like to point to a article or presentation from the open data science conference in Boston recently Kyle Kelly who’s here and Andrew ottawan who’s the CTO of O’Reilly I’ve worked with them both in this project over the past year or so but they had a great presentation called on demand analytic and learning environments with Jupiter and in this they explored a couple of themes one is the notion of computational narratives that they were describing of where we can leverage code in the context of media and learning experience and into it speaking from the perspective of being an o’reilly author before we always try to put the code first you know that’s what you’re taught by your editor is that when you start out the first paragraph of the first chapter show some code get people to dive into it start to feel comfortable and then back up from there and start to explain what’s going on in the code so we like the idea of integrating code as media into the narrative and there’s a lot of reasons for using these narratives sometimes they’re books sometimes they’re actually being used to surface insights there’s some type of notebook that’s it’s showing leaderboards or other other structure of a community they may be a white paper or a sponsored paper they may be an executive dashboard so we see a number of different types of consumers of this my experience on this started a couple years ago when we were in 2013 we we pitched doing a new type of book called Jason if math and the thesis there was teaching advanced math to business executives in particular so that people in business would understand sort of the why and how of leveraging more advanced techniques in data science essentially to fund big data products projects and you know we find that there’s a very big demand for people who haven’t taken three years of calculus but still want to understand what is it about graph algorithms what is it about matrix factorization how is this going to affect how I have to manage supply chain in a billion dollar business unit and the way that we approached this was to use ipython notebook for the examples and take code showing math but then tying it very directly to a use case that would be familiar starting from use case first in the presentation and contextualizing the problem so that they don’t have to have three years of calculus before they actually see a use case and while we were doing that we we did run into you know very interesting things what we were going out to conferences and doing tutorials a lot of it was more on the operations side but we had a lot of learnings and I got to work with Kyle

Kelly and Andrew I don’t wanna on this one of the big learnings was this I thought it Stanford years and years and years ago and I’ve taught sense at other schools but for a lot of my career I had been a software engineer or engineering manager and you know then going into leading data teams and doing more than eight a scientist role I found myself in the role of being an author and teacher with this tool chain that depended on authors using git and docker and ipython notebook and software engineering practices that authors typically don’t touch so it was very interesting to go back and forth and a lot of that dialogue with Kyle and andrew is we’re trying to use that to redefine our workflow for authors one of the things we were doing was to be alpha testers for this if you haven’t seen it I think it’s very compelling nature magazine used tenpin be and I Python notebook essentially the same tech stack the same code base they have an article here where they’re taking their peer reviewed articles and they can ship them out here’s the data here’s the code here’s the article if you want to run this you know feel free and the infrastructure will support you know 10,000 plus people online at a time running their own container it leads to repeatable science it really is in my opinion a lot of the essence of science and Jupiter combined with dr. and few other things in the tech stack are allowing that kind of open science repeatable science there was also an article sort of our thesis statement from O’Reilly it’s by Andrew auto on that came out in May talking more about our internal platform that now we’re starting to expose and Tim O’Reilly made some notes about that here it’ll basically looks like this it’s called Atlas and it’s where authors can go in and put in HTML or markdown or a ski dock or other types of content now also Jupiter notebooks we can bring in a bunch of different content organized it as chapters and sections etc and it’s a very collaborative environment kinda like Google Docs you could think of so editors are in there are you know changing things are making comments of what needs to be changed and then it’s all backed by get so the authors can also be putting in content via pushing and get and get is already kind of getting out of the comfort zone for a lot of authors there’s a lot of learning has to happen but when we start to combine Jupiter get doc Earth Eve and other open source projects along with it it becomes a pretty interesting tech stack but it’s also a fairly unfor engrams for writers I mentioned this already but there’s something called phpbb and this is an open source project it’s on github the idea is to be able to have essentially Jupiter cells as a service there’s a way you can take parts of a notebook and publish them as HTML as a webpage so I like to think of it almost like a three-tier model it’s almost like saying middleware were the books of the middleware but what gets provided by thieve is actually HTML that’s for the presentation and in this way you can do things that you want to and that you need to in terms of branding for instance is very strong and publishing sometimes what we’re publishing is for another entity and they need to have their logo they need to have their marketing funnel tied into it and also it’s a little bit better in terms of instrumentation and some of the operational aspects of it phebe is a Jupiter is a moon of Jupiter by the way and so our proof of concept then is called beta and if you go to o’reilly sorry beta or oddly calm / learning in particular we’ve got some good examples some of these actually did not start out as articles they started out as as I pregnant notebooks and so for instance here’s one Jake just had an excellent talk and let me bring up one recent article so this was a you actually from earlier this month Jake had article about pivot tables in Python and here if you go and you can run it and you can see it’s got a kind of a different skin it’s got a very different sort of branding and look and feel but these are notebooks and you could have interactive graphics with controls etc again we’re just kind of blurring the lines between what would be classically our content platform and em project Jupiter okay now as far as the tech stack on you know

there’s the authoring process where you’re iterating with co-authors bring in content from different sources working with the editors etc then there’s the production process where you go through QC and get everything ready to do pre-press or in this case get it packaged as a Jupiter notebook in a docker container and then we can use Thebe to surface that out to the web pages of course there’s some type of cloud infrastructure underneath this typically so the question though is Billy what’s the difference between authors currently create materials and this new world where they’re using Jupiter docker Thebe get various types of clouds to supported etc okay so for some examples I wanted to point part of the talk was to try to point out some good examples of using notebooks in education and also some other examples beyond Jupiter to try and compare and contrast I was really excited to see what’s going on with Microsoft in terms of Cortana’s use of Jupiter I had been working at data bricks prior to o’reilly and we’re doing a lot of work of course with notebooks there I’m very much in a similar similar kind of effort also if you haven’t seen it I highly recommend something called code neuro this is led by Jeremy Freeman out of Howard Hughes Medical Institute janelia farm and this is a really good one they have a basically a hackathon for some of the world’s top neuro scientists they get together for a weekend twice a year and they are working with Jupiter on top of spark primarily but then also using d3 and things like that Matt Hamblen is from New York at a company he’s worked on lightning and thunder along with spark as a visualization layer really high performance data visualization layer that leverages d3 and oj s etc see what they doing is they have these intense microscopes and they’re looking at huge networks of neurons firing in real time off of rats are using lasers so they don’t you know they don’t hurt the rats per se but they’re able to get real-time feeds and process them and redirect what they’re doing in the microscopes so it’s actually one of the larger use cases for or larger data rate use cases for spark streaming for instance but also for data visualization and again the way that they are sharing science is based off of Jupiter another good example one of the other people involved with this conference is olga vodnĂ­k at UCSD they have a project they’re called Yale lab and again they’re they’re bioinformatics is being formulated around using an iphone notebook workflow very very similar idea um Lori knows over there are definitely looking forward to keynote tomorrow and I think that out of some of the I want to put together a list of examples and so I want to highlight some that are my favorites if you haven’t seen these I think there’s definitely some some great examples throughout the web there’s also a shared doc that we have we’re collecting uh you know where we’re seeing Jupiter hub being used in academia also just different use cases for Jupiter in education and also an industry to you know taking a look at some of the companies are being really innovative about this and so I’ll leave links about that up there I’ve been working with Fernando and others on this or just trying to to curate it let’s say now in terms of compare contrast with other types of notebooks I definitely you know give a lot of credit to our mark down I work for data bricks and I’ll show you an example here if you haven’t seen so data bricks is where you have notebooks that actually maintain the state of the analytics you’re working on and then the clusters the spark cluster is there in the cloud but the clusters can be attached to detach within seconds and it’s quite different than usual practice for at one point eyelid a team that had the largest how do instance in ec2 and I worked very closely with Amazon on the the lead up to elastic MapReduce and the problem that you see using cluster computing the cloud is that if you have to launch hundreds of nodes on VMS it could take you hours just to get the cluster up and running um so by leveraging linux containers see groups are you know you can instead bring that time down to seconds and it might be that part of your analytics workflow requires a huge cluster 45 minutes and then the rest of your workflow you know you need one or two notes aab just to be able to do some visualization or some

some queries and we see that across the industry is a very typical pattern so the depression is trying to leverage that it’s not Jupiter it’s their own code base but I used this for about a better part of a year teaching in conferences teaching at universities also in private corporate training teaching people to think of applications by using notebooks and then having them backed by cluster computing the cloud here’s one where we take a look at meetups for spark and pull down the data so that we can put together leaderboards if who’s doing what meetups where who’s sponsoring them what the topics are just trying to get some analysis of you know really what what does that community look like so I’ll leave that up another one I’d point2 if you haven’t seen is Andy petrella and it’s called spark notebook and it’s not to piter’s use some of the protocol there’s definitely some some commonality there I’ve done some teaching in Amsterdam with Andy and this is all Scala based but it’s very interesting I a he’s iterating on this very rapidly he’s done very nice integration with docker with Mei sews a very nice integration of visualization packages definitely one to track and you know a couple others as well I know that Toby was around her earlier on if he’s here now but friend of mine who was at IBM formerly was sharing with with me about IBM Knowledge anyhow which is using ipython notebook hosted in the cloud softlayer by IBM and of course mathematic is a great example of they’ve been doing this for a long time mathematically notebooks have had been out there commercially let’s see how we’re doing for time uh there’s a few things that are on my wish list for what I would really like to see with notebooks and I hope to be able to help I hope that our team is able to help bring some of this about but for us in terms of publishing video is really important if you look at the demographics over certain age most people don’t want to buy books anymore they buy a kindle and if you look under that age they don’t buy books anymore they watch videos I have two daughters who are 11 and 10 and they go and watch YouTube for anything they need to learn I I caught them watching they were trying to modify minecraft so they need to learn how to redo the manifests on jar files and they watched hours and hours if not days of how to redo jar files and I told them look I can just give you a build script but they’re like no no dad get out of here so you know kids love video and we have to take that into account and meanwhile we have a lot of conferences all over the world we have this huge amount of video content if it could just be structured and tied together with other types of learning materials there’s also the social context of mentoring and bringing in experts on a problem for some period of time there’s the notion of learning paths how can a subject matter expert structure of a program of learning and of course i’ll i’ll use the M word there are the remote learning environments so that leads me to MOOCs and we just had a very interesting experience with about a hundred thousand people signing up for a couple of mooks that we did or I I was tangentially involved let’s say the UC professors really did the work hanging around folks who study MOOCs there are some some difficulties I think that they’re excellent it for learning at scale but they do have some drawbacks number one it’s really costly for authors to go and create a MOOC it’s also very difficult to instrument and a lot of this especially in industry they want to see some sort of performance what are we getting out of this what’s the engagement and with EDX you’re just behind a wall you can’t really instrument it all that well there’s very low completion rates typically typesafe is a counterexample I understand they have like about a ten percent completion rate which is actually fairly rare and and the other thing is that the social context if you’re trying to teach thousands of people at a time even if you’re trying to teach hundreds of people at a time let me draw that back because I’ve done that a lot you need to have TAS you need to have somebody to help engage in the dialogue you can’t just have one speaker and thousands of people if you’re trying to a large MOOC you end up having to try to arrange a small army of TAS and that’s complex in itself so you know there’s a future learning summit at Stanford recently and I thought it’s hilarious I’ll paraphrase Peter Norvig here but you got up and said that MOOCs were having huge troubles because of the search engines and we were like wait a minute aren’t you from a search engine

company but it was a really good point that you need to get people to want to interact with the learning materials and generally that’s because of social context so i’m not sure that i really believe that a MOOC alone is going to be the path forward for learning because unless it has better social context i could talk more about rubric particularly ways to leverage social context but maybe that’s for another time that’s a favorite topic now if you haven’t seen it this is a really great example yes sir Abu Mustafa out of Caltech does a machine learning program there it’s not EDX they did I believe one year in EDX but they they had been doing something prior to that which is arguably more advanced and really leveraging the notion of inverted classroom or you might hear it called flipped classroom where you know the students have a lot of materials in advance they can work through and then essentially they’re they’re trying to build a funnel where the instructor really has the engagement at points where it’s very critical and can have a lot of outcome a lot of benefit and in it is very much kind of the opposite of what you think of when you’re building a lesson plan for a classroom of 30 people there are other issues out of their pedagogical issues in terms of trying to match the content to the learners needs and Andrew had a nice paper about this actually a lot of interactive examples it’s called patterns of code as media arm and as far as like learning to program I think there’s some great examples there they go across a spectrum where is the person on learning curve where are they in terms of being an doctor how well do they know the material not just did they pass the 10 questions on the pre exam so looking at it in terms of more dimensions it’s really crucial and so you know for that what our team is looking at is really an exercise in high-dimensional data arm taking a look at the spectrum across a number of different dimensions of how do people engage with content and what do they need and why are they doing this not just throwing an exam at them but trying to understand where they at with respect to the topics and therefore what experiences they have to that note I think that there are some very good ways to use machine learning for this if you had the right instrumentation if you haven’t seen there’s great work out of Stanford you’re a leskovec out of MDS etc and but more more to the point when it was postdocs Julie McCauley was there and Julian’s from Australia and when he got to grad school in the US he believes that the beer was horrible compared to Australian beer so he started doing machine learning work to try to understand why and and he really came out with some interesting results it dove tailed with other work that you’re I was doing but basically they said was with a few different signals and not that much data but with the right data you can actually measure the distance between a person coming into a community and starting to learn about it and you can actually get a pretty early trajectory of whether or not they’ll stick around just even looking at the language they use and the difference in the language usage patterns as the person learns and as the community evolves and you can start to detect with actually very good reliability when the person is ready to leave in relatively far advanced of them actually leaving and if you’ve been involved in online forums you’ve all seen this we’ve all seen this but they came up with recommender systems time-based recommender systems to actually quantify this and measured distances of a learner with a community where they’re both evolving and I think this is fascinating and I trying to put this in practice again in terms of open source communities etc I also want to make the point that learning curves are forever it’s sort of a Sisyphus kind of example the idea is particularly in data science my experience leading data science teams is that learning curve is the hard problem and the thing about data sciences you’re breaking down silos your your your transgressing and crossing boundaries that if you did not get the data across those boundaries you would never get the insights that’s that’s the big difference between the prior generation of business intelligence and what some of the success stories and data science have proved but you’re always having to learn about these different domains the different processes all these different tools and actually budgeting for that is really one of the hardest things of being a manager in data science but perpetual learning curve is the problem the way to address that is to borrow from engineering software engineering notions of say continuous integration or continuous deployment and think about continuous learning there’s a great brief article by Whitney Johnson also building on one Mendes of looking at learning curve not just as the

traditional sigmoid transfer curve but actually looking at as a continuum for for a continuous process and be equal to measure and quantify when is it that you needed to jump from one curve to another so highly recommend that and that leads to this point when you look at the sort of bowling alley curve that Geoffrey Moore my talk about your it really comes back from everett rogers the diffusion of innovation decades ago or should the year i was born but you know you’ve seen this probably where there’s the innovators the early adopters is the bowling alley where they lead into the early majority and then the late majority and the laggards and typically when we talk about putting the other a conference and being the program committee for talks we want to get the innovators we want to get them doing keynotes we want to give them doing the really hot topics but I want to pose the question in terms of industry in terms of community where are the experts on this graph are they the innovators and I’d love to talk about that more maybe afterwards I know we have time constraints but I don’t believe that the innovators are the experts and I think there’s a continuum and you have to look at sort of a multi-dimensional perspective of where is the learner visa V the topic and the community and the experts show up in a lot of different places but you can pinpoint it and we do see that you see that in our sales figures and I hope to be able to surface more about that looking ahead um beyond books because we have to be moving beyond books in my pin and Beyond kindle I’ve gone through the experience of actually putting the other mobi from Python scripts manually and having to deal with all the different xml conflicting xml formats we need to get beyond the notion of view pubs and kindle particularly to be able to do any kind of interactive content that’s running code are leveraging cloud etc beyond MOOCs are you know I believe that the the path forward will be leveraging learning paths continuous learning and verdict classrooms I didn’t talk much but computational thinking is very much out of jeanette weighing and all out of CMU it’s very much the kind of Rubik that I try to leverage when i’m teaching also learner segmentation and a lot of multi-dimensional analysis for machine learning of how do you fit the content with the learners and interestingly more and more we’re hearing that it’s not about an individual person learning it’s about their social context once they get paid to go to a class how well do they perform how does their team perform afterwards or maybe you’re putting an entire team through a class for sending an entire team to a conference but it’s the team outcome that’s ever so more important than an individual’s test score and at the same time though we still have to do things like certification and I leading that and trying to understand where do developer certificates fit into this picture because we have to make it a comprehensive whole and finally I’ll say that we’re very eager to work with new authors on this this is a green field and what we’re doing with notebooks is essentially displacing or helping to evolve what had been done with white papers what had been done with conference talks what had been done with MOOCs and we’re inventing this as we go along with everyone else here at this conference but I believe that Jupiter is having an enormous impact and displacement on these prior modes of media and going out and talking with early customers they’re extremely eager to do this nobody knows exactly what it’s going to look like but we’re eager to help invent that future together and I would invite you to be participants please let us know if you’re interested armed with that thank you very much any questions yes yes yes yes that’s great well there’s the badges I mean there’s the notion of badges and certainly better experts in that than me armed cash in a couple things there that I would like to say one is I mean a lot of people when they think about MOOCs the second thought they have is get a study group together and that’s one thing I’ve been trying to analyze and really map out is you know where some this is coming from a lot of its ad hoc but some of it’s becoming more formalized through meetups and things like this so we’re actually looking to try to sponsor or facilitate study groups for difficult topics I know some of the books that come out of O’Reilly I know of one in particular I want I won’t name it but it the examples are so difficult that people have to have a study group just actually run the code and so I’ve seen these thriving study groups as a result and I think that once you get past the beginner audience the introductory

courses you get past even some of the intermediate you get in the advanced topics it’s absolutely crucial and so the social context really reinforces I got to get past this hurdle or I’m blocked and I’ll give up another one that I guess I would point out maybe not as much badges but kind of the gamification I mean chief Devlin and others are really good at this but um Masha said ova at salesforce who does like training for security at salesforce has some great talks about this where the problem that they had if i can paraphrase it right is when you’re working in enterprise and you’re building these in ginormous Java projects you know a build will take like 60 minutes so for Salesforce when an engineer changes some code goes to you know commit and run through tests they’ve got an hour open what are they going to do are they going to go watch YouTube probably or reddit or something like that so what they came up with is let’s have sort of what they were using actually are you familiar with egg cast is a sort of mukesh kind of platform I believe Microsoft is very much involved in this too but it casts was one of the platforms that are using and they basically would structured so they knew they had that our window while maven or whatever was running and they they came up with video-based instruction tied with get repos so you watch a certain amount of video that would show you how to do best practice and implementing a security feature they give you an app that runs but it you know has security holes you have a time window to be able to fix that check it in pass through tests and then they have all these leaderboards on different dimensions so it’s totally gamified so I mean rather than go play a video game they basically give you one and that’s kind of the badge notion I think it’s kind of asymptotically out there um yeah I I had I haven’t really had a lot of direct hands-on with the other areas of gamification but I have had a lot of hands-on with group projects and the peer review the one that I would point you there I’m blanking his name but I can look it up after in the break there was an instructor out of Australia who did an intro robotics course that was targeted more third world Africa Asia Southeast Asia and they had largely it was based off of peer review the inverted classroom model is about groups assessing other groups and sharing with them and they’re even Kickstarter’s for you know groups in Africa to get the the source materials and so they sort of gamified the pier parts of it um I thought that was pretty interesting to I can get up sorry I don’t have a link to that I can grab them questions yes Oh certainly yes excellent excellent school well I mean we have a that is a a mode of learning a type of channel or actually a collection of channels that we have and it has mentoring aspects it has video aspects it has several different aspects in some ways it’s a head then maybe some of the other channels we have I think that mentoring is key it’s just the economics of mentoring have always been kind of problematic I participated in various pilots over time and I would love to be an independent consultant living high on the hog by being a mentor from remote just never quite worked out for me I think that you know OST is definitely a big part is picture because we’re seeing that content itself is becoming more and more about training and training itself is becoming more and more about consulting light so there’s this whole spectrum or marketing funnel if you will where it isn’t just about writing a paper anymore it is much more interactive and it is much more socially engaging and I think the OST has a very good model for that right there-there’s layer so just to repeat that that’s about software carpentry there was a spreadsheet that I had a link to that’s actually you know talking with them and that was actually some of the examples we were linking to and I you know really that follows after kind of guild models and initiative models and I I love that I’m also involved in something called data guild out of Palo Alto where we had some similar notions of leveraging more the social dynamics as opposed to vc funding yeah I do think that peer teaching is really crucial for what it’s worth my teaching fellowship at Stanford was just set up computer science peer teaching it’s now called residential

education and I think it got it helped to propagate out to MIT and Dartmouth and others and I’ve just seen this really work well I mean it it’s still going 30 years later oddly enough um how can we leverage that though online and you know I I do have concerns I mean sure you’ve all seen what’s happened at reddit and other places recently Cora is another example so whenever we have something remote anonymous we get in these online forums there’s just such enormous gaps for bullying or other types of distortions I mean we see it with our five-star ratings for conference talks we see atrocious things happening last week was actually a extremely unfortunate example of that so I have to understand what are the mechanisms for social reinforcement for actual beneficial behavior if you will but I think suffer coffin carpentry is awesome I’m just sort of thinking out loud about about that kind of pure relationship at distance does that g yeah yeah yeah I absolutely I mean I again very much Greenfield I don’t claim to be the expert in this but I have seen those kinds of programs and I believe that there was some over skype i also i personally i participated in some language training like pure language training over skype and and it’s great and I and I look forward to that I also see that kind of behavior with my kids interacting with friends online through skype and Instagram and whatnot so no I’m really hopeful that I mean a lot of the social channels can be can be built out it just I i think that when i see you know chat forums like we use for webcast for instance they’re fairly flat we need to be able to make that richer that’s a good one I think we’re actually blown past our time for the talk so in the interest of keeping the video link that the sort of the same way we’ll go ahead and maybe soccer but please encourage I want to come down ask questions to Paco in a person also thank you