The Vault API – Part 4: Job Server Customization

all right let’s begin since we’re a bit behind here sorry about the technical difficulties but welcome welcome to the special Valentine’s Day edition of the vault API now when it comes to romantic you really can’t beat the job server it’s really the most romantic thing and actually while we’re at it I mean if you’re looking for the perfect Valentine’s Day gift this year don’t go with chocolate cirrhosis or something like that why not give the gift of vault this Valentine’s Day anyway a bit about me my name is Doug Redmond I’m a Gemini I like long walks in the beach my hobbies include computers computer programming and vault by the way if here if you’re viewing the recording just pretend it’s Valentine’s Day then the jokes will make a lot more sense um anyway let me kind of go over the the series so far so part one was just kind of an overview of some of the programming rules the different areas of customization from like a high level and then parts two through five I drill down on two specific topics so part two talked a lot about communicating with the vault sir sorry um communicating with the vault server part three was about customizing the voltage for application you know how to add custom commands and things like that today’s session will be about the job server and I put the skill level at intermediate because dealing with the job queue I wouldn’t call it hard but it it it can be a stumbling block if you’ve never really dealt with this sort of thing before and even some of the professional programmers stumped on some of the concepts so so yeah that this is you know I’d call this an intermediate class and then part five is about the event hooks that we’ve put in to the vault API and that one is more geared toward professional programmers and yes I am recording the session all the sessions are recorded I will be posting them all when the series is completed so and and I’ll be posting everything the the sample code I go over the slides the video it’ll all be up there online on my blog okay a bit about the class today so the goals of the class I want to kind of describe the architecture of the job server I want to kind of show you how you would put your own jobs on to the job queue and then how you would basically take those jobs off the queue and execute them the requirements for this class you have to know what vault is I will talk about if you’re not familiar with job server I am actually gonna kind of explain what that is but still it’s self Alif you know what vault is and the features and things like that also some level of programming experience like I said this this is a bit more in-depth or at least the concepts are a bit more difficult than in the past couple sessions but still if if you’ve been programming for a while you should be able to pick up on it so this is kind of a brief outline of this session so I’m gonna start with an overview as always so I’ll talk about vault and then I’ll kind of drill down into the job server I’ll do a video demo again I’ll I’ll be playing a video showing you how you would build this this customization from scratch and then I’m gonna go deeper into the job server and and the various programming things you need to worry about when you’re when you’re customizing this and then we’ll do Q&A at the end so there should be a questions box and you can type in questions at any time so I I won’t be able to get to them until the end but feel free to ask questions at any time

or at least type it in okay moving along okay so let’s start with with an overview here so we’ll start with the vault architecture this is the same kind of diagram I’ve been showing for all the sessions and in this case we are going to be customizing the job processor which is a utility that’s part of the client that is basically for taking jobs off the queue and executing them so that’s the component we’re plugging into now the feature is called job server but if you notice this box is on the client side and that’s just it’s just because that us as developers sometimes the technical meaning is different than the user facing meaning so from the users point of view you know when when they’re doing something and they’re putting you know they’re doing actions and things are going on the queue in the background in the users mind it’s happening on the server but us as programmers we know that it actually doesn’t happen on the server it happens on client side and it’s a special client called job processor now of course you can run job process around the server but you know from from kind of a technical point of view job processor is a client to the server so again this follows the same rules that I’ve established before in terms of server communication all communication with the server is going to go through the web services API so we’re still all those rules that I’ve told you before still apply here the job queue doesn’t use any type of backdoors or anything we’re still doing client-side programming okay so the point of this whole session really is to explain how you would create your own job types put them on the queue and then take them off the queue and execute them so those are kind of all the things I’m going to be going over here now to talk a bit about the the feature itself the job server feature so this feature is available in vault workgroup collaboration and professional it’s not available in base vault but the idea is you can take work and you can distribute it to other computers through the use of a centralized queue and that key was maintained by the server so the the queue is actually on you know a DMS and you interact with it through the web service calls now I should probably explain what I mean when I say job right well one way to define job is well it’s a thing you put on this queue but but the real meaning is it’s supposed to be a kind of a well-defined unit of work so like out of the box we have a couple of them of jobs predefined so we’ve got ad whiffed generate job which will take a CAD file and generate aid with for it I believe there’s a property sync job which will take vault properties and write them into the CAD file and I think we’ve got a revision block job for I know it’s for inventor files I’m not off the top of my head I don’t know if it supports other types but the idea is you you have your revision data and you can write it inside of your inventor drawing in in the you can create a block inside of the drawing and it will fill out all the revisions for you so so those are kind of examples and the ideas well if you want to do more you can kind of define your own now here’s a part that actually stumps a lot of people even people who have been programming for years and years and decades when you do this customization there’s two parts to it part one you have to get the job on the queue part two is taking the job off the queue and doing something with it if you do one of these steps and not the other you you don’t have anything right if you only do step one the result is gonna be a bunch of jobs sitting on the queue forever and if you only do step two you’re gonna have code that never executes because no jobs ever go on the queue for it to handle so you know it it sounds kind of easy when I when I say it but when you’re programming it it does

feel a bit weird it’s kind of like mailing yourself a letter instead of doing the thing right away right let’s say you want to generate a PDF file right normally you would just code up the logic for for doing that but when you’re doing when you’re dealing with a job server you have to write some code to create the job and then you just turn around immediately and write the job or you write the code to actually do the work so when you’re writing it up it feels kind of weird but of course you have to know in your head okay these things are going to be running on separate components and at separate times and on separate computers and you have to kind of mentally get that break in there but but but yeah this is this believe it or not this is the biggest stumbling block is people will forget one of these two steps usually it’s the first step people forget about so let me talk about the first step getting the job on the queue and this is the easiest part I don’t really have many slides on this so there’s two ways of doing it way number one is you use the web service API you go to the job service and you call the add job function now I’ve been showing you you know various customization techniques so you know where you where you make this add job call you can do it for many places right you can maybe have your own exe that can put things on the job or maybe you’ve got like a custom command involved Explorer that could add a job and then you know on Thursday when I talk about event hooks you can actually do this from your events so there’s a lot of options as to when to call this code but the basic idea is you you explicitly talk to the server and you you create that job the second way to get a job on the queue is to do it on a lifecycle state change so as part of the SDK there’s a utility called the life cycle event editor and you use that to configure the server and tell it which jobs are going to get fired on which life cycle transitions so let me talk about that a bit more so here’s kind of what it looks like and in this example I’m kind of showing a file you can do you can do this for files items and change orders but so in the case of file you have to select the life cycle definition at the top so I’m saying for the flexible release process whenever we go from work-in-progress to release I want this job to go on the queue this job that that the type is autodesk custom job type so what that’s going to do is through this tool you you can figure everything when you’re done you’re going to hit there’s like a commit button so you write the changes to the server and the server is going to remember that and what’s going to happen is that any time a file goes through this transition work-in-progress to release the server is going to queue up a job for you you don’t have to call add job explicitly the server will add the job ok so just some more bullet points so I mentioned this tools in the SDK the source code is also in the SDK so if you kind of want to if you want to explicitly make these settings and you don’t want to force the user to have to use the tool you can you can look at how it’s done in the in the SDK again all of my communication with the server’s through the web service API so you know feel free to to borrow that code if you want to so yeah I mentioned this this utility doesn’t actually create the jobs itself it just tells the server when to create jobs so the server will fire will create the jobs for you also another kind of nice thing is that it’s part of the server operation so let’s say that the life cycle transition fails then it’s not going to queue a job and and this is guaranteed either either the transition works and you get a job or it fails and you don’t get a job you’re never going to be in this weird state where you get a job but the operation failed or something like that so that’s kind of nice you have kind of

a guaranteed delivery mechanism ok so that um that’s all I have on getting the job on the queue the rest of what I’m going to talk about is mostly about executing the job so the recommended way to do that is to write an add-in to this job processor Exe utility which is part of the client install you’ll just get this automatically if you look in your your client Explorer folder you’ll find job processor exe sitting there so we’re going to build an extension and this is this should start looking familiar right now this is going to be a dotnet DLL and inside of it you have the freedom to do pretty much anything however we do discourage you I and the reason is because job processor exe is supposed to run as a service it’s not really a service but you’re supposed to think of it like a service so the idea is that your vault admin is going to start this thing up on a computer and then the computer’s just going to run on its own forever so if if you’ve got your plugin and your plugin pops up a dialog well guess what you’ve just locked up job processor because no one’s going to be there to click Okay on it so please don’t have any UI but everything else you can do connecting to other systems you know writing to files on disk all of that type of thing you you have the freedom to do that and again because we’re in dotnet you’ve got your choice of languages that you want to use so you know I’ve talked about this thing called a job so let’s get a bit more technical if if you’re working in code what you know what does that look like hopefully you can read the text there it’s kind of small but I kind of printed out what the class looks like so the class is called I job and it’s got some parameters I’m not going to go over all of them but there are some ones that are really important so job type is very important this basically is it indicates what task you want to do and this jobs get routed based on their type so if you wanted to write you know something that creates a PDF file you would you would make sure that the job type indicates that and then when you write your handler you you have to look for that string that you set so so you had job types just a string that you set and it’s used by the vault framework to route the job to your handler now as far as vault the vault server is concerned it doesn’t really care what string you put in there we do recommend though you put your company name first so that way you don’t collide with any other customizations that might be in place but but yeah this the server really doesn’t care about it and and this is going to be a theme for a lot of the job queue you can you can put things on there but the server is not really going to do any type of enforcement for you the other thing that I want to point out on here is this thing called params which in this object is a dictionary it’s it’s basically a set of name value pairs and these name value pairs tell the handler well that they provide information that the handler needs to do the job and what what that is exactly is different for each job type so again if we’re talking about a PDF generator what would we need in in here well your we would need some way to find the file that we want to generate a PDF on right so you’d probably have the vault file ID in here or maybe the path to the file or something some way to uniquely find that file and that would be a required parameter if if you didn’t have that your handler would just have to throw an error right let’s see what other parameters might we have maybe there’s different PDF you want to have like the size of the page or something that could be a parameter but maybe it’s not a required parameter maybe it’s optional meaning if if if a job comes in and that parameter is not there well you’ve got some default settings and and you don’t have to throw an error so so that’s just kind of an example of what you’re meant to do

with this so um and and it’s perfectly fine to have no parameters to it did you know it depends on what you’re doing so you know the this the set of parameters is not fixed to something you have complete control to define and again the server is not going to care about it is you know it’ll just kind of store the data and then give it back to your handler but it’s not going to do any type of validation check or anything on this data okay a lot more just kind of the filling in the gaps here going going deeper into all the little intricacies so I mentioned before job processor exe is supposed to be a service we would have liked to have done it as a real window service but there were some technical reasons we couldn’t so it’s it’s it’s own exe that has to be launched so job processor works by polling which means that every you know it’ll go idle for a certain period of time then when that time is up it’s gonna check the queue if it doesn’t get anything on the queue it’s gonna go back to sleep for another time period the time period is 10 minutes by default if it way sup and there’s jobs on the queue it’ll just execute them one by one until there’s no more jobs on the queue and then it goes back to sleep if you want to change the interval you can do that in in that job processor config file but it’s always that the quickest you can make it is one minute now if you want you can have multiple job processors running on a network so that’s perfectly allowed and if you’re doing bulk operations sometimes that’s that’s a good idea like you’re doing a you know it’s the weekend you’re loading a bunch of CAD data in there you want a bunch of dwith files so you take all of these idle computers you have and you you start up job processor and then they can all just chug away at whiffs for the weekend or something like that you can specially you can specimen specialize your job processors so that each processor handles different types so again in that config file let’s say you want something that only does with generation you can comment out all the other job types so that it only does with and then he you know that that processor is nice and specialized okay here’s one that that nobody’s really happy with when the job processor is running it’s going to consume a license when it’s idle it won’t but if it’s actually doing something if like if it’s creating a whiff in order to create that to whiff and then check it in it has to be logged in to vault and like I’ve said in my in a couple of other sessions when you log in it consumes a license because the job processor follows the same rules that every other app follows a license is going to be consumed for this so that’s something to be aware of and then my last point on this is that you for a given computer you can only have one job processor running at a time you can’t just create four of them and and do things in parallel its computer can only have one running okay so a couple of things about using the queue I’ve mentioned before that the vault server doesn’t really care what you put on the queue that’s it’s it’s your burden as a developer to make sure that the code that puts the job on the queue matches the code that takes the job off the queue if like I said if you if you’re cueing me up the job and you make a typo that’s it’s not going to get routed properly or if you forget to put a parameter on there that’s critical you’re gonna get a bunch of errors so the burden is on you to make sure all

that is lines up at runtime okay the thing about the queue is it’s asynchronous you when you queue it up when you call a job it’s gonna go immediately on the job but you don’t know how long it’s going to sit on the queue maybe you’re first in line which means that the next time job processor wakes up it’s going to get to it or there might be a thousand things in front of it on the queue and it could take days before it gets to your job so keep that in mind if if you’re doing something that needs to happen right away then don’t use the job queue job queue is for kind of distributed tasks or tasks that don’t need to be in real-time and this last bullet is another one that’s kind of hard for some people to wrap their head around the order is not guaranteed so let’s say you put job a on the queue and then later on Joby goes on the queue right so aids before B but doesn’t actually mean that a is gonna get executed and completed first before be like I said there might there’s many job processors on the network one of them could grab a another one could grab B and the one that grabs B is just a faster computer and and you know it gets done quicker right so here let me give you an analogy let’s say you’re at the bank right and every bank I’ve been to you’ve got a single line and then you’ve got multiple tellers and they the tellers just grab the next person in line so you’re in line at the bank there’s a person in front of you so you’re second in line and you know they’re there’s a bunch of tellers so my question is which one of you is gonna leave the bank first you or the person in front of you the answer to the question is I don’t know because you know that depends on what the person in front of you is gonna be doing at the bank maybe the person in front of you is gonna be doing something that’s gonna take a long time they’ve got a big jar of pennies and they’re gonna count them out one by one and you’re just gonna cash a check or something also you know maybe the guy in front of you gets the slow teller right the the new person who doesn’t know how to do anything yet and you know just constantly call the manager and be like well how do I do this thing and it takes forever right and and and when it’s your turn maybe you get the expert who’s much quicker and then you’re out the bank first so it’s the same type of thing with the job queue you you just because something goes in first doesn’t mean it’s gonna actually complete first so if you’ve got something that is order dependent in other words you have to do something to completion before the next thing can happen don’t do it by queueing up two jobs so involved we actually have this case and in in the default jobs there’s this thing where you can sync properties and then generated with because you want those properties to be in the do if so what we do is we we create one job one job goes on the queue that’s that syncs the properties when that job completes it queues up ad with job and then and then it’s done handling that job so it’s kind of a chain one job queues up another job and then later on that do if generate job is going to get executed by a different set of code so that’s kind of how you can do orders you can basically have one job queuing up another job it’s a couple of other ways you can deal with it too okay so let’s say you’re a vault administrator how do you actually manage this job queue we’ve got built-in UI for that so if you go to to vault explorer and I you know I show this in the demo you you there’s a window where you can see the job queue so you can do a bunch of useful stuff you can see what’s on the queue you can delete things off the queue manually if a jobs failed you can resubmit it so so this can come in handy if you know you fix the problem or maybe the network went down or something and that’s why the job failed and then later on when the network comes back up you just resubmit the job and then it’ll work hopefully the next time through and

you can also view metadata about the job with this with this queue window so this is what it looks like not much more to say there there is kind of a hidden feature though let’s say you get into this case where your handler grabs the job which will put it into a processing mode it’ll say it’ll say processing in in this job queue window so so you’ve reserved the job and then you crash the hole or maybe the power goes out or something the the job processor dies and you never get to you don’t get to mark the job as success or failure it’s in this limbo state how do you recover from that we’ve got a command that’s hidden you have to edit the menus and I kind of show you how to do that in the screenshot there’s this reset to queue button and what you do is you you make that command visible and then you run that on the job and that will basically move it back into a pending state so that’s how you handle that case it’s kind of I don’t know why we chose to hide that button but someone thought that was a good idea so okay moving on to a couple other just miscellaneous topics things you might run into okay let me talk a bit about multi site and multi workgroup these are really big concepts they can’t go into them that much in this session but the quick definition is that multi-site means you’ve got multiple a DMS instances running inside of a web server and you’ve got multiple file stores but with multi-site they share a single database then we’ve got this concept of multi workgroup which means you actually have a fully replicated database so you’ve got one install of sequel server in New York for example you’ve got another sequel server installed in Hong Kong and they synchronize data periodically so that’s multi workgroup you can also mix and match between the two just to make it ultra ultra complex so you can have replicated data bases with replicated sites very confusing and the way jobs work is is kind of counterintuitive it when I first started working with it I got stumped on this a bit so when you reserved a job you can only reserve it from the site that cued it up so when you know when you queue up a job with that job you’re connected to a site and that job is bound to that site and can’t be reserved from any other site why we did this I’m not going to go into why we did this we did we do have a reason for it but you know it’s when you’re programming it up it’s just going to be something that that might cause you some frustration so that’s on the reserve now when you look at the job queue it’s going to show everything regardless of what site cued it up and there’s no way to really tell which jobs are from what site so that’s kind of confusing as well because you can you know you can it for example if you have a bunch of jobs queued on another site you can see them all in the job queue you can start up your job processor but it’s not going to do anything because it can’t get any of those jobs and you’re gonna wonder why it might be because of this this site issue and and it’s the same at the API level you call good jobs by date it gives you this list of things but that doesn’t mean you can reserve those things and process them I had the is kind of on the to-do list to give a bit more information in these cases but for vault 2012 there’s no easy way to tell what jobs come from what sites now I should mention because I had to deal with this and it made me quite angry I built something called cue tools which is designed to mitigate some of this pain so cue tools it does a lot of things but related to this multi-site

thing you can actually have this thing move jobs from one cue to the other so for example I created this this job handler that sends out email notifications and our internal vault is replicated to five different databases you know I didn’t want to configure job processor five different times I only wanted to do it for the one that’s local to me so I hooked up cue tools and cue tools would basically if if any of these notification jobs were created from let’s say the Singapore office that would get copied to the office in the United States and I’d only have to deal with one job processor so that’s kind of one way you can work around it is is you can get cue tools and and I should mention that this is this is on my blog which means it’s not really an official Autodesk product it’s just kind of a sample app so if if you use this it kind of just make sure that it works for you and everything but but I haven’t really heard of many issues with it okay another top priority when you add a job to the queue you’re gonna give it a priority value and let me explain what this is for right this is for cases where you’re doing a bunch of bulk operations but you want them to happen in the background right you’ve loaded a bunch of files they need drifts created for them you know up front that this is probably going to take a week or so you just want that thing to chug away in the background in the meantime people are still using vault they’re still adding new data and you want to make sure that the stuff they’re doing gets processed relatively quickly so what you can do is you with priority and it basically is a hint as to which jobs are more important than other jobs so that way the user stuff can get executed it basically gets to cut in line is what priority does but again going back to that bank analogy you still things can still get executed out of order because of how you know the multiple processors get set up or the fact that these processors can get specialized so don’t use this as an ordering mechanism just use it basically to kind of distinguish between things that that are more important than other things okay yeah so the way it works is that the lower the number the higher the priority so priority one is the lowest number allowed so that means is that’s going to be the highest priority I should have put this on the slide deck but the default priority that that we have in in vault twenty twelve is that if it’s like a user operation like a check-in and the check-in triggers like add with job that’s going to go in at priority ten so that’s kind of our little default number and if you do things through an autoloader that’s going to go in as priority 100 so those are kind of the baseline numbers that we’ve kind of established at at Autodesk so you can kind of fit your priorities and based on that so if you think your thing is more important than a check-in operation with Twitter a shin you can have it a priority higher than 10 or if if you think it’s more of a background you can put it in as already 100 so that’s kind of what we’ve we’ve tried to establish once you cue the job you can’t change the priority on it of course you could always write a customization that takes the job off the queue and then puts it on the queue again with a different priority that that’s possible but there’s no way to edit it without taking it off the queue like that okay here’s a nice little tip that I recommend when you write this extension you can actually put more than one extension type in a single assembly so let’s say you want to do a custom command and vault Explorer and you want to do a custom job Handler you can put those in the same DLL yeah so why this is good is because you’re gonna have a bunch of logic but like I said before

you want you want your code to be in sync right when you when you add the job of a certain job type you have to make sure that that text is exactly the same as what the handler is looking for well if they’re both in the same DLL that makes it really easy you can just reference the same exact string or the same class I find it’s good to actually create your own class that represents that job and then that class basically is in charge of queuing itself up and and handling itself so very nice technique very useful so yeah you put it all in one one DLL one reset config file an example already exists in the SDK if you go to the job processor API samples it has this exact thing it’s got a custom command and a custom job handler all in one package so that that kind of leads me to the my my slide on extensions the fact that when you’re extending vault functionality when you’re writing this DLL and this DLL is being loaded into vault there’s this common set of rules you have to follow right the assembly attributes the deployment to program data the V set config file these all are things you have to do now job processor extensions have that extra step where you have to go into job processor exe config but other than that it follows the the same rules that a custom command follows so that again that kind of consolidates things a bit okay another thing that I just feel I should mention we recommend you use job processor but that is not a hard requirement because everything goes over web service functions and we’ve we’ve opened up that API technically you can do everything without job processor you can just talk to the queue directly to to pull jobs off the queue and you know there are times you might want to go that route and in fact I’m currently working on a project that it uses the job queue but we’re not plugging into job processor so I’ll just kind of list some advantages and disadvantages so so one of the nice advantages of job processor is that it’s a centralized it provides centralized administration so if you can imagine that you’re the the administrator of a vault it’s much easier if you only have to deal with one program right then then having five different services running that you always have to you know kind of maintain and and and if something fails you have to kind of hunt down okay well what service broke or what application broke if everything goes through job processor it’s only one single spot it’s one log file it makes it very easy to find problems and maintain and everything so job processor has built-in logging in in it so if something goes wrong you notify job processor job processor writes it into its log file so all of that taken care of and it forces kind of a best practices approach in other words job processor is gonna give you things one job at a time and you have to think in isolation if you’re building your own app you’re gonna get really tempted to do things like scan the whole queue and collect up like like you know if you’re expecting jobs to get queued in Paris like job a and job B will always get queued together alright you’re gonna kind of be tempted to kind of scan through and collect up different jobs and handle them together and stuff and that’s not really how you’re supposed to use a queueing system so if you go through a job processor you’re kind of forced to go this way and and I think that’s a good thing but anyway but you know there are some disadvantages so so let me talk about those of it I mentioned before it’s not a window service which is kind of a big downer because you know you’d really like to be able to do the you’ve got that nice control panel window where you can start and stop the services and and monitor everything and do command line operations so you don’t get any of that with job processor at least not involve twenty twelve job processor is only

going to connect to one server at a time job processor doesn’t really do much in terms of parallel processing like pulling multiple jobs off the queue and spawning other threads to kind of handle things in parallel it’s it’s basically a single threaded application so another thing I probably should have put on the slide is that job processor consumes a license when it runs and that’s one of the big factors in the project I’m working on the thing I’m doing is read-only so I don’t want to consume a license if I don’t have to if if I did it through job processor I would be forced to consume a license because job processor handles the login for you so you actually can’t tell it oh don’t don’t you know login is read-only for me there’s no way you can do that with job processor whereas if you’ve got your own service you you have that ability yeah anyway that’s mostly it for my my talk those are kind of the main things I’ve run into and and kind of the stumbling blocks if you’re getting started if you want more information about you know Vault programming you know the same slide is in my other sessions except for the hearts of course um the sdk is a great resource I’ve got my own blog which is where I’m going to be posting this video the Autodesk developer Network supports everything I’ve been talking about so you can use that help online if you get stuck and there’s a discussion group for vault API topics so anyway I hope you found this informative at this time I would like to do questions so if you’ve got a question please type it in now let me see if I’ve got any questions