 Summer Institute on Advanced ComputationAugust 20-23, 2000 College of Engineering & CS Wright State University Dayton, Ohio 45435-0001 |
|
DR. MIRON LIVNY: I was wondering whether you [Garcia] were talking about me or somebody else, but thank you for your very kind introduction. I must say that I'm a little bit nervous because this is the first time I am giving a talk right after dinner. I know usually I don't like to have to be the first speaker after lunch in conferences, and we'll see how it goes.
I also don't know how much time I have, given that originally it was 8:30. So I'll try to get to 8:30, and then we'll see what we should do after that. I also decided to make it light on technical details and to make it maybe more heavy on concepts and ideas, and at least I will be very happy if, by the end of the evening, I'm able to share with you the ideas and the concepts behind what I am going to talk about commodity computing. And I'll be around here tomorrow, so I'll be happy to talk off-line with whoever is interested in talking about the nuts and bolts of what we are doing. I have enough slides here to keep you occupied at least till midnight.
So, what I would like to get you thinking about and get you motivated about and to go back to your organizations, or go back to your labs, is with this idea that since computing is everywhere, can we make it usable by anyone?
So there are two aspects here. One is how do we really put together or access or bring together this computing, which is everywhere; and the other one is how do we make it to be used by anyone?
And one of my goals in what I'm doing here is to bring every scientist to use one order of magnitude more computing than they are using today, okay? Now, I'm not concerned with those who are using a million CPU hours a year; I'm concerned about those who are doing less.
And my feeling is, is that those in science and many organizations, people don't think aggressively enough about computing because they don't think they can get access to it. And I believe that if you take any scientist out there or any engineer out there, they can do today one order of magnitude more computing than they think they can do. So computing is everywhere, and we want to make everybody use it.
So let me give you an example of what group of scientists that isn't powered by commodity computing can do. And this is hot from the press, and the end result even made it to a small story in Science Magazine. So several months ago we were proud to say, to announce to the world, that the nug28 was solved.
Now, the problem itself is not that important; it is a quadratic assignment problem that the optimal solution, in order to find it, you have to look at half of 28 factorial combinations. And those of you who know some math, will quickly realize that unfortunately half of 28 factorial is not 14 factorial.
So this was announced by a group that involved some Midwest scientists, and after four days and eight hours using commodity computing in Wisconsin, New Mexico, and INFN, Italian Nuclear and High-Energy Physics Group, using on the average 200 CPUs, nug28 was solved. And that is by these scientists coming together and saying let's do it.
Now, the original problem was the nug30 problem, which was actually presented to the scientific community 32 years ago. And this one, as you can imagine, had thirty factorial combinations. How? Because it's symmetric. So this group came in and said, Okay, let's do it. And they took 600 processors in Wisconsin, a 190 processors in Georgia Tech, and the list goes on; and down here, processors on an origin 2000 at NCSA and some more Linux nodes at Chiba City in Argonne. They simply went around and said we would like to use your available computing power or the existing computing power to solve this problem that has been out there for 32 years. We have some smart algorithms, but we still need the computing power.
And just to give you the feeling that it works, in the first day they were already using over 900 machines. There is software behind it, there are tools behind it, but I claim that anyone here can actually do this. You don't need the approval of anyone to get access to these resources. You don't have to do anything. Just get yourself organized and you can do this kind of computing. And in the first day, they had already visited three billion nodes on the branch involved. What they are doing is they have a big tree, and they branch around and they go here and they go through and they cut and they go here, but this tree is pretty big.
Now one of the things that is very important to understand when you are dealing with commodity computing, is that it breaks, and having it falling apart is part of game. When you start using it, when you are working with commodity computing, then it has to be built in to break at points and the recovery from it, because that's the way you make progress. So it's not this world that you can come out and put it in the corner and do it once and everyone is holding it and nursing it and making sure that it works and if it breaks you start from scratch. Here you are moving on. You are moving on.
You are sort of building an agile computing environment. Whatever you can do you, you grab, you use it, and you move on. And since this was built the right way, they were able the recover and keep going. And these are actually e-mails that were sent, and you can see this all happened June this year to move on with the computation.
This is a graph which is fascinating, which is the number of workers that was used by the computation of over 6000 seconds, the first six thousand seconds. And you see here -- and for those in the back who can't see -- it goes up to a thousand. The scale on the left is a thousand. And you can see that the number of workers is changing; up and down and up and down, 300 here, two hundred here. They come; they go. That's okay because they want to do the computation. They have a lot of things to do on the tree, and if they can get some workers they use them. If those workers go away -- and we'll talk later, I hope, if I have time, about this master worker paradigm, which has been unfortunately given the name of embarrassingly parallel. But this is the way to go, and you make progress, and you get your work done, and if you do it right, you can solve it. And these guys, in less than a week, consumed 11 CPU years to solve the problem.
Now, there was a lot of engineering in the algorithm because if you wanted to solve the problem by checking all the combinations, they claim that if you use the fastest CPU, single CPU you can think of, it would be a 150 times the life of the universe to visit 30 factorial, or half of 30 factorial. So this is science in the analyzation area that was done by commodity computing and was empowered by the approach, by the software, by the way of thinking. There's no rocket science here. There's smart science here, but there's no rocket science here, and everyone can do it.
So let me try and take you through a simpler example and try to see how commodity computing can be applied to you. And I would actually strongly encourage you to try this at home, okay? This is stuff that you are supposed to do. Don't wait for anyone else to do it for you.
So a typical problem that I think a lot of experimental scientists are facing out there is, I have what I will call a job parallel master worker application with 600 workers, I mean 600 potential workers, and how can I benefit from a commodity computing.
And here is an example. I have a function that has three parameters to it, and I would like to explore 20 values of one, parameter 10 value of the other, and 3 values of the third, and before you know it you have 600 combinations that you have to check.
And it can be your hardware design; it can be your software that you are building and you want to check if it does what's it's supposed to do on 600 different series or inputs or what have you; or it can be a grid computation that you have a grid like here that you have a value and you want it to compute a function. And each computation is small. It's about three hours on a typical workstation. It doesn't take a lot of memory, right? Today 128 megabytes is what you find on something as primitive as a laptop. And it does a little bit F I/O. A few megabytes here a few megabytes there.
So this is what you have. And the question is: Do you need a supercomputer for that, or can commodity computing do it for you? In other words, how many scientists are out there, who when faced with that will say, no, this is something I cannot do. This is 600 times 3. 1800. Almost 2000 CPU hours. This is not something that a small scientist in a small lab can afford doing. This is something that only somebody who has access to a supercomputer can do.
So everything in life goes, if you want to do something and take advantage of what's available around you, you have to get organized. So create 600 input files so you have everything ready with the right X, Y, Z values, and also write a script. Not only to create them, but also write a script to collect the data when you are gone, all right. Because if somehow you can get all these 600 jobs, or these 600 workers do the work for you, then you would like to have a small script that comes in the end and collects these laid eggs and has a final dish ready for you.
So what you have here is very simple, but it's possible to understand it because it's a master work application. It's a parallel application. Because you did a fork potentially of 600, then you do a join at the end. The fact that each of them is a job, fine. But don't let anyone come to you and say, oh, this is not really parallel computing. This is not the real stuff. We are doing the real stuff. Cause I can show you later that doing this right is not pretty.
But if you think about it this way, then if you take your small machine that you have on your desk and you say, okay, I will use this as my commodity computing environment, and I'm willing to install some software that will manage it for me. Then I can give this work, this computation of 600 sets, and then modify it when I'm done to this software. And I don't have to worry about what will happen if something will break or crash or whatever, because I have a software that will manage it for me.
And the reason why I'm saying go on long vacation, is that typically if you have something like this, you will start writing these scripts and then you will be concerned about the scripts crashing and you will put something in wrong and every time you wake up in the morning you will log in immediately to see what is going on. And there's no reason for that because there is software that can take this burden away from you and will do the work.
Now, if you trust this software that I'm calling here a "Personal Condor," then it will keep an eye on your jobs. You can even tell it to tell you mail when something went wrong, or it is finished with a certain amount of jobs, or whatever you want. It can even implement the policy of the order in which you want to do it because you might say that out of these 600 runs, some of them are more important and I would like to see earlier, and the other ones I want to see later. It makes sense when you're doing some type of a great computation. You can specify.
It can even help you, and that is second point here, to decide when this work should run on your commodity computer when, you know, because you need it for other purposes and you don't want this background computing to compete with the other work you have to do.
It's the same way within a car. You want your air conditioner to get disengaged when you need to go faster; you want your background computing to go away when you have to do something else, and this is what a scheduler like Condor can do on your machine. And it can add also fault tolerance, because if you link it with the library it will do check-pointing for you and stuff like that. If you take the jobs and put this work into different environments, you get a lot of benefit even if you're just using your local commodity resource.
But that's just the beginning of the story. But before I go into the story, I'll show you a small movie. So you have your workstation, you turn it into a Personal Condor, use send to it your 600 jobs in a queue, and it's running on your machine. Okay. So that's step No. 1 of commodity computing. And you will say, okay, what's the big deal. You promised me thousands of machines. But this is very important, because by taking your work and putting it into the hands of a system that you trust and can manage your jobs, now you can move beyond the boundaries of your machine to other places. But you need something that is reliable, trustworthy that will do the work for you, because if it's all a bunch of scripts and rubber bands and duct tape and all this kind of stuff, it will go nowhere.
Step number 2. You have heard a lot about grids, and tomorrow you will be grided in the afternoon to death. And you want to call it build your personal grid, or you want to call it build your personal Beowulf or your personal cluster, or whatever it is. So you have your machine and you have it running and you're happy. And now you start moving around in the organization and you see a machine over there that's really not used all the time, machines in the classroom, nodes in a Beowulf downstairs that is not used all the time because they are trying to fix the fancy network that they put to connect it. Don't take it personal.
And you have an O2K, and, you know, wow, it's running at 80% utilization. This is great, but there's 20% of the nodes that are available for this kind of funny work we are talking about. If you can install of these resource, this software, that allows you to access them, then you can build your own small commodity computing environment. It goes beyond the boundaries of the machine that you have on your desk. The only bad news here is that you have to shorten your vacation because it will take you less than two months to compute these 600 jobs.
So, here are the other machines that you installed the software, and without any changes to what you have done, you started on your machine a month ago, it's making progress. Now, you've added these other machines, and it suddenly moves faster.
In the example that I showed you earlier, sometimes the machines in Georgia Tech where there, sometimes they were not there. Any time they came in, there were more workers. When the link to Georgia Tech was disconnected or a machine dropped or whatever, they were gone, that's fine. We are willing to live in this world, right? When we have a bunch of things to do, we look for people to do them. If it's really important we give it to two or three. If somebody doesn't come back fast enough, we give it to somebody else. Somebody resigns in the middle, it's not the end of world. We have to build or computing models.
How does this magic Condor system do all this? And this is a little bit of technical details, but the important thing is to get a feeling for the components here. So on one side, the top, you have the application, and at the bottom you have the resource. And really the only thing that matters here is how do you take the application who wants to do work and the resource, which is somewhere out there on your toaster oven, on your palm computer or on your super computer, it doesn't matter where it is, and you bring them together so they can do work, right?
So unfortunately we have to put all these layers in between to do the management and to make sure everything works according to the right policies and also to make sure they find each other. So the application needs an agent that represents it. You know, we are now in the agent-computing era. Everything is an agent. The truth is that a lot of these are daemons, but we cannot call them daemons anymore because everything has to be an agent today.
But we have an application agent. And the application agent represents the application. It says what the application wants, what it needs, what's it throughput. The application is owned by a customer. A person who may own many applications, and therefore it needs an agent. So one way to look at it is the application agent is working at the level of tasks, and the customer agent works at the level of jobs. So whatever is coming from the application goes to agent, the agent goes to the customer agent, and the customer agent is looking at all these children that it has to feed. This is coming from above.
Coming from below, the resource has somebody who is managing the resources. It can be an operating system; it can be a batch system; it can be the fire department. I don't care what it is. It needs a manager to decide who can go, who can do it, when, what.
What is important to understand in commodity computing is that every resource has an owner. The days that all the computing of the university was owned by the computing centers are over. So we have always to think of every resource in the world it has an owner, and the owner has ideas, has policies, has constraints. And it doesn't matter whether you agree with that or not, this person is the owner of machine; and therefore, has to be represented by an agent that carries out the will, the policy, the spirit, whatever you want to call it, of the owner.
Now, between these two you put an environment agent, and we'll talk about it, I hope, which is really a matchmaker. Because what you have on one hand, you have these applications running around saying we want to do things. Who can do it for us? And on the other side you have these owners saying, you know, I want people to use me but under these constraints and all these kinds of things. And what happens is, is that therefore you need the information from here and here coming to the environment, and the environment brings these guys together and connects them and then goes out of picture, exactly like look a good matchmaker. The two of you are a match, now go to the corner and resolve your problem. I'm out of here. And then eventually, as it percolates down here and it percolates up there, the application is connected to the resource, and happiness.
Now, if you build everything this way, then the story that I am telling you can evolve the way that I am telling you, and eventually it even works, as I showed you in the very beginning.
So this is another representation of this architecture in the Condor environment. Some of this will, I think, be introduced later during the institute, and you will see some examples of Condor, and you will go through the tutorial. But the idea is on an executing machine you have the resource agent and then the customer agent, or the submitting machine, and the submitting machine has here a description of the resources it's looking for, and an environment agent, which can be on another machine or any of these machines, is on the one hand the collector. He's getting the information in all the time.
There is some similarity between this and the BCDS, the META, the Data -- no. They found it in Globus. It's called the BCDS, right?
AUDIENCE MEMBER: Right.
DR. MIRON LIVNY: Do you know what it stands for?
AUDIENCE MEMBER: Better Computing Directory Service.
DR. MIRON LIVNY: Better Computing Directory Service. It actually collects all the time the information about what is going on, and then we have a negotiator, which is really the matchmaker, that brings them together. So these are the components of the software if you get to the point where you are looking under the hood of what Condor does and you start reading, you know, install this, install that, so you know what's happening.
Now, I'll give you a brief feeling of what's happening once there is a match. Because what we have to do, is we have to take the application that's running here and put it on a machine that has to execute on the other side. So once a match happens between the customer agent and the owner agent, they talk and they say are you really who you are, and do I want to deal with you and all these kinds of things, and they start processes underneath them, and the application agent is running on the submission side. Here is another layer that I didn't talk about earlier, which is an execution agent. Those of you who are familiar with Unix can look at it as a remote card. This is really a piece of software that's representing you on that machine, and that allows you to run the application underneath it.
Now once this is in place, the application agent is moving the execution file over here, puts them on the disk here and starts them to run here. This is a limitation of Unix because you cannot start a process; you cannot exact a process over the network. It must be from this. But once it starts running the I/O and the check-pointing information, which is the state of the computation as it was the last time it was running, is communicated from here to here to the application process.
Because what is very important in a commodity computing environment, as I pointed out in the very beginning is, the machine is gone. The worker just decided to switch to a startup. What do I do? Can I get what this person did, save it, and give it to somebody else when I find another worker, or are the units of work small enough that I don't care what I lost. But if the unit of work is big enough, you would like it be able to checkpoint back here and then make sure that the application at restart can get the work that was done; otherwise, you are not going to make progress forward.
And this allows you, this architecture, to run on any machine regardless of whether it has access to your files. So as long as you find a CPU that fits your requirements with enough memory and the right architecture and the right operating system, happiness. Because this infrastructure that we provide makes you run here and feel at home because we generate for your surroundings, that everything around you looks as if you are running at home.
So that was Step No. II, and I hope by you now you have some kind of a fuzzy feeling of what are the layers and the stuff in between that support this merger. But if you believe that all that I told you so far works, I'll show you how we can go bigger and bigger, further and further away from your single machine that we started twenty-five minutes ago.
Okay. So, I know that this is not what you were taught in kindergarten. You're not supposed to take advantage of your friends, but I think in this case it's okay. Let's assume that somebody else did what you did and created this kind of a Condor. So you have your own environment, and your friend has his or her own environment. Now, what you can do, is you can take your customer agent and what we call flock, because we're now creating a flock of Condors to another environment. If I know how to talk to one environment, if this environment is not good enough for me, I can talk to the other one as well, right? And another one and another one. That's the way the next thirty people did what they did. They built their own personal Condor, and they talked to Wisconsin. And when Wisconsin didn't give then enough, they went and talked also to Georgia and whatever. They don't care where the CPUs are coming from.
And if you configure yourself to flock, then you get into this picture. There is a friendly Condor pool out there. You flock to it, and some of your jobs start running better. Note that this pool is not the same color as this one because this is not part of your Condor pool. It is a friendly one that allows you to talk to them. If they don't allow you -- but know that it's just a matter of permission. You don't need passwords. You don't need accountants. Just they have to agree to talk to you at the level of the machine, because nothing that I described so far requires you to be yourself when you are running on the remote machine, because you're not doing anything as yourself. Everything that you do as yourself is on the submitting machine.
So, now there is this infinite amount of computing that the grid is bringing us. So can we go beyond what we have done so far and use resources that are not under our immediate control, like the machine next door or the machine in the classroom? Not under control of a friend, but they're out there in this thing that's called a grid. So if you want to do this, you have sort of to add some functionality to your Condor, and that's what we'll call Condor-G, which is Condor with some grid-enabling technology.
And I think this will make much more sense to you by the end of the session of Globus tomorrow afternoon when you understand what does it mean to submit jobs to the grid.
But they idea is that there is a grid out and it has resources. And the question is: How can I bring these resources into the framework that I described to you earlier? Because if I can bring them into the framework, then suddenly they become also candidates for my workers, and that's really all I care about. So by bringing in Condor-G to your environment, you can submit grid jobs, which means submit a job to your personal Condor and these jobs will transform themselves to run on a grid resource.
And then we have something that we call a glide-in, which really follows the principle of if you want recursion in mathematics. Because recursion is very nice, right? It gives you a lot of power. What we are going here is, I'll talk about it in a minute, is we are using grid universe jobs in order to glide-in Condor into grid resources. We are also using what I mentioned earlier, the BCDS, which is the directory services, in order to help these jobs and submit them and to find their destination.
What's the idea of the Condor-glide-in? The grid provides us with mechanisms to get our jobs running on grid resources. So I can submit my job to a resource in the grid, and what makes things more interesting there, is that when I get to a resource, it's not that I start running them right away, because there can be a scheduler there that decides when I will run, so the grid will take my job and will put it in a queue waiting to run whenever my turn will come on a remote machine.
The glide-in idea is saying, okay, that's fine. But what I want this job to do for me is not to run my applications, because I don't know when it will happen, and I don't know which application I want to run at this time. I don't know which worker is important for me. I don't even know if I will have a worker at all. So what I want this job to do is to really turn this resource into a Condor resource, like the one that I had before, and come back to me and say, hey, I'm here. What do you want to run?
So my grid job is not the work that I want to do, but the ability to transform the resource that I got into a resource that looks like any other resource in my Condor pool, and then I can make a decision. I want to, I don't want to, what should you run on it? Whatever. This is the idea of the glide-in.
And the big advantage of this is, rather than sending my dear work out into the wilderness not knowing when it will run, I keep everything at home. And what I'm sending out is these scouts to find me a resource, and when they actually find a resource, they call back home. The owner level is -- the owner agent is sending me these messages, and then I can do a match and I go over there.
So what is involved in, you know, in adding grid resources into your commodity environment, or in your Condor pool, or whatever you want to call it? First of all, you need to get access, which are accountant certificates on grid resources. Because the model here is different, you cannot run on a grid resource unless you have an account in one of these resources and you have a certificate that allows you to authenticate yourself on this resource. And I'm sure tomorrow afternoon there will be a lot of discussion about what it really means and how do you go about it.
But once you have access to these resources, you can submit, and I'm putting here 599 jobs, which are really glide-in jobs, then you are saying, okay. I know for sure one job will run on my machine. I would like to get out of the grid 599 additional workers.
Now, let me point out that you don't know when they will show up. You don't know for how long they will show up. And part of what may happen is that when they show up, you will say, sorry, I'm not interested in you anymore. But that's okay. You want to go out there, you want to get in the queue, stand in line, so whenever they will deliver to you something, you'll be there ready to take it.
The trick of computing on the grid is always to have work waiting because you always want to be ready when something shows up. Every time something showed up and you were not ready, it's a missed opportunity. It's very different from the way that you do computing on this protected, isolated machine that they call supercomputer because then you got the block and it's for two hours and you know exactly what's happening and after that. Here, no. Here the time you are always waiting for something to happen, and you want to take advantage. You didn't take advantage, you missed. If you can do it, then probably it can be done with your computing in two and a half months in an afternoon.
So here is the picture. Here is the grid, and this grid can have resources which are managed by LSF, which is run by system PBS, which is the portable pack system from NCSA, and it can be another Condor pool that is not friendly. They don't know about you. But they found out about the grid and they were somehow were willing to give you an account by whatever mechanism.
For example, the NCSA alliance is going step by step to the point where people will get an account on all the resources in the alliance via one certificate. So you don't have the password there because the certificate is actually filling in for the password.
So what you do is, one again, you submit all your jobs locally. And that's very important. All your stuff, all your jobs, everything is sitting here. You're in charge. You don't depend on anyone in terms of management. You depend on the resource, but as long as your machine is working, you are fine.
You connect to the grid, to the Globus grid, and you get your glide-ins running there. Right? Because the jobs that you send out eventually will run on these nodes and these are the glide-in jobs; this is not the work itself. Once they run, they transform themselves into machines that look like the machines in your Condor pool, and then your jobs can run and you see how your computing environment is growing step by step.
Are there any questions or comments before I go to the next step? Okay.
What I would like to do is share with you the way I see today. If we have time at the very end, I will tell you about the history of the project. I have been doing this work for over fifteen years now, and not everything was clear fifteen years ago, but being scientists, we have to look all the time at what we are doing and try to generalize and understand how the thing that we actually do to solve a problem come together in the form of concepts. So these are concepts that I believe in them today as I look back at what we have done over the last fifteen years.
So the first thing is what I've been talking about already, and it's the fact that hardware is commodity. It is cheap. It's dynamic. You don't have control over it. Anyone in this world today owns computing power. And it's not only that the system is distributive, we have distributive ownership, which is a concept that we have been believing in from the very early stages of our project and what is really differentiating us from anything else that was done then. It is really the understanding that every resource has an owner, and you have to understand the owner if you want to build a commodity computing environment.
It's heterogeneous by definition. Even if you buy today a homogenous system, one of the components breaks, you have a heterogeneous system. Because you replace the components it's different. And if you're running in an environment where you're willing to work with anyone, you don't have control over what they buy and why they buy it. Whether it's because they liked the salesman of a given vendor, or they gave them a better deal, or they're stupid and they bought the machine just because of the name. It doesn't matter. The system is heterogeneous.
Not only is it heterogeneous; it's evolving all the time. Because the machine next door will change. The machine on the grid will change. They upgraded and they moved it and changed it. It's changing all the time. And you have to be prepared for these changes. And part of these changes mean -- like we have just discovered a month ago when one of our users at Xerox said, you know, we are running on Sun. We tried to evaluate on Linux. We would like to run Condor on Linux. Oh, sure, we are running on Linux, no problem. They installed it on RedHat 6.2. And they ran Condor status, which is, I don't know, maybe 1500 lines of code, and the machine hanged. I mean, the kernel hanged.
This is dealing with software evolving. How do you go back to a customer who is using your software and the machine hangs as a result of it? Now, it's obvious to anyone who has taken a basic course in operating systems knows that no application caused the machine to hang, right? It's only because there is a bug in the Linux kernel, and if you go three times and try to establish a connection that's asleep between them, the kernel will hang. This is dealing with software and hardware that's evolving, and if you're not prepared for it from the beginning, you will get nowhere.
This is a quote from my dissertation, and to be honest, I'm still surprised that I wrote it in '83. And what I feel is extremely important here is the concept of a community. It is not the system. The system is sort of a top-down organization. A community is motivated by the interests of the components to come together in order to accomplish something; and therefore, I believe in this grassroots approach to commodity computing. And I always get very nervous when I hear approaches that are top-down, because the only way that this will succeed is if all the parties involved see a reason to be part of it.
In the exactly the same way that communities of human beings are coming together, or for anyone, or whatever you want. Each individual component there has an interest of participating, and if we can build our software and build our computing environment in such a way, I think we are going to win. Because eventually the sizes that we are talking about, look at the commodity aspect, is driving us into the community direction rather than in the system direction. We are losing control over what we have there unless the control is coming from the local interests of the members of the community.
So if you buy this concept of community, you realize that once again, if you're looking at the way communities grow up over the years. I mean, not grow up, but evolved over the years, that communities have matchmakers, and matchmakers are a crucial component in a community because they are sort of providing the glue to bring the members of the community together. Because the members of the community have interests, they want to provide and they want to consume, and the matchmaker is the one that brings them together.
What's interesting is that if you take a look at classified ads in newspapers, which is a matchmaker, right, because it brings the sellers to the buyers. If you look at them, you realize that they are basically symmetric, because you cannot distinguish between a seller and buyer. Because both are interested in finding each other. And if you look at the personal classifieds, there are constraints both sides.
And any approach, which is typical in resource management and computer systems, to say that only the requester has constraints, falls apart. Because if you think about it, every resource has constraints as well; access lists, authentication, scheduling priorities. All these are basically constraints on the resource in terms of saying, I don't want to deal with you now. I don't want to deal with you ever. I hate you.
Now, if you cannot bring this together into the matchmaking process, you can go into a queue that says, okay, fine. I will get you into the queue but I will never serve you. What good does it provide? So you need matchmaking if you want to build together.
So, one way to look at what we have been doing for fifteen years is that we have been using matchmakers to build computing communities out of commodity components. And matchmaking is the cornerstone of it. And if you look at what is really involved in matchmaking, you realize that there are four main elements in this protocol. One, is how do you advertise yourself? How do you send out to the matchmaker a description of who you are, what are your requirements, what are your preferences?
And we have developed something that we call Classified ads, ClassAd, which is a language that allows you to do the above. And the semantics, it also has some other nice features. It includes, for example, the ranking capability. Because not only to you want to say who you are looking for, you want to say also if you can find several; these are the ones that I prefer.
Then we have the Matchmaking Algorithm, and that is how does the matchmaker take the information provided by the parties and comes up with a match? And we have just recently published the results of the new work that is dealing with what we call Gang Matching. Because we don't match only two, but we can match a whole gang. Because we see more and more the need of forming an alliance not only between a requester and a provider, but a whole bunch of requesters and providers in order to get the computation going.
Once the matchmaker provides the match, it has to notify the parties. And there are a lot of issues here because we live in a distributive environment which is not reliable and a lot of things can go wrong. Now remember, that everything that is happening, even at the advertisement step, but definitely the matchmaking and the notification, is based on old information. And that is very fundamental to understand in any kind of distributing system, that whatever you do, there is no guarantee that it's going to actually happen because you are using old information.
And the way I like to explain to my class in Introduction to Operating Systems, is when you take a chair and you want to sit on the chair, you pull it over, you turn your back to it, and you sit down, right? And I'm sure that every one of you had the pleasure of ending up on the floor because somebody pulled away the chair. Because the information that you had about where the chair is, was old information by the time you were sitting down.
That happens to you all the time in a distributive system. So all of these protocols have to be built in a way that I'm not there, I'm not interested in you, I found something better, I don't care about you anymore, and everyone has to recover from it. And once there's a modification, you have to come in and do a claiming protocol in which two parties actually get engaged in the relationship.
Now, if you think a little bit about it, you can bring in the revocation of the relationship or the match as part of this because you can always continue to advertise and look for a better match while you are engaged in a relationship, and I don't want to get into the moral issues of this practice.
You heard earlier about my commitment to high throughput computing. So after several years of doing this kind of work -- and I have to remind you, that there was a period in our scientific history that parallel high performance computing was it. It sometimes is a little hard to believe how quickly things change, but at that time I was looking at what we were doing and looking at the high performance computing activity and I said: What's different between us?
And what's different is that we are realized pretty early that a lot of scientists and engineers are not interested in instantaneous performance, but they're interested in sustained performance. They don't measure things by what can happen in a second, which is the way that you measure the performance of high performance computing, right?
Because if I ask you how big is your high performance computer, you will say ten kiloFLOPS. FLOPS: Floating Point Operation Per Second. Most scientists are interested in how many scenarios they can do in a day; how many wind patterns they can do in a week; how many instructions sets they can simulate over a month; or how many crystal configurations they can do in a year.
Almost in most cases they have more work to do than computing available, so the question is: How much can they do by the time they have to reach a decision? How much computing can they do by the time they have to publish a paper? It doesn't matter what the constraint is. And that is more a question of throughput than a question of performance.
If you agree to that, you realize that you're in the business of a 24-7-365, because the question is: What can you sustain? And I'm sure all of you will agree with me that you cannot take what you can do in a second, multiplied by the number of seconds in a year and say this is what I can do in a year.
Reliability, robustness, portability, all these kinds of things are extremely important here in order to go from flops to flopy. I'm interest in flopy. I'm interested in people who come to me and say I need a million CPU hours for the next year. How can I deliver that? Rather than, I need to do one computation tomorrow morning. It's a heroic effort. We'll do it, and then everyone will go and take a vacation.
So how do we get to the point were we can really doing high throughput computing? By the way, this is a term that I coined in '96, and I see now more and more people claiming that they're doing high throughput computing and it's good.
So the first obstacle, if you want to deliver a large amount of computing, is what we talked about ownership distribution, and that's a matter of sociology. And anyone who is trying to solve this kind of computing problem with hiding latency and nanosecond communication lengths, as my mother said, should go home and help his mother to peel potatoes. It will go nowhere. You have to understand you have to deal the humans with egos, with sensitivities; and if you cannot satisfy that, you will go nowhere.
Now, if you got the owners on your side, you still have a problem with the customers and that's one of the reasons why I'm here this evening. A lot of scientists and engineers who are out there, don't see the power of this kind of computing. They still believe that they need high performance hardware, supercomputers, to do their work. There's too much of an obsession with efficiency in the scientific community. When you think about this kind of computing, efficiency is not the issue.
I just came back from JPL a week ago where we are trying to build an ocean simulator using this kind of technology, and what I was trying to convey to them was that, look, if each node of your computation is running at half capacity, so you have a factor of two slowness, but what you have is another thousand machines, which is two orders of magnitude.
Efficiency is a matter of factors. Commodity computing deals with orders of magnitude. And if you understand that, you will realize that you can do a lot of computing by not being efficient. But we are still, very, very concerned about efficiency. If we are not running at 99% utilization, it's a criminal act. But we are missing a lot by doing so.
So how can we get this into the awareness view of scientists? How can we also convince scientists that you can do very, very good research on machines that you can buy at K-Mart? Not the biggest, greatest machine in the world, but something that every high school child can use. And you'd be surprised how much of a problem that is.
The third problem is size and uncertainty. We as a community don't have a very good track record in building robust software. We, the university, are not doing a good job in training our students to build robust software. Most of our students leave the university assuming that a program that compiles works. What do you want? It compiles, so why are you still complaining? We don't know how to build robust software. We don't understand the case of sophisticated algorithms on the robustness and maintenance of software. And the size and uncertainty of what I described to you earlier today requires robust software.
In these kinds of systems I can guarantee you that everything that can go wrong, will go wrong. And if you can imagine debugging a system that has a thousand CPUs all over the world -- please don't do it tonight because I'm not sure how well you will sleep. It's a nightmare. So you're very careful in the way you write your software, you're very careful in the kind of algorithms you put in because robustness is critical.
And a very close cousin of robustness is portability. If you will think back fifteen years ago and remember all the silver bullets that were promised to us that will be it. You know, if you use this, this is it. And I'm willing to include even Java today as one of the silver bullets. We went from write once, run everywhere; to write once, run never. That's the perspective of people talking about Java. And if you write your software and you are making it very much dependant on the latest and greatest, portability becomes a tremendous problem. So you have to be very, very careful.
We don't use in our system threads, because threads are extremely non-portable, and there are a lot of problems today on Linux with threads. And you cannot build a system and then implement a thread packet for every obstacle. We are running on 14 different combinations of hardware and software, and if we had to also carry along a thread packet on all of these, we would have been dead in the water many, many years ago.
The last one is physical distribution. I don't think this is a big issue. We have networks, and we understand them. We can improve the networks. We can improve some on the bandwidth. I think there are a lot of people working on it. We have to be aware, but I don't think it's a fundamental obstacle in where we are headed. Let me mention that and we will do another check on how we are doing time wise.
So, I believe that these are the four basic mechanisms that are needed in order to do high throughput computing. Let emphasize and emphasize again, mechanism is the key not policies. Because everything I told you there has nothing to do with how smart the policy is. If you have a good mechanism, I'm sure you can come up with a decent policy in half a day or day, implement it and be happy. Any attempt to come up with sophisticated policies will break your mechanisms.
Matchmaking I talked about it. Check pointing I mentioned it briefly earlier. You must have a mechanism that allows you to say use it as long as it's available, and then let's save what is there so we can continue somewhere else.
Remote I/O is fundamental because you must bring your data in and out of the execution site, and if you limit yourself to run only on sites that have your fastest amount of OSM AMAFSL(Phonetic), you are losing a lot of computing power.
Asynchronous API. You must have asynchronous API for managing your resources because you are all the time looking for more stuff to come in. You want more workers. You have more resources. You have to send them out and they will come back to you whenever they are ready. And if you think about your application and you are managing your application, you have information coming in from your worker saying, I'm done. I need more work. I have to do that. And then you have information coming in from whoever you asked for additional resources, they all have to come in through a single asynchronous interface, so you didn't have to listen with one ear here and the other ear over there. They should all come, the messages, the signals, the callbacks, whatever you are using, so you have a uniform user management interface.
Okay. My next concept is Master-Worker, but I'm afraid that you will start throwing things at me. I mentioned already that Master-Worker computing is not embarrassingly parallel. If you want to do Master-Worker right, it's a hard task. And I would prefer that everyone would call it naturally parallel; it also sounds more of the nineties than the eighties.
But it will take away from it the stigma that this is bad or you're not supposed to do it, right? Nobody is going to come home and when asked what did you do today say I did embarrassingly parallel work today, right? It's a very important computing product, very powerful, and nontrivial.
Okay. Let me tell you a little bit about the bird. I was asked earlier why did you call it Condor. So the first paper that we published on this in '88, the title was, Condor, The Hunter of Idle Workstations. So Condor is a name; it's not an acronym. And if you take Master-Worker computing and you take high throughput computing and you take commodity resources, Condor is the system we have developed to deal with all that.
So as I mentioned already earlier, we started quite a while ago. In '86 we had the first production system running in Wisconsin. In '92 we published a paper that summarized our experience of running a quarter of a million jobs. It was originally developed for Unix, and today we have it running on Unix, Linux, and NT.
And here this is an interesting challenge of how to maintain one software free, for all the three systems. We worked very hard to make it happen because all our versions are native versions. We are not trying to make Unix look like NT, or NT look like Unix. It's based on matchmaking technology. It's running all over the world.
Just in Wisconsin, I'll give you a little bit of chart there, we have 1300 CPUs. And if you want to look at it, play with it, complain about it, it's all on the web site of our department at slash Condor. (www.cs.wisc.edu/Condor.)
So here's what happened to our Condor resources at Wisconsin. And the red is what we have at computer science, and the blue is what we have outside of computer science. And in '88 I still remember we were very, very excited when we crossed the 100 line. So we started on the floor, which was basically just computer science at the time, and then we expanded. In '94 we got engineering on board. In '99 we crossed the 600 line.
Part of this increase was the fact that we were able to the take advantage of SMPs on desks. So rather than using 1 CPU on the machine, if it had more than 1 CPU, we were able to turn it into an independent CPU that we can schedule, and that gave us a boost. Here is when we got also the NT machines on board.
But there was also a growth, a growth on where it is on campus and how many machines are coming in from different resources, and that also included 200 CPUs that came in from a dedicated cluster in computer science. And I believe you can do the same wherever you are if you believe in it and if you want to go and make this kind of technology work for your community.
Here's a snapshot. This is actually pretty old. It is October of '98. It's two years ago almost. But this is all on-line. If you go and visit our web site you can see real-time status of the pool. What's interesting here is this is the availability of machines. At the time we only at 410 at computer science. The red shows the number of machines that are in owner state.
Every owner of the machine can be fine using the power of logical expression when the machine is in owner state and when it can be used by others. And what you see here very nicely is the daily cycles how at night it goes down to about fifty machines, and during the day it can go up to roughly 300 to the red.
You see this is the week, this is the weekend. This is the week again, weekend, and so on. Here we have some kind of failure in the system for half a day or we upgraded it. I don't remember.
Now, if we just focus on the red, you first of all see here that the number of machines in owner state was never above 300, which means that even at the peak of the day when everyone is working very hard, over 100 CPUs out of the 400 were available. So it's not just a matter of using the resources at night. Yes, at night it went down to fifty or sixty, but there's a tremendous amount of computing available during the day because people are on vacation, they attend meetings or seminars, or whatever. So if you are after them, they are there.
The green, which gives you also the upper line, is the number of machines that are used by Condor. And, obviously, since I have this slide, this was a very good month and we used almost everything. Where the blue is, is the idle. You can see the statistic here that the owner worked 35%, Condor worked 60 and idle was 4%. So this is out there for you to grab.
Viewed as real numbers, this is roughly two years. We delivered to our community 450 CPU units. And when I say real users, we have sort of jobs that are background jobs with code cracking or 30 at home that we keep in the system just to make sure that it's used all the time, and we use it for testing to see how low priority and high priority interact.
And here some numbers. Some users use half a million, 610 hours, and some users from a business school use 1000 hours. That's okay. Because this person had been fighting on an old VAX that they had in the business school to do 10 hours of CPU, or 100 hours of CPU. So remember, in order of magnitude, if it's from 100 to 1,000, from 1,000 to 10,000, from 10,000 to 100,000, this has a tremendous impact on the scientist.
Usually a scientist who gets an order of magnitude increase in computing cannot handle it. They have to change the way they think. They have to change the way they work with the data. Orders of magnitude are not trivial, but if we can give them a push, they will learn how to take advantage and they will come back and they'll take another order of magnitude.
What we can also do using this technology, as I showed you earlier, is we can let people from the outside run. So these are MIT, Cornell, CalTech. CalTech is about roughly at 100,000, and it goes and goes because there is a huge amount of computing just in this small Condor pool that we have in Wisconsin.
I assume that you will see some of these features in the presentation of Condor later this week. It gives you local control over the jobs because everything is stored in your machine. So we don't come to you and say you cannot submit more than a 1,000 jobs, because everything is on your machine.
We have users who submit 10,000 jobs in one keystroke and go home. That's okay. The largest number that I heard so for is 48,000 that a group of biologists did in order to do gene matching. Because they take every fragment of the gene, turn it into a job, submit it and go home. They don't want to worry about it. Let the system take care of it.
With this priority scheduling you can decide who is running first, whether it's among the users, whether it's among the machines, whether it's among jobs that belong to a single user. We have the job preemption. If you can re-link the application, you can use check-pointing and you can do remote I/O. These are the two last points. So sometimes people make the wrong assumption unless you can re-link your application, you cannot use it on the Condor this is not the case.
If you cannot re-link your application you are limited because you cannot do check-pointing and you cannot do remote I/O. So you're limited to whatever resources that you are using that can give you a long enough period so you can finish the job or where you can access your files.
What we can do is we can provide you with these additional capabilities to do remote I/O and check-pointing. And you can do a lot of smart things about selecting the resources. You can say I want this machine, but if I wait for more than five hours, I'm willing to go to a machine that is more slower and stuff like that.
I mentioned that we can manage large amounts of jobs. We have the capability of managing jobs with dependencies with what we call the DAGMan, which is Directed Acyclic Graph Manager. Mainly, you can take a large number of jobs that form a DAG, and then let the system execute them based on the dependencies and it will do some nice things for you.
And this was also mentioned earlier in the introduction. We have this support for Dynamic Master-Work applications using PVM. The application I showed at the very beginning, the nug30 used the Master-Worker library that we developed to manage the work. So the communication between the master and the worker was all done via the Master-Worker, but not via PVM but via file. Because the granularity and reliability requirements dictated a less efficient communication but one that worked. Because if you think about the way the Master-Worker can communicate, they can send messages over the PVM connection, or they can have files that communicate.
The nice thing about files is that the master can crash and the workers can still do their stuff and the master can come back and collect the work. In a PVM situation, if the master crashes, bye-bye computation. We have to start the whole thing from scratch.
So can you think of a master that runs into trouble, for whatever reason, and has 900 workers and because of a small problem in the master, all these 900 workers have to be called home, all the resources relinquished until the master comes back and then we have to start the whole thing again. It's a tremendous penalty.
By the way, MPI is not mentioned here. You will hear about MPI. MPI has a fundamental problem when we talk about this kind of computing because it's not dynamic. MPI requires you to have a fixed number of processes for the duration of the computation. You come in and you say I need 10 processes. That's it's. Cannot increase; cannot decrease. This is not the way to do commodity computing.
Here's a way to submit a Master-Worker application with 1,000 workers. Assuming that you have created a directory for each of your workers and you put into it a file called in, which contains the input, and you expect to see the results in a file called output; and if there are any errors it will go to errors, and you would like Condor to log for you everything that happens in the job.
Then you tell Condor that the executable name is worker, and the requirement is that you are looking for a Linux machine that has more than 64 meg of memory, and you want to rank it based on the KFLOP ranking of the machine. Then you say queue 1,000 and you are done. Yes, you have to prepare the directory. You have to put in the data. We cannot do the work for you. But after that, we will do our best to do this kind of work.
But that was a job-parallel application. Here is a task-parallel application. Let me use that as an example of the tricky part of doing Master-Worker computing right. So here's an application that we got from material scientists at Oak Ridge. What they do is they integrate over the energy at 31 points in the material for a given potential, get the result of the integration, compute the new potential, and do it again.
So what you have here is you have a Master-Worker application that has asynchronous points, you do some kind of computation and off you go. So these workers, right? Compute the energy at this location. And in this case we said that the master is going to do the integration and is going to compute at the end the total energy. So this is the centralized activity. Then you go out. Then you come back.
Let me ask you, how many workers would it take for this computation? Obviously, you don't want to take more than 31, right? Or maybe you want to take more than 31 because you want to have a couple of hot stand-bys because you may lose a worker, right? Because you have your 31 pieces of work to do.
How many workers you should go to the system and ask for is not a simple question, because it depends on a lot of issues; depending on how much you have to pay for a processor; what level of efficiency you want to run; how long does it take you to get it. There are a lot of questions here, and these are all part of embarrassingly parallel computing.
Now, let's assume that you have a certain number of workers. How are you going to give out the work to the workers? In what order? Think about it in reality. You have a list of things to do; you have a bunch of workers. How are you going to assign work to workers? Not a trivial question -- problem.
So here's what happens when you take something like this and you run it under Condor and using some of the visualization tools that we have developed. And in this case we were running it to the PVM application using message passing.
So this is the visualization, and the quality of the bitmap is not great, but at the top you see the entire computation. The total time to compute it was six hours, and on the Y axis you have the logical worker ID, because remember workers, you may lose them, right? Somebody was an employee and disappeared and you hired another one. So you recycle your worker ID's; otherwise, the worker ID's will grow constantly.
And you have 36 times 31 worker tasks that you have executed. And what you see down here is one cycle of the computation. So you have 31 pieces of work that you give out here. Actually, I have an explanation of this. The green is the first allocation that you give to a worker; then the blue is the second one; and the red was the third allocation; and the black ones are workers who took a hike during the cycle and said we are not interested in you anymore.
What you see here right away is there's an interesting problem, that depending on how you give out the pieces of work, you have all this white space here, right? And there is actually a theoretical result that shows that if you do the greedy approach and if you give up work, the biggest first and the smallest last, in most cases you are optimum. Which makes sense because you want to give out the big pieces first, and then with the small ones you will fill in whatever is left, so you will get it more packed.
What is interesting here, and is again not trivial, is how do you deal with the fact you may have the probability of losing a machine. Because if you lose a big piece, then everyone has to wait for this piece to be redone depending if you have check-pointing or not.
So I'll just leave it as a question that you can think about throughout this week is: What is better, N plus one machines, where I tell you that I will take one away from you randomly, or N machines that are yours for the entire cycle?
So if you look at this curve that shows you the number of workers, there's one important thing to see here right away. Unlike the approach that you take on a supercomputer, here you start running right away. You don't say, hey, give me 25 workers. You say, okay, can you give me one? Let's start working. Another one will show up, fine. Come in. We'll give you some more work do to. Another one, one is going away, fine. We start right away. We don't wait.
One of the things that sort of happened in the high performance computing era was everyone was focusing on execution time. Turnaround time, as Computer Theory 101 will teach you, is the sum of computing time and execution time.
But nobody looked at the question: How long do I wait to get on a machine if I ask for 560 nodes. Oh, yeah. When I get there I run beautifully, but I may wait for three weeks to get on it. If I could build my application and I run all the time, I can do pretty well by running right away. And this is what's happening here. And the total number is changing depending on the mood of the system.
That red is how many machines were idle within the cycle. And you see that there is quite a bit of red here because of the way that the scientists gave out the work. Because if you look at the distribution of the execution time of the tack and the function of the location and the material, the boundaries are clearly much more expensive than in the middle. And the scientist gave them out in this order. And a simple algorithm that will look at the work and will say, you know what, I see a correlation here between the location and the size, let's change the order. We have shown that you can save about two hours off computation.
Nothing sophisticated, but be careful with the way you do it. Even if you would have given them out randomly, you would have done better than the way the scientists did here. But they had the big machine. They didn't care.
So, we have customers who do 5,000 jobs, 1,000 tasks, run for 6 months, run for 4 weeks. It's a real system. It's being used to do real computing. And things happen large scale for a long time and it all works. And it all works on commodity computing.
What's important to keep in mind here is that in a system like this, you have big events. Anyone of you who is doing or thinking about queuing theory and exponential arrivals and things happening, this is not happening here. The blue is the number of total of jobs in the system. Somebody submits 500 jobs. Somebody looks at the results of the first 5 jobs and removes 300. A job that was running with 300 nodes finishes and has higher priority, lower priority. You get these big events, and you have to deal with them.
So nothing is sort of smooth and nice and happening in a friendly manner. And therefore you have to build your scheduling and your algorithms and the application to be prepared for, oh, all my workers are gone. Oh, I have another 500 workers. Let's use them. And if you are not agile or flexible, you are missing your opportunities.
Okay. I think what I will do is I will jump to the end and give you sort of my last message here and maybe open the floor for a few questions. As I said, I will be around tomorrow, so if you feel that you had more than enough for one day, there's no need to get into any kind of discussion here.
It works, and from my experience looking at all the places where we got Condor to run and to be productive, it needs one person who is really committed to it. Who is really concerned about the fact that there are scientists, or there are engineers, or there are analysts, or whatever you call it, who are not using computing because they think computing is not available. And computing is there. It just requires somebody to bring all the pieces together and show some of these individuals the light.
Now, after having said that, I will put here a disclaimer and say whatever I talked about this evening is no a silver bullet. It's not going to solve all of the computational problems. I believe, however, that it can address a very wide spectrum of problems, and if you are creative, it can work for you. Now the good news of this is, that for the part of the community who is concerned about the big machines, if this is adopted by many scientists, it simply will reduce the load in the big machines and leave them for applications that need the big machines. So it's a win/win situation.
Anyone brave enough to ask a question?
DR. OSCAR GARCIA: That was as daring remark. But I do have a question that I wouldn't be able to sleep tonight if I didn't ask. It has to do with check-pointing. How is the check-pointing done? Deterministically, or do you have some sophisticated algorithm, or is it random, or do you put it in? How does it's happen?
DR. MIRON LIVNY: Check-pointing happens at two instances. It happens at the time that you have to vacate the machine, and then when the person comes to the -- you know, depending on the policy. Because we have owners that say --
DR. OSCAR GARCIA: That's the interrupt time. I wouldn't call it --
DR. MIRON LIVNY: But it's still check-pointing, and we have go do it in a constrained amount of time. But we have owners who say when I come back to the machine, no check-pointing, no nothing. Kill it right away. I want the whole thing no messing around.
We also do periodic check-pointing, which is not sophisticated at all and not optimal at all. I think, at the moment we do it every 3 hours. But if you ask me why 3 hours and not 2 and a half, I don't have a clue. And if anyone wants to take the task of looking into the check-pointing activity, we'll be happy to collaborate.
One of the things that we do struggle with is the impact of check-pointing on network consumption, and we are trying to schedule that so we don't get too much traffic on our network. Because some of the checkpoints we are moving around are 150, 200 megabytes of data, so these not small puppies.
DR. OSCAR GARCIA: That's good enough for me. I have other questions, but I'd rather see if somebody else wants to ask any.
Right here. Tony.
AUDIENCE MEMBER: I have a question. I'm tempted to make a comment, but I would like to give you the privilege to answer a question, if you will.
DR. MIRON LIVNY: You can make a comment. That's okay.
AUDIENCE MEMBER: The issue is the big picture. The long-term picture. You told us what the benefits could be at this moment. What happens if you succeed in getting interested too many people in doing that?
DR. MIRON LIVNY: Too many people in terms of customers?
AUDIENCE MEMBER: Too many people in terms of the CPUs, hours.
DR. MIRON LIVNY: Again, I'm going to give you an unscientific answer, but how should I say it? I have been in the trenches for 15 years, and I'm not concerned about it. Because I think the rate at which computing power increases, we will always have more out there than customers ready to use it.
I have seen a lot of scientists who came to me and said give us access and we'll use everything that you have. They did it for two or three weeks, and then they disappeared. Because as I pointed out earlier, when you can do an order of magnitude more computing, you have to pause and you have to regroup.
What bothers me is, as I showed you here, I have 450 years here that I delivered; only 170 of these were what I would call real science. I want to get these other 200 years, or whatever it was, used by scientist that today do less science because they don't have computing.
And I was at JPL. They don't use any of this. They have thousands of machines there. So I think the potential is in the two to three orders of magnitude. If we reach this point, come and talk to me, but I am not concerned at all. And I definitely don't think -- and we are not talking about technology here, we are talking about energy -- that we should stop now because of this fear. Let's get there, and I'm sure that we will find a solution.
AUDIENCE MEMBER: Well, I would not suggest that you stop you now because what you are doing is very useful; it's only going to help increase the usage of the available computing power. But my point is that if you get people to count on that computing power that you are conjuring for them, if everybody starts doing that, the power is going to disappear because if you go back to your second or third Condor slide you have the thread in the green. If you flip it over, you have just to use -- the green area is about two times the red area, that's two orders of magnitude. You use that and you're done. But that doesn't mean you shouldn't be doing that. It's very good.
DR. MIRON LIVNY: I know that I'm still, just in the University of Wisconsin, I probably have another two to three thousand CPUs that I'm not using. And then we can go beyond that. And by the time we will be there, the speed and the availability of these CPUs -- the chemistry department is putting a cluster in the physics department. It's growing continuously. But you might be right, and then we'll have to find some new solutions.
DR. OSCAR GARCIA: One more question right here.
AUDIENCE MEMBER: I was just going to respond to that. There's really no difference between computation and any other resource that we have. And to suggest that we don't encourage people to use all the resources that we have because at some point we'll have to start competing with them, is kind of counterintuitive to the way we've been doing things all along.
For example, if you look at electrical power. You know, in the summers we have power outages because we use more power than there is available. Does that mean we should tell people only to use 50% of the power that's available? No. Of course not. The idea is get more power, or make it, you know, allow supply and demand to make the price change to the point where we basically prioritize who uses it and who doesn't.
In terms of scientific computing power, you know, if you get to the point where you have a 100% utilization of all of your power, what does that mean? It means you have to start prioritizing. And you know, we've certainly been in situations over the last few decades we've had computing power where we've had to ration computing power. Rationing is not a terrible situation; it basically means that you have more work to do than you have resources to do it. You just have to make choices. I don't think that we should aim for anything less than a 100% utilization of the resources we've got.
DR. OSCAR GARCIA: One more question.
AUDIENCE MEMBER: The obvious question about the client's computers is that if the owners are concerned about privacy, about using the machine. Can it be done? What's the situation there? How can we persuade them this is not intrusive?
DR. MIRON LIVNY: The question of security and privacy is a challenge. And, you know, in the same way that by putting your machine on the network you expose yourself to a threat. Participating in this brave world increases the threat. There's no doubt about it.
And what I know is that the National Security Agency has been using Condor inside, and I don't see them opening their machines so that we can run our gene searching on their machines. So there's going to be -- if you want fragmentation in this beautiful world, it's not going to cover all the machines because of security.
And there are various tools and mechanisms, approach and enhancement that go into the system to address these issues, and it's always the question going back to the sociology of what the end user is going to be comfortable with.
What's interesting is, it's not only who is running on your machine. There is another interesting question of: How much do I trust the results that are coming back from your machine? So once again we have asymmetry. If I buy from you, I don't know, a pound of meat. I paid you with money. Is this good money? Is the meat that I got good meat?
And it's an interesting question, and there is, for example, for some -- obviously companies with codes that they don't want to share with anyone, are they willing to run it on your machine. Can you break the code into pieces so that I can run on your machine without revealing the big picture?
So, yes, these are all issues. There are also issues of, as I pointed out in the beginning with the Linux example, if, you know, I put something running on your machine and something happens on your machine, the first thing you will do is blame my software. And we have examples.
Condor now runs at that NASA Ames, and they actually rolled it out very carefully a bunch of machines at a time. But they made an announcement that they are going to roll it out over the next couple of months. And I think 85% of the complaints they got were from people who didn't have Condor running on your machine. Because now everything that was different on their machines, you know, if the coffee maker had a problem.
So, yes, these are all interesting issues that, you know, that's the reason we are scientists and we are not taking no as an answer. I mean, we have to push forward. So to tell you today, and that is part of my -- you know, I don't want to push you people to invest and not invest in some of these new dot coms, but there are these companies that are now saying they will give you ten dollars and you will run everything. I think that we have still a lot of problems to solve before we can do that.
But we have to go into larger and larger groups and do it, and I think we can do a lot of it without being concerned about the security issues within the boundaries of organization or across organizations that trust each other.
You will here tomorrow a lot about the security issues from the Globus side, and we are doing as much as we can to bring their technology into our environment to use the authentication of Condor from Globus. We have a project that's actually done here with our friends at the Air Force base on adding a Kerberos. So there are things that can be done. There are things that can be added. Will it be as secure as having a machine in a file date cage disconnected from -- no.
DR. OSCAR GARCIA: I think it's been a very illuminating and broad presentation. We want to thank Mr. Livny for his excellent remarks. Look forward to seeing you tomorrow. Good night.
| last revised:11/21/00 05:24:21 PM |
| editor: pmateti@cs.wright.edu |