Summer Institute on Advanced Computation August 20-23, 2000 College of Engineering & CS Wright State University Dayton, Ohio 45435-0001

The Globus Project: Infrastructure for Computational Grids Mr Lee Liming Project Manager, Argonne National Laboratory   August 21, 2000. 1:00 p.m.  This site contains the "web-based proceedings" of Summer Institute on Advanced Computation that focused on high- performance high-throughput  Cluster Computing. Slides

DR. OSCAR GARCIA: I'm delighted to introduce to you the Project Manager of the Distributed Systems Laboratory in the Mathematics and Computer Science division at Argonne National Laboratory. Last year we were very fortunate. We had one of the Argonne leaders in MPI, and we were very pleased with the broad introduction that he gave us to the latest and the hottest on MPI.

Lee Liming has also been, before his recent participation in the Globus team, has supported collaborative activities at Argonne and the USC Information Sciences Institute, where he has supported Globus collaboration as well as in many other sites.

Now that I think everyone is here, I see the last newcomer is here, let's welcome Lee Liming.  

MR. LEE LIMING: Thank you. I am here today to talk to you about the Globus Project, and Globus has been mentioned already a number of times in Miron's talk yesterday, and earlier this morning. And Miron's talk last night was kind of a very conceptual talk, and this morning we will kind of get into some of the theory of parallel computation, and I'm going to do something that I think is kind of an intermediate position somewhere between them.

I'll start out talking about kind of our grand vision for what is behind the Globus Project, and then over the next few hours I'll drill down into some of the realties that we've created out of that grand vision; in other words, we've taken pieces of that vision and made them real. We've done that in the form of software that we call the Globus Toolkit, and so I'm going to start out at kind of a high level; but don't fear, by the time we're done you'll probably know more details then you wanted to. So let me just go ahead and get started.

I want to start out with a few "what ifs," okay, to kind of give you an idea of why we're doing all of this wild work. So what if in the year 2000, rather than the current situation that we have which is where we have a few power companies around, everybody instead had decided to keep their own generators.

So every house, every office building, every university building or campus had their own power generators, and they had to maintain them all themselves. They had to buy them or build them themselves. If you are a small organization, maybe a family, you just bought a generator and maybe put it in the backyard and every so often filled it up with some, you know, whatever fuel you were using. I don't know, maybe you have wood-burning power generators, you have gas generators, you have whatever.

Some people needed more power, so they bought bigger generators and it cost them more money, but they needed it and they were willing to pay for it, fine. If you have enough money, you can buy as big a generator as you want. And then there are some organizations, maybe big government installations or military units, or whatever, that actually have to build their own because there's nothing big enough on the market because nobody else wants anything that big. It would be a pretty strange situation to be in and probably not very comfortable for people who don't have a lot of money to build or buy generators.

So here's another scenario: What if, rather than having, as we have, a few phone companies that you can go to, everybody decided to build their own telecommunication systems for their own organizations. So families might be completely out of the loop here. They have no hope of possibly building communication systems to talk to their relatives in other states and other cities or whatever.

Small companies maybe run things between offices. Maybe you have cups on strings and you talk to each other from room to room. People actually used to do this centuries past. They would have little ways, you know, little things they would talk through, and pipe the sound through the different buildings. Very primitive from our standpoint, but it used to be reality.

Maybe if you had an organization that had offices in different cities, you'd run lines between your organizations. Eventually, you'd probably get hundreds or thousands of these lines between cities, and you'd have a big trunk, you know, not cables, but actually like whole bundles of cables, each one corresponding to a different company or different organization or government agency, or whatever. No sharing between these things. You bought what you could afford, and if you couldn't afford anything then you were just completely out of the loop.

Guess what? It's the year 2000 and that's pretty much exactly what we're doing with our computational power, with a few exceptions. So the two scenarios I gave you before, we kind of think of as completely outlandish. We would never do that. It's the year 2000. How could we possibly get into a situation like that?

Guess what? We're still actually doing that with computational power. If you are a family and you want to do computation, you buy a computer. You buy a home PC or Mac or whatever you want. If your kid wants to run video games, then you buy Nintendo or Sega, or whatever the latest Sony variant is. If you are a university, you probably have bought large computational infrastructures. You probably put in your own networks. If you're a company, you probably did a lot of this too. Government agencies probably did the same things. Everybody goes alone in these things.

So the question is: Why aren't we sharing? Why don't we have a computational infrastructure like the telecommunications infrastructure or like the power infrastructure? This gets to a concept that we are addressing with the Globus Project. It's a very broad concept. Many organizations are pursuing this vision, the vision of the computational grid. That's exactly what the computational grid is; it's a common infrastructure for computation.

The vision here is that we will have a pool of computational resources that anybody can plug into, the same way as you plug your telephone into the telecommunications infrastructure. You plug your devices that need electrical power into the power infrastructure, the power grid, and the idea is that you should be able to plug an interface device into the computational infrastructure and gain access to it. It's basically built for whatever economic model ends up surfacing for that model.

So you don't have to build your own infrastructure. It's going to build one for you. We're going to be sharing resources. You don't care whose computer you are using you now, it's just something that is out there somewhere. We don't even know where it is. It's just out there. And we basically pull in as much of that computational power as we need to, to do the work and we pay for it. Okay. Enough said there.

So, this is an evolution of the computational collaboration that has been going on for decades now. Ever since computers were created, we've had different levels of collaboration in using the computational resources that were available. In the early decades, and, of course, it goes before '75 -- actually, before '75 I would almost argue there wasn't a whole of collaboration, other than, if you want to use my computer, you come to me and sit in the, you know, input/output office and submit your jobs here. And if you won't come to me, then you can't use my stuff.

This area from about 1975 to 1995, I'd call it collaboration. Lots of different organizations worked together. They had different computers, but it's really hard. It's really hard to get the resources to work together. It's not clear how to share data. Maybe you have systems that don't have the same data formats, or maybe -- I remember, you know, trying to share floppy disks with people that would have different types of PC's. There were all these issues you had to think about, and it was just so annoying that most people didn't even bother. They just bought a big enough computer, so they didn't have to use anybody else's and lived that way.

I would also argue or propose that between '95 and 2005, we're kind of in the era, right about now in the middle of it, of the era of virtual organizations. Of course, there are a lot of different ways to look at these time periods. I'm just throwing out some generalities here.

In the era of virtual organizations, what happens here? Well, we start to build systems that pool resources, that start to make it easier for us to share data, to share instrumentation that we have that can connect to computers so people can access it remotely. We start to learn how to manage the various differences that there are between computer systems, so that in order to write a program that runs on five different computers at five different organizations, you don't have to know all the details of those computers and how they were configured and what the policies on them are and what scheduling systems there are on them and what software they're running, and things like that. So the idea is we're basically in the period where we're learning how to do this well.

And I would argue that soon, and I just threw out 2005 just for fun, soon we'll be in an era that we can actually call the computational grid, where computation becomes, as I sort of suggested earlier, a commodity, and it's something that can be bought and sold by anyone. You can use computational services with standard devices or standard interfaces.

And in red here, the ultimate goal being that organizations and individuals usually won't need to build their own computational infrastructures. There will always be people who will have to build their own, just as there are always people who have to have their own power generators for backup or even primary purposes. Just as there is always going to be organizations that have their own telecommunications services, there will always be people who need something better than they can get on the open market.

But by and large we want it to be the case that people do not have to actually own their computers in order to do computation, and you're not actually afraid of using somebody else's computer rather than having your own that you can depend on.

So why do we need this? I would argue that sharing resources is better than not sharing resources for a number of reasons. One, economies of scale. We can reduce the overall investment required to create a large supply. That is, with everyone buying their own computational power, there's an awful lot of spare cycles out there. And Miron and the Condor group are working to help us utilize those spare cycles, and that's wonderful. We need that to be universally true.

In other words, we would like to see computational power be a pool that everyone can use and is so ubiquitous and so available that everyone just assumes it's there and will use it, and we'll get rid of all these barriers that we've built up around or own computers so that essentially it can be used as one common pool. The argument, of course, being that hopefully we will get better utilization and we'll overall be spending less money in building this resource.

There's also economies of scale in terms of the human expertise that are required to set up and maintain, administer, monitor, and all these things, for the computers that are available. People spend an awful lot of time right now working on their home PC's and getting them to work just right, or not. You spend an awful lot of effort in computing centers getting everything to work just right. People are doing the same exact things all over the country, all of over the world with their computing centers. They're having the same problems, and so we're actually not taking advantage of economies of scale in the terms of everybody running their own computers and encountering the same problems. And wouldn't it be nice if we could share that expertise?

The other reason I think why sharing is better is because of this concept of Strategic Reserves. That is to say, right now if there were some natural disaster that required the government to come up with some kind of great plan to get us out of harm's way -- and there are plenty of situations like that right now. You could look at the forest fires in the West of the country. You could look at the situation that's going on right now in the -- is it the Black Sea or Baltic Sea? I can't remember which one it was. Basically north of Russia with the Russian sub that sank.

There are all kinds of things that have happened where it would be nice if we could do some simulations and say: Is this response actually going to work? Is there something better we could be doing? If we were to discover there was an asteroid coming towards the earth, we wouldn't have a clue right now what to do about that because there would be no way that we could actually simulate enough cases where we could actually find an optimal solution. The computing resources just aren't there.

And so if we had a common pool, then we could actually prioritize the use of that pool at strategic moments and say it's to everybody's common good that we use these computers together to solve this problem. We could not do that right now. We do not have an infrastructure that would allow it. Even if the government declared martial law and said we are, you know, taking all of your computers through eminent domain, and we are all going to use these things too now. It would never happen. We could never marshal all the resources necessary to make computers work together like that because there's no infrastructure for doing that. We just don't have the ability to do that.

Another reason I think we need the grid is because sharing resources enables a new kind of computation. In other words, right now our use of computation is limited conceptually based on what we think we can do right now. New types of computation that could be possible include on-line instruments; the ability to control and receive output from instrumentation that is unique, only available maybe in one place in the world, and make it available to people all over the place.

Some early examples of this include the UARC Project, which was a large collaboration centered at the University of Michigan, where, let's see, how do I put this? Instruments that observed phenomena in the upper atmosphere -- that's what the UA in UARC stands for, Upper Atmospheric -- were connected to computers. They were physically located in Greenland, but for various reasons climatologists and space scientists and earth space scientists did not want to go to Greenland every three months or so in order to use these instruments. So it kind of limited the amount of research that could be done with the instruments, and it also limited the amount of collaboration that could be done between the scientists who used those instruments.

So what they did was they took those instruments and hooked them up to computers, hooked the computers up to networks, hooked the networks up to nice client workstations that had nice interfaces, and they created an upper atmospheric co-laboratory, which meant the researchers could sit at their desks at their home institutions and actually view the data coming out of the instruments.

They could actually control the instruments; where they were looking, what sorts of phenomena they were looking for, what wavelengths and all sorts of things they were looking for. It allowed them to communicate with each other. All the different researchers who were out there could chat with each other in a little chat box.

These sorts of things can be done. They have been done. But they're not done routinely because there's no infrastructure right now that makes it trivial to write an application to do that.

Collaborative engineering is a similar thing, but it's basically more focused on simulation rather than instrumentation. The ability to simulate a design and have that design be simulated on some high-end computer that can actually run through the design and simulate what is going to happen. Maybe do optimization studies, maybe simply render what the mechanical parts of the thing looks like so it can be viewed by people, connect that simulation or that rendering device up to this sort of co-laboratory, and you have people being able to work together on design even if they're not physically located in the same place, looking at the same computer screen and using the same supercomputer.

Parameter studies and, I think there's another one down here. Well, parameter studies, I guess, and to some degree simulation and data intensive computation are all things that Miron talked about last night. These are things we can do but we don't do very often because we think it's hard. The tools that we are talking about this week are supposed to be tools that allow us to make these things easier and allow us to use existing resources that we have now. Not just buying better computers and bigger computers and running software on them, but doing it on the computers we've got today.

Browsing of Remote Datasets. There are organizations -- NASA is a good example. CERN is a good example -- that have huge datasets. They have either instruments or they have -- well, usually it's instruments, sometimes simulations, that generate very large datasets. These datasets are stored on very large storage systems. And right now the people who can use them are essentially limited to people who have access to those storage systems.

It would be very nice if we could make those datasets be accessible to people using their PC's on their desks without special software, special clients, special this, special that, all this kind of stuff, and have it actually be available. It would be nice for people to be able to find out what datasets are available, what they mean, what is the Metadata associated with, and have this stuff be available on a more community-oriented basis rather than it being focused on a very specific small research community. It would be nice to have some of it be available in a more broader sense. So these I would propose are some of the reasons why we need computational grids.

On-line instrumentation. One of my goals in this presentation is to prove to you that these are not just pie-in-the-sky ideas, but are actually things that are possible. And the Globus Project has done about as much work in actually building applications that use these concepts and the tools that we've developed as we have actually building tools and kind of proposing concepts.

So this is an example of one of those things that we've actually done. This was the Department of Energy's X-ray Source Grand Challenge. We actually took on this grand challenge within the Department of Energy to make the advanced photon source, which was an x-ray source, a very brilliant x-ray source at Argonne National Lab, accessible remotely so that it could be used by people who don't physically go to Argonne and also actually make the process of using the advanced photon source a more interactive experience.

So rather than sending your sample to the advanced photon source technicians and having them putting up the thing, have them generate the x-rays and shoot through the sample at various angles and then eventually ship you back a huge pile of data; instead, we'd like people to be able to view the data as it's being generated by the instrumentation.

So that's exactly what's happening here. The advanced photon source, this picture is inside and outside of the advanced photon source, was connected to a bank of computers that produced tomographic reconstruction of the data sent. They took all the raw data coming out of here and produced essentially 3-D datasets that represented what the thing was actually looking at and what it was showing.

This was sent both to archival storage and also to desktop and virtual reality clients. So in other words, you could either be sitting at a desktop and looking at kind of a 2-D display or maybe a lower resolution 3-D display, or you could be standing in a virtual reality cave or in front of one of our smaller I desks, which is kind of like the desktop version of a virtual reality cave where you kind of sit in front of it like an easel and put on your 3-D glasses and you have a similar experience but it's much cheaper, and view the data as it comes out of the advanced photon source.

This kind of experiment is possible and doable. We did this. We did it for about fifteen minutes at a time, but we did it. It took a lot of effort to make it happen because we don't have the infrastructure yet, but it's possible and we've demonstrated you can do this sort of thing. And with the proper infrastructure you could probably do it whenever you feel like it. It's just a question of making sure we put those tools in place.

Collaborative Engineering. This was actually an application that was developed at the University of Illinois, Chicago Electronic Visualization Laboratory. It's called CAVERNsoft. It was used in a case study with Caterpillar in an effort to engineer a better tractor. The idea was to allow people at different sites to work in a tele-immersion environment, that is, basically work in 3-D in the virtual reality cave, to have the simulation or the model of the design available to them and be able to see each other and talk about things they see in the design. So that rather than having to build models, which can take days or months or weeks, instead they could actually have the thing be created in virtual reality and interact with it.

This CAVERNsoft software is available and can be used, and it can be used with some of the software we provided to take advantage of computers at various locations to do the simulation and rendering necessary to support this.

Distributed Supercomputing. This is kind of one of those way out there things probably for most of you. But basically the idea is if you have different organizations with supercomputers, you shouldn't be limited by what any one of those can do, you should only be limited by what all of them can do together.

So in this particular example, SF Express, this was a defense application. What you're seeing in the upper right-hand corner is a battleground simulation. And the challenge here was to keep track of a very large number of vehicles in a battleground scenario and keep track where they were, where there were going, what their velocities were, what their status was, are they in trouble, that sort of stuff, and to do this for a whole battleground. Which is nontrivial, because in a battleground you can have tens or even hundreds of thousands of different units that you have to keep track of.

In this particular case, SF Express set a record for the number of objects that could essentially be kept track of during a simulation. There were more than a hundred thousand different objects that were kept track of during this particular run, and it was all kept track of in the computers, the high-end supercomputers. It was rendered in one of the other supercomputers and displayed on a display somewhere else entirely. So essentially you were able to, you know, run a simulation on a very high-end machine, render it on a graphics machine and display it on your desktop machine, or wherever you happened to be. I think in this case that did it, again, in the cave.

A lot of issues here. For example, Resource Discovery and Scheduling. In order to keep track of which resources are available, which nodes on which supercomputers are available for processing and assign them for use.

Configuration. A little bit of background on this application. The Defense Department already had the software to do this stuff. Caltech had already even parallelized it so it could be run on more than one computer. The problem was, the reason we that got involved and the Globus Project with this, was it was very difficult to set all this stuff up. In other words, the code was available to simulate it all. The parallelization was done. They knew how to run it on different machines.

In order to get it started up, what did they have to do? They had to go to each machine in the environment, have a Telnet window to that machine; they had to log into each machine; they maybe had different accounts on every machine, so they had to have a little cheat sheet that keep track of which account to use on which machine; they had to start up a program at the same time on all these Telnet windows so they would all start at the same time and be able to initialize and talk to each other; they had to watch the output of all the little programs they started up and see if any errors happened while it started up, and if it did, they had to go in and modify and change it, whatever.

This was a real problem. It took a long time to get a run started up properly and going and everything working fine and it's all talking to each other and happy and then take care of the all the little errors as they come up. One of the things they needed was the ability to have a common configuration and handle faults, fault tolerance, in an automated way, so people don't have to do this. The job submission is important. The message-passing is important. Of course, we have some solutions for that, so that's important.

Miron talked last night about high-throughput computation or computing. You have a large number of jobs that you need to run over a certain amount of time. You want it to get done as quickly as possible. You want to submit the jobs and have something, some schedule or some job manager, personal job manager if you want to call it that, take care of all that and get it through the systems as quickly as possible.

Some of the challenges here are, again, resource discovery, which nodes are available and how am I actually going to connect to them to use them. Data access. If each one of those jobs requires data input, you have to stage that data on the machine before it runs and you have to get the output back from it. Scheduling is important. The ability to make sure that your jobs actually run on these machines rather than being stuck in a queue for a long time and perhaps never getting done by the time you need it.

Reservations. It would be nice to be able to say that I want to know that I'm going to have a certain number of nodes at a certain time so that I can actually get this work done by a certain deadline. Security is important when you have large numbers of machines, especially if the machines are owned by different people. You want to make sure when you give other people access to your machines, you know who you're given access to and you're not given access to anybody else.

Miron mentioned last night you have the issue of security in terms of, you know, if I submit my job to a machine, I want to make sure that I trust the machine I'm giving the job to not to give me back bogus data on the other end. So I want to make sure I know really who I'm talking to.

Every system has it's own accounting system. We can't get around those. Owners will not let us get around the accounting system, so we have to make these things all work together in code management.

What you're seeing here is output from a piece of software called Nimrod-G, which was developed at Monash University in Australia. They have some interesting features in here. The ability to specify a deadline for when all the jobs have to be done. The ability to say how much you're willing to spend on the overall job or overall work. Given that each one of these things may have accounting systems that charge you for how many cycles you use on the computer, you have to know whether you want to submit to a machine. Maybe it's too expensive to submit to that machine. You probably as a user don't want do deal with all that stuff. You just want to say, I have this job. I need to get it done. I'm willing to spend no more than fifty dollars, and go for it. And then the ability to see the available machines down there and what state they're in, and how much you've used on them and so on and so forth.

The last kind of example I want to show you, just to prove this is useful stuff, is Problem Solving Environments. This is kind of a hot topic lately. The idea is that people who are using high-end resources don't want to deal with high-end resources. They're too complex and they're just not interesting.

What you would like to do is have user friendly interfaces, common interfaces that your familiar with and that actually speak to you about your work, have metaphors in there that make sense to you. If you're doing mechanical engineering or high-energy physics, you want to see mechanical engineering or high-energy physics things on your screen. You don't have want to be talking about schedulers and resource managers and accounting systems and job submission scripts and things like that.

So the idea is we would like to be able to build web portals or customize applications that use the supercomputers on the back end. So basically these applications that you can run on your PC know how to submit jobs to large resources out there and do it for you without you having to worry about that. So maybe you can use things like Nimrod-G to control costs and things like that but not actually be confronted with the interface for that. Have it be built into the application instead. So, again, remote job submission, monitoring and control, resource discovery.

Having data available to anybody who is using this application. In my earlier example of the upper atmospheric researchers -- I guess they weren't actually upper atmospheric researchers. The scientists who were looking at the upper atmospheric instrumentation, they want to be able to share this data and use it together and comment on it together and point out things to each other. Security and accounting are all things that come up with these types of applications.

So these are all things we ought to be doing but aren't because it's too hard right now, and the idea behind the Globus Project is to make it a little bit easier. Or, rather, that's the idea behind the grid.

There are three key words that go along with the grid; dependable, pervasive, and consistent. Dependable means if you have a grid you ought to be able to count on the performance and functionality that you are going to get from the grid. When you plug a device into your power plug, you expect you're going to get exactly the same kind of power you would from any other power plug. It's not going to be less, it's not going to be more, it's not going to be faster cycling, it's not going to be slower cycling, it's going to be the same thing.

And you have to depend on that, because otherwise you can't create a standard device to plug into it. It's got to be consistent, that is, you use the same interface. Pretty much the same thing I just said, but in a slightly different way. In other words, it works the same. It doesn't matter which part of the computational grid you're using, it looks the same to you. It might be a big computer; it might be a small computer. It looks the same from an interface standpoint.

Pervasive. In order for this to be successful, you have to be able to plug into it anywhere. A power grid would be no good to us if it were only available in certain neighborhoods but not others. You have to be able to go anywhere that you want to work and be able to plug into this thing. We can't just have this be available at the national laboratories like Argonne and Lawrence Berkley. We can't have it only be available on major university campuses. We can't have only be available in the commercial sector. It's got to be available everywhere; otherwise, we can't work together the way we are all starting to do now and have been for a while.

What are some of the technical reasons why we're not doing this? This is where I start outlining some of the things I'll be talking about in the next few hours. There are basically six reasons I want to point out now. There are probably other reasons too, but these are the ones I'm prepared to talk about later on, so I'll tell you about them right now.

One is Security. Whenever you have public resources you always want to be sure that you are safe when you are using them, and the resource provider wants to make sure that they are safe in letting you use their resources or letting you be in that space. So the whole idea behind security is you want to have guarantees about basically what interactions you're going to have with other people or other resources out there when you deal with the grid.

You want to make sure, for example, people aren't snooping on you to find out what kind of work you are doing; or, people aren't impersonating you, pretending to be you and using resources that maybe have bills associated with them. You want to make sure other people who are using the grid can't step on your data or step on your computational work that's going on disrupt it, change the results or whatever. There have to be guarantees on security.

Information Services is another problem; finding out about what's available on the grid. If all you did was you took your computer and you plugged it into a socket somewhere, how is your computer going to find out about all the various resources that are out there that it could use. Information services is key when you're talking about a large computational grid.

Resource Management. That is, once you know about all these resources that are out there and once you feel relatively secure that you can use them safely, how do you use them? What are the interfaces that you use to start a job on a supercomputer? What are the interfaces you have to use in order to hook up to the data output of the advanced photon source at Argonne? There are all these intricacies that constantly bedevil us as we're trying to build collaborative systems, and we have to get over that problem so that resource management becomes easier.

Data Management. Getting the data to the jobs, or, perhaps, in some cases, getting the jobs to the places where the data is if it's easier to do that than the other way around; that is, making sure whatever data we need in order to do our computation is either transportable or is at least accessible in some way to the processes that we start that need to do computation on them. That is a surprisingly difficult problem.

Communication. Allowing jobs to communicate across the grid. Once you know how to start up jobs at a thousand machines, what good does that do you if they can't talk to each other? I suppose in some cases that would be fine.

In the case Miron was talking about last night with high-throughput computation, you could probably ship off data files and bring back some other data files when you're done, and that's all you need. But what if you have an MPI application, some application that has to pass messages between the different processes in order to do computation where intermediate results over here are going to effect the computation over there, or where you have to basically have communication going on while the computation is happening?

You have to have some kind of way of communicating, and it has to be a standard way. It can't be different depending on what type of computer this is or that is or whatever. It has to be some kind of standard way of communicating.

The final issue is Portability; the ability to write applications that run on any kind of system. And this is probably where all these things kind of coalesce is this area of writing applications. You have to be able to write software that can depend on a common environment and that knows how to use all of the solutions above, all the five things above, and also that isn't going to be stuck on little operating system differences or little differences in the way it handles TCP/IP or something like that.

These are what I would layout as the six major issues that the Globus Project is addressing. We address each one of these to a greater or lesser degree depending on how long we've been working on the project and what phase we're in, what particular problems seem to be big for our collaborators right now.

One of the things I should mention about the Globus Project is we are very focused on applications. So if a particular application that we're working on has a problem in one of these areas, we'll focus on that area for a while to solve the application problem, and hopefully it will be applicable to other applications later on. So rather than trying to take a top-down approach and address each of the six issues in order or all at once or something, we kind of go where the problems are and work on them and then move on.

I'm going to skip that slide. I think I've talk enough about that. So to convince you that the work we're doing is necessary, now that I've laid out some problems, you might say they are solutions for the problems out there, and why don't we just use the solutions that already exist? For example, there's DCE, the Distributed Computing Environment, there's Corba, Jini. All these different examples of software environments that are designed to solve these problems.

The nice thing about those systems is they have very rich functionalities, so it's usually pretty easy to write applications in those environments. It's not too difficult to write a Corba application once you have Corba installed everywhere and figured out all the bugs in it and everything. Once you've got all that working, it's a nice environment for writing applications.

DCE is a good environment for writing applications. Lots of features available. Security, resource management, storage. All that kind of stuff is available. It's all common. It's all simple to use, and a pretty easy paradigm.

Jini is like the latest, great thing from Sun and the Java consortium, and the idea there is to make it easy to do all these things also. The problem with these is that they're very complex. And in particular, there is no global control over all the resources. In a perfect world where there was one organization that ran all the resources you were interested in, then that organization could say we're a Java shop, and you're going to use Jini for all your work. Well, that would be fine.

Unfortunately, we don't live in a perfect world, and if you really want to do collaborative computation, you're going to have to deal with organizations that are using DCE and organizations that are using Jini and organizations that are using Corba, and a bunch of other systems out there.

And so the problem is what do you do when you have to leave the safe environment that you started out in and you go out to another organization? For example, I've written an application here, and I want to use it with software over there that was written with Corba, and I only know how to do Jini stuff. Then you have to have some kind of a translator between Jini and Corba. They're different semantics, and it's very complicated. That's one of the problems.

There are also in some cases performance difficulties with these. These systems are designed to optimize on inoperability; to optimize on simplicity or at least producing a common environment. So Jini, for example, based on Java, the idea is you want to be able to write once run everywhere and so you have this interpreter in there. You can compile it, but it's still not completely down to object code and so there are potentially some performance difficulties there, particularly related to communication between processes and things like that.

So, okay. We can take another tack. We could say, okay, fine. Fine. So the one solution solves all isn't the best way to go. Why don't we do the bottom up thing? Take a look at the Internet, the web, and all those things. They're very popular. They're everywhere. Everybody is using the web. If you write something for the web, you're pretty much sure it can be used pretty much anywhere. If you're doing TCP/IP stuff, there's hardly any networks out there anymore that don't speak TCP/IP one way or another.

Again, the good news is these things are very simple, so it's easy to deploy these things. So it's really easy to put up a web server, really easy to tell people to use web browsers. That's fine. The problem is that there is a lot of missing functionality there. These are simple technologies, very powerful. It's easy to leverage them, but you have to do a lot of work on to top of it. You can't just take a web browser and a web server and build a collaborative environment. You have to add a lot of stuff to it.

So while the ones upstairs here give you a lot of functionality you can use, it's difficult to get it out there and get everybody using the same stuff. Whereas this stuff down here, which is very simple and generic, you can get it everywhere, but then you still have to do a lot of work. So it's kind of a dilemma which way do you go.

Well, the way we are going with the Globus Project is to use standards and commodity technology in the infrastructure wherever possible and wherever it makes sense. We basically are looking at things like the LDAP protocol for information services; SSL/TLS for secure communications; X.509 for secure identities; HTTP and FTP for transport, data transport; XML and SOAP for data representation.

All these different standards that are really -- well, they've probably been buzz words at one time or another, maybe still are, and have made it into some of the commodity technologies that are available, but people really haven't done a whole lot necessarily or used them to their fullest potential. These things give us leverage.

And then in order to handle the people who are using Jini and Corba and DCOM and things like that, we provide interfaces to these things. So while the core infrastructure may not have all the functionality that they provide, we essentially provide glue between the infrastructure, which is very generic, and the rich higher, level functional services that people need in order to develop good applications.

So the project as a whole, looks like this. There are four main thrusts here. Research. Basic research into grid technologies. So the various issues I listed up there before, those six issues. The Globus Project is a research project that's at Research Institutions ISI at Argonne, so our primary focus is research.

Software Development is the next step. We are actually developing software that provides the core services for grids, for computational grids. So the idea is you can take our software, deploy it on your systems, and you have the raw materials for a grid. You can take you're commodity technologies then like Java and Corba, DCOM, whatever, and build applications on top of this and have systems that will work, or applications that will work together, on a grid.

Not satisfied with that yet, we have to go on and actually build production grids and test beds, so we are also involved in several smaller and larger scale grid development efforts. There are actually people now, organizations, that are building computational grids. Just as there were originally people building power grids, there are now people building computational grids and trying to make this stuff work. I'll tell about some examples of those in a few minutes.

Experimentation With Real-Grid Applications. This is probably the most important aspect of our work, and that is, that we were actually helping people to write applications that use the grids that they're building, and we're coming up with all sorts of issues that come out of that effort. So in other words, rather than just building the environments and say here, come and use it, we actually are using the applications to drive the test beds and say in order for these scientists to do their work, you have to provide these services. We've got to get these into the production test beds.

Well, in order to get them into the production test beds, we have to develop software. Well, in order to develop software, we have to do research. So you see it all kind of feeds itself and goes through the cycle there. So it's very important. We take a user-oriented view of this stuff. We start out with the science and then move it towards the research that needs to be done to support it.

So here is kind of a simplified architecture for grids. The Grid-Services Architecture. This is a model we've been using to explain grids for a number of years now. The idea is you start out at the bottom. If you've ever been working in networking, this may look very familiar to you, this kind of layered approach to things. It's the same sort of thing.

We have this grid fabric, which you can't see because it's too dark. Oh, well. And down here we have things like operating systems, instrumentation, control interfaces, schedulers, data transport. All the things you buy off the shelf. This is kind of the infrastructure or the fabric.

Then you have the Grid Services; security, portability, fault detection, data management, information services and resource management. These are the things that make these resources work together.

Then you have on top of that Application Toolkits. These are also called frameworks. These guys are things that you build on top of the infrastructure that make it easier for you to write applications. So, for example, you might have a high-throughput application toolkit that makes it easy to write high-throughput computational applications. You might have one that focuses on data-intensive computation. So, for example, the high-energy physicists who are getting tons and tons of data out of their supercolliders would use the data-intensive toolkit to write some of their data management applications.

Collaborative Design. Remote Visualization. These are all the kinds of generic categories of applications you might want to write that would use the grid. So we need to have toolkits here that do that.

Finally on top of that you would have Applications. Real applications that real scientists or engineers or people just in general use. Maybe Gamess, whatever. You know, things like high-energy physics data analysis in one case, collaborative engineering, climate studies and so on.

The boxes that are highlighted here kind of give you an example what it would look like if you actually wanted to use one of these applications. For example, if you want to do a high-energy physics data analysis, what would probably be involved at the application level is, of course, the application. And the application toolkit would probably be a toolkit for doing data-intensive work.

At the grid-services level you'd probably have to involve several different grid services; information services, resource management, data management, and fault detection. Turns out that high-energy physics people are not that concerned about security because nobody would be interested in their data anyway. It's all very esoteric and meaningless to other people. Portability is not such an issue. They have enough money to buy the same kinds of machines all over the place. They don't really care.

So mainly they're interested in these sorts of things: The ability to detect when one of their jobs failed and they have to go back and redo it; being able to handle large amounts of data; being able to control large numbers of machines; and, being able to be aware of where all those machines are and where the data is and how to make it all fit together.

Then, of course, at the fabric layer you're dealing with things like data transport, your network cables that go from machine to machine and supercomputing center to supercomputing center; schedulers that schedule all the work to happen; the operating systems that all the software runs on. Things like that. So these would be the things that would be available for one type of application. Other applications will use different modules within each of these layers.

And basically our sense is, after working in this area for five or six years now, that these are the types of services that are going to be used by all of these applications, some to a greater or lesser degree depending on the application.

So this is the area, the grid services area, where we've been doing most of our work. Some of our work is in application toolkits because we have to support applications otherwise we don't know what to put into the grid services layer, but most of it is up here.

So who are some of the people who are working on the Globus Project so you have an idea of who to talk to later. Argonne National Laboratory and the University of Southern California Information Sciences Institute. These are the two organizations that kind of started working on the Globus Toolkit and the Globus Project.

NCSA. The keep changing what all the initials stand for. I guess now it's the National Computational Science Alliance based in Urbana-Champaign and the San Diego Supercomputing Center are both major contributors to technology that goes into the Globus Toolkit.

We also have organizations that are helping us by being prototype sites. For example, NASA -- and I'll talk about these a little bit more -- the Trilab, Department of Energy, Department of Defense Laboratories, and other organizations; and then lots and lots and lots of collaborators. More collaborators than sometimes we can even keep track of. So if any of you are collaborators of ours and we forgot about you, I apologize to you now.

But we have lots and lots of collaborators out there who are giving us application examples, scenarios, requirements, telling us what they need. We're working with them to develop applications. They are using the applications in real life or they're demonstrating them, writing papers about them. All sorts of things like that. It's a very exciting field, and a lot of people are finding it very interesting.

There is also an organization called Grid Forum. The Grid Forum is kind of the community or body that is helping to develop standards for computational grids. So if you're looking for the grid standards, the Grid Forum is the place to go. The Globus Project is doing grid research and software development, and Grid Forum is really the standards body where people are in communication with each other about what the standards need to be.

So, again, the Globus approach is to take a toolkit of services, addressing key problems. We want to make it into kind of a bag of services so you take what you need when you need rather than taking the whole thing all at once.

This is a key thing here: We do not provide you with a vertical solution; that is, we provide you with that grid-services layer. You will probably have to build your application on top of that, and you will certainly need to provide the computing infrastructure underneath it. We can't give you that. And so it's important to realize that when you get the Globus Toolkit and install it on your machines, you do not have a high-energy physics data processing grid. You've got to build applications on top of it.

Let's see. We focus on the interdomain issues. So while applications like Condor, for example, or MPI, toolkits that allow you to use your clusters efficiently and effectively focus mainly on, you know, one organization's resources, our focus is on how do you get resources at different organizations to work together, all the policy issues and security issues and information issues and things like that. That's where we basically see all of your problems coming up.

Distinguish Between Local and Global Services. I don't even know what that means. I don't care.

You probably heard of the IP Hourglass or the TCP/IP Hourglass. The idea being you have many, many services on top of the hourglass that use all the services. You have probably many, many different types of hardware and software at the bottom. And what you really need is a core set of services in the middle, some narrow set of services that link up all the applications up there and all the technologies down here. And if you can come up with the right set of services, then you have a very powerful, powerful model there, or powerful architecture. Very important things.

Went to keep participation costs low. That means it should be easy to use this stuff and it should not create barriers to your use of the resources. You shouldn't have to completely throw out everything you've got for security on your system right now and replace it with our stuff. That would be bad. So we don't do that. You'll see how we do it in the next few minutes.

Enable Local Control. As Miron was lecturing us last night, it's incredibly important to remember that every resource has an owner. You cannot dictate to the owner how their resources will be used. They dictate to you. They dictate to the rest of the environment, and so that's another one of our design principles.

Support for Adaptation. It basically reflects the fact that you can never really know what the situation is out there on a grid. There are just too many things to keep track of. It's going to be changing all the time. The minute you find out that a machine is available, somebody else sneaks in there and uses it and takes it away from you. So you have to be adaptable. All the stuff that you do has to end up being able to tolerate changes in the environment, faults, whatever. Some machine goes down for no apparent reason, doesn't matter why it went down, it went down, you have to deal with it. So those are all some of our design principles.

This is taking that same architecture diagram before and pointing out where some of our modules actually fall into these things. We have tools like MPICH-G, which is a grid-enabled version of MPI that fits in this applications toolkits area. If you have an MPI application and you link it with the MPICH-G version of MPI rather than whatever version you've been using up till now, your MPI application is grid aware. That's really all you need to do. It suddenly becomes grid capable.

DUROC is a co-allocation framework, which basically means that you can allocate a whole bunch of computers all at once at the same time. In other words, you don't get some of them sometimes and other ones later on, and this one comes in, you know, five hours into the computation and this one pops out. It means you get all of them all at the same time. So it's basically a way of taking all of the schedulers and making them work together.

The Grid Services. This is where we have most of our stuff. Nexus and Globus I/O Communications. MDS is our information services. GSI is our security. I'm going to go through all of these. You don't have to memorize anything. GSI-FTP for data transport. The Heartbeat Monitor for fault tolerance. GRAM for Resource Allocation and Management. GASS, Globus Access to Secondary Storage, which leads to many different jests in our group. It's a fun acronym to play around with.

So essentially this is what I just said. We provide in all of these, I guess in this case we have seven different areas, we have tools that are available to solve the problems that show up in those areas. We're always redeveloping some of these or developing new ones to fill in some of the gaps that we find when we try to build applications with them.

What's going to be coming out in the near future, just so you can be aware of probably what we don't have right now, for data transfer we are currently working on -- probably actually next week we'll be releasing an alpha release of a new version of FTP called GSI-FTP, which is a secure, high-performance protocol for data transfer. It's basically the FTP portal with protocol extensions.

The FTP specs have room for protocol extensions in it, and so we're taking advantage of that to produce a protocol that's both compatible with everything that you already have probably, assuming that you have standard Internet tools, but that also adds important things for a grid-like security. Very important. We want to have something that has really hard security in it rather than this ID password stuff, clear text security, that just doesn't work for people.

We want high performance. That means you have to have be able to have parallel data channels. You have to be able to have automatic window and buffer size adjustments and things that happen that really optimize the performance on a high-speed network link. Turns out FTP just isn't that efficient in a lot of ways for some very simple reasons, and we can fix that. So GSI-FTP is something we're working on.

Replica Management is also something we're working on. It is the ability to have multiple locations for storing datasets on a grid, and then you can choose the one you want based on whatever is closer or has a high-performance network connection between you and it, use the information services to find out where the replicas are and probably also the network conditions between you and there, and then your application can easily pick the best place to get data from. And then replica management means you can create these replicas, you can move them around to optimize performance. You can do all kinds of stuff like that.

Notice that these things are both in the area of data management. That is because our collaborators right now tend to be in areas that require data-management solutions, and so we've been working a lot on that this year.

We also experimental prototypes in the area of Advanced Reservations and Quality of Service; that is, the ability to reserve a network link for a period of time and say I need to have 50% of your OC-12 link. It's very important. If you have an OC-3 link, I have to have 90% of it from this hour to that hour tomorrow night because I'm going to be transferring some data and I've got to have it; or, we're going to being having a video conference and we've got to have that bandwidth available. So we're working on that.

Distributed Events in Logging. That is the ability when you're using a grid to find out what's going on out there in your application world somewhere. You don't even know where your application is running, but you need to know what's going on. And so you can subscribe to particular information channels that tell you things that are happening with the machines your application is running on.

I promised you I would tell you a little bit about the production grids and test beds we are using. We have production deployments using the Globus Toolkit at the National Science Foundation's PACI's, or Partners in Advanced Computational Infrastructure. This is a program that the NSF is funding which basically says let's build some real grids. Let's get people using them. Let's find out some of the problems that come up when you do that. And so that's what's happening.

Two efforts based at San Diego Supercomputing Center and NCSA here in, or nearby in Illinois, are actually building grids using a consortia of universities in the area and other supercomputing centers that are available. They are actually using the Globus Toolkit as middleware to make these things happen.

The NASA Information Power Grid. NASA has a number of different research centers around the country, and they are trying to link them all up so they can use them more efficiently together, rather than have each center work independently.

The Department of Energy's ASCI, or Advanced Strategic Computational Infrastructure, which is basically the people who do nuclear simulations so that we don't have to set off bombs all over the place all the time to test things out. They have a lot of computational power across three different labs in the southwest of the country, and they want them to work together so they don't have to have people moving from lab to lab as the high-powered computers get installed at different places.

More recently the European grid has been vamping up with the Europeans building their own European union grids, and we're helping them out with that too. There are a lot of different test beds that are going on as well.

You can ooh and ah over the pretty pictures, but they really don't mean anything. They are just some of the things I was just talking about, application experiments. I'm going to skip over most of these.

A couple that are interesting, though, recently are the particle physics data grid, which is essentially building a data management, data storage and management grid for particle physics research; and the earth's systems grid, which is doing data management for climate resources. So you have large climate datasets, and you have to have access to them as well.

So these are a couple of the things we have been using for some of our data access and management technologies. A lot of the other ones are a little outdated.

Before we go into the actual technologies and start giving you the details about them, how do you get this stuff? For most of you, I think, the answer is probably going to be talk to your system administrator first. You might actually, at your institutions, already have Globus available somewhere or have access to an institution that has it.

For example, if your academic institution is a member already of the Alliance or NPACI, which are the two NSF funded test beds I was telling you about, then you probably already have the ability to get a grid account from those organizations, and your system administrator hopefully will know about that. Maybe you'll know about that because you got an e-mail a few months ago or a year ago or something like that saying, hey, you can get your grid account now, and you didn't know what to do with it. Well, now you know. Or at least you will in the next few hours have a good idea of what you can do with that grid account.

So give your system administrators a call and find out if there's any possibility that you might have these things, particularly if they have a large supercomputing resource of some sort, whether it's a cluster or an SP2, or who knows what. They may very well have looked into the idea of getting Globus on those machines, or getting some sort of software, Condor or something, available on those machines.

If you don't have that luxury and you actually have to do that yourself, then here's how you do it. Go to the Globus Project web site -- there's a screen shot there of what it looks like -- click on the Globus Toolkit. That's this blue button here. It will be blue when you put your mouse over it like I'm doing now. It won't be blue until you do that, so it won't be quite so highlighted there. But click on that button.

Once you do that, click on this download, download the Globus Toolkit link here. When you do that, you will have the ability to download the Globus Toolkit. It's a software package you download. You'll probably be running it on a Unix machine, so you'll uncompress it and you'll untar it and you'll get a whole bunch of source code and you'll do some build stuff.

You can learn about what's in the latest version, view the release notes. There's a system administrator's guide for the latest version so that you can actually have this manual. It's about this thick, and it tells your system administrator all they would ever want to know about the Globus Toolkit, including how to build it and install it, which is important. And then, of course, for system administrators, all the things like what does it do to my system, and what do I have to have to do in order to make it work and everything. And there's also some patches available. So this is where you want to be. This download to the Globus Toolkit page.

That's basically what there is to it. It's all described in the manual. There's a quick-start guide for you to get started using it, whether you got it from your system administrator or whether you installed it yourself, you can start using it that way. You probably won't know what to do with it until we go through the rest of the presentation.

For more information on it, in other words, if you're not satisfied, or if you have to leave early, or if, you know, you don't get what you need out of the rest of this presentation, go to www.globus.org. We have all sorts of research papers there, technical papers, white papers. We have tutorials, manuals. There are a couple of mailing lists that you might want to subscribe to, to find out what's going on with Globus. If you decide to install it, you certainly want to subscribe to our announce list so that you know about new releases and things that are available.

The discuss list is a good way to find out what other people who are using Globus are encountering and what problems they are encountering. For example, I had this problem on a Cray, or, you know, when I try to run this on RedHat 6.2 something bad happens. It's a good way to find out what you might want to be looking at. All the software, all the API's, a lot of the Globus Toolkit is software libraries that you link your applications against. All the API documentations.

You also might want to think about now, if you haven't already, attending Supercomputing 2000. It's really the conference to be at to find out about what is going on in the advanced computation world. It's in Dallas this year, and it's in November. So you might want to jot that down. Take a look at www.sc2000.org  and consider going to that. We'll be there. Miron will be there I'm sure, right?

DR. MIRON LIVNY: Right.

MR. LEE LIMING: Right.

DR. MIRON LIVNY: What did you say?

MR. LEE LIMING: He will. And you'll find all kinds of other things there that will be useful.

I have to give you the plug for the book. This is written by my boss, my boss and my friend, Carl Kesselman. Ian Foster and Carl Kesselman. Basically they used it as a textbook in courses that they've each taught. I know Ian has. It really lays out kind of the whole vision for the grid and some of the things that you need to have in it. I believe Miron has a chapter; is that right?

DR. MIRON LIVNY: Yes.

MR. LEE LIMING: So it's a good book. You'll want to take a look at it if this stuff interests you.

For the rest of the session, the rest of the afternoon, I'm going to take six sections, and I'll go through them as quickly as I need to, or as quickly as you allow me to, I guess. Hopefully, you'll stop me periodically and ask questions, otherwise, I don't think it will take the rest of the afternoon.

We'll go through each of those basic Globus services and show you what they do, how they work, how you would use them, give you some idea what it would feel like to use them. If it really interests you and you want to go the next step forward, what you want to do probably is get the Globus Toolkit somewhere. Either use an installation somebody else did or get your own, and then go back to www.globus.org/tutorial  and get some of the examples there and actually try it out for yourself, and then you can start building applications.

Are there any questions about the overview, which took a very, very long time, but hopefully it gave you a lot of information? Any questions about that?

I'm going to propose that we break for the next five minutes so you all have a chance to stretch and shake off some of the after-lunch tired eyes and all that. We'll meet back here absolutely no later -- I will start talking at 2:30 whether you are here or not, even it's an empty room.

Let's go back to one of the application examples that I told you about earlier, Distributed Supercomputing. The idea is you want to be able to seamlessly from the desktop, sign on once to some interface and then be able to submit jobs to different supercomputers and hopefully have them work together nicely.

There are some examples of this that we've done. We've worked with the ECC Project; Cactus, which does astrophysical simulations; HotPage, which is out at one of the PACI's out at San Diego Supercomputing Center, the NPACI group; NCSA's Chemical Engineering Workbench, which is where scientists and chemical engineers could look at molecules collaboratively and simulate them and use the high-performance computational capabilities to simulate what would happen with molecules if you did various things to them and stuff; WebFlow and LSA.

A whole bunch of examples here in desktop supercomputing. What you see over on the right is essentially a button for every one of the supercomputers available to this particular user. These happen to be supercomputers that are available at Argonne National Laboratory, Caltech, University of Chicago, and UIUC.

It shows you, for example, how many nodes total there are on those machines, how many nodes are free, and gives you some totals down here and a button that you can press if you want to submit a job to that resource. The idea being that what you probably don't want to have to do a bunch of gobbledygook to log into the machine, run a script, submit to the machine and all that kind of stuff. You want to have a nice web interface to it.

Another example is WebFlow. WebFlow is essentially a dataflow interface where you say I want to start out with this data source, run it through these filters, and then get my output here. And your output might actually be some display, like this display of a molecule here.

You have a whole bunch of controls that control what these filters do. It changes the simulation, and it goes off to the supercomputer, submits the jobs, gets it all there, brings the frames back and displays it for you, and it all happens seamlessly as if you just had a really powerful workstation at your desktop. Again, we need to have some grid services like authentication, process creation and management, and some common services like that.

What are the challenges here? The first challenge is security. How do you authenticate yourselves at the remote site? Not only how do you authenticate yourself to each site, but how do you arrange things so you only have to type in a password or do something once, and then you can use all the resources out there without having to deal with them individually?

Resource Specification. How do you locate and request a resource? How do you know there is a machine out there? How do you submit jobs to it, staging of the code and data. How do you take the program you want to run and get it on all those machines in the first place? How do you get the data files there? And then finally the computation. How do you start and manage the computation?

These are all the issues that come up when you want to do applications in distributed supercomputing or problem-solving environments. So we need to have a single sign-on for all resources. The idea is you don't want to have your users keeping track of a bunch of different passwords and accounts for different machines even though they're getting them from different organizations and everybody has their own standards for things. And you also want to make sure you never, ever send a plain-text password over the network. It's the death of any security system to send a password right over the network where anybody can sniff it or snoop it out or intercept it or whatever.

You want to have a uniform interface to all kinds of scheduling mechanisms. Every supercomputer seems to have its own type of scheduling. Some of them use PBS, some use Condor, some use LSF. There's a whole bunch of them. Some use Unix fork to start jobs. Those probably aren't very powerful supercomputers or else they'd probably be wanting to use something more sophisticated, but you never know. The thing is that we don't want to have to learn all the details of how to submit a job to these machines.

Finally, we have to have support for file staging, remote I/O and all of that. So let's look at security. Let's find out about how the grid handles security. The grid authentication model is done on a user basis. That is, each person authenticates as themselves. So we basically have accounts based on people. We don't have accounts based on organizations or projects or things like that. They may be associated with an organization or a project, but every person basically has their own credentials.

We can't prohibit the communication of plain text passwords. It just isn't allowed. Most sites will use conventional account mechanisms. That means that if you get access to a supercomputer, you are probably going to have to have the people at that center create an account for you, and that's what you'll end up using. Even if you don't know that you're using it, the account has to be there, and it has to be set up the way they set it up, and it has to be accounted for. All your cycles are going to be kept track of, and they'll send you a report at the end of the month saying you used 200,000 dollars' worth of cycles or whatever, and you'll go, whoa, how did I that? But anyway, there it is. Ideally, your application would tell you this before you do it.

And then sites may use generic grid accounts. That is, if, for example, NCSA had a supercomputer and they say, well, I just want to take everybody at the University of Wisconsin and treat them all as the same user. It's perfectly possible for them to do that. Just because every user has a credential, or grid credential, doesn't mean that the accounting mechanisms have to treat it that way. So the local resources can have accounts that look very different from the resources that are used for the grid.

The Grid Security Infrastructure, GSI, this is the grid, basically the infrastructure for security on the grid. It's based on Public Key Infrastructure technology, or PKI. I don't know if you've heard of PKI before. If you have, this will be a bit of review; if not, it will be something new and interesting I hope.

This is the same technology that's used to authenticate on the web. When you go to order something from Amazon.com and you type in your credit card number and it says, oh, by the way, you happen to be using a secure channel, you're actually using PKI. And what you're actually doing in that case is authenticating to a web server. You're basically being sure that the server you're talking to is Amazon.com and not somebody else so that not only is your credit card going over an encrypted channel, where basically your browser and the server's browser can talk to each other using some code that they agree on, kind of dynamically, but you also are pretty sure of who you are sending your credit card to and it's not somebody else.

And what happens is that we have a grid map file which maps from the grid identity, the identity you get from the people who run the grid, to the user account on a particular machine. For example, I might be Lee Liming or L Liming on a machine at Argonne, and I might be Liming on a machine at the University of Michigan, and I might be Lee L somewhere else. The grid map files handle all that. They take my user credential, which says Lee Liming at Argonne National Laboratory Mathematics and Computer Science and all this gobbledygook, and we've verified through so and so trusted authority that this is Lee Liming.

It takes that credential, turns it into Lee L off at NCSA, and they know who I am then. So it's kind of a two-step thing. You get your grid account, get your account at the resources you want to use, and the map file takes care of the mapping between them.

Let's have a brief review, one slide worth, or maybe two slides' worth of what is PKI. It starts out with mathematics. The idea is symmetric keys. In particular, these keys are, or these two numbers really, are picked via a very particular algorithm. And the algorithm is such that you can take one of these keys and encrypt data using it, and through some magic of mathematics the other number that you have can be used to decrypt that data.

The trick is the second number can't encrypt data, and the first -- basically, the second number can't encrypt data that can then be decrypted again using that same key. In other words, you have one-way encryption. You can encrypt using the first key, decrypt using the second key, and between those two things you actually have a security infrastructure.

So we call the first key, the one we encrypt with, the private key. You keep it to yourself. Only you can use that key to encrypt data. Then you give everybody else in the world your public key and say use this to decrypt whatever I send to you. So if I get a message from you and I can take your public key and decrypt the message, then I'm pretty sure that you must have been the person who sent me the message because only you could have the private key that could to decrypt that message. So we have a key pair public/private. They're very specific and they have special properties.

Then we have something called an X.509 certificate. This is a standardized version of a certificate. What it has in it is somebody's subject name, their identifier, their identity. It's usually pretty verbose. It will have a name, an organization, a bunch of information about you, nothing particularly interesting. It won't have your social security number or anything like that in it. It will have something about who you are and where you are from. It will have your public key. So when somebody gets your certificate, they get your identity, or some version of it, and they get your public key so they can decrypt stuff that you encrypt with your private key.

They also get this certificate signed by. What that really means is encrypted by a certificate authority. A third party out there that both of you trust. I trust Verisign. You trust Verisign. Verisign says I'm Lee. Verisign says you're so and so. As long as we have these signed certificates from Verisign that say we are who we are, then we can be sure when we talk to each other that we're talking to the right person. That's the whole idea behind PKI or Public Key Infrastructure.

How does this work in practice? Well, if I want to talk to a computer out there, a supercomputer, what do I do? First of all, I'll send my certificate over to the supercomputer. It will have software that receives my certificate and sends me a challenge string. It will say take this string and I dare you to encrypt it, and you say, okay. I'll take that dare. I'll take my private key and I will encrypt your challenge string, whatever you sent to me in that using my private key. The computer at the other end gets that. It uses the public key that was in my certificate that I sent at the very first step to decrypt that. If it's the same thing as what it sent out, it knows I am who I say I am, or at least it knows that somebody else, whoever signed my certificate, says I am who I say I am. Hopefully it's somebody we both agree on.

The trick here is to keep your private key private. It's very important that only you have that, because if somebody else gets it, then they essentially have assumed your identity and they can also encrypt data and fool the other end. So private keys are very important.

It's very important to protect your private key, so we have to introduce something else called a user proxy. The whole idea behind the proxy is to make it harder for somebody to get your private key. We do that by not using it very often. If you don't use your private key very often, there are less opportunities for somebody else to get at it.

We minimize the exposure by creating user proxies. It's a temporary credential you can use when your using the grid. What really happens when you use the grid now is the very first thing you do is on your local machine, before you talk to anything else on the grid, you create a proxy. You take your private key and you use it to create a proxy. That proxy is essentially another identity that you have used your key, your private key, to sign.

So you say, I'm printing this new identity and I'm saying it's me. It's not really me, but I'm saying that it's me, so it's okay. And the other end then can -- so when you authenticate with someone, you can use your proxy instead of your own private key, and you can use that proxy credential as a replacement or essentially a proxy for you.

The final piece of the puzzle here for grid computation and security is this concept of delegation. Remote creation of a user proxy. So not only do I want to be able to create a proxy at my end of the line, I want to be able to create a user proxy on the supercomputer. Why do I want to be able to do that? So the supercomputer out there can communicate with other computers on which I've started jobs.

Otherwise, let's say I started a job on Supercomputer 1 and Supercomputer 2. They both trust that I'm who I say I am, but they don't trust each other. If I can create a proxy over on Supercomputer 1 and a proxy over on Supercomputer 2, they can use those proxies then to talk to each other and prove to each other they are both processes created by me. That sets up the ability to have these triangles of trust across the grid, and that's very important.

So this is what the picture looks like. You have the user up here. User creates a proxy on their own machine. They have a Globus credential then at that point, a grid credential. They can use that credential to authenticate to Supercomputer 1. Use a piece of software called GRAM to start a process. It could, for example, use the GSI, the Grid Security Infrastructure software to get a local, maybe a Kerberos credential, if that's what that site uses, which then can be used to communicate with another site that you've basically done exactly the same thing on. This site might use public key cryptography over here.

The fact that you've authenticated on both these machines and created proxies on both these machines mean they can talk to each other and believe that they are talking to essentially proxies of you.

Any questions about that?

AUDIENCE MEMBER: Is there anything special you do or does it happen automatically when you create your own proxy and send something off?

MR. LEE LIMING: In this particular case you would use Globus software to submit the job to the remote machine over here. When you do that, it has GSI software built into it, so it takes care of all the sending a certificate over, creating a remote proxy, all that kind of stuff. Setting up the remote communication. So basically it's all kind of built in to the tools we provide. Does that answer your question?

AUDIENCE MEMBER: Yes.

MR. LEE LIMING: Did you have a question also?

AUDIENCE MEMBER: Yes. My question is, I guess maybe I'm missing something here, but I envision this grid of being this huge number of computers. Do I have to go through this process on every one of these computers that I'm going to have in my grid to get them to communicate if I had, like, a thousand computers out there?

MR. LEE LIMING: What you have to do is if you a thousand computers, you have to get one credential for you, and then the administrators of each of those thousand machines are going to end up getting certificates for each one of the machines. So maybe thirty organizations are going to get, you know, thirty certificates each, I guess, roughly, for their sites.

AUDIENCE MEMBER: Is that automated in the software? I mean, when I go out and --

MR. LEE LIMING: When you install the Globus Toolkit you have the opportunity to get those certificates for the machines that you are installing on, so it's part of the installation process. For a user, you will use software on any machine that has Globus installed on it. Basically run a little command, Globus cert request, that basically says I want a certificate, and it will tell you to type a little bit of stuff or whatever, send off the request, and you get the certificate and you plug it into your account. You basically run another tool to install it in your account. That's all you have to do.

It's pretty much the same thing you do when you get a certificate for a web browser. I don't know if you've ever done that or not, but you can, if you want to, get an ID from Verisign to prove you are who you say you are when you are talking with one of these servers out there. There aren't that many servers who care about that now, but there may be in the future. You can actually use the same certificate for your web browser that you use for Globus.

AUDIENCE MEMBER: How long are the certificates good for?

MR. LEE LIMING: It depends on the certificate authority, the people who sign your certificate. They have different policies depending on their concerns about security. In most cases they last for a year, sometimes two years, and then you get another one. So you basically go through the renewal process.

AUDIENCE MEMBER: I was thinking they only last for a day.

MR. LEE LIMING: The proxies only last for a certain amount of time. Right now our software lets you set it to whatever you want. So if you don't really care that much about somebody intercepting it, you figure, oh, it's pretty strong anyway. It's all cryptography. You can say create a proxy that will last for a whole year. Or you could create a proxy that lasts for thirty minutes if you're really, really concerned about security or something, or it's something very sensitive you want to do.

Any other questions? Before you run Globus applications you have to install Globus, obtain a GRID certificate and key, set up your environment so that Globus knows where to find it. That's basically running this cert request and then installing the certificate. You also have to contact the sites that you are going to run your jobs on and say put me in your map file. That means let your computer know that when my certificate comes over the wire, it's okay to start up a job.

So, for example, you might go to the administrators at Argonne and say put me in your map file. Here's my certificate. Stick me in there, or my subject name. Stick me in there. Or you might go to NCSA or the Alliance or NPACI or the NASA information power grid administrators and say I'm collaborating with you guys. Let me use some of your computers, and they'll just stick your ID into their map file so then their Globus software will recognize your certificate.

Notice that from a resource provider standpoint this gives them control over who can use their resources. it's not just anybody who has a grid account, it's only the people they put in their map file. It's kind of the first line of defense of security for them for control over who can use their stuff.

Every time that you log in, essentially to the grid, you create a proxy. And as long as that proxy is there, you're essentially assigned into the grid and you don't have to do any more authentication ever until this --

AUDIENCE MEMBER: Who keeps track on how much resources you can use? Is it built into your certificate?

MR. LEE LIMING: The question is who keeps track of how many resources you use and is that built into your certificate.

AUDIENCE MEMBER: Allowed to use.

MR. LEE LIMING: Allowed to use, right. The answer is only the resource provider can do that. So basically each resource provider can basically say I am going to limit my system to only starting so many jobs for this person when their certificate comes over the wire. It's not something that is built into the system right now. The resource provider would have to actually configure their system to do that. So maybe they would configure their scheduler or maybe they would configure the -- you know, we have shell scripts that get run when a job is submitted to your machine. You can put something in there that intercepts the request and says this person has already used up their allocation. They can't use anymore. That doesn't prevent you from going to some other resource provider and using their resources.

Remember, the whole idea here is that there is no central point of control for the grid. There are points of control at every single resource that are out there. Every resource owner can control what happens to their resources, but nobody can say, you know, I'll only allow you to use five grid sites and that's the maximum, no more. Who would that be? Who would have the right to say what you can use on the grid? It's basically up to the people who own the resources to say how much they will let you use. Does that make sense?

And then finally the Globus quick-start guide, which is available, again, through our web site. It actually goes through every one of these steps and says this is what you do to start using the grid.

This is what your certificate will look like. It's pretty ugly. Basically that's a bunch of encrypted data in there. The public key and the modules and exponent and all that kind of stuff. We don't even care about that. One thing you might care about is the fact that it has time stamps in the certificate for how long the certificate is valid. When does it start being valid and when does it stop being valid. That's the whole expiration date thing when you have to renew your certificate. In this particular case it looks like it was given out for one year from April 22,1998, through April 22, 1999.

If you try to use your certificate on a system that thinks it isn't that starting date yet, your certificate is not going to work. If you use it on a machine that thinks it's later than that, it's not going to work then either.

What we see up here is NTP. That's the Network Time Protocol. That is highly recommended. That will allow you to keep your system clock synchronized with all the other system clocks on the grid. We really recommend that people install NTP on their machines rather than relying on their system administrator to set the date properly.

In fact, it turns out when we get installation questions about: Why isn't Globus working? I installed it and now it's saying I don't have any certificate authority. What's going on? The problem usually is that the date is set wrong on the computer. And so that's probably one of the first answers we'll send out is: Did you check your date? Is the time the same on the machine you're submitting the job from and the machine your submitting it to? Chances are there's a clock skew there and they're off from each other.

So what does it look like to log into the grid? The very simple tool we provide with the Globus Toolkit is called grid-proxy-init. Grid-proxy-init asks you for a password. So you might say to yourself, well, wait a minute. I thought the whole idea behind this PKI stuff was I don't have a password. Why do I have to type in this password? Where do I set this password? What's all this stuff about?

The password is used to encrypt your private key. You actually encrypt your private key so even if somebody else can get into your file system on your computer and get it, it's encrypted. So the first thing that happens in grid authentication is you decrypt your private key, just for a minute. What happens then is the private key is decrypted in memory. It stays in memory. You create a proxy and then you throw away the private key information again. You only leave the encrypted version there on the disk. So basically you get at that private key using encryption technology, use it to create a proxy, and then throw it away.

So now you have a proxy. Again, once you have your proxy, now you can use other programs, which I think might be on the next slide, to get the information about the proxy. How long is it valid for and that kind of stuff. And you can, with grid-proxy-init, set how many hours you want it to be valid for. How much security you want. How many bits you're going to use in the proxy's encryption. Get help about the command itself. What are the options and things like that. That's the basic tool.

Now, if you're using somebody else's grid, like if you're using the Alliance's grid or if you're using NASA's grid, they might have another tool for you to use besides grid-proxy. Maybe it's a web-based tool. This is the one we ship with the software, so it's probably on your machines. Whether they tell you to use it or not, you know, you might get other instructions from them on how to do that.

So you would use grid-proxy-info to get information about your proxy. If you don't know for some reason which proxy you're using, which shouldn't be the case, if you don't, you could use grid-proxy-info to print out information about your proxy. What you'll probably find is -- actually, not this.

Let me do that. Let me see if I can actually do that for you. I'm logged into a machine at Argonne right now. If I do grid-proxy-info -subject right now, I'm hoping I will get an error. Yes. Why did I get an error? Because I didn't do this first. I kind of tricked you there.

So I have to type in my password, which I just violated security. I'll probably have to change my password even though you didn't see anything. But it went over the network between here and Argonne. I think it was encrypted because I'm using an encrypted communication between here and Argonne. Probably not a concern. What it did was decrypted my private key and gave me a proxy.

Now if I do grid-proxy-info -subject, there we go. All that information came in. Remember what the subject is? The subject name is my identity. That's who I am. Let's look at this. What does this say? It's an old one. It says country is US; organization is Globus; organization more detailed, I guess, is Argonne National Laboratory; organizational unit is mathematics and computer science division; CN is Lee Liming; and CN is proxy.

A little bit different here. There's a CN = proxy on the end. That basically means this is not really Lee. This is Lee's proxy. If we were to actually look at my certificate, you would not see that CN = proxy on the end of it. But I'm not going to give out my real certificate to the other machines on the grid because I don't trust them quite that much, so I'm only going to give out a proxy.

This right here has been a source of consternation to a lot of people. It starts out with country = US. When somebody in Italy gets the Globus Toolkit and they create a certificate for themselves, it says country = US. They're like I don't like that. You Americans. And so we actually changed this so now it says O = grid. So we kind of, I don't know if you want to say we freed ourselves from the United States or something, I don't know, but basically now when you request a certificate from our certificate authority it will say O = grid. It will start out with that instead of C = US. That's an example of what happens when you do grid-proxy-init. So now I have my proxy there and I can do stuff.

What does the grid map file look like? Your slides, in case you can't see something, are towards the back of your book. So if you want to look at that more closely, you can do that there.

The map file basically has subject names. All these things here, just like you saw. Doesn't have the proxy on the end. You don't need that because it's kind of assumed that everything you are going to get is going to be a proxy, and then the local user name. So basically you have a subject name and local user ID.

Some when my certificate comes over the wire, it's going to find Liming has the user name on this particular grid map file probably, and it's going to use that account when it runs a process for me. So all the local accounting stuff works the way it normally would. I just don't have to log in with an ID and a password. I log in with a grid instead. So if any accounting information is generated here, it's going to say ITF instead of all this stuff about Ian Foster. So all the accounting information will be done under his local user ID.

So what does it look like when you start a job? The first thing that is going to happen is you use your certificate and your key to get a proxy, a proxy certificate. The client workstation here is going to send that with a job request to something on the remote machine called a gatekeeper. I'll talk more about this a little later. This is more the resource management side of things than the security side.

But the gatekeeper is a special process which takes basically all of the stuff out of that communications channel about the security certificate. Takes the certificate out of there. Checks the map file. Says, okay, who is this person? Do I recognize the subject name? Is there a local account for them? And if they're not found in the map file, this request is just going to be ignored.

If there is an entry in the map file for it and there's is a particular user ID for it, it's going to potentially create a proxy on this side for that ID and then it will actually start the job up.

It will also take its own certificate and key, the machine's certificate and key, the machine's identity or secure ID, and it can use that then to send back its certificate and say, here's my certificate. I'm going to start sending you encrypted stuff. You can send me encrypted stuff. I know who you are because I have your public key. You can have secure communications then, encrypted communications.

Right now encrypted communications is not part of what is shipped with the Globus Toolkit because of export regulations from the past. Now that those export regulations have gone away, hopefully forever, we can probably start putting that back in there. It's actually something you can download separately and stick it into your version of Globus, but now we'll probably stick it into the main toolkits so you can get it.

Any questions how that works? Okay.

All right. Globus-job-run. Very simple command. It works just like RSH or SSH. All you do is say Globus-job-run, a host name and a program. Assuming that you have your proxy, assuming that you are in the map file for the remote machine, and assuming there's nothing wrong with the command you gave, it will run that command on the machine and send the output back to your screen. And, again, we'll be talking about this a little more later.

So technically, theoretically, if I go to globus-job-run -- this is the same machine I am on, so I don't know if it's going to be a problem or not. This is a very roundabout way of doing an LS command. There we go. So what happened here was that I submitted a job to the machine called pitcairn. The job was bin/ls. Through all the magic of Globus and everything it did all the security, it made sure I was authorized to use the machine, it did all the communication stuff, and it even redirected the output from this command that I ran remotely, or through a remote connection, to my output on globus-job-run.

Some a whole bunch of things just happened here. Looks incredibly unimpressive, but really a lot of stuff just happened. That machine could have been half way around the world at a machine that was run by some other organization that knew very little about me but trusted me enough to do an LS.

AUDIENCE MEMBER: Do you need some sort of a path?

MR. LEE LIMING: Command search path?

AUDIENCE MEMBER: You specified /bin/ls. If you would have just said LS, would that work?

MR. LEE LIMING: I don't think LS would, but if there were a command in my home directory there I think I would probably be able to do that. You don't have to know what your home directory is. You can say just say tilde slash whatever.

AUDIENCE MEMBER: The effective path of dot or, well, maybe tilde?

MR. LEE LIMING: I think so. I think it goes to your home directory, yeah. There are other ways to submit jobs. You probably wouldn't be using something this simple, probably. This is kind of good for debugging and showing you things, but there are other ways to do it. In fact, in most cases you are probably not already going to have your program on the remote machine. You are going to have to stage it there, and so I'll show you how to do that.

AUDIENCE MEMBER: The way it's set up, am I able to be do an LS in arbitrary directories? Is there any kind of change root involved, or does the person coming in have enough trust that they can look at arbitrary directories on that machine?

MR. LEE LIMING: It will use the account that is in the map file for you. So whatever you're authorized to do on that machine with that account, you'll be authorized to do it with globus-job-run.

Any other questions?

So what's being used here? What are the names associated with all of this stuff? GSI, Grid Security Infrastructure. That was used to authenticate. GRAM was used. That's the Globus Resource Allocation Manager. That was used to create the remote process and deal with all the local resource managers.

If that pitcairn machine that I submitted to only allowed jobs to be submitted through LSF or PPS or Condor, it would have happened without me even knowing about it. It might have taken a little bit longer for me to get the output because it would have had to go through the queue, but that's all. I wouldn't have had to know anything about how to submit the job. If I remember correctly, pitcairn just uses the Unix fork, so it just starts a job for you. But that's beside the point. I wouldn't have to know the difference.

GASS, Global Access to Secondary Storage was used to redirect the standard output of the command way over there wherever that was, in this case it was the same machine, back to the console of my program that was running, my submission program. So a whole bunch of different things came together.

So the key points here are you have a single sign-on. You can create remote processes securely with hard authentication. It's pretty secure. And you can add this remote job submission capability really easy. It's applications that you probably already have. In a lot of cases all it is going to do is change RSH to globus-job-run and make sure that you do a grid-proxy in there before you start the program up.

You can also use software libraries that we've written to build a set of the programs that you link against those libraries. So you could link against a library, and instead of doing the fork system call to start a job up, you could use globus-gram-job-submit or something like that to submit a job from a system call or a library call. So that's the key.

We just went through the first of the six topics. If you're looking at the clock and going, oh, no, we're going to be here till eight o'clock, I'll just tell you that these get faster as they go along because you've already gone through a lot of this stuff, and by that time you'll have seen enough that we won't have to go through quite so detailed for each one. You'll notice that the next section kind of gets a little shorter than the one before it, and eventually we'll actually hit 5:30. We'll probably have time for questions I hope, so we'll see how we do.

The next topic to go through is Information Services. What is the whole issue around information services? Well, if you remember way back to the introduction I did an hour or so ago, information services was the second area I talked about. The need to find out what resources on a grid are available for my use.

You might also want to know something about those resources, like what kind of machine it is. Is it a powerful machine, or is it a really rinky-dink machine that isn't going to do anything for me? Is it a really powerful machine that has tons of people using it and I'll never get anything into the queue, or a machine I'm actually going to be able to use? And also things like what kind of software does it have installed on it and stuff like that. I might need to know that stuff. I might have some requirement that I have to have a certain third-party software package installed or something like that, and I can't stage it myself so it's important that it be there.

So information services are what's going to tell us this information. Information services can also be used by administrators to kind of evaluate what the state of the grid is it any given point. How much of the grid is up and running. How much of it is in trouble. Maybe there's a network partition somewhere and you can't get to a bunch of machines. Maybe there's a machine that crashed and they have to fix it. Maybe there's a machine that has ten nodes taken off line because of maintenance or something. It's a good way of kind of keeping track of what's going on with that.

In the future we hope that information services will be the basis for adaptability and optimizations in the application itself based on what's available. So, for example, an application might find out there aren't that many graphics resources available to me, so I better do all my research in low-resolution mode. That way I can get the same amount of computation done and the user interface will still work, it's just going to be lower end.

I'll know to do that because I looked in the information services and I see there aren't that many graphics engines available. Or, maybe you'll be in a situation where your application that normally runs fairly slowly can be done really quickly, and you want to tell the user you have the option of using a really high-powered supercomputer for your application. Do you want to spend a thousand dollars and make that happen and get the job done today, rather than three months from now? And let them know. Let them have the option so they can adapt to the situation. You'd never be able to do that if you didn't have the ability to search for resources out there and get that information. So that's what we're trying to do with information services.

It's very important we have a general infrastructure for information. It can't be that you have to give five or six different commands to find out everything's that going on. You have to be able to ask a single system what is happening on the grid and get a simple answer back.

So here's a potential application, and actually there are a number of application groups that are going after this goal. There are some that have actually had some good success here. The idea is to have a Resource Broker. What is a resource broker? A resource broker is basically a service you can send a job to and say, I don't know what machines are available. I do know how much computing power this job is going to take, so I'll describe the job to you and you go off and run it somewhere. And maybe you tell me where it went and maybe you don't. All I care about is getting the job done.

A resource broker would do that. You might use Condor as a resource broker. You might use GRD as a resource broker. GRD is another one that's commonly used for that. So there are different sorts of things you might use for resource brokers. Having a generic resource broker or one where you can just say there are machines coming and going and all the time, some are really high-powered and they have really peculiar behaviors under certain circumstances and things, and, oh, by the way, I'm this particular type of user and I can only run my jobs on machines that I'm authorized to and only a certain subset of machines are able to do that, and all this stuff. It can be a really complex tricky business. So resource brokering can be really difficult to do.

One of the things that's important for a resource broker is to be able to ask this box over here, which we call the Metacomputing Directory Service, what resources are available? What are the restrictions on them? What are their configurations? The resource broker needs that information in order to know what to do.

It also needs to know things like not only what machines are available, but what network links connect me to those machines? What are the speeds of the network between me and there? If I have to stage a terabyte of data to a supercomputer in order to get my job run, you can be certain that I want to know what the network speed is between here and there in order to transfer a terabyte of data. It's not going to happen overnight, unless you have a really good network connection and really good software to get the data from here to there.

So it's very important that you have information services not only about the computers, but also about the other pieces of the environment like network and other field factors that might factor into your job. I think that's all we need to say about resource brokers right now.

What are the things that are useful to have in information services? Characteristics of compute resources like, for example, the IP address. I have to know how to get there in order to effectively use it. Software available on the machine. Perhaps that would be of interest to you. Maybe, maybe not. Maybe you have to have a certain math library. Maybe you have to have a certain application package installed there.

System Administrator. Who administrators this machine? If I submit my jobs to this machine and they never get through, maybe the user interface is going to tell me, oh, by the way, you might want to call John Doe over there and tell him to get off the stick and get their machine running because it's not taking any jobs. What networks is it connected to? OS version, load. How busy is the machine? That sort of stuff. It might be things that are interesting. So these are sort of the candidates for the pieces of information you might find in information services.

Characteristics of a network. Bandwidth and latency protocols, the topology of a network. All these things are things you might need to have in information services. Maybe the end users, the people who actually run the jobs, don't care about this stuff. But the resource broker that they use, or the application they use might need to know this stuff in order to adapt. It might need to say, we're not going to get to high-speed data transfers today, so I'm just going to do everything low resolution because there's just no other way to do it.

And then characteristics of the Globus infrastructure. Things like resource managers, hosts, what things out there are going on. If you're going to talk to this machine, what resource manager is going to control it? What policies are on it and things like that.

What do we provide? Well, the generic name for what we provide is the Grid Information Service, or GIS. The GIS provides access to both static, that is information that doesn't change very often, and dynamic information. For example, the state of a queue on a particular computing resource. So static and dynamic information regarding system components.

It can used as a basis for configuration and adaptation, figuring out which machines to run jobs on, which machines to turn into a dynamic Condor pool through a glide-in, which machines to use, which network links to use to get there. All that sort of stuff. How does your application want to adapt, if it does at all?

There are some communities that are very, very focused on adaptation. We're working with the DARPA Quorum Group an adaptation for them, which really means how do they optimize their resources to make sure that incoming missiles don't hit your warship? Adaptation is key there. It's very important to be able to adapt to different conditions.

Requirements and Characteristics. Well, I think I basically said almost all of that. Another thing that's key here is decentralized maintenance. That is to say, there is not necessarily any central body that maintains all the resources on your grid. In fact, if it's going be a real grid, the grid, that's going to be true, probably for sure. But there has to be some central place where you can get information about it. So everybody has to report information, not necessarily take the configuration from that central source, but they at least report it so other people can get to it. That's the GIS.

What does the Globus Toolkit provide? It provides an implementation of a GIS that we call the Metacomputing Directory Service. Metacomputing basically meaning computing on a metasystem or a large conglomeration of machines. Directory service basically meaning information.

So MDS maintains information in a directory structure. This is not a requirement for a GIS. The directory structure just happens to be the way we did it. So with us it's basically a hierarchal system. You start out at the root, everything there is to know about the grid. Go down to maybe organizations that belong to the grid. Go down to machines within the organizations or networks within the organizations. Go down to specifics about them. Maybe even go into the queue of the machine you're looking at. So it's done in the hierarchal tree structure. That's what it looks like.

It's distributed across many servers. There is no one single machine that has the whole directory tree about the grid on it. It's all distributed. So pieces of the grid keep their own information, and when you ask for it you go to those machines and get the information. The key, though, is that you have to know how to get to that information, that it exists, and there's a way of getting to it.

And each server, each type of information server, can be optimized for its own particular function. For example, you might have an information server running on a particular machine like a PC. Maybe it's a PC that happens to belong to a Beowulf cluster. The piece of the MDS that belongs on that machine is probably focused on retrieving specific pieces of information about that machine. You're going to ask, what is the load? How many processes are in it? How much memory is on it? How much disk space is free? It's not going to be all that concerned about indexing and information retrieval and doing all kinds of high performance searching and all that.

You might have another server central to the organization, maybe each organization has one of these guys, called an Organizational Information Server. Its job is to help up find machines within the organization that are going to suit your needs. That particular piece of the MDS, the Metacomputing Directory Service, has to be optimized probably for searching. Find me a machine of a particular type. It's not going to care about retrieving particular details about each one because you are going to go to the machines to get that information. So it's all distributed, and it's all kind of optimized for what's needed when.

The information comes from many different sources. This is a requirement of the GIS. You have to be able to have information put into the directory by anybody. An application that's running on a machine has to be able to publish information to the information service saying, I'm half way done. I'm three-quarters done. I'm 90% done. I'm done. You have to have each computer be able to say, here's my queue. I have this many jobs running. I have this many nodes free, come and start more jobs on me. You have to have information about all sorts of things to be available in the information services.

Remember, this is dynamic information. It can be changing at any time. It can be changing very rapidly from second to second. You can constantly be changing that information. That's one of the requirements. And the information has to be available anyway. It does not mean that it has to be accessible by everybody.

You can have access control on your information so that not everybody can see the contents of the queue at the ASCI Lab at Sandia National Laboratories where they're doing nuclear simulations. They don't have to publish the queue information. They might want to publish the queue information to authorized users, or they might want to publish a summary of the queue information, like 50% utilization, but you don't know what that particular utilization is and who's using it. So you have to have that kind of capability too. The key is it's got to be a source where you can get the information from anywhere if you're authorized to do that. It can't be segregated.

What are some of the features of MDS in particular? Well, there is the White Pages function, which is look up a piece of information that you know exists. I know that there's a machine called pitcairn@mcs.anl.gov, and I know that machine probably has a load average. Tell me what it is. You have to be able to do white pages look up, just like you could look up a person in the white pages of the phone directory. You know the person exists. They probably have a phone number. What is it?

There is a Yellow Pages service that is searching for computers or other resources of a particular type. In other words, find me a match. I have this kind of requirement. Find me something that meets my needs. It's essentially a searching function.

AUDIENCE MEMBER: If I find a match, how easy is it going to be for me to get authorization to run on that computer?

MR. LEE LIMING: It's a totally separate issue. Not part of information services problem.

AUDIENCE MEMBER: I realize that. It's part of my problem, and I'm asking you do I need to have, you know -- I mean, what kind of obstacles am I looking at for that?

MR. LEE LIMING: Let's say you find a machine that's out there and it's the perfect machine. You want to use this machine, but you don't have an account on it. One of the things you probably find in the directory services is who runs this machine. What's their phone number. Call them up. Hey, I want to use your machine. I'm so and so. I'm here in Dayton, and I want to use your machine. I'm working on this project, and it's a really cool project and you want to help me out, right? And they add you to their gridmap file, and you're authorized to use their machine.

AUDIENCE MEMBER: The fact that I'm trying to use Globus, is that going to buy me anything with them?

MR. LEE LIMING: It will mean they don't have to educate you on how to use their machine. That's a big win. In other words, they don't have to send you the user's guide. They don't have to tell you how to use LSF to submit jobs. They don't have to tell you what your account ID and password are.

So basically if you say, I'm a Globus user, and I want to use your machine. They'll say, oh, great. All we have to do is create an account and put you in the map file. We don't even have to tell you what your ID is. We just have to get your certificate from you and use it to set up the machine.

They might have to do some policy stuff. They might have to put in some limitations on how much you're allowed to use the machine, but they probably have a standard way of doing that already. The alternative being, of course, in the old days they would have to give you all the documentation for their system and answer all sorts of questions from you. This kind of takes all the administrative stuff out of the loop. Does that answer your question?

AUDIENCE MEMBER: Yes.

MR. LEE LIMING: Any other questions?

One other important point of information here about information is that the data that you find on the directory services often has an expiration date or some other measure of uncertainty about it. You're going to get, for example, the utilization of the machine. Load average maybe. It will probably have something attached to it saying, by the way, this is only valid for the next fifteen seconds, so then you have to ask again. So it will have a time stamp or something saying, I took this measurement at such and such a time. It's valid for the next N seconds or minutes or hours or whatever. So that way you'll know when it's a good idea to come back and get more information.

How do we build this information service? What are you installing when you install the MDS on your machines? We based the MDS on the LDAP protocol. LDAP is the Lightweight Directory Access Protocol. You can get LDAP servers from practically every computer vendor out there. You can get it from Netscape, from Microsoft. If you get Microsoft's active directory, it's LDAP. If you get Novell's directory services, it's LDAP. Most academic institutions use LDAP for their campus directories.

When you go to the web site and search for a person on the web site, you'll probably get some message back about LDAP. It's very commonly used and very commonly known. So we use the Lightweight Directory Access Protocol as the implementation of our directory. It's a standard data model. We basically say what sorts of information go in here. There's a standard query protocol. Why did we pick LDAP and not something else? It has white pages functionality, yellow pages functionality, and it's used widely. So basically it meets our requirements, and it's used widely.

What else do I have to say about this? We use the protocol LDAP. That does not mean we use the LDAP servers that are commercially available. We actually use the open LDAP library, which is an open-source version of LDAP, and we've written our own LDAP servers specialized for specific functions.

So we have things called GRIS's, GRIS. Grid Resource Information Service, which basically run on each machine and publish information about the machines. So you can go to the GRIS on a machine when you want to know like, what's the load average? How many people are using it right now? How much memory is on it. It's a specialized tool.

We have something called a GIIS, or Grid Index Information Server. That means you can go to this service and find out about other machines that are out there. It's a search service essentially. It's an index service kind of like Lycos or Yahoo! It basically goes out and crawls the grid and finds information and indexes it for you. So specific tools built on this LDAP protocol.

If you already have a server, if you already have a Netscape directory server or active directory, you can use that as your MDS server for your organization. We give you the information on how to set it up to do that and accept information to be published and all that.

What does the LDAP directory look like? I'm going to show you live data in a couple minutes, but what I want to tell you about now is that the directory is object oriented to some degree. They use an object model. They call them object classes and entries. That means if you want to put information in the MDS, you first define an object class.

What information is going to being in this particular type of entry? What am I going to call the entry? What are the possible values within each field of the entry? You basically put a schema in there and then you just start putting in entries of that particular object type into the directory. It also says what things are allowed to fall underneath it in the directory for that particular object class.

You might have an object class for a computer. You might have an object class for a network link. You might have an object class for a queue. You might have an object class for software. You might even have an object class for Globus software and find out what version of Globus is running on that machine. All these classes, and they're all defined already for you. You can also create your own classes and start sticking stuff in there too. So it's extensible. You can add stuff to it.

It's also organized. Not only is it defined using these objects, but it's also organized in a directory structure or Directory Information Tree, DIT. What that means is that everything's organized in a hierarchal tree structure, and in order to get to any particular piece of information, you have to walk the tree to get down to it. So the position in the tree is actually the name of that particular piece of information or the address of it.

How many of you have ever used the gopher service? Anybody remember gopher? You have basically this name of a piece of a file and it's this big long thing. It's actually the location in a directory. It's the same thing. LDAP is just the refined version of that sort of idea. The URL's are kind of similar too because the file system is directory oriented, hierarchal. Same thing.

Sample Object Classes. Compute Resources, Network Interfaces, Performance Data, all this stuff, it's all kind of defined as objects. Each object in a server is uniquely determined by it's distinguished name or DN. The DN is the name of an object in the tree. It's basically the path that you take to get to it.

For example, in this case we're talking about a computer. This is what a distinguished name, or if you were using security terminology, you'd say the subject name actually, of what a computer would look like. You read it backwards. You start out with O = grid, organization = grid; DC = EDU; DC = SDSC. What does this look like? It's the domain name of the machine. It's basically the domain name broken into components.

What we've done is, by default, the name that you give for you machine and the position for its information in the directory structure is based on the domain name service. So my machine pitcairn@mcs.anl.gov would be O = grid; DC = gov; DC = ANL; DC = MCS; and then pitcairn.mcs.anl.gov would be right there underneath all that stuff in the directory. So we basically took a name space that already existed for machines called DNS and are using it as our directory structure.

So the DN gives you a global name for your resource. Because DNS is already a global system, you get a global name space for all the information you'd ever want to have about a machine on the Internet.

The host and port uniquely name an LDAP server. So once you know the name of the machine and once you have the host name and port number on that machine, you can create a URL. And a URL for MDS, or for GIS, is an LDAP URL. Specify the protocol LDAP, just like HTTP is for the web, and then a host and a port.

So maybe it's pitcairn:6666 or something like that, and then your DN. That is, what position in the directory tree do I want? You've got a URL for an entry in the directory information tree, or, in the MDS. So it's basically a URL for a piece of information in MDS. So that can be used as the global name for information. So you can say the queue for my machine is available at this URL and give it out, and you can be guaranteed that pretty much anybody in the world can get to that if they have an LDAP browser.

Any questions?

What are the components used to build up MDS? OpenLDAP. I already mentioned that we use the OpenLDAP library, which is basically software you can get from a web site out there. It's open source. It's been reviewed by tons and tons of people, so we trust it fairly well. New versions come out periodically. We use that to build our own tools. Netscape, Oracle, Novell, Microsoft, everybody has an LDAP server out there that you can use if you want to just build a tree without using our tools.

There are tools for populating the information in the directory and for maintaining that. So tools for searching, tools for putting information in, tools for looking information up. These are all available.

There are API's. If you want to write a program that does something special with the directory services, we provide -- actually, OpenLDAP provides their API's so you can use the OpenLDAP library yourself to do all this stuff.

And then there are various tools for manipulating MDS contents. Things like a program that runs a queue stat program on your machine, gets the contents of the queue and pushes all that information out into MDS. Or if you want to, you can make it so when somebody comes to the GRIS on your machine, the Grid Resource Information Service, it runs the queue stat and then responds through LDAP. It's only done when you need it to be done. Not all the time. Various tools like that.

We've already talked about Grid Resource Information Server and Grid Index Information Service, so I won't go over that too much here except just to say the GRIS is mostly white pages stuff. If you want information about your machine to be available, put the GRIS service on it and people will be able to come to your machine and get that information using LDAP.

You can put a URL out there for it, and it will be available to people. So basically you just pop up your own LDAP server. You got it for free. It was easy to do. There it is. You basically made your machine part of the grid, at least in terms of information services.

The GIIS you'd use if you had a whole bunch of machines and you wanted people to be able to search for machines. So you bring up a GIIS server, which is also shipped with the Globus Toolkit, and you can have your GRIS's then, your individual machines, register with themselves with the GIIS. So you can say when you start up your machine, start up the GRIS and tell the GRIS in a configuration file, tell the GIIS about yourself and from then on anybody who searches in the GIIS can get information from that GRIS. A lot of GIS's here, but that's what you get.

Referral Services. Referral services are basically a term that we use right now for anybody who is running any LDAP server but putting grid information into it. So basically you can bring up Microsoft Active Directory and put in LDAP URL's at the leafs of the tree that point to GRIS's or GIS's or whatever else you want to point to, and it refers people to there for information about your grid. You could basically build up your own tree however you want to organize it and then refer to pieces of the grid.

The only thing I want to point out about the GRIS further is that it runs on a well-known port number. We worked with the Internet Assigned Names Authority to define a port number for GRIS, so when you go to your SC services file on you Unix machines or when your system administrator says what port should I run the GRIS on, you can say run it on the standard port, the one that IANA says you should run it on. The one that they say no other application on the Internet will ever use but the GRIS.

So the nice thing about this is that if you know a host name and you know that a host name is on the grid, you know what port number to go to get to its GRIS server. It's the standard port. It's going to be the same for everybody. Just like you don't have to say go to port 80 when you go to a web server, right? How often do you type in host.www.apple.com:80. You don't have to. It's going to be on 80. You know it's going to be on 80 because everybody puts it on 80. Same thing is true for the GRIS.

So when I did globus-job-run, I didn't have to do specify a port number. I just said run it on pitcairn and it knew what port number to put it on because it's a well-known port number. So your system administrators will be happy that you're not conflicting with somebody else's port numbers or anything like that.

The one thing that's important about GIS's, is you don't have to use them just to list all the machines in your organization. You can use them for other things. Like you can list all the machines that you're authorized to run jobs on. Make it GIS for that group of machines. That way you know whenever you do a search in the GIS for a machine, you're going to be authorized to run on it. You know because you put it there. Because you put it there after they added you to their map file.

You could make a GIS for all the machines that are RS6000, and you know that when you get a machine out of it, it's going to be a RS6000. You can make a GIS for all machines that belong to organizations that are part of the GrADS Project, for example. If the GrADS Project is a research project that you are working on and you want to know all the machines that you can run jobs on because as part of the GrADS Project you make a GIS for the grads project. I'll show you the GIS for the GrADS Project in a couple of minutes to see how this is actually being used in real life.

So basically you can use a GIS for exactly that, to index a certain number of machines. You say here's the index of machines or whatever. You can put network information in it. Whatever you want to.

Here's a key point: How do you make a big tree out of all this information? We've got thousands and thousands of machines out there that are parts of grids right now. How do you build a DIT that references all of that? In other words, how do you get your GIS's to know about machines? There are some options. We're not sure yet which is the right option. It depends on what you want your grid to do.

You can either make your GRIS's register with a GIS, or several GIS's, when they start up. Maybe my GRIS when I start up my machine will register with Argonne and it will register with the GrADS Project, and it will register with the Alliance so that my information about my machine appears in all of their directories.

You can also make it GIIS configured. You can configure it to look for specific machines. You can say, here is a list of machines that I want you to list information about. Go out and get it. And then you don't have to register. The GRIS's don't have to register. So basically you can be proactive about it, where you can force a machine to be on the GRIS -- or, the GIS, even if it's doesn't want to be. There's nothing to stop it, I guess, unless you access controls or something on it.

The other thing you can do is you can point your GIS server at a commercial LDAP server, like maybe you university's campus directory, and look for grid machines in it and see which machines you can find. That's another way you can do it. That's kind of the web crawler way of doing it. The web search engine method of doing things.

So any of the things are possible. Which ones you pick depends a lot on what you need. I don't know what you're trying to do, so you have all these options available to you.

Then you have the more difficult question, which is, how do I know what GIS' are out there? How do I know which directory services to go to? I guess the analogous question on the web is: How do I know where to go to find a search engine? You know, if you're a complete newbee and you don't know about Lycos, you don't know about Yahoo!, you don't know about Go or any of these things, which ones do you go to? That's tricky.

You have a lot of options here. Most of them are recursive. Most of them say, well, you go to so and so. And you say, well, how do I know about so? And they say, well, you go to so and so. And you say, how do I know about that? It's kind of a chicken-and-egg problem.

There are a couple of ways we thought about for doing that. One of them is the domain name service. Again, it has the ability to specify services. So you might go to Argonne's name service and say, where is your GIS service and it will tell you. That might be a good way to do it. We haven't really decided yet. This is still ongoing research. We're still figuring out how to tell people where GIS servers are.

For now we just build it into the applications. We say, if you want to run the GrADS application, you have to go to the GrADS server. The GrADS server, here is the host name and port number for the GIS and just build it in. That's probably good enough for today.

There are several tools that you can use with the MDS to help you browse, search, all that kind of stuff. There's a Java LDAP browser. The URL is provided here. It's in your notes. If you want a Java LDAP browser, go to this address and download it. Since it's in Java, you can run it pretty much on any machine. It's a nice graphical user interface. Basically, it let's you say, I want to connect to the LDAP server. Here's the host name and port number, and you can browse the directory tree underneath that and find all the stuff that's in there.

Web-based Browsers. There's a web-based browser, which I'll show you right now, at www.globus.org/mds. So anybody can come to this address, type in a server name and a port number, and start browsing the MDS at that location or the directory tree at that location.

There are various libraries that you can code against that let you do LDAP look-ups. Various libraries you can use for searching an LDAP server. There are even translators to translate from the way it's stored on a directory server and text-based files. So you can basically take a text-based file, convert it into the LDAP object data types and update an entry on a directory server somewhere out there, or you can do the reverse. You can look something up, translate it into text, and dump it into a file.

Let me actually show you what it's like to browse. This is really primitive right now. You know, ideally you'd want your application to do all this for you. You wouldn't want to have to sit here and muddle around with this yourself.

I'm going to click on the demonstration web interface. We have configured here a few places that might be interesting to start out at when browsing the MDA. There's the GrADS research project. There's the directory service for the NCSA Alliance. That's one of the big grid test beds. There's the organizational server for our division at ANL where we listed machines that we have that run Globus. There's a particular GRIS server. Remember I told you about GRIS servers that just run on one machine and tell you about that one machine. Here's one that runs on pitcairn. Here's one that runs on a machine at ISI.

Why don't we go to the Alliance. What happens here is it automatically fills in the server name, the port number, and the base that is the particular location in the tree I want to start out at. In this case C = US and O = Globus. They're using the old style naming at the Alliance. They haven't upgraded to the O = grid thing yet. You can also specify a refresh rate so it updates your screen as things change. But we're not going to do that.

Let me submit this. Wow. That was awfully fast. I didn't expect it to come back that fast. This is all the information in this table about that particular level. O = Globus; C = US. So what's in there? It says we have a top-level organization entry, and all it really says is a bunch of gobbledygook that I don't really care anything about at all. It's got some ACL's, Access Control Lists -- or, ACI, Access Control -- I can't remember what that stands for.

The interesting stuff is down here. This is the directory tree itself. I can go up a level to O = grid, or I can go down a level to various organizations. They also have some experimental stuff in here; groups, people, special users, or they actually talk about users of the Alliance grid. They did that themselves. We didn't give them that stuff. They did that themselves.

So let me go up and find an organization. We're going to look at NASA. I hope this works. Okay. So now we're at the NASA level. Notice our object class has changed to Globus top Globus organization. We have the name of the organization and a bunch of gobbledygook.

And down here we have a bunch of suborganizations. Ames Research Center, Glenn Research Center, Langley. Let's go to Ames and find out what they're talking about here. Look, they have a domain name nas.nasa.gov. That's really helpful. And we found out about some machines they have there. Evelyn, Toring. HN means basically Host Name; NN means Network Name.

This is a network interface right here, or probably a router, and they're going to give us information about that performance of that router. But I'm going to skip that and go onto Evelyn. Here you'll start finding some interesting information. A whole bunch of things about this entry. It's got all this information in it. Resource name. Host Evelyn. Host name evelyn.nas.nasa.gov. Canonical system name. That basically is what type of machine is it. It's a MIPS SGI Irix 6.5. Here's the host ID. It's a workstation.

Here's the operating system. It has 8 CPUs on it. Here's the load. The current load average, one-minute, five-minute, fifteen-minute averages. Here's the subject name for the machine, so when I connect to it I can check its security credentials against the subject name and make sure I've got the right machine. There is a time stamp here.

It looks like year and month and day and probably hour, minute, second, is what I'm guessing. Z probably means it's universal time. That's when this information was last updated. Notice they are advertising a load average for July 11, 2000. That's probably not going to be too helpful. I'll probably have to go back to that machine and find more about it.

Underneath that we have a whole bunch of interesting information. SW = globus. I can find out what kind of software Evelyn is running. Evelyn is running Globus 1.1.2. Our current release is 1.1.3. That's why they use C = US, O = globus, rather than O = grid, because they're using a version that's one version behind. This is NASA. They're not going to use the latest version. Are you crazy?

They have a whole bunch of configure information. This is how they build the Globus software. They used these configure options. So if it's important for debugging purposes, you can get that information.

I think you are starting to get the picture. We can get a lot of information about these machines. Here's a queue. PBS queue. What does it tell me about that? Probably not -- oh, default queue. All right. Tell me more. So what they're telling me, I think, is that there's nothing in the queue right now. It's empty.

Okay. So, Miron, start some jobs there. They have an unused machine at NASA.

DR. MIRON LIVNY: It's not the only one. I guarantee you that.

AUDIENCE MEMBER: How would you go about -- like, you say that's back to July. How would you go back to getting the information --

MR. LEE LIMING: Actually, is it? Let's see.

AUDIENCE MEMBER: I saw the date there.

MR. LEE LIMING: This is even later than that. This is June. I don't know. This machine hasn't been updated in a long time.

AUDIENCE MEMBER: It says July.

MR. LEE LIMING: July. You're right. I'm sorry. July 11th. It was unused on July 11th.

AUDIENCE MEMBER: Right. How would you go about getting that information for today?

MR. LEE LIMING: In this particular case, because they're running Globus 1.1.2, you probably can't. If they were using Globus 1.1.3 they would have a GRIS out there that would be running on that node and you'd be able to go directly to the GRIS, which would actually be referenced right here. In fact, it would probably be updating automatically if they were using the GRIS and the GIS and all that stuff.

In other words, you probably wouldn't have the problem with the latest version. If you did have a problem, you just go to the GRIS on the machine and you can say, I know it's evelyn.nas.nasa.gov. I know the port number is 2194, I think that's the port number, or 2145. I can't remember. Whatever the GRIS port is. Go directly there and ask it, and it will probably even run queue stat at that moment and tell you what's in the queue, rather than kind of giving you an old thing like this.

That was a big difference in 1.1.2 and 1.1.3. Major improvements to the directory services. Made it a lot easier for people to keep the information up-to-date. I think it's good you saw this example, because you see how important it is to check the dates and things like that and find out that sometimes they're not so up-to-date.

I'm not going to take the time, but I could go to the MCS version at ANL or the GrADS Project and show you that they're like within, like, five seconds of now or something because we're running the latest version.

Here is the Java LDAP browser. It's running under X Windows, but it's running in Java and basically you can see here pretty much the same stuff. O = globus, C = US, a bunch of organizations listed. For each organization you can view more information about the entry. All the stuff we just saw in the CGI browser shows up here. This is just a screen shot.

AUDIENCE MEMBER: If I install Globus on my machines, do I know about these other machines automatically, or do I have to add these servers?

MR. LEE LIMING: When you install Globus, you'll have the server side of the directory services available to you to run to provide information about your machines. You'll also have the client-side tools, which will allow you to query other machines. You can start up a GIIS at your site, an LDAP server. You can even put references in there to other machines at other sites if you wanted to, or you can use the client tools to go directly to their servers.

There is no global server, just like there is no global web server. You kind of have to know. You kind of have to be part of a community.

AUDIENCE MEMBER: I thought maybe Globus might run something.

MR. LEE LIMING: We used to. In fact, up until, I think it was a end of June beginning of July. July 1st, I think, was the date we finally shutdown the last of the Globus worldwide MDS servers. So we used to do it. You're just a little bit too late. But we stopped doing it because it got to be too much information and nobody using it. It was a lot of work to keep those machines up and running and not crashing when too many people used them and things like that. So we stopped doing it.

It was also a pain because in order for everything to show up there, people had to register with us. So we were telling people when you install the Globus software you have to register yourselves with our projects, and it could take a couple days for us to get you added. It was just a pain, so we stopped doing it.

All right. Grid-info-search. Important tool here. This allows you to search for information in the directory services. So the defaults for grid-info-search are a host name, and in this particular case these defaults are wrong. Grid-info-search is going to be configured based on how you installed the Globus Toolkit on your machine. When you configure Globus at your site, you are probably going to start up a GIS somewhere and that will be the default for your grid-info-search command on that particular machine.

You can also always specify another host name if you want to use somebody else's GIS, but we'll give you the default. You can specify a host name and port number, a place to look in the tree, how long you want the command to keep asking for information if the server doesn't respond immediately; and, do I want to search the entire subtree underneath that, or do I only want to search the directory object itself, or do I want to search just one level below and no more?

So when you do that, the output ought to be -- let's see if I can give you an example. Here we go. In this case we're using grid-info-host-search. That's a special variant that searches the GRIS rather than a GIIS. In other words, it searches only on one host. You specify a bunch of things you're looking for, and you should get output from it which specifies a DN, a distinguished name. In other words, what thing in the directory matched my search, and then all the information that is there. Object class, release date, blah, blah, blah.

If more than one thing matches, you'll get more than one of these types of output. It will just come first one, second one, third one, fourth one, and so on and so forth. You could write a shell script really easily, which took the output of this and then did globus-job-runs to start jobs on all the machines that come back.

You could do a grid-info-search for all the machines that are of this processor type, that have at least 4 CPUs, that have at least 256 megs of RAM, and I guess that's it. And it will give you a bunch of distinguished names, which will probably include the host name there. Write a shell script that parses that little bit out and do globus-job-runs on all those machines. You've just made yourself a little resource broker, a very simple resource broker.

What have you done? You basically took a program that looked for something in the MDS, got a list of machines and then used the resource management stuff to start your jobs. It's basically the definition of a resource broker. Very simple resource broker, but that's what it is. So this is not, you know, this is not really rocket science stuff. It's just basically cool to know it exists and you can use it.

AUDIENCE MEMBER: What you just described is a broker going out and looking at machines that you may not be authenticated on.

MR. LEE LIMING: Depends on where you start. If, for example, here you just used the defaults, grid-info-search and no options, no host name no port number, it's going to go to your local grid information index server. So it's going to be the one that you run, the directory server that you run, and it will look starting at your organization and go down.

So it will only do machines in your origination. If you specify something different, let's say you specify I want the Alliances LDAP server, and I want to start at the top of the Alliance, everything that's inside the Alliance, you might very well get machines that aren't at your site and you might not be authenticated to use them. You might do a globus-job-run on them and it will come back and say you can't do this. Sorry.

AUDIENCE MEMBER: (Inaudible.)

MR. LEE LIMING: That's not how the GSI works. We don't do trusting in the GSI. The only thing we trust is we trust the certificate authorities that sign our certificates, and we trust the map file that the system administrator put in there that says you're authorized to run on this machine. So one of the things that your resource broker, that you are theoretically writing, would have to do is check to make sure did that job actually execute or not. If not, I have to send it to some other machine.

AUDIENCE MEMBER: I was thinking about the Condor environment. If you set up a Condor pool --

MR. LEE LIMING: Condor pools might show up in the GIS too. You might do a search and what you get back is a Condor machine that is a job manager, that has a job manager under that subtype Condor. You submit to that thing, and who knows where it runs the job.

If you're authorized to run with that personal Condor, or with that Condor job manager, you might actually be able to start on a 100 machines or a 1,000 machines, and it will tell you in the GIS. It will say this is a Condor pool. How many nodes does it have? 2,400. You'd be like, whoa, I hit the jackpot. And sure enough, you submit to it and your authenticated to it, you can start up zillions of jobs on that machines, or at least queue them. Get them in the queue. So you can actually do that. You can start that out. We'll see that when we get to resource management.

You can filter the output. So in this particular case, we're specifying I want all object classes. Object class = star, and I only want the distinguished names and the host names. Makes it easier for your shell script to parse the output. So here's a DN; here's a host name. It always displays the DN. It's kind of annoying sometimes. Maybe you only want the host name. But it always prints out the DN, so you just ignore that.

If there's an object that matches your search but doesn't have a host name, it will still list it. So, for example, here we have service job manager, host name = pitcairn, blah, blah, blah. It prints out host name = pitcairn. That's good. Here there's a distinguished name that has the host name in it, but this particular object happens to be a queue. It's a queue on that machine. There's no host name associated with that queue, so it doesn't print one out. So you have to be a little careful.

You want to make sure that you're only getting things -- in this case you might have changed your query not to be object class, but to be object class equals, I think it's compute resource is the name you're looking for. What you really want to know is, I only want compute resources, and then they'll probably all come back with host names.

I'm wondering if this will actually work. Let me see if I can do this example. What your seeing right here is whether my copy and paste skills actually are up to spec. So what are we doing? You'll see this in a second when I hit return. Let me see if it runs. I'm a little concerned that it might not have the right defaults for -- grid-info-search minus help. This should tell me what my defaults are. It's going to look on pitcairn. It's going to look on port 6666. That's where the Argonne GIIS runs. It's going to start at the O = grid level.

So in theory that should work. What I'm going to do first is try this out with something I'm a little bit more confident about. I'm going to go back to that web browser. I assume I'm going to go back to that web browser. There I go. I'm going to go back here. This is the same machine pitcairn 6666, and I'm going to look up what I can look up there. Here we go.

Notice that we said object name DC = mcs.anl.gov grid. So I started out way down there in the middle of Argonne. Here are all the machines that are available. If I click on Yukon, let's see what happens. I am getting back good information. That's good. Let's take look at that time stamp. I made myself a liar. Last update -- oh. That's the last update for the host information.

The queue information presumably is more up-to-date than that. Let's make sure. Fork. Queue = default. I'm just checking since we're already here. No. Looks like it's still out of date. What version are we running? I don't know. I just wanted to check to make sure that I was probably going to get some useful data out of this thing.

Let's go back and run that command again. It worked. So basically, what did I get here? I got all the host names. That's what I asked for. Object Class = Globus compute resource. All the Globus compute resource entries in the directory. Asked for the DN, distinguish name, and the CPU load 1, 5, and 15.

This basically gave me the load averages of all the machines in Argonne in the MSC division that are running Globus. Useful information to have if you're writing a resource program. I think. I'm sure Miron could say it's probably not useful to know that. You just put it in every queue anyway. That gives you an idea what can be done with this.

MDS and Security. Some operations, particularly write operations, the ability to update the software on the directory service, requires authentication. So again, we use, right now -- actually, we don't. Right now, with version 1.1.3 of Globus, we just use really rinky-dink -- I think it's a password to update the information in the server. Not a very good way to do it. We'll be adding GSI based authentication soon, which, I think, means in our next release. I'm not positive about that because ISI is doing that. We're not doing that at Argonne, so I don't really know for sure.

The idea is you want to use GSI authentication for updating, and you want to set access control in the server for what each subject name can do. That's the summary. MDS provides the information needed to perform dynamic resource discovery and configuration. So dynamic information stuff is what MDS does.

Any questions about the MDS? That's the last of our big long sessions. Really because security and information services are like the bedrock that we have to have on the grid. Any questions? We're getting into that very quiet, sleepy mode.

Resource Management. Let's go back to an example, another example application. Distributed Interactive Simulation. This is the SF-Express. Remember the battlefield simulation I told you about, where we had four different supercomputers running. We already had the code parallelized. We already had the code available to run on different supercomputers.

The key thing we added here was the ability to manage those supercomputers. Start the job. Submit the job. Stage the data. Get the output from the jobs. All that we want it to happen automatically. How do we make that happen? So resource management is key for this. We need to be able to specify what required resources do we need and what are the constraints. That's resource specification.

We need to be able to co-allocate those resources. It's no good if we only have one supercomputer to run on. If we want to do 100,000 objects on our battlefield, we know we need at least four all at the same time running and talking to each other. You have to have co-allocation.

Executable Staging, Remote Data Access, I/O, all that kind of stuff we'll talk about in the next session. They all should be integrated with higher level tools like MPI and the globus-job-run and globus-job-submit and all those tools. Condor, grid, PBS, GRD. All these things should be able to use this resource management service. That's exactly what we are doing.

How does resource management work from this perspective? If we run globus-job-run on a client machine, what's really going to happen behind the scenes? We are going to go to a machine and talk to its gate keeper. We are going to send our certificate over there. We already went through this security stuff, so I'm not going to review that. You're going to go to gatekeeper, and he's going to say, I know who you are. It's okay.

It's going to start up what is known as a job manager. The job manager is a process that manages a job running on this resource. Then, depending on what kind of a machine it is and how it's been configured, it will use a local resource manager -- maybe it's the Unix fork command, maybe it's LSF, maybe it's LoadLeveler -- to start processes for you. Whatever processes you said to start.

So you go to the Globus gatekeeper, you go to the job manager. That gets started up dynamically for you. And then you use the local resource manager, local queuing system, whatever it is. That's basically the general model.

What are the components here? The components are job specification language, or Resource Specification Language, RSL. This is our current best effort, this thing, is to create Resource Specification Language. Basically it allows you to say what you need. The Globus resource allocation manager API, if you want to use an API.

That is, if you want to write a program and link against a library and call function calls, we'll allow you to do this in an application, or you can use things like globus-job-submit, globus-job-run, globus-run, and things like that. The tools that come with Globus to do it.

Doesn't matter what these things are in terms of what type of queuing system they have. It could be LSF, could be Condor, could be GRD. You don't know. You don't care. All you care about is you're authorized to use the machine. You might care what processor type it is so you know that your program will actually run, but that's it.

This layered architecture allows things like resource brokers and co-allocators and things like that to be defined terms of GRAM services. You can tell Condor how to submit a job via this GRAM interface. Or, you can tell the GRAM interface how to submit a job to Condor. You can stack these things on top of each other. Make bigger and bigger management resource systems.

I'm not going to guarantee to you that's going to work well. There are some combinations that will work well, and there are some combinations that won't. You have to be careful about what you pick for these layers. It is entirely conceivable that you can have a Condor personal job manager running on your local machine which uses the MDS to find machines out there, uses the glide-in mechanism to send the Condor server software off to those resources, basically turn the resource into a Condor pool, submit jobs using GRAM, again, to machines within that pool, which then queue jobs in LSF in those machines. It's kind of a contrived example, but you can do it and some people actually are. That's entirely possible to do that.

So what does this look like? You have your application. It uses RSL to talk to a resource broker, perhaps. This is not the way you have to do it, but it's one way to do it. The broker, which may be personal Condor or Condor-G might speak LDAP to the information service, get a bunch of nodes, send the information to a co-allocator, which might be anything, DUROC, whatever, that's one that we wrote, which would then go out to GRAM on all those machines and deal with the local resource managers to actually start the jobs running.

Notice there are arrows that go from the each of the GRAM's back to applications. That's the job manager. Common notation for exchange. RSL. RSL is basically a language that you can write little scripts in that basically say, I need a computer. It's got to have so many nodes. It's got to be such and such an operating system type. It's got to have this much memory, and all that kind of stuff, and, oh, at the same time I need to have this other computer. And I also need to have this other computer. And you can do all that on RSL. You basically specify what resources you need for your job. There are some API's provided for manipulating RSL's. So if you want to have some other format and you want to convert it into RSL, we provide you with some libraries to do that.

I think it's probably easier to show you some examples. Here's an example: &. We start out with &. What do you put after &, a bunch of things in parentheses. Basically this means all these things have to be true in order for the resource to match what you need. (count> = 5)(count<=10) I have to be able to create 5 to 10 instances. (max_time=240)(memory>=64) I have to be able to at least run a job for 240 minutes, and I have to be able to use at least 64 megabytes of RAM on that machine. My executable name, by the way, happens to be myprog. That's the name of the program.

That's basically what it means. Create 5 to 10 instances of that program, each on a machine that has 64 megs of memory and that's available to me for four hours. That's what RSL does for you. It allows you to specify these things.

You can do or's ("|"). This is basically, I want to run this program, and it's got to be either on a machine that I can have five instances with 64 megs of RAM each, or ten instances with at least 32 megs of RAM. I can split it up farther if I have do. You can do that with RSL.

So what's going on here? Your client machine talks to MDS, locates resources. You get resource information using the GRIS. So once you know where the machines are, use the GRIS to find out about them. It gives you candidates. That will probably talk to a local resource manager, so here's an important point: GRAM talks to the directory services. The GRAM component of Globus publishes information about the resource into the directory services. That's how that information gets in there. It's from GRAM, the resource manager and stuff. That's where the queue information comes from, the load information, how much memory does this machine have. That kind of stuff. That all gets into the GRIS via the GRAM stuff.

You submit stuff using GSI, it goes to the gatekeeper and creates a job manager. This is a Globus piece of software. The job manager will parse the RSL and start up the processes. Notice this arrow back from the job manager back to the client. You can use the job manager to find out rudimentary stuff about how your job is doing. Is it still there? Is it done? What's going on? Really, the key here is rudimentary. It's really, really, rough stuff. Probably not enough for a really production robust system.

That's one of the areas we're looking at next with Globus is how do we change this job manager thing so it provides better information. Like, oh, your job crashed an hour ago, or something like that. Rather than it's not there anymore, which is basically what it says now. So you kind of have to say, well, either it's crashed or it's done. I don't know which, but it's there. You probably know by that time whether you got any output or not, but maybe you don't. So maybe it's important you find that out.

The queue information and things like that all go into the local resource manager, which goes back into the GRIS which goes up, so there's lots of different paths of information here. Any questions about this diagram?

All right. You want to be able to do multirequests. What is a multirequest? It's basically co-allocation. That means you're allocating more than one resource at the same time. Somehow we have to fiddle with the schedulers and do whatever we have to do in order to get all of these things at the same time.

That's done using a software component within the GRAM thing called DUROC. You can read about that in or web site. Basically, I just want to tell you it's there and it allows you to use the + operator in your RSL, which basically means I want this, oh, and at the same time I also want this kind of resource. They've got to be the same time basically.

So "&" is used say these are all the things about a particular resource that I need. "+" is used to say this is a different resource, another resource. And we nest the parentheses appropriately, so this parenthesis and this one match up, and this one and this one match up and all that. You have to do that syntactic stuff.

You can take your globus-run or globus-job-run command and do some more stuff and submit RSL to it. So rather than the simple host name and job name I put in there, you can give it an RSL string. My executable is called this. It has to run on a machine that is this type that has this much memory and has this many processes, whatever.

And what will happen is globus-job-run will take the RSL and parse it, it will go to MDS, it will find the right stuff. It will submit the job to the job managers, and the job managers will take more RSL and do that kind of stuff. You can specify in your RSL executable staging and things like that. The program name is this, stage it for me. That's the next section. We won't get there quite yet, but it's coming up in a couple minutes. So you can do all that in these handy little tools. So it's knitting together a lot of stuff.

I will say something about this. This is kind of neat. So you can say with this +, I want to talk to this particular resource right here. I know exactly which machine I want it to go to. I want it to go to flash.isi.edu. I want it to use the fork manager. That is, you might have a choice. Do you put it through PBS? Do you put it through LSF? Do you put it through fork? What do you put it through?

In this case I specify executable my_app1. I want one instance of it. In this case down here I want a different machine. I want my_app2 and I want two instances of it. You can kind of specify complex stuff here. Different programs. Maybe they're different architectures. Maybe it's the same program but it was compiled for a different architecture, or maybe it's totally different programs and they talk to each other or something. I don't know. But it can all be put here and there, so.

So what are your options for submitting jobs with just purely Globus? Globus-job-run for doing interactive jobs. That is where you get the output from them. You can redirect standard input and standard output, so you can actually type stuff into your job while it's running.

Globus-job-submit, very similar to globus-job-run. The difference is it does it in the background. What does it give you? It gives you a URL back once you start a job. What does that URL have it in? Your output. So, basically, when the job is done, you can check it with these commands, plus some of the tools that are here. Globus-job-status. When it's done, you can go to the URL and get the output. So it's kind of disconnected.

And globusrun. Flexible scripting infrastructure. Basically globusrun knits together a whole bunch of features. I'm not going to go into it right now. You can probably read the instructions for it on the web sit.

Other people are building better interfaces than we are. This is key. We in the Globus Project do not specialize in writing fancy user interfaces or job management stuff or anything like that. A lot of other people are doing much better work in this area. Condor is a big one. Condor-G, PBS, GRD. The HotPage is a web portal or a web interface to the NPACI GRID out on the West Coast of the United States. So you can go to HotPage and submit jobs through it using a web interface. Much nicer than doing globus-job-run and things like that.

These are all better options, I think. They all have different applications. Condor-G, for example, will restart your job if it fails or if something goes wrong. It will remember you submitted your job and go back and check on it. PBS and GRD have similar functions to that. You can't beat a web interface usually. It's nice to use. There are some that are application specific; for example, ECC, Cactus, the other web portals that are being developed by other scientists. People write their own interfaces for job submission.

When you do a globus-job-submit, that's the batch mode. You can do globus-job-status, globus-job-cancel, globus-job-get-output, and globus-job-clean, to clean up after a job if there's any clean up stuff you have to do. These are all things you can do with the batch mode.

Globusrun. Again, you can use RSL with globusrun. It's very powerful. This language you can use to specify all the resources you need and co-allocation and all that stuff. Globusrun is also hard to use because you have to learn RSL to use it. You saw the examples. It wasn't that hard, but it's also not trivial to type these things in.

Starting jobs up. Submitting jobs. All that kind of stuff. One of the really great things about this stuff is you just don't have to remember anymore whether the machines use LSF or PBS or Condor or whatever. You just submit one command and away it goes. It's very nice.

The next section is on Data Access, and this is where we talk about all that executable staging and standard in and standard out. We've had this for a coupler years now in Globus, so it's fairly stable. We're always making little tweaks to it, but it hasn't changed a whole lot recently.

Remember that SF-Express example? Remember I said staging is important because of all this work you have to do to start the job up. Here we're going to see how we do that. We need executable staging. Remote data access. These things have to be important. They have to be integrated with all the other tools.

So what is Globus Access to Secondary Storage? What is GASS? To most people it looks like RSL extensions. Not only do you specify in your RSL what the executable name is, but you specify where to get it. Where to get standard input. Where to get standard output. Maybe even some extra things like transfer these ten files as well. Things like that.

GASS is also available as an API. You can link your programs with the GASS API. Globus_gass_ and a bunch of functions, open, close, read, write. All that kind of stuff. What does that do? It basically let's your program access files on the machine you submitted the job from or arbitrary other machines if you want to. And I'll show you the details of that in a second.

So basically what it means is, if you have a file, pretty much anywhere on the grid, including the machine you're using as a client to submit stuff form, you can access it with your Globus job if you use this API. You can use it the same way. You can use open/close; read/write; F read/F write. All that kind of stuff. F print f. All that stuff you that you could use with normal Unix file I/O, you can use over the grid. There are no special fancy commands. It's pretty much the same you would normally program. You do have to re-link, and you do have to change some of the code, but it's there.

What's the architecture? Basically this is really primitive stuff. It's basically staging stuff. It's not fancy I/O. It's just staging. You have the GRAM, just like you normally do on a resource, and it manages all the jobs that get started. It starts a job manager and does all that kind of stuff. Has a gate keeper and everything.

You create a cache on the machine. You create a place to put files. And then based on whatever the RSL said, you transfer files from the machine that submitted the job into this cache. And when you start your job up, you point it at this cache and say use this stuff for your standard your input. Use this file for your standard output.

When the job is over with, you take any output that went into standard output and ship it back as the standard output of the globus-job-run command. It's pretty basic stuff. Transfer the input files. When your done, transfer the output files.

And in your code, what would it look like? Normally you'd say FD = open and you'd get the file name. Here you say globus_gass_open and specify a URL. What will happen is the software that submits the job on the machine will fetch that data from that URL, whether it's HTTP or FTP or whatever, and stick it into that cache and you can use it in your program.

Your program doesn't have to know about all the complexity. It just has to have this command in there. When you close it and your job finishes, it will take it from the cache and send it back to wherever it got it from, assuming you still have permission to it and everything.

File naming. We use currently with GASS the HTTPS protocol. That is the secure HTTP. This is basically the same HTTPS that you use to talk to secure credit card sites. It's the same HTTPS that you use with the GSI, the Globus Security Infrastructure. Same credentials, same security, everything. It's all the same.

Host name, port number, and then the file name. This can be either like this, with a period/output data relative to the directory that all your jobs start up in and relative to your home directory, the place where the Globus cache is, or it can be relative to some account's home directory like tilde bester/myjob.

This can be any web server. You can point it to any web server. You can stage all your files on a web server somewhere, and then in your RSL say get your standard input from this file in a web server somewhere. Or you could do it through GASS and have GASS start up a little-bitty web server on the machine that you submitted your job from, and it gets it from there. Kind of tricky. Currently we support HTTP without security and HTTPS. The next release will support both FTP and gsiftp, which is the grid version of FTP.

These parameters can be added to your RSL with GASS; executable, standard in, standard out, and standard error. Executable and standard in get loaded onto your machine when you start the job, the remote machine that is, and standard out and standard error get pushed back to the submitting machine when the job is done, and the cache is cleaned up and all that is taking care of.

Here's an example. Executable equals, and here's a URL. It's going to fetch the executable from that URL. Standard in is that URL; standard out is that URL; standard error is that URL. All of these things get redirected into the program, so you don't ever have to copy the file over to the machine you're going to run it on. You never have to install binaries onto machines in order to run them. You can use GASS to do all that work.

Take your executable on your local home machine, my laptop right here could be such a machine, and I could say globus-job-run or globusrun, specify this RSL here. It's going to take the binary off my machine, put it onto that machine, take my input files, stage them out there. It will run on it. Take all the output files and put them back on my laptop when it's done. Really handy.

If you've ever done a big parameter study that has like hundreds of jobs and each input file is just a little bit different, it's so much easier to set them up on your machine, submit the stuff and have it take care of all that stuff, all the copying the files around and copying the input and gathering it all together into a big directory or whatever. You don't have to go looking on a hundred machines for where are all my output files. It's really nice.

That's just the user command. The tools that you can use. The programs you can use. There's also the API, the library. The library allows you to do this under program control, the program that you're submitting. The program that you're submitting can do a globus_gass_open(); globus_gass_close(); fopen(), fclose().

Once you get back the descriptor from that, you can do all the normal F print F, and F get S and all that kind of stuff will work on these descriptors. The only trick is it came from the cache on your machine, and it will be sent somewhere else when it's done. But your program doesn't care about that. You don't have to have any code in your computer for doing the file transfer. It's all built into globus_gass_open() and globus_gass_close(). You don't have to do anything but call that routine and link to our library. Very straightforward.

One nice thing about globus_gass_open() and globus_gass_close() is it only copies files when it needs to. If you start up three different jobs on a machine and they're all using the same input file, you don't copy it three times. You only copy it once. What's going to happen is, it's going to look in the cache. When you do a globus_gass_open(), it will look in this cache. Is the URL already there? If it is, it will use it. If it's not, it will download and put it in. So the first time it happens it will get it. The second time it will use the one that's already there. The third time it will use the one that's already there.

When you do a close, it checks to see, did you change the output file? Did you change the error file, the standard output or standard error. If you didn't, it's not even going to bother sending it back. If you did actually write something to it, then it will transfer it and it will clean it up afterwards.

Notice this remove cache reference. What does that mean? If you had three processes that were all writing to the same standard output, there's going to be a reference count on that. It's going to say there are three programs using this file. So the first program will close and it will say, okay, am I done? And it will say, there's still two other people using it. So it will wait till all of them finish and the reference count goes to zero. Then it will finally say, has it changed, yes or no, then ship it off if it has. So you only transfer it once. So you kind of try to optimize a little bit on the input and output there.

Globus-gass-server. You can actually run this program on any machine that has Globus installed on it. What does it do? Starts up a server, a very simple file server. In this case, since we support HTTPS, it's a very simple secure web server. It's kind of neat. You can start up a simple web server and connect to it and start getting files. So you can get use this as a generic tool without even using globus-job-run or globusrun.

If you ever need to just start up, you know, start up a little server, get some files off your machine and shut it back down again, there it is right there. When we add FTP to this, you will have a very simple FTP server; or, you might even have a server that can speak both HTTP and FTP, in which case you can connect either way. Different port number, but it doesn't matter. A very simple, lightweight server.

-r, -w, and -t mean basically do you allow reads? Do you allow writes? And tilde means do you handle things like tilde Liming, tilde bester, tilde whatever to get into people's home directory. There's a help also. So when you get Globus and download it to your machine and build it and install it, type globus-gass-server dash help and you can find out what you can do.

This is kind of the big picture. It's really big. So what happens here? Let's say we are using a program called MPI run. MPI run is basically a command to start up an MPI job. In this case it happens to be MPICH-G, which is a grid-enabled version of MPI. You want MPI run on your client machine, your desktop machine.

It's going to have an RSL string in it. It's going to create an RSL string, actually. It's going to take your user proxy certificate, which you got before because you did grid-proxy in it, right? You always do grid-proxy in it first. Then you do your MPI run. It's going to submit via RSL in multirequest, because this is MPI, and we're starting up a whole bunch of different jobs, probably on different machines.

It will give it to globusrun. Still on your machine. Globusrun runs on your machine, not anybody else's machine. It's going to use DUROC to handle the co-allocation, the coscheduling, to make sure all of the machines are available. It's going to use the GRAM client here to talk to the gatekeeper and the job manager and start up jobs on remote machines.

It will start up a gass server on your machine so that these guys, the jobs you start up, can get input and output. Can get input and write output. It's going to use Globus I/O, which is the next topic of discussion, to communicate and do message-passing. So your message-passing will be done using grid communication libraries to do high performance grid stuff.

AUDIENCE MEMBER: This GASS stuff here, that's even if you don't have any GASS open and coded in your program --

MR. LEE LIMING: I think this is assuming that you want standard input and standard output for your MPI job.

AUDIENCE MEMBER: This does the standard input and standard output without GASS open and stuff?

MR. LEE LIMING: Right. Without GASS open. You just write your program to read from standard input and write to standard output. You don't have to change your code at call. In this particular case, if you have an MPI application, all you have to is re-link against MPICH-G, which is part of the standard MPICH distribution that you would get if you got MPICH. Re-link against that MPI library, you don't have to change a line of your code, and you suddenly have a distributed MPI application. You can run a grid.

The whole idea is try to make this easy so you don't really have to do a lot. In this particular case, what you have to do is, if you don't already have it on your system, get the Globus Toolkit. Build it. Install it. Get MPICH-G. Build it. Install it. Or MPICH, I guess, and it will have the MPICH-G stuff in it. And re-link against that library and away you go. Of course, you have to get certificates and install them, and you have to get into peoples' map files. There is a little bit of set up you have to do.

So that's the big picture. Next section is communication. I'm just going to go right through. I don't think we have enough time for a break, or I'd give you another five-minute break. We're going to whip through these last two sections. I think they're pretty short. I think we have a total of twenty slides, at most, left, and I can easily do that in thirty minutes. Unless you all ask me lots of questions, which you can do too, and we'll just skip something.

IPC. Interprocess Communication with globus_io. There used to be, I don't know if you really care about this or not, but there used to be an older communication library for Globus called Nexus. Nexus achieved some notoriety for being really cool. This was even before Globus was out. It was a multiprotocol. It allowed multicast, so you could do all kinds of multicast, distributed broadcasting, network messages and things like that. Pick the best protocol for the conditions on the network. All kinds of wild stuff like that.

The problem with Nexus was you had to learn how to program with Nexus. Nobody liked to do that, so we replaced it with something called globus_io. The idea was keep the interface the same as you would have used any other time using any other library. You know, the standard open/close, read/write. All that kind of stuff. You want to be able to do things like multiplexing, listen, accept, connect. The same stuff you do for all your other network programing.

How many here do network programming of any sort; if you've written a program that used the listen call or the connect call? You have. You have. I figured there was some CS people out here.

I don't know if this will actually be of particular value, so let me just breeze through it, and some of you will get something out of it some of you won't. The key thing is: Why do we need anything other than standard sockets, standard Unix sockets and things like that? Security. Very important. Standard sockets and things like that do not give you security. They don't give you the GSI. They don't send your certificate over the wire automatically. The don't do all that checking stuff. They don't do all the authentication. It's very important to have a grid security capable communication library. That's the main thing you have in this.

The other thing is it's pretty high performance. We've actually done a lot of work tuning the globus_io calls to be the best possible performance we can get. It's about as good as TCP. Across a whole range of uses, it's about as good as TCP itself, raw TCP, so it's very important.

One of the other things about globus_io is it works really well on Windows NT. So Windows NT is different from Unix in that it has some fancy programming paradigms. One of them is called Completion. Do I even mention that here right now? No, I didn't. We want to make it easy to run on Windows NT, which is a little bit different from Unix in the way it uses sockets and file descriptors and things like that.

On Windows NT you can't treat a file descriptor that is for a file on your machine the same you can treat a network socket. You can on Unix. On Unix they're kind of interchangeable. You use the same data structure for that. So we have to kind of mask that difference so that your program can run on NT or run on Unix without any problems. You also want to make it efficient and all that stuff. That's what we're doing here.

So if you know anything at all about networking and Win32, you would probably know about completion ports and the idea that they kind of work like callbacks and have neat features in there that makes it high performance. Optimizes your use of a network. So this stuff is designed to run that paradigm. Even if you're using Globus, you get the benefit of using this stuff.

This is a little bit of what it looks like, and this basically shows you the programming interface. And you are probably going to having to program if you want to use globus_io. There are no user-level tools for this stuff. You have to be a programmer. Once you create a network connection, you have something called a handle or a file descriptor -- I'm sorry, a socket ID, a descriptor. You do an open or something like that and accept, connect, and it will return an ID.

You can take that ID and call something like this: Globus_io_attr_set_secure_authentication_mode. And all you have to do is pass the file descriptor and this identifier and automatically your connection becomes GSI authenticated.

So in other words, you will authenticate with the other end. You don't have to do anymore programming than that. Open up your connection, set the attribute for security, boom, there you go. It uses the credentials that you've got in your proxy. So you better make sure that you've got your grid-proxy-init done if you're doing this stuff, otherwise this is isn't going to communicate with anything. It's very important.

You can set secure delegation mode. What does that mean? Well, you either allow proxies that were created by proxies or you don't. So, in other words, I have a proxy. I use it to create a proxy. Now, is that proxy of a proxy recognized by other things or not? This is what controls that. You can say either none, limited, in that you allow proxies of proxies, but you don't allow proxies of proxies of proxies.

In other words, you allow one more level of proxy creation than you would otherwise. That's necessary for some applications. You have to be able to have a proxy create another proxy, but that's it. No more. After that, if that proxy credential gets out and you use it to create another proxy, you won't recognize it anymore. And then you can say full proxy, which means you'll just accept any proxy that comes your way as long as it has the original name in it somewhere you're okay.

Secure authorization mode. This says: Who will I accept credentials from at the other end? You have several options. One is self. That is, I'll only recognize the other end if it authenticates as me, but if it's somebody else I don't want to talk to them. So somebody else must have gotten ahold of the control line. I don't want that.

Or you could say identity, which basically means it's even more strict than that. The first one is kind of like me and any proxies of me; the second one is it has to be exactly the same certificate. So if you start out with a proxy, it's got to be a proxy at their end. It can't be the originator. It can't be a proxy of a proxy. It has to be exactly the same thing.

Callback. This basically means take the subject name that's out there and tell me what it is and I'll decide if I want to talk to it or not. So basically you can make your own access control stuff out of this. You can make your own filters that say, well, accept me or Joe or Ian, but I won't accept Steve.

The Secure Channel. That basically means is the data that's sent across this connection going to be encrypted or not. There's a difference between secure authentication and encryption. You can make sure that both of the ends of the communication lines are who you think they are but still have all the communication go across in clear text, which is the default; or, you can say, I want every piece of data going across this line to be encrypted so that only me and the other end can read what that data is. There are several ways to do that. One way is to use the GSI. One of them is to use SSL, which is the same protocol used on the web. So a whole bunch of options here.

Remember what I said about encryption export controls? How normally we don't ship the encryption code because of all the export controls on encryption. You wouldn't be able to use these two options if you didn't get the additional little piece that let's you do encryption. But it's freely available from our web site if you're within the right domain names and things. We're going to make it available to everybody. As soon as the Argonne lawyers are happy, we'll probably make it available. They're kind of slow in making that happen, but that's okay.

This is basically saying after you get all these attributes set and everything, you can use all of these functions to do standard things; select, cancel, close, read, write. These are the things you would normally use if you were doing Unix socket communications.

We add the globus_io part of it for compatibility reasons. So that no matter what platform you're on, it works the same way. You can still use standard Unix, read, write, connect, listen, all that kind of stuff. But if you're not running on Unix, it's not going to work for you. So we add the globus_io version so that you can always link against and it be sure to use it. So it's a portability feature basically.

Any questions about communication? I skipped over a lot of that because there weren't that many of you who said you did that stuff. Any questions about that?

Partibility Features. This is the very last piece, and it's probably the one we focus on the least because it's the oldest and everybody knows that portability is important. If you're doing grid stuff, you have to be able to write applications that will run on a whole bunch of different types of machines. So we provide in the Globus Toolkit some portability features that allow you to do some common things that very often work differently on different machines. These are the things that annoyed us a lot when we tried to write programs on different platforms, like the Globus Toolkit. So we wrote set routines that allowed us to basically mask the differences between different types of computers.

For example, one thing is we have a standard way of activating and deactivating modules. If you're writing a program and you want to us the GASS calls or you want to use the GRAM calls or the GSI calls, you can activate and deactivate the module of the Globus Toolkit.

You say, why do we need to activate it? Why don't we just link against it and run it? The problem is that a lot of these libraries use threads and have state associated with them, like they keep track of things. So in order for you to make sure that you don't leave memory links around or that you don't, you know, start something up or re-initialize something that's already in use, we provide you with an activate and deactivate routine for each module.

So if you, for example, have GASS code three places in your code, you can activate it, run your GASS stuff and then deactivate it. If, while you are doing that, another piece of your code is also doing GASS because you have threads in your program and you're running all over the place, you've got concurrent stuff going on. In other words, you don't have to keep track of that yourself. You can tell Globus activate, do the stuff, deactivate, wherever it is in your code. It doesn't matter how concurrent your code is, it will always work right because you always say when you're going into Globus code and when you're coming out of it. And these things are done in a thread-safe manner so you don't get into race conditions or anything like that.

We have a portable thread library, so you can do threads on different platforms. Threads are notoriously different on different machines. Even Unix machines have subtle differences between threads on this machine, threads on that machine. You can link on some machines against different thread libraries even. Like IRIX, for example, SGI machines, give you different thread libraries you can link against. They're different. How do you know which one you are supposed to be using?

We provide you with a portable thread library, and the cool thing about that is it uses whatever threads are available on your platform already. If you don't have threads, it emulates them. So you're pretty safe when you're using the Globus threads library. You're not going to have a performance hit, because if there are threads available on your machine, it uses them. Just passes everything through. Very, very, teeny-tiny little overhead. You won't even notice it.

If there are no threads available, you can still use thread functions. So you can still get concurrency, or at least emulated concurrency, in your code which is important for lot of applications on the network.

Thread-safe and portable libc wrappers. So you have threads. You have concurrency in your application. Now you want to say get host by name. You want to use a system call to look up a machine's IP address. Guess what? Get host by name is not thread-safe. So if you use that in one thread and you use it in a different thread to look up a different machine, they're going to interfere with each other because the operating system isn't thread-safe. And it says so. If you look in the man page it will say: This is not thread-safe. Don't use this concurrently. Make sure you put semaphores or locks around this code.

So we provide you with an alternate version of libc that is thread-safe. So you can call globus get host by name instead of get host by name, and it's thread-safe and you don't have to worry concurrent issues and things like that.

Timed and periodic callbacks. This is if you want to be notified every five minutes while your program runs and have a certain routine called every five minutes. Maybe it's a heartbeat or something, I don't know. Maybe it's going to send out a status message back to some program that's monitoring you. You can do that with these calls.

You can basically say call this routine every five minutes. And until you say stop calling this routine, it will keep doing that over and over again. Really handy stuff to have around. Everybody wants to do this stuff when they get into distributed computing, but there's no standard way to do it on all these machines, so we provide you with some standard ways to do it.

Data object and error object management. That means that if you are doing error handling in your code, we give you routines to create an error object. That basically means a specialized class or type data structure that contains information about the errors. So you can say this is not only an error object, but this is a Cactus error object from the Cactus application. It might contain different information, application specific information about the error, which then can be printed out by a user interface.

Finally, some very common data structures. Link lists, First-In-First-Out queues, URL's. Data types that everybody wants to work with, but maybe you don't have the best library to do these things with. Maybe it's not efficient or something was done a little bit wrong and you get a lot of overhead for it. We provide you with a standard set of good, well-written, well-reviewed libraries for doing these common data types.

The rest of Globus uses these features, and your programs can use these features if you want them to. You're not required to. You certainly can use any fifo code you want to, or any link list code you want to, but these things are provided for your convenience and they're all thread-safe. They're all well tested. They're all as high performance as we could possibly get them. We found the best fifo implementation. We found the best listen and select or select code when you're multiplexing input connections and things like that. So you're pretty sure to get good stuff here.

Globus module activate and deactivate I already told you about. It basically allows you to activate and deactivate the functions in a particular Globus module. So if you're going to use globus_io, the first thing you do is globus module activate then you specify globus_io as the module you are going to activate. Then you start calling globus_io calls. Then you do globus module deactivate and then specify globus_io. It takes care of all the nasty thread issues.

Threads. We provide you with Globus thread, Globus mutex, Globus condition. So if you want to do mutex operations, mutual exclusion, if you want to do condition variables, things that you've been hearing about in the other lectures, this is the code that will do that for you.

If you want to use threads but there's no thread support on your platform, you can use it with Globus. We can't guarantee it will be fantastic performance because you really only get good performance from native threads. But if this is the best you can get, then go ahead and use it. Why not?

Globus supports pthreads, cthreads, Solaris threads, and sproc. If you know about threads, you know there's a whole variety of them out there, and you can use any of them. When you build and install the Globus Toolkit, it will give you libraries that are linked against those various varieties of thread support, and you can support nonthreaded applications.

Globus libc I told you about that. Callbacks. I told you about that. This is basically the things that call you periodically or when certain things happen. When events happen, whatever happens, you get a signal.

Here are some of the convenience things; fifo's; hash tables; link lists; a symbol table. I'm not sure what the difference is between a hash table and a symbol table is; strptime, which is basically a time format I believe; URL's. If you want to parse or construct URL's, there's an object for doing that; and then the error types and objects.

We're done. Any questions? I'll bet everybody is really exhausted. I am. Well, you know where www.globus.org is. On that web site you know where the discussion lists and announce lists are. You have the notes in your handouts, and you have it everything that's on the web site, including the software itself.

I urge you to get it. Try it out. Take a look at some of the documentation. Ask other people who are using it. I think you'll find it's good stuff. Thanks very much.