The scheduler and the queue are two essential parts for a cluster. Together, they transform a group of networked machines into a cluster, or at least something closer to one. They're what allow users, working only on the head node, to submit "jobs" to the cluster. These jobs are transparently assigned to different worker nodes, and then - without the user needing to know where the jobs were - the results are deposited back into the user's home directory.

This process requires software in two different roles: the resource manager, responsible for accepting jobs to the queue and running jobs on worker nodes, and the scheduler, responsible for deciding when and where jobs in the queue should be run in order to optimize resources. I'll be using Torque for the resource manager and Maui for the scheduler. Both of these are open source projects.

Note: Torque has a built-in scheduler that can be used instead of Maui. However, Maui integrates seamlessly and provides more options and customization than Torque's scheduler.

Parte 1

Installation

Before setting up Torque and Maui, DNS must be working. If that's not an option, this requisite can be "cheated" around by setting up /etc/hosts on the head node with an entry for each of the nodes and then copying this file out to each of the worker nodes. (See the Cluster Time-saving Tricks page for help with the copying.)

Torque needs to be installed in two parts. First, a pbs_server is set up on the head node and configured to know where all of the worker nodes are. Then, each of the worker nodes are set up to run pbs_mom, a sort of client, that will accept jobs from the pbs_server and run them on the worker node. A basic queue for Torque also needs to be configured.

Maui is installed only on the head node, and needs to be set up to interact with the pbs_server. It does not communicate with the worker nodes, but instead talks to them by way of the server.

- Troubleshooting Torque and Maui, se algo der errado com a submissão de processos, consulte

Use and Features

After both are installed and working properly, you might want to look at

References

Parte 2

About Torque

From the Cluster Resources page on Torque,

"TORQUE is an open source resource manager providing control over batch jobs and distributed compute nodes. It is a community effort based on the original *PBS project..."

Because torque branched off from PBS, it still retains a lot of the old commands and names. PBS stands for portable batch system, and from here, I'll still call it torque, but commands may have "pbs" in them rather than "torque".

Installing Torque

Antes de instalar torque, instale todos os compiladores necessários compilers, senão aparecerá erros nos que estão faltando.

Baixe a versão mais recenete do torque em http://www.clusterresources.com/downloads/torque/

Baixe o arquivo no diretório e descompacte

cd /usr/local/src
wget http://www.clusterresources.com/downloads/torque/torque-3.0.1.tar.gz
tar xvf torque-2.2.1.tar.gz

Mude de diretório e veja o help do configure

cd torque-2.2.1
./configure -help

Usaremos

./configure --with-default-server=<your server name> --with-server-home=/var/spool/pbs --with-rcp=scp</code>

--with-default-server specifies the head node, which will run the server torque process. Be sure to replace your server name with your actual head node's hostname!
--with-server-home sets the directory where torque will run from. /var/spool/pbs is by no means standard, but it's the paradigm I'll be using. Others use a directory like /home/torque. I don't like confusing my processes with users.
--with-rcp=scp sets the default file-copying mechanism. Technically, scp (for secure copy) is the default, but if you don't specify it and scp isn't found, it'll move onto trying to find the next one, which we don't want.

If the ./configure finishes successfully, you're ready to move onto the next step. If not, address the issues before running the command again. When it does finish successfully, it will end with a line like config.status: executing depfiles commands, but no message about being finished. Next, run

make

A lot of what looks like gibberish will scroll by, and it may take somewhere around five minutes. Again, it will finish without a confirmation message. The last part of the script finished on mine with

make[3]: Leaving directory `/usr/local/src/torque-2.2.1/doc'
make[2]: Leaving directory `/usr/local/src/torque-2.2.1/doc'
make[1]: Leaving directory `/usr/local/src/torque-2.2.1/doc'
make[1]: Entering directory `/usr/local/src/torque-2.2.1'
make[1]: Nothing to be done for `all-am'.
make[1]: Leaving directory `/usr/local/src/torque-2.2.1'

Finally, you're ready to run

make install

You won't get a confirmation message for this, either, and it'll finish similarly to the way the last one finished. To make sure it was installed correctly, try using which to locate one of the binaries, like this:

gyrfalcon:~# which pbs_server
/usr/local/sbin/pbs_server

If it can't find it, double check that the binary was installed with ls and grep:

gyrfalcon:~# ls /usr/local/sbin | grep pbs       
pbs_demux
pbs_iff
pbs_mom
pbs_sched
pbs_server

If it's there in /usr/local/sbin, but which doesn't find it, you'll need to edit /etc/login.defs. Locate the line for ENV_SUPATH and add /usr/local/bin and /usr/local/sbin to it. The line for ENV_PATH should be right below it; add /usr/local/bin to it.

Configuring Torque

To start the torque server running on the head node and create a new database of jobs, issue

pbs_server -t create

Now, if you run ps aux | grep pbs, you'll see the server running. However, if you run a command to list the queues and their statuses,

qstat -a

you'll see nothing because no queues have been set up. To begin configuring queues for torque, we need qmgr, an interface to the batch system. You can run

qmgr

to start it up in an interactive mode, or enter the commands one at a time on the command line:

qmgr -c "set server scheduling=true"
qmgr -c "create queue batch queue_type=execution"
qmgr -c "set queue batch started=true"
qmgr -c "set queue batch enabled=true"
qmgr -c "set queue batch resources_default.nodes=1"
qmgr -c "set queue batch resources_default.walltime=3600"
qmgr -c "set server default_queue=batch"

Additionally, you can run commands to set the administrators' e-mail:

qmgr -c "set server operators = root@localhost"
qmgr -c "set server operators += kwanous@localhost"

Sanity Check

At this point, running qstat -q to view available queues should give you something like this:

gyrfalcon:~# qstat -q

server: gyrfalcon

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
batch              --      --       --      --    0   0 --   E R
                                               ----- -----
                                                   0     0

Excellent, we have a queue called "batch" and it's empty. You can also view your qmgr settings with

qmgr -c "print server"

Time to try submitting a job to the queue. First, switch over to a different user account (don't run this as root) with su - <username>. Then, try to submit a job that just sleeps for thirty seconds and does nothing:

echo "sleep 30" | qsub

The purpose of this is to see whether the job shows in the queue when you run qstat after submitting it. Below is a script of my testing it.

kwanous@gyrfalcon:~$ echo "sleep 30" | qsub
0.gyrfalcon
kwanous@gyrfalcon:~$ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
0.gyrfalcon               STDIN            kwanous                0 Q batch

Excellent, the job shows up! Unfortunately, though, it won't run... the state is "Q" (I assume for "queued"), and it needs to be scheduled. That's what we'll install Maui for later.

Introducing Torque to the Worker Nodes

Now we need to tell the pbs_server which worker nodes are available and will be running pbs_mom, a client that allows the the server to give them jobs to run. We do this by creating the file /var/spool/pbs/server_priv/nodes. With your favorite text editor, add each worker node hostname on a line by itself. If they have more than one processor, add np=X next to the line. Mine looks like this:

eagle np=4
goshawk np=4
harrier np=4
kestrel np=4
kite np=4
osprey np=4
owl np=4
peregrine np=4

Which that, configuration on the head node for torque is done.

Installing Torque on the Worker Nodes

Now we need to install a smaller version of torque, called pbs_mom, on all of the worker nodes. Move back into the directory we untarred earlier, /usr/local/src/torque*. There's a handy way to create the packages for the torque clients. Run

make packages

and they'll be created for you. This time you'll get a confirmation message:

Done.

The package files are self-extracting packages that can be copied
and executed on your production machines.  Use --help for options.

You'll see some new files in the directory now if you run an ls. The one we're interested in is torque-package-mom-linux-*.sh where the * is your architecture. We need to copy that file to all the the worker nodes. You can either copy it over to a shared NFS mount, or see my Cluster Time-saving Tricks on how to copy a file to all the nodes using the rsync command. I'm copying it over to my NFS mount with

cp torque-package-mom-linux-i686.sh /shared/usr/local/src/

Once it's on each worker node, they each need to run the script with

torque-package-mom-linux-i686.sh --install

You have a couple of options for doing this on each node. You can ssh over and run it manually, or you can check out my Cluster Time-saving Tricks page to learn to how to write a quick script to run the command over ssh without having to log into each node. If you're going with the second route, the command to use is

for x in `cat machines`; do ssh $x /<full path to package>/torque-package-mom-linux-i686.sh --install; done

Before we can start up pbs_mom on each of the nodes, they need to know who the server is. You can do this by creating a file /var/spool/pbs/server_name that contains the hostname of the head node on each worker node, or you can copy the file to all of the nodes at once with a short script (assuming you've created a file at ~/machines with the hostnames of the worker nodes as outlined in the Cluster Time-saving Tricks page):

for x in `cat ~/machines`; do rsync -plarv /var/spool/pbs/server_name $x:/var/spool/pbs/; done

Next, if you're using a NFS-mounted file system, you need to create a file on each of the worker nodes at /var/spool/pbs/mom_priv/config with the contents

$usecp <full hostname of head node>:<home directory path on head node> <home directory path on worker node>

The path is the same for me on my head node or worker node, and my file looks like this:

$usecp gyrfalcon.raptor.loc:/shared/home /shared/home

Again, this file can be created on each of the worker nodes, or you can create it and copy it over to each of the nodes. If you're using the latter technique, assuming you've created a machines file with all the host names, and you've created a config file, the command to run from the head node is

for x in `cat ~/machines`; do rsync -plarv config $x:/var/spool/pbs/mom_priv/; done

After you've done that, pbs_mom is ready to be started on each of the worker nodes. Again, you can ssh in to each node and run pbs_mom, or the script equivalent is

for x in `cat ~/machines`; do ssh $x pbs_mom; done

Everyone Placing Nice on Torque

Finally, it's time to make sure the server monitors the pbs_moms that are running. Terminate the current queues with

qterm

and then start up the pbs server process again

pbs_server

Then, to see all the available worker nodes in the queue, run

pbsnodes -a

(I don't know why this command doesn't have an underscore.) Each of the nodes should check in with a little report like my node peregrine's below.

peregrine
     state = free
     np = 4
     ntype = cluster
     status = opsys=linux,uname=Linux peregrine 2.6.21-2-686 #1 SMP Wed Jul 11 0
     3:53:02 UTC 2007 i686,sessions=? 0,nsessions=? ,nusers=0,idletime=1910856,totme
     m=3004480kb,availmem=2953608kb,physmem=1028496kb,ncpus=8,loadave=0.00,netload=18
     0898837,state=free,jobs=,varattr=,rectime=1200191204

Ready to continue? Move on to installing Maui, the scheduler.

References

Parte 3

About Maui

The Maui Cluster Scheduler, or just Maui for short, is a cluster scheduler from Cluster Resources. Maui needs to be installed on just the head node, and then Torque is used to submit jobs to this scheduler. Maui manages the clients by way of the pbs_moms.

Installing Maui

To get Maui, first visit http://www.clusterresources.com/downloads/maui/temp/ and find the most recent version of it. At the time if this writing, that happens to be the 27-Jun-2007 snapshot. Copy the link for the location of the file. From /usr/local/src/, issue the following command for the most current file:

wget http://www.clusterresources.com/downloads/maui/temp/maui-3.2.6p20-snap.1182974819.tar.gz

Next, untar the file with

tar xvf maui-3.2.6p20-snap.1182974819.tar.gz

Move into the directory that that just created with cd maui-*. We're ready to run ./configure (as part of the Source Installation Paradigm, which you might want to check out if this seems unfamiliar to you). We'll add a number of arguments. To see all of the possible arguments, type ./configure -help. What we'll use is this:

./configure --with-pbs --with-spooldir=/var/spool/maui/

--with-pbs makes it compatible with Torque
--with-spooldir sets it to use /var/spool/maui as its home directory

If it finishes successfully, you'll see a message and a confirmation, as shown below.

configure: NOTE:  link 'docs/mauidocs.html' to your local website for access to 
user and admin documentation
NOTE:  latest downloads, patches, etc are available at 'http://supercluster.org/
maui'

configure successful.

Next, run

make

If it finishes without an error, the make was successful. Finally, run

make install

and again, if it finishes without an error, that's a success. In order for mine to work, I had to edit /var/spool/maui/maui.cfg. (If you didn't change your spool directory during ./configure, yours will be located at /usr/local/maui/maui.cfg.) You should have a line like

#RMCFG[HEADNODE] TYPE=PBS@RMNMHOST@

where HEADNODE is your head node's hostname in capital letters. Comment out this line by adding a pound symbol, #, in front of it. Then create a line below it:

RMCFG[headnode] TYPE=PBS

where headnode is your head node's hostname in lowercase letters.

Starting Maui

Now maui can be started up on the head node. Maui installs the executable to /usr/local/maui/bin, so you'll want to add that as part of root's path. To do this, run

export PATH=$PATH:/usr/local/maui/bin:/usr/local/maui/sbin

(To make this a permanent addition, add the above line to your ~/.bashrc file.) Then run

maui

You won't get any output from it, but running

ps aux | grep maui

should show maui running now. In addition, running showq should show give you a nice view of jobs in the queue waiting to be scheduled. Currently there are none.

gyrfalcon:/var/spool/maui# showq
ACTIVE JOBS--------------------
JOBNAME            USERNAME      STATE  PROC   REMAINING            STARTTIME


     0 Active Jobs       0 of    0 Processors Active (0.00%)

IDLE JOBS----------------------
JOBNAME            USERNAME      STATE  PROC     WCLIMIT            QUEUETIME


0 Idle Jobs

BLOCKED JOBS----------------
JOBNAME            USERNAME      STATE  PROC     WCLIMIT            QUEUETIME

Sanity Check

By this point, you should have both torque and maui installed. Great! Continue onto the sanity check to make sure they're talking to each other.

References

Maui - PBS Integration Guide

Parte 4

Torque/Maui Sanity Check: Submitting a Job

A job is one particular instance of running a particular script or program of code. You won't want to run a job as root, so first, on your head node, become one of your users. (For instance, su - kwanous.)

Jobs are submitted to the job queue run by torque, which maui monitors and will then schedule, and torque will tell the pbs_mom client running on the worker node that maui picks to run the job. Jobs are submitted to torque with the qsub command.

Test: Sleep Job

An easy job to submit and monitor is just a sleep command.

As one of your users, enter the command that will create a job that simply sleeps for 30 seconds, as shown below:

echo "sleep 30" | qsub

Immediately afterward, run the torque command qstat to see the job appear in torque's queue, and then the maui command showq. You can even run

pbsnodes | grep -v status | grep -v ntype

to see which node the job is running on. A script of my output is shown below.

kwanous@gyrfalcon:~$ echo "sleep 30" | qsub
6.gyrfalcon

kwanous@gyrfalcon:~$ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
6.gyrfalcon               STDIN            kwanous                0 R batch

kwanous@gyrfalcon:~$ showq
ACTIVE JOBS--------------------
JOBNAME            USERNAME      STATE  PROC   REMAINING            STARTTIME

6                   kwanous    Running     1     1:00:00  Wed Jan 23 14:00:24

     1 Active Job        1 of   28 Processors Active (3.57%)
                         1 of    7 Nodes Active      (14.29%)

... snipped ...

Total Jobs: 1   Active Jobs: 1   Idle Jobs: 0   Blocked Jobs: 0

kwanous@gyrfalcon:~$ pbsnodes | grep -v status | grep -v ntype
eagle
     state = free
     np = 4

 ... snipped ...

peregrine
     state = free
     np = 4
     jobs = 0/7.gyrfalcon

Approximately thirty seconds later, the job should finish running. If you run qstat and showq again, you should no longer see the job (6.gyrfalcon, in my example) running.

Sleep Job Results

In the home directory of the user you've submitted the job as, you should now see two files, something like:

STDIN.o3
STDIN.e3

where 3 is the job ID. The file ending in .o# is all of the output in the form of standard out that came from the job. .e# is all the output from standard error. For our sleep job, both of these should be empty. sleep doesn't give any output to standard out or standard error.

Test: Standard Output vs Standard Error

Qsub can also take input in the form of files. These files can give all sorts of specifications to torque about how long the job will run and what resources it needs. (To learn more about qsub submission files, see Torque Qsub Scripts.) We'll write just a simple one. Open your favorite text editor and enter the contents of my Standard Output/Error For Loop Script and save this file to submission. This script has a simple for loop that runs from 1 to 10. If the number is less than 5, it will print a statement to standard output. If the number is greater than or equal to 5, it will print a statement to standard error.

Submit the job with

qsub submission

where submission is the name of the script file.

Job Results

Again, you should have .o# and .e# files in your home directory, but this time they should start with the name of the file submitted to qsub (submission). This time, they should have content in them. Your output file should have the first four lines, which were printed to standard output:

1 is less than 5
2 is less than 5
3 is less than 5
4 is less than 5

and your error file should have the last six, which were printed to standard error:

5 is greater than or equal to 5
6 is greater than or equal to 5
7 is greater than or equal to 5
8 is greater than or equal to 5
9 is greater than or equal to 5
10 is greater than or equal to 5

Hmm...

If you didn't get the results described on this page, visiting the Troubleshooting Torque and Maui page might be of help.

Minicluster:Torque

Índice

Parte 1

Installation

Use and Features

References

Parte 2

About Torque

Installing Torque

Configuring Torque

Sanity Check

Introducing Torque to the Worker Nodes

Installing Torque on the Worker Nodes

Everyone Placing Nice on Torque

References

Parte 3

About Maui

Installing Maui

Starting Maui

Sanity Check

References

Parte 4

Torque/Maui Sanity Check: Submitting a Job

Test: Sleep Job

Sleep Job Results

Test: Standard Output vs Standard Error

Job Results

Hmm...

Menu de navegação

Ferramentas pessoais

Domínios

Variantes

Visualizações

Mais

Pesquisa

Navegação

Ferramentas