·
4 min read
Useful Tips and Tricks for the BU SCC
Basic Commands
| Directive | Description |
|---|---|
-l h_rt=hh:mm:ss | Hard run time limit in hh:mm:ss format. The default is 12 hours. |
-P project_name | Project to which this job is to be assigned. This directive is mandatory for all users associated with any Med.Campus project. |
-N job_name | Specifies the job name. The default is the script or command name. |
-o outputfile | File name for the stdout output of the job. |
-e errfile | File name for the stderr output of the job. |
-j y | Merge the error and output stream files into a single file. |
-m beasn | Controls when the batch system sends email to you. The possible values are – when the job begins (b), ends (e), is aborted (a), is suspended (s), or never (n) – default |
-M user_email | Overwrites the default email address used to send the job report. |
-V | All current environment variables should be exported to the batch job. |
-v env=value | Set the runtime environment variable env to value. |
-hold_jid job_list | Setup job dependency list. job_list is a comma-separated list of job IDs and/or job names which must complete before this job can run. See Advanced Batch System Usage for more information. |
-t a-b | Runs an array job with task numbers a through b. This is one way to parallelize a job. To access the task number, use $SGE_TASK_ID. Then, you can query this variable inside a script to index a list, etc. |
-l mem_total=#G | Request a node that has at least this amount of memory. Current possible choices include 94G, 125G, 252G ( 504G – for Med. Campus users only). |
-l mem_per_core=#G | Request a node that has at least these amount of memory per core. |
-l cpu_ARCH | Select a processor architecture (sandybridge, nehalem). See Technical Summary for all available choices. |
-l cpu_type=TYPE | Select a processor type (E5-2670, E5-2680, X5570, X5650, X5670, X5675). SeeTechnical Summary for all available choices. |
-l gpus=G/C | Requests a node with GPU. G/C specifies the number of GPUs per each CPU requested and should be expressed as a decimal number. See Advanced Batch System Usage for more information. |
-l gpu_type=GPUMODEL | Current choices for GPUMODEL are M2050, M2070 and K40m. |
-pe omp N | Request multiple slots for Shared Memory applications (OpenMP, pthread). This option can also be used to reserve larger amount of memory for the application. N can vary from 1 to 16. |
-pe mpi_#_tasks_per_node N | Select multiple nodes for MPI job. Number of tasks can be 4, 8, 12 or 16 and N must be a multiple of this value. See Advanced Batch System Usage for more information. |
-l eth_speed=1/10 | Ethernet speed (1 or 10 Gbps). |
-l mem_free=#G | Request a node that has at least this amount of free memory. Note that the amount of free memory changes! |
-l scratch_free=#G | Request a node that has at least this amount of available disc space in scratch. |
Requesting Memory
-pe omp determines the number of notes and -l mem_per_core determines the amount of memory on each core.
| Job Resource Requirements | qsub Options |
|---|---|
| ≤ 16 GB | -pe omp 4 |
| ≤ 32 GB | -pe omp 8 |
| ≤ 64 GB | -pe omp 8-l mem_per_core=8G |
| ≤ 128 GB | -pe omp 16 |
| ≤ 192 GB | -pe omp 28 |
| ≤ 256 GB | -pe omp 16-l mem_per_core=16G |
| ≤ 256 GB | -pe omp 28-l mem_per_core=9G |
| ≤ 384 GB | -pe omp 28-l mem_per_core=13G |
| ≤ 512 GB | -pe omp 28-l mem_per_core=18G |
| ≤ 1 TB | -pe omp 36 |
Job Submission Commands
qsub filename.shsubmits the jobqstat -u usernamechecks on the job status and allows you to find the job id`qdel jobiddeletes the job with the provided jobid
Array Job Notification
To get notified when an array job is done, submit a job like this:
$ job_id=9156754
$ echo 'date' | qsub -N "Array_job_${job_id}_done" -m b -l h_rt=00:00:05 -hold_jid "${job_id}"
where job_id is the id of the main array job you submitted.
