Cheat sheetsBU SCC Tips

BU SCC Tips

Tips and tricks for the BU SCC.

These tips are mostly taken from the BU SCC documentation site.

Basic Commands

Directive	Description
`-l h_rt=hh:mm:ss`	Hard run time limit in `hh:mm:ss` format. The default is 12 hours.
`-P project_name`	Project to which this job is to be assigned.
`-N job_name`	Specifies the job name. The default is the script or command name.
`-o outputfile`	File name for the stdout output of the job.
`-e errfile`	File name for the stderr output of the job.
`-j y`	Merge the error and output stream files into a single file.
`-m beasn`	Controls when the batch system sends email to you. The possible values are – when the job begins (`b`), ends (`e`), is aborted (`a`), is suspended (`s`), or never (`n`) – default
`-M user_email`	Overwrites the default email address used to send the job report.
`-V`	All current environment variables should be exported to the batch job.
`-v env=value`	Set the runtime environment variable `env` to `value`.
`-hold_jid job_list`	Setup job dependency list. `job_list` is a comma-separated list of job IDs and/or job names which must complete before this job can run. See Advanced Batch System Usage for more information.
`-t a-b`	Runs an array job with task numbers `a` through `b`. This is one way to parallelize a job. To access the task number, use `$SGE_TASK_ID`. Then, you can query this variable inside a script to index a list, etc.
`-l mem_total=#G`	Request a node that has at least this amount of memory. Current possible choices include 94G, 125G, 252G ( 504G – for Med. Campus users only).
`-l mem_per_core=#G`	Request a node that has at least these amount of memory per core.
`-l cpu_ARCH`	Select a processor architecture (sandybridge, nehalem). See Technical Summary for all available choices.
`-l cpu_type=TYPE`	Select a processor type (E5-2670, E5-2680, X5570, X5650, X5670, X5675). SeeTechnical Summary for all available choices.
`-l gpus=G/C`	Requests a node with GPU. G/C specifies the number of GPUs per each CPU requested and should be expressed as a decimal number. See Advanced Batch System Usage for more information.
`-l gpu_type=GPUMODEL`	Current choices for GPUMODEL are M2050, M2070 and K40m.
`-pe omp N`	Request multiple slots for Shared Memory applications (OpenMP, pthread). This option can also be used to reserve larger amount of memory for the application. N can vary from 1 to 16.
`-pe mpi_#_tasks_per_node N`	Select multiple nodes for MPI job. Number of tasks can be 4, 8, 12 or 16 and N must be a multiple of this value. See Advanced Batch System Usage for more information.
`-l eth_speed=1/10`	Ethernet speed (1 or 10 Gbps).
`-l mem_free=#G`	Request a node that has at least this amount of free memory. Note that the amount of free memory changes! Request a node that has at least this amount of available disc space in scratch.

Requesting Memory

-pe omp determines the number of notes and -l mem_per_core determines the amount of memory on each core.

Job Resource Requirements	qsub Options
≤ 16 GB	`-pe omp 4`
≤ 32 GB	`-pe omp 8`
≤ 64 GB	`-pe omp 8` `-l mem_per_core=8G`
≤ 128 GB	`-pe omp 16`
≤ 192 GB	`-pe omp 28`
≤ 256 GB	`-pe omp 16` `-l mem_per_core=16G`
≤ 256 GB	`-pe omp 28` `-l mem_per_core=9G`
≤ 384 GB	`-pe omp 28` `-l mem_per_core=13G`
≤ 512 GB	`-pe omp 28` `-l mem_per_core=18G`
≤ 1 TB	`-pe omp 36`

Job Submission Commands

qsub filename.sh submits the job
qstat -u username checks on the job status and allows you to find the job id`
qdel jobid deletes the job with the provided jobid

Array Job Notification

To get notified when an array job is done, submit a job like this:

$ job_id=9156754
$ echo 'date' | qsub -N "Array_job_${job_id}_done" -m b  -l h_rt=00:00:05 -hold_jid "${job_id}"

where job_id is the id of the main array job you submitted.

Using VS Code

The use of VS Code is highly discouraged by the SCC people, and they would much prefer that we all use OnDemand. However, it is pretty annoying to have to use OnDemand in the browser to do all development work. They did provide a workaround, but it is a fair amount of work.

On SCC OnDemand, request an interactive session using the “Desktop” interactive app with whatever resources you need. Wait for this session to start.
In the interactive session, Load the vscode module using the command module load vscode/1.79.2 in the terminal
In the interactive session, Create a tunnel using the vscode command code-tunnel in the terminal
Accept the agreement, cancel any keyring pop-ups, and write down the 8-character authentication code.
Login your Github account and enter in the authentication code.
In the interactive session, accept whatever the default tunnel name is (in my case it was scc-wl2).
On your local machine, open VSCode and install Remote – Tunnels. Select Remote Explorer icon in the left dashboard and new section called “Tunnels” should appear. Under Tunnels select Sign in to see the registered tunnels. Sign into your Github account and available tunnels will now be down as online/offline.
Select the tunnel name created in your interactive session (e.g., scc-wl2). This should connect your local VSCode to this interactive session on the compute node.