If you are not accustomed to using a Linux command line interface (CLI), we recommend familiarizing yourself with introductory material such as 12 Commands Every linux Newbie Should Learn OR Linux Command Line Basics. The ability to navigate and manage files at the Linux command line is important in order to work effectively.
To connect or log in to the cluster you will need to use NCSU Campus Network and Secure Shell (ssh). If you are new at SSH please see: SSH 101. People who are off the NC State campus are reminded to use the Virtual Private Networking (VPN) service when connecting to the campus network. Please see NCSU VPN for more info.
Once at a command line you can connect to the cluster using commands like one of these:
ssh username@brccluster.statgen.ncsu.edu
Login Nodes:
you can connect to the cluster using SSH to any of below machines. These are just different names for the same machine.
For security reasons, you are encouraged to using passwordless SSH with public authentication keys. (It's also more convenient once set up.) Please follow easy steps mentioned on these articles. How to Setup SSH Passwordless Login. 3 Steps to Perform SSH Login Without Password Using ssh-keygen & ssh-copy-id
Note : From October 18th 2021 OIT blocked the default SSH port (port 22) at the campus gateway. You should connect to the campus VPN and then use SSH to get to the cluster. Along with SSH other protocols like SFTP, SCP, and SSHFS are also affected by this change.
For Job and Resource management we use SLURM workload manager software.
Here is the brief overview of some of the slurm commands:
* sbatch - Job Submission * scancel - Job deletion * squeue - List all jobs in queue * sinfo - List all nodes * scontrol show nodes - Show information about nodes * squeue --start - Job start time * scontrol show job <jobid> - Job information * sacct -u <username> --format=JobID,JobName,MaxRSS,Elapsed - View details of all jobs by user * showq - shows cluster state * myslurmaccts - see which accounts and partitions you can use * lsload - Get information on node state
The environment module system is used to manage the user environment, allowing you to load and unload software packages and set up environment variables for your jobs. It also controls which software is in your path, when you log in to the cluster. For more details please refer: Environment Modules.
To get the list of available software modules execute module avail command. As of 30-Jan-2023 below is the list of available softwares.
|user_name@sisko:[~]> module avail ------------------------------------------- /opt/modules/modulefiles -------------------------------------------- abyss/2.1.5 dot igv/2.10.0 perl/5.34.0 samtools/1.13 abyss/2.3.2 eigensoft/7.2.1 java/openjdk-8 picard/2.25.6 seqtk/1.3 augustus/3.4.0 exonerate/2.2.0 java/openjdk-11 plink/1.07 snap/2013-11-29 bamtools/2.5.1 fastp/0.21.0 java/openjdk-20.0.2 plink/1.90 sratoolkit/2.11.0 bbtools/38.90 fastqc/0.11.9 jellyfish/2.3.0 python/2.7.18 stacks/2.64 bcftools/1.13 fftw/3.3.10 julia/1.6.1 python/3.9.5 STAR/2.7.9a bcl2fastq/2.20 fseq2/2.0.3 locuszoom/1.4 R/3.5.3 temposeqcount/1.4.1 bedtools/2.30.0 gatk/4.2.0.0 LTR_retriever/2.9.0 R/3.6.3 TRF/4.09 blast+/2.11.0 gcta/1.24.7 mafft/7.481 R/4.0.5 Trimmomatic/0.39 blast+/2.14.0 gcta/1.94.0beta maker/3.01.04 R/4.1.0 upc++/2023.3.0_smp bowtie/1.3.0 GeneMark-ES/4.68 matlab/R2021a RECON/1.08 use.own bowtie2/2.4.4 GenomeTools/1.6.2 module-git RepeatMasker/4.1.2 vcflib/1.0.2 bwa/0.7.17 globuspersonal/3.1.6 module-info RepeatModeler/2.0.2a vcftools/0.1.17 cd-hit/4.8.1 globuspersonal/3.2.2 modules RepeatScout/1.0.6 wiggletools/1.2 cutadapt/3.1 go/1.16.4 Ninja/0.98-cluster_only RMBlast/2.11.0 diamond/2.0.11 hmmer/3.3.2 null Ropen/4.0.2 dida/1.0.1 hyphy/2.5.31 perl-threaded/5.34.0 samtools/1.12
There are many ways of dividing up and managing a cluster. Partitions are a means of dividing hardware and nodes into useful groupings. Please find more details at Partitions
* Run Jobs - Submit jobs using sbatch command. Use -p to specify the partition.
sbatch my_job.sh sbatch -p bigmem my_job.sh
* Submit Many Jobs - One of the way to submit many jobs is to submit multiple copies of same job.
sbatch -n 2 my_job.sh
* Stop Jobs - Use squeue to get the job number and use scancel to cancel it.
squeue -u your_username scancel job_number
* Check Job and Cluster Status - sinfo provides an overview of the cluster's current state, including details about node availability, node states, and partition configurations.
chris@node0:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST all* up infinite 4 drain node[5-7,11] all* up infinite 9 alloc node[1-4,8,12-13,15,17] all* up infinite 4 idle node[9-10,14,16] short up 2:00:00 4 drain node[5-7,11] short up 2:00:00 10 alloc node[1-4,8,12-13,15,17,91] short up 2:00:00 5 idle node[9-10,14,16,92] gpu up infinite 1 alloc node91 bigmem up infinite 1 idle node92
* Get Mail Notifications - You can get email notifications about the job status. –mail-type values are: BEGIN,END,FAIL,ALL,REQUEUE
sbatch --mail-type=ALL --mail-user=chris@statgen.ncsu.edu wait.bash
The GPUs are accessible through compute nodes in the gpu SLURM partition. You can send a script to this queue using simple commands like sbatch -p gpu myscript.sh. More information about executing your scripts with GPU is available here GPU Computing.
Feel free to send questions or report problems to: help@sciences.ncsu.edu
When sending a request for help please be clear that you are requesting help with an issue on the BRC cluster and provide as much detail as you can.