If you are not accustomed to using a Linux command line interface (CLI), we recommend familiarizing yourself with introductory material such as 12 Commands Every linux Newbie Should Learn OR Linux Command Line Basics. The ability to navigate and manage files at the Linux command line is important in order to work effectively.

To connect or log in to the cluster you will need to use NCSU Campus Network and Secure Shell (ssh). If you are new at SSH please see: SSH 101. People who are off the NC State campus are reminded to use the Virtual Private Networking (VPN) service when connecting to the campus network. Please see NCSU VPN for more info.

Once at a command line you can connect to the cluster using commands like one of these:

ssh username@brccluster.statgen.ncsu.edu

Login Nodes:

you can connect to the cluster using SSH to any of below machines. These are just different names for the same machine.

1. brccluster.statgen.ncsu.edu
2. captainsisko.statgen.ncsu.edu
3. brccluster.cos.ncsu.edu

For security reasons, you are encouraged to using passwordless SSH with public authentication keys. (It's also more convenient once set up.) Please follow easy steps mentioned on these articles. How to Setup SSH Passwordless Login. 3 Steps to Perform SSH Login Without Password Using ssh-keygen & ssh-copy-id

Note : From October 18th 2021 OIT blocked the default SSH port (port 22) at the campus gateway. You should connect to the campus VPN and then use SSH to get to the cluster. Along with SSH other protocols like SFTP, SCP, and SSHFS are also affected by this change.

For Job and Resource management we use SLURM workload manager software.

Here is the brief overview of some of the slurm commands:

  * sbatch  - Job Submission
  * scancel - Job deletion
  * squeue  - List all jobs in queue
  * sinfo   - List all nodes
  * scontrol show nodes       - Show information about nodes
  * squeue --start            - Job start time
  * scontrol show job <jobid> - Job information
  * sacct -u <username> --format=JobID,JobName,MaxRSS,Elapsed - View details of all jobs by user
  * showq   - shows cluster state
  * myslurmaccts              -  see which accounts and partitions you can use
  * lsload  - Get information on node state

The environment module system is used to manage the user environment, allowing you to load and unload software packages and set up environment variables for your jobs. It also controls which software is in your path, when you log in to the cluster. For more details please refer: Environment Modules.

To get the list of available software modules execute module avail command. As of 30-Jan-2023 below is the list of available softwares.

|user_name@sisko:[~]> module avail
------------------------------------------- /opt/modules/modulefiles --------------------------------------------
abyss/2.1.5      dot                   igv/2.10.0               perl/5.34.0           samtools/1.13        
abyss/2.3.2      eigensoft/7.2.1       java/openjdk-8           picard/2.25.6         seqtk/1.3            
augustus/3.4.0   exonerate/2.2.0       java/openjdk-11          plink/1.07            snap/2013-11-29      
bamtools/2.5.1   fastp/0.21.0          java/openjdk-20.0.2      plink/1.90            sratoolkit/2.11.0    
bbtools/38.90    fastqc/0.11.9         jellyfish/2.3.0          python/2.7.18         stacks/2.64          
bcftools/1.13    fftw/3.3.10           julia/1.6.1              python/3.9.5          STAR/2.7.9a          
bcl2fastq/2.20   fseq2/2.0.3           locuszoom/1.4            R/3.5.3               temposeqcount/1.4.1  
bedtools/2.30.0  gatk/4.2.0.0          LTR_retriever/2.9.0      R/3.6.3               TRF/4.09            
blast+/2.11.0    gcta/1.24.7           mafft/7.481              R/4.0.5               Trimmomatic/0.39    
blast+/2.14.0    gcta/1.94.0beta       maker/3.01.04            R/4.1.0               upc++/2023.3.0_smp  
bowtie/1.3.0     GeneMark-ES/4.68      matlab/R2021a            RECON/1.08            use.own              
bowtie2/2.4.4    GenomeTools/1.6.2     module-git               RepeatMasker/4.1.2    vcflib/1.0.2        
bwa/0.7.17       globuspersonal/3.1.6  module-info              RepeatModeler/2.0.2a  vcftools/0.1.17      
cd-hit/4.8.1     globuspersonal/3.2.2  modules                  RepeatScout/1.0.6     wiggletools/1.2      
cutadapt/3.1     go/1.16.4             Ninja/0.98-cluster_only  RMBlast/2.11.0        
diamond/2.0.11   hmmer/3.3.2           null                     Ropen/4.0.2          
dida/1.0.1       hyphy/2.5.31          perl-threaded/5.34.0     samtools/1.12 

There are many ways of dividing up and managing a cluster. Partitions are a means of dividing hardware and nodes into useful groupings. Please find more details at Partitions

* Run Jobs - Submit jobs using sbatch command. Use -p to specify the partition.

sbatch my_job.sh
sbatch -p bigmem my_job.sh

* Submit Many Jobs - One of the way to submit many jobs is to submit multiple copies of same job.

sbatch -n 2 my_job.sh

* Stop Jobs - Use squeue to get the job number and use scancel to cancel it.

squeue -u your_username
scancel job_number

* Check Job and Cluster Status - sinfo provides an overview of the cluster's current state, including details about node availability, node states, and partition configurations.

chris@node0:~$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
all*         up   infinite      4  drain node[5-7,11]
all*         up   infinite      9  alloc node[1-4,8,12-13,15,17]
all*         up   infinite      4   idle node[9-10,14,16]
short        up    2:00:00      4  drain node[5-7,11]
short        up    2:00:00     10  alloc node[1-4,8,12-13,15,17,91]
short        up    2:00:00      5   idle node[9-10,14,16,92]
gpu          up   infinite      1  alloc node91
bigmem       up   infinite      1   idle node92

* Get Mail Notifications - You can get email notifications about the job status. –mail-type values are: BEGIN,END,FAIL,ALL,REQUEUE

sbatch --mail-type=ALL --mail-user=chris@statgen.ncsu.edu wait.bash

* Job Scheduling

* Job Dependencies

• Resource Allocation
• Enforce CPU Core Count
• Enforce Memory Allocation

• Transfer data to /tmp

The GPUs are accessible through compute nodes in the gpu SLURM partition. You can send a script to this queue using simple commands like sbatch -p gpu myscript.sh. More information about executing your scripts with GPU is available here GPU Computing.

• JupyterLab
• R and BLAS Libraries
• Graphical Linux Apps (A.k.a X Applications)

• Cheat Sheet
• Citation
• Cluster Utilization
• Data Protection and Retention
• Cluster Schematic
• Hardware

• OIT High-Performance Computing Services
• OIT Research Storage Services
• NCSU Remote Access Service(VCL)
• Department of Mathematics HPC Cluster
• Department of Statistics Virtual Teaching Lab(AVD)

Feel free to send questions or report problems to: help@sciences.ncsu.edu

When sending a request for help please be clear that you are requesting help with an issue on the BRC cluster and provide as much detail as you can.