======= BRC Documentation =======
===== Connecting and Logging in =====
==== - Command Line Interface ====
If you are not accustomed to using a Linux command line interface (CLI), we recommend familiarizing yourself with introductory material such as [[http://www.pcworld.com/article/214370/12_commands_every_linux_newbie_should_learn.html | 12 Commands Every linux Newbie Should Learn]] OR [[linux_command_line_basics|Linux Command Line Basics]]. The ability to navigate and manage files at the Linux command line is important in order to work effectively.
==== - Log In ====
To connect or log in to the cluster you will need to use **NCSU Campus Network** and **Secure Shell (ssh)**. If you are new at SSH please see: [[ssh_101|SSH 101]]. People who are off the NC State campus are reminded to use the Virtual Private Networking (VPN) service when connecting to the campus network. Please see [[https://oit.ncsu.edu/campus-it/campus-data-network/vpn/ | NCSU VPN]] for more info.
Once at a command line you can connect to the cluster using commands like one of these:
ssh username@brccluster.statgen.ncsu.edu
**Login Nodes:**
you can connect to the cluster using SSH to any of below machines. These are just different names for the same machine.
- brccluster.statgen.ncsu.edu
- captainsisko.statgen.ncsu.edu
- brccluster.cos.ncsu.edu
For security reasons, you are encouraged to using passwordless SSH with public authentication keys. (It's also more convenient once set up.) Please follow easy steps mentioned on these articles. [[https://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps/|How to Setup SSH Passwordless Login]]. [[https://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/ | 3 Steps to Perform SSH Login Without Password Using ssh-keygen & ssh-copy-id]]
**Note : From October 18th 2021 OIT blocked the default SSH port (port 22) at the campus gateway. You should connect to the campus VPN and then use SSH to get to the cluster. Along with SSH other protocols like SFTP, SCP, and SSHFS are also affected by this change.**
===== Using the cluster =====
For Job and Resource management we use [[https://slurm.schedmd.com/documentation.html | SLURM]] workload manager software.
Here is the brief overview of some of the **slurm commands**:
* sbatch - Job Submission
* scancel - Job deletion
* squeue - List all jobs in queue
* sinfo - List all nodes
* scontrol show nodes - Show information about nodes
* squeue --start - Job start time
* scontrol show job - Job information
* sacct -u --format=JobID,JobName,MaxRSS,Elapsed - View details of all jobs by user
* showq - shows cluster state
* myslurmaccts - see which accounts and partitions you can use
* lsload - Get information on node state
==== - Environment Module System ====
The environment module system is used to manage the user environment, allowing you to load and unload software packages and set up environment variables for your jobs.
It also controls which software is in your path, when you log in to the cluster. For more details please refer: [[environment_modules | Environment Modules]].
To get the list of available software modules execute **module avail** command. As of 30-Jan-2023 below is the list of available softwares.
|user_name@sisko:[~]> module avail
------------------------------------------- /opt/modules/modulefiles --------------------------------------------
abyss/2.1.5 dot igv/2.10.0 perl/5.34.0 samtools/1.13
abyss/2.3.2 eigensoft/7.2.1 java/openjdk-8 picard/2.25.6 seqtk/1.3
augustus/3.4.0 exonerate/2.2.0 java/openjdk-11 plink/1.07 snap/2013-11-29
bamtools/2.5.1 fastp/0.21.0 java/openjdk-20.0.2 plink/1.90 sratoolkit/2.11.0
bbtools/38.90 fastqc/0.11.9 jellyfish/2.3.0 python/2.7.18 stacks/2.64
bcftools/1.13 fftw/3.3.10 julia/1.6.1 python/3.9.5 STAR/2.7.9a
bcl2fastq/2.20 fseq2/2.0.3 locuszoom/1.4 R/3.5.3 temposeqcount/1.4.1
bedtools/2.30.0 gatk/4.2.0.0 LTR_retriever/2.9.0 R/3.6.3 TRF/4.09
blast+/2.11.0 gcta/1.24.7 mafft/7.481 R/4.0.5 Trimmomatic/0.39
blast+/2.14.0 gcta/1.94.0beta maker/3.01.04 R/4.1.0 upc++/2023.3.0_smp
bowtie/1.3.0 GeneMark-ES/4.68 matlab/R2021a RECON/1.08 use.own
bowtie2/2.4.4 GenomeTools/1.6.2 module-git RepeatMasker/4.1.2 vcflib/1.0.2
bwa/0.7.17 globuspersonal/3.1.6 module-info RepeatModeler/2.0.2a vcftools/0.1.17
cd-hit/4.8.1 globuspersonal/3.2.2 modules RepeatScout/1.0.6 wiggletools/1.2
cutadapt/3.1 go/1.16.4 Ninja/0.98-cluster_only RMBlast/2.11.0
diamond/2.0.11 hmmer/3.3.2 null Ropen/4.0.2
dida/1.0.1 hyphy/2.5.31 perl-threaded/5.34.0 samtools/1.12
==== - Partitions ====
There are many ways of dividing up and managing a cluster. Partitions are a means of dividing hardware and nodes into useful groupings. Please find more details at [[partitions|Partitions]]
==== - Job Management ====
***** [[running_jobs|Run Jobs]] - Submit jobs using **sbatch** command. Use **-p** to specify the partition.
sbatch my_job.sh
sbatch -p bigmem my_job.sh
***** [[submitting_many_jobs|Submit Many Jobs]] - One of the way to submit many jobs is to submit multiple copies of same job.
sbatch -n 2 my_job.sh
***** [[stopping_jobs|Stop Jobs]] - Use **squeue** to get the job number and use **scancel** to cancel it.
squeue -u your_username
scancel job_number
***** [[job_and_cluster_status|Check Job and Cluster Status]] - **sinfo** provides an overview of the cluster's current state, including details about node availability, node states, and partition configurations.
chris@node0:~$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
all* up infinite 4 drain node[5-7,11]
all* up infinite 9 alloc node[1-4,8,12-13,15,17]
all* up infinite 4 idle node[9-10,14,16]
short up 2:00:00 4 drain node[5-7,11]
short up 2:00:00 10 alloc node[1-4,8,12-13,15,17,91]
short up 2:00:00 5 idle node[9-10,14,16,92]
gpu up infinite 1 alloc node91
bigmem up infinite 1 idle node92
***** [[get_mail_notifications|Get Mail Notifications]] - You can get email notifications about the job status. **–mail-type** values are: BEGIN,END,FAIL,ALL,REQUEUE
sbatch --mail-type=ALL --mail-user=chris@statgen.ncsu.edu wait.bash
***** [[scheduling_jobs|Job Scheduling]]
***** [[job_dependencies|Job Dependencies]]
==== - Resource Allocation ====
* [[resource_allocation|Resource Allocation]]
* [[enforcing_core_counts|Enforce CPU Core Count]]
* [[enforcing_memory_allocation|Enforce Memory Allocation]]
==== - Data Transfer ====
* [[copying_data_to_the_nodes|Transfer data to /tmp]]
==== - GPU Computing ====
The GPUs are accessible through compute nodes in the gpu SLURM partition. You can send a script to this queue using simple commands like ''sbatch -p gpu myscript.sh''. More information about executing your scripts with GPU is available here [[gpu_computing|GPU Computing]].
===== Software and Programs =====
* [[running_jupyterlab|JupyterLab]]
* [[r_and_blas_libraries|R and BLAS Libraries]]
* [[running_x_applications|Graphical Linux Apps (A.k.a X Applications)]]
===== More details about cluster =====
* [[cheat_sheet|Cheat Sheet]]
* [[citation|Citation]]
* [[cluster_utilization|Cluster Utilization]]
* [[data_protection|Data Protection and Retention]]
* [[cluster_schematic|Cluster Schematic]]
* [[hardware|Hardware]]
===== Useful Links =====
* [[https://hpc.ncsu.edu/main.php | OIT High-Performance Computing Services]]
* [[https://oit.ncsu.edu/my-it/teaching-research/research-storage/ | OIT Research Storage Services]]
* [[https://vcl.ncsu.edu/ | NCSU Remote Access Service(VCL)]]
* [[https://wp.math.ncsu.edu/it/high-powered-computing-cluster/ | Department of Mathematics HPC Cluster]]
* [[https://statistics.sciences.ncsu.edu/resources/it/virtual-teaching-lab-avd/ | Department of Statistics Virtual Teaching Lab(AVD)]]
===== Contact Us =====
Feel free to send questions or report problems to: //**help@sciences.ncsu.edu**//
When sending a request for help please be clear that you are requesting help with an issue on the BRC cluster and provide as much detail as you can.