https://brcweb.statgen.ncsu.edu/ 2026-05-18T20:52:07+00:00 https://brcweb.statgen.ncsu.edu/ https://brcweb.statgen.ncsu.edu/lib/exe/fetch.php?media=wiki:dokuwiki.svg text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=cheat_sheet&rev=1708959933&do=diff Cheat Sheet Submit a job This command submits a job to run “in the background”. Output is written to a file which is named slurm-NNNNN.out by default, where NNNNN is the job number slurm assigns to your job. sbatch [-p partition] [-c ncores] [--mem=NNNG] [--exclusive] scriptname text/html 2025-03-11T02:25:13+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=citation&rev=1741659913&do=diff The cluster consists of 43 individual nodes, providing a core count of 1,180 CPU cores and 14.3TB of total RAM. The cluster is a heterogeneous configuration comprised of 38 compute nodes, five large memory nodes, and 3 GPU nodes. All nodes are connected via a local high-speed, low-latency interconnect. This cluster has access to one petabyte of active storage and another petabyte of space for remote backup. A tape library is used for archiving old files. The system is divided into two parts phys… text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=cluster_schematic&rev=1708959933&do=diff text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=cluster_utilization&rev=1708959933&do=diff Cluster Utilization There is a cluster monitoring page here: <http://transporterroom3.statgen.ncsu.edu/ganglia> This page gives a quick look at the cluster's current state and how heavily used it is. (The page does have some quirks though.) text/html 2024-10-17T17:33:26+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=copying_data_to_the_nodes&rev=1729186406&do=diff Copying Data to the Nodes If you have a job that reads the same data file many times, or makes many “random” accesses to a data file, it may be more efficient to have that data locally on a node than compete with other users to access the file server. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=data_protection&rev=1708959933&do=diff Data Protection and Retention The data on all the cluster volumes (/home1, /home2, /home3, etc.) are mirrored to other hard-disk volumes. * /home1 is mirrored to space in the NCSU HPC Center. * /home2, /home3, /home4, /peng_1, /pengseq, and /gt4sp_1 are mirrored to text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=enforcing_core_counts&rev=1708959933&do=diff Enforcing Core Counts Running cluster jobs have access to exactly as many cores as requested in the job submission. If you submit a job requesting 1 core, and start a program that uses 10 threads, all 10 threads will be time-sliced on that 1 core. Your job might run something like 10 times slower than you expected it to! text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=enforcing_memory_allocation&rev=1708959933&do=diff Enforcing Memory Allocation On the cluster, slurm is configured to treat memory as a “consumable resource”. That means that you should be careful to specify how much memory your job needs. By default your job will be allocated 8GB per core that it has requested. By default 1 core is allocated, in which case 8GB of memory would be allocated to your job. If your job attempts to use more than its memory allocation text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=environment_modules&rev=1708959933&do=diff Environment Modules Environment modules allow you to control which software (and which version of that software) is available in your environment. For instance, at the time of writing, the cluster has 4 different version of standard R installed: 3.5.3, 3.6.3, 4.0.5, 4.1.0. When you first log in and try to run R the text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=get_mail_notifications&rev=1708959933&do=diff Mail Notifications You can be notified when your job completes (or fails!) using the --mail-type and --mail-user options. sbatch --mail-type=ALL --mail-user=chris@statgen.ncsu.edu wait.bash --mail-type values are: BEGIN,END,FAIL,ALL,REQUEUE You might want to use just END and FAIL: text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=gpu_computing&rev=1708959933&do=diff GPU Computing SLURM partition The GPUs are accessible through compute nodes in the gpu SLURM partition. You can send a script to this queue using a command like sbatch -p gpu myscript.sh. Hardware The gpu computing node has two K20 GPUs: $ srun -p gpu lspci | grep -i nvidia 05:00.0 3D controller: NVIDIA Corporation Device 1028 (rev a1) 42:00.0 3D controller: NVIDIA Corporation Device 1028 (rev a1) text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=hardware&rev=1708959933&do=diff Hardware Picture by Kevin Dudley. Summary * 32 general-purpose compute nodes (16, 24, or 32 cores each) * 6 special-purpose compute nodes * 4 Big memory machines * 2 GPGPU machines * 9 File servers * More than 1PB of storage * Nodes are connected to the file servers, and to each other, by 10Gb ethernet text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=job_and_cluster_status&rev=1708959933&do=diff Job and Cluster Status squeue * Lists all jobs in the queue. sinfo * Shows which nodes and processors are in use. scontrol * Shows more detailed job information. sstat * Shows maximum memory use of a job (among other things). sinfo chris@node0:~$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST all* up infinite 4 drain node[5-7,11] all* up infinite 9 alloc node[1-4,8,12-13,15,17] all* up infinite 4 idle node[9-10,14,16] s… text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=job_dependencies&rev=1708959933&do=diff Job Dependencies Using sbatch or srun you can make the start of your job dependent on the successful completion (or failure) of a prior job. This could be useful if you have some data preparation to do, followed by your analysis. This code uses srun -d afterok:NNNNN text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=linux_command_line_basics&rev=1708959933&do=diff Linux Command Line Basics There are a number of good resources on the web. Here's one of them: <http://linuxcommand.org/index.php> Commands you should know Changing Your Password Note: this does not apply to users who log in with their Unity Id and password. passwd There is a 12-character minimum length requirement on the cluster. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=mysql&rev=1708959933&do=diff MySQL A MySQL server is running on file server 1. It can be accessed using the mysql client program from any node, or using any other software with a MySQL interface. You should ask one of the system administrators to get a database and associated user permissions created for you if you need it. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=mysql_management&rev=1708959933&do=diff MySQL Management On the main cluster MySQL is installed on file server 1. The MySQL databases are dumped to disk weekly and the dumps are backed up to the HPC backup space. New Database and User To create a new database for a specific user, log in to the file server and run mysql (the MySQL root password is configured in .my.cnf, so you will log on automatically). In the text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=partitions&rev=1708959933&do=diff Partitions Partition == Queue SLURM allows the definition of different partitions. A partition is a subset of all the nodes in the cluster. Partitions are not mutually exclusive, so a node can belong to multiple partitions. Every compute node is assigned to one or more partitions. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=resource_allocation&rev=1708959933&do=diff Resource Allocation In our slurm configuration both cores and memory are consumable resources, meaning that slurm keeps track of how many cores and how much memory has been allocated to jobs on each node. A node is considered full if either all the cores are allocated, or all the memory is allocated. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=running_jobs&rev=1708959933&do=diff Running Jobs You should submit all long-running, or computationally intensive jobs to the compute nodes. You can write and test your code, and make trial runs of programs, format data, etc., on the head node, but please don't bog it down. <color blue>In general, the command you should use to run jobs on the compute nodes is text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=running_jupyterlab&rev=1708959933&do=diff Running JupyterLab These notes apply to JupyterLab 1.0 run in a Singularity container. JupyterLab is a web-based interactive development environment for Jupyter notebooks, code, and data. JupyterLab is flexible: configure and arrange the user interface to support a wide range of workflows in data science, scientific computing, and machine learning. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=running_x_applications&rev=1708959933&do=diff Running X Applications You can run an X Windows application on the head node e.g. igv and fastqc. * Run an X Server on your client machine (e.g. XMing or MobaXTerm on Windows). * Make sure “X Forwarding” is enabled. * -X on ssh. * Option in putty (Connection / SSH / X11). text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=r_and_blas_libraries&rev=1708959933&do=diff R and BLAS Libraries R uses BLAS (Basic Linear Algebra Subprograms) to perform many of its operations. There are several different implementations of BLAS each having different performance characteristics in different situations. The default BLAS on the cluster is Atlas (Automatically Tuned Linear Algebra Software). The Atlas libraries on the cluster use a single thread to perform all operations. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=scheduling_jobs&rev=1708959933&do=diff Scheduling The cluster is running SLURM's scheduler with: * “Fair share” job priorities. * Simple round-robin node selection. The “Fair Share” algorithm assigns a priority to submitted jobs which is inversely dependent on the amount of cluster CPU time consumed by the user submitting the jobs in recent days. These priorities apply only to jobs waiting in the queue, they do not affect jobs which are already running. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=ssh_101&rev=1708959933&do=diff If you use Linux or MacOS X, terminal applications are included with the operating system. Windows users will need to download a client such as PuTTY or some other ssh client. e.g. MobaXterm, Openssh for Windows, or Windows Subsystem for Linux. Secure Shell is a standard, encrypted means of connecting to remote servers. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=start&rev=1708959933&do=diff BRC Documentation Connecting and Logging in - Command Line Interface If you are not accustomed to using a Linux command line interface (CLI), we recommend familiarizing yourself with introductory material such as 12 Commands Every linux Newbie Should Learn OR Linux Command Line Basics. The ability to navigate and manage files at the Linux command line is important in order to work effectively. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=stopping_jobs&rev=1708959933&do=diff Stopping Jobs Use the scancel command to kill your jobs. scancel -u chris Will kill all jobs for user chris (you can only kill your own jobs). scancel NNNNNNN Kill a specific job. Get the job number from squeue, or from the output of job submission. text/html 2024-02-26T15:05:33+00:00 Anonymous (anonymous@undisclosed.example.com) https://brcweb.statgen.ncsu.edu/doku.php?id=submitting_many_jobs&rev=1708959933&do=diff Submitting Many Jobs You may need to run many (possibly similar) tasks on the cluster. You can do this by submitting each job separately to the cluster, techniques for doing that are discussed below. You could also do it by submitting a small number of jobs (possibly even just one) and have those jobs execute the many tasks you need to run. You should try not to submit thousands of separate jobs into the queue.