Discworld Field Guide

The Gates 🚪

There are two ways to pass through the gates of your laptop on Discworld:

Through a webportal
Via the command line

🔓 You must be on campus or behind the UNSW VPN 🦁🔒 to access any of these services

These ways of accessing the GPUs serve different purposes:

Provides convenience and a visual multi-program interface during an interactive session. But, by its very nature is less flexible (some programs aren’t supported, many don’t even have an interface).
Involves some learning and involves installing a suitable terminal on your laptop, but is far more flexible and opens up the possibility of more powerful calculations, ‘set-and-forget’ longer running jobs, and more effective job queuing and control.

To access the webportal, go to portal.discworld.analytical.unsw.edu.au

To login to Discworld through the terminal:

🍏 MacOS: Start the Terminal app

🪟 Windows: Install the “Windows Subsystem of Linux” (WSL2). If new, try Ubuntu 🟠

UNSW Research Technology Services has a guide here at the bottom of the page Operating System ➔ WSL

🐧 Linux: this will have an inbuilt terminal. Use your favourite.

Login into a command-line Discworld shell securely (ssh):

Passwords: Access via zID and zPass. You do not set a new password

🗝️ In a Discworld new user onboarding session, we will set you up with a securely encrypted key access per device; the same public-private key access that protects the internet servers every day. The world is moving away through passwords, like passkeys are now replacing password fields in web logins.

We would prefer you use this secure device-to-device key.

ssh <zID>@discworld.analytical.unsw.edu.au and you’ve opened the door to Discworld, we gave you the key during onboarding 🗝️

`Command` masher ⌨️

One thing that makes Linux systems seem hard to work with is your possible commands are invisible. You’re given a black mirror without the buttons to show you what to press to do the things you normally want to do.

Luckily, Linux people old and new recognise this & have compiled lists of common things to do.

For example: https://www.geeksforgeeks.org/linux-unix/basic-linux-commands/

Please think before you type. If you’re new, familiarise yourself with the common commands below before branching out to more advanced use.

These allow us to keep the cluster nimble and flexible while also providing ‘easier’ commands for common software use.

The first hurdle is learning to work with your files without clicking on them. Feels weird at first, but you’ll generate thousands of files in CryoEM so it will absolutely save you time.

Basic navigation and file movement commands are: ls, pwd, mkdir, cd, cp, mv, rm, grep

If those seem to be missing letters, that’s because they are. You’ll end up typing so often you don’t want to spell out list ./my_data or print-working-directory over & over again.

rm is DANGEROUS. It is a COMPLETE DELETION.

We provide a saferm that will ask for confirmation, if you’re a beginner. Don’t get used to it, other systems will immediately execute a delete and the file will be gone.

Discworld `command` shortcuts

📂 Folders are called directories in Linux land
🧭 paths are as /full/path/to/file. The slashes / separate folders, like in your file browser.
There are some shorthands: .=‘here’, ./=‘here folder’, ../=‘folder above’, ~=’home folder’
🏠 cd /home = cd ~; takes you home quickly
↩️ cd - will take you back to the last folder you were in
⬆️ Up arrow (↑) goes back through your shell commands
📍 whereami prints the working directory (the one your command line prompt is in)
❔ which tells you where a program is located. You can then tell the other programs

which is useful after loading a program. e.g. module load sbgrid/relion
which relion then gives a full path often nested according to software details: /programs/x86_64-linux/relion/5.0.0_cu12.2/bin.capsules/relion

⏪ Cntrl + r starts a ‘reverse search’; lets you search for old commands by a particular word
🚫 Cntrl + c cancels whatever action you were doing
👋 exit (or quit) will get your out of most programs, including logout from Discworld

📃 list -al will give you all files, including the ones hidden by the magic . in front
It also shows permissions: who can read, write (hence overwrite/delete), run/execute files

rwxrwxrwx size username date filename
u g o

Be very careful who you share ‘group’ or ‘other’ write access with

In the above read/write/execute letters are grouped first-to-last by user, group and other

group will typically your lab group, but can be made from any collection of users
other is anyone else on Discworld!

You can control the permissions (of your own stuff):

chmod – change (file) mode. Another ch command; to change file permissions
chmod works on the corresponding u,g,o letters I’ve listed under the left-listed permissions

Use chmod+ to add permisions, chmod- to remove, chmod= to set them to something new
chmod g-r myfile will remove the ability for anyone in the group to read that file.

🕐 list -snew will sort your files by the newest ones accessed
🃏 * is the ‘wildcard’ and will fill in the slots letters should be:

cryo* matches cryosparc, cryoEM, cryo-EM, cryotomo, cryogenic_hibernation_capsule, and so on.

* is handy, but be careful.

📦 disk-use shows you the ‘disk usage’ of a folder – that is, where all the file storage is going
🔎 search will help you locate where a particular file is: just give it the 📁 directory to search
📑 search-word will look for a word (or a string of letters) inside the list of files given to it

🐈(👁️) cat (also linked as show) is used to show a file on the terminal: cat file.

It’s give that nickname because you can concatenate two files together with it cat file1 file2 > file3

🐧 Linux land is full of brief historical puns.

🔚 tail --follow live view of the bottom of a file

Very useful for slurm-<jobID>.out job log progress

✘✓ find-replace is done by sd (newer sed), it modifies the file ‘live’
sd old_word new_word file.txt

Be sure to save a copy before running a ‘find-replace’, e.g. with cp file file.bak

↩️ extract unpacks all sorts of compressed files
↪️ compress creates a the standard Linux compressed file format (.tar.gz)
↔️ difference finds the different lines between two files (i.e. what changed?)
🔺 delta compare two files on a character level
✂️ cut & paste are useful to work segments of files. Be sure to back up
𝄜 If handling complex tablulated data, try using DataFrames from your language of choice

🎨 We’ve used colours to make common file types easier to see

run unset EXA_COLORS to remove the colours

Even more helpfully, you can read what the commands do while in the command line.

tldr — Too long; didn’t read 📃

We have installed the user-friendly ‘too long didn’t read’ community manuals. tldr command will provide one line on what the command does and use examples. If you want a complete manual of a command it’s: man command.

🥱 Please don’t ‘tl;dr’ this guide, or the manual pages The manual pages examples will often tell you what you need for simple uses without having to wait for an admin to become free.

Most command line tools you can also add --help after them to see what they do.

Basic command line (`zsh`) shell improvements.

We’ve moved to a modern user-friendly shell by default. This gives you colours, autocomplete etc. For command shell scripts from elsewherei, you will need to tell the command prompt to run the older shell: bash script.sh
We’ve but hello_discworld.sh in your /home, you’ll see it specifies at the top that it uses bash to run the script.

Have a try with bash hello_discworld.sh on the command line, since it’s tiny. Please queue big `program`s.

If you really don’t like zsh you can ask us to change your default shell with chsh (‘change shell’)

🦀 We’ve swapped some classic commands for new ones. For example du (‘disk usage’) secretly runs the dust (‘disk usage [programmed in rust]’) implementation through aliases (how we gave you custom commands).
We think these are nicer and faster, but if you want to change any of these substitutions, feel free to edit your personal aliases https://www.howtogeek.com/439736/how-to-create-aliases-and-shell-functions-on-linux/

📝 Writing to files is common task. Use the edit <file> command. edit is secretly micro

cp file file.bak is a quick way to create a second copy of a small non-critical file. Anything more important should go through a proper back-up process.

What’s nice is you can easily share scripts from others and make your own, check out:
python fasta_fixed-width_to_single-line.py --help
It uses the powerful international BioPython module that understands biological files in a cleanly.

Discworld standard `s`cheduler and custom ‘helper’ commands

Since all our users are doing CryoEM work, our admin team have set up some automated scripts that can be called as if they were out-of-the box shell commands. Standard scheduler commands remain

sdesktop — start a visual desktop backed by cluster resources.

This is done through the portal

sdraft [job_name] — custom Discworld script; directly edits a batch job template

Batch job files are just shell commands with info for resources at the top.
We can call them [job_name].slurm to keep track of what you want to run. Discworld will colour them.

sbatch — send a job file to the cluster to run in the background
sinteractive [resources] — request an interactive shell with resources
scancel — cancel a job based on its JOBID

JOBID is a number not a name, to make it easier for the scheduler track

sacct — print accounting for you jobs’ resource usage
squeue - see the state of your job in the queue.

See STATE CODES, R is running PD pending 👍

sinfo - get information about the status of the cluster
scontrol [action] [job_ID]- advance control over what a submitted job should do

Requesting resources 🫲

🤖 Just ask for what you need, with a bit of overhead so your job won’t run out of time or RAM

Put the resources you need as --flags after sinteractive

Resource	Meaning
`--job-name="descriptive-name"`	Purely for your records.
`-time=1:30:30`	Time duration to hold onto resources. (`hr:min:sec`)
`--mem=20GB`	Your RAM. That is, the memory needed to keep the program running. You just have to measure the consumption in a test job to avoid your program crashing.
`--cpus-per-task=4`	How many central processing units to run your program on in paralllel.
`--gres=gpu:<model>:1`	Since GPUs are expensive, you need to ask for the number as generic resource (`gres`). `<model>` is optional; if you know if you need a big one or a small one you can specify here. Otherwise `--gres=gpu:<num_gpus>` is fine.
`--array=<indexes>` `--array=0-N` `--array=0,5,20-25` `--array=0-25:5` `--array=0-25%5`	🪄 Advanced users may set up many many jobs differing by one parameter. Arrays excel here. Run subjobs number `0`➔N Run jobs number `0, 5, 20➔25` Run jobs with step size `:5`, i.e `0, 5, 10, 15, 20, 25..` Run jobs `0➔25`, but only run maximum 5-at-at time, to avoid overwhelming the queue Please read the SLURM Array manual page

Data at the door 🗂️

Do not pull your CryoEM particle data down to your /home drive. The cluster already knows how to access the data and handles transfers, you just occasionally need to tell it where to go looking.

The Electron Microscopy Unit gathers & manages the raw data from the instruments. It is classified as Raw data (see below) and stored on the central university’s storage solution. Discworld mounts data remotely to process it, not store it.

Raw data - This is generated from the instruments at the Electron Microscopy Unit. They handle permissions and backup of the raw data
Processing data - This lives on the university’s ‘Elastic Storage Solution’. It doesn’t live on Discworld (it’s in a different room) so it needs to be mounted (like a USB) by giving the programs this file path structure to your data:
/data/ESS/<cryo_instrument>/<lab_head_lastname>/<user_zID>/(raw|live|processing)
Program data - This is in a pooled /home drive, which through filesystem magic lives in a network of shared, expensive flash drives. As such it’s suitable for installing programs and working environments (conda should resolve quickly). It is not suitable for CryoEM data
Finalised 3D maps – these should be deposited in the appropriate scientific database (e.g. EMPIAR) and/or deposited to UNSW’s Research Data Archive ‘glacial’ storage. The Data Archive is managed under your supervisor’s Research Data Management Plan.
⁉️ Unsure? - Doesn’t fit the above classes?
Contact an admin at sbf+discworld@unsw.edu.au
For hard classification cases talk to UNSW Research Data Management support team

🌐 Globus [In Development] - UNSW has a Globus site license. Globus excels at large data transfers, and makes data points discoverable (like EM data connected at another Facility). Please use the Globus portal and all the data sharing permissions it grants you. It has many nifty features.

🔄 rsync — when in doubt this is a safe, resumable way to remotely sync data between Linux systems.

What to do when things go wrong 🤕

Notice within 2 weeks
DON’T PANIC

💾 We save a snapshot of your /home drive to another storage server every night. That means if you alert us within two weeks we can start the process of ‘rolling-back’ to an old save state. We only keep the last two weeks otherwise the user storage will balloon.

📜 Recall The Laws of Discworld /home is for your tools not your data

GPU Alley👨‍💻🛣️

SLURM (Simple Linux Utility for Resource Management) provides a single master queue for a variety of computational work, ensuring a fair quality of service for everyone.

The fairshare algorithm is used. To stop other people’s smaller interactive jobs being entirely delayed, the scheduler naturally brings forward jobs that have been waiting a long time, and schedules really large jobs to start when the resources are less busy (the middle of the night or the weekend).

We’ll keep our queue as simple as possible, as such we can’t write in exceptions to your personal deadlines. Your jobs will still run efficiently, but in the share the queue autodetermines is fairest. Accordingly, plan your work with suitable wait time if you have a deadline in the future.

Future improvements 🚧

Please try to be patient and respectful with our administration and development team. We try to roll out new features to you as fast as we stably can, so require testing time.