Maintance Thur 2026 Nov 12

For this exercise, we will be processing a small subset of a milled yeast dataset. The data was acquired by the Stefan Raunser group from the Max Planck Institute in Dortmund, Germany. A special thanks to Tatjana Taubitz, Elisa Lisicki, Sebastian Tacke and Stefan Raunser for making the data available. The dataset was collected on a Titan Krios dataset and the original subset contains almost 100 tilt-series of FIB-milled yeast cells.

This guide will cover steps from processing raw images such as those obtained directly from the microscope to density maps obtained by subtomogram averaging.

Due to time constraints, we will not be running the full workflow in this workshop. We have prepared pre-calculated results for several steps to speed up the process and reduce computational burden. All principles are applicable to the entirety of the data. Instructions for the full workflow are included, with instructions to indicate changes for running the time-constrained version we will be using.

This exercise is intended for use with Relion 4 (5.0.0_cu12.2) with Napari (0.4.18), Warp (2.0.0dev36), AreTomo3 (2.2.2), IsoNET (0.3), TomoTWIN (0.9.1), and Imod (5.1.3). We will also use the visualization software UCSF ChimeraX version 1.10_rhel9.

(For Relion jobs run in Amazon Web Services (AWS), the slurm errors about _handle_getpw and _handle_getgr can be ignored.)

SBGrid CryoET ChallengeBank DOI: 10.15785/SBGRID/1255

If you have not already downloaded the data, open a command terminal window and type:

rsync -av rsync://challengebank.sbgrid.org/10.15785/SBGRID/1255

When not specified, parameter values can be left at their defaults.

Step 0. Create your project directory

In the terminal window, navigate to the working directory for your project:

• cd /shared/data/projects/$USER/ or cd /shared/data/projects/tttuserNNNN (replacing the NNNN with your user number)
• Check your current directory contents by typing ls. You should see several directories, e.g. 1_warp_scripts, 2_isonet_scripts, etc.
• Navigate to the first directory: cd 1_warp_scripts

Step 1. Create the settings file.

First, we need to create the settings file that defines our input data and where it is stored.

#!/bin/bash

WarpTools create_settings \
--output warp_frameseries.settings \
--folder_processing ${PWD}/warp_frameseries \
--folder_data /shared/data/projects/${USER}/raw_data/subset3/ \
--extension "*.eer" \
--gain_path /shared/data/projects/${USER}/raw_data/subset3/reference.gain \
--eer_ngroups 42 \
--angpix 1.56 \
--bin 0 \
--exposure 2.41 \
--tomo_dimensions 4400x4400x2200 > warp_create_fs_settings.out 2> warp_create_fs_settings.err

In your script:

--folder_processing

We have provided the path to the files on our system. However, you would need to provide the path yourself ordinarily, and make sure to include /warp_frameseries on the end of the path. (The script will create a subdirectory titled “warp_frameseries”. You do not need to make it yourself.)

--folder_data

You need the path to your working directory. We have provided this for you.

--extension

Define the file extension.

--eer_ngroups

Enter the number of eer subframes to sum for motion correction. Our default is 42 because most eer images have ~250 subframes in them, which would make 42 the equivalent distribution of a tiff image with about 6 subframes (the average amount).

--angpix

Put the calibrated pixel size in angstroms per pixel.

--tomo_dimensions

Put the tomogram dimensions in unbinned pixels. (When this is unknown, 4400x4400x2200 is a good starting point if your data was collected on a Falcon camera. You can measure the tomogram thickness in Imod after reconstruction and adjust as needed).

To run:

• ./01_warp_create_fs_settings.sh

Time for one tomogram (CPU only job): A few seconds. After successful completion, the log should show "123 files found"

Step 2. Estimate 2D motion and the Contrast Transfer Function.

We need to estimate the motion of our specimen in 2D prior to CTF estimation and correction. Corrected averages will be written out to the /warp_frameseries/average directory as .xml files. There will be one .xml file for every fraction in the tilt-series.

#!/bin/bash
#SBATCH --job-name=warp_motion_ctf
#SBATCH --error=warp_motion_ctf.err
#SBATCH --output=warp_motion_ctf.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-mixed

echo "SLURM: ${SLURM_JOB_ID}"

WarpTools fs_motion \
--settings warp_frameseries.settings \
--grid 1x1x5 \
--averages

WarpTools fs_ctf \
--settings warp_frameseries.settings \
--voltage 300 \
--cs 2.7 \
--amplitude 0.1 \
--defocus_min 3.0 \
--defocus_max 8.0 \
--range_min 30.0 \
--range_max 7.0 \
--use_sum \
--grid 2x2x1

To run:

• sbatch 02_warp_motion_and_ctf.sh

Time for one tomogram (4 GPUs): ~10 minutes. After successful completion, the log should show 123/123, 00:00 remaining.

Step 3. View histograms to assess results.

This step is technically optional but we advise not to skip it. It will print histogram plots to the terminal window that you can use to evaluate if the jobs have run correctly. It will inform you how many frames were estimated to contain what degree of CTF fit (e.g. 13 frames at CTF fit of 4.2 - 4.6 A).

#!/bin/bash

WarpTools filter_quality \
--settings warp_frameseries.settings \
--histograms > warp_histograms.out 2> warp_histograms.err

To run:

• ./03_warp_histograms.sh

Time for one tomogram (4 GPUs): Nearly instant.

Read the contents of the .out file to view the histograms printed to the terminal.

Step 4. Import the tiltseries.

Tell Warp which files belong to which tiltseries by importing them so that you can manage your data downstream.

#!/bin/bash

WarpTools ts_import \
--mdocs /shared/data/projects/${USER}/raw_data/subset3/ \
--pattern "*.mdoc" \
--frameseries warp_frameseries \
--tilt_exposure 2.41 \
--dont_invert \
--override_axis 84 \
--output tomostar \
--min_intensity 0.8 > warp_ts_import.out 2> warp_ts_import.err

In your script:

--mdocs

We have input the path to where the mdoc file is stored.

--pattern

We have input the file extension of the mdoc file.

--override_axis

Put 84 in this case.

--tilt_exposure

We have input the per-tilt electron dose exposure (not including any pre-dosing, which we don’t have, in this case).

To run:

• ./04_warp_tiltseries_import.sh

Time for one tomogram (4 GPUs): Instant. After successful completion, the log should show "Successfully parsed 3 MDOC files, 0 failed".

Step 5. Create tilt settings.

Create a .settings file that tells Warp how to process the tilt series. Produce metadata files (STAR files) based on the original mdoc files (output by SerialEM) that Warp can use to import and manage your data downstream. This is where it becomes important to change the tomogram dimensions if you wished to reconstruct tomograms with a smaller or larger Z-height later (for example, if your dimensions truncated your region of interest).

#!/bin/bash

WarpTools create_settings \
--output warp_tiltseries.settings \
--folder_processing warp_tiltseries \
--folder_data tomostar \
--extension "*.tomostar" \
--angpix 1.56 \
--bin 0 \
--exposure 2.41 \
--tomo_dimensions 4400x4400x2200 > warp_ts_settings.out 2> warp_ts_settings.err

In your script:

--tomo_dimensions

(XxYxZ), replace the Z value with the height of the tomogram you’d like in unbinned pixels. It must be larger than the biological lamella, and even larger still to account for the uncorrected tilt axis offset. In this case, 2400 is sufficient for now.

To run:

• ./05_warp_create_ts_settings.sh

Time for one tomogram (4 GPUs): Instant. After successful completion, the log should show "3 files found".

Step 6. Align tilt-series using AreTomo3.

Align tilt-series images using AreTomo3. The AreTomo3 wrapper in Warp automates running AreTomo3 by preparing inputs, passing parameters, and reintegrating outputs into Warp’s workflow.

#!/bin/bash
#SBATCH --job-name=warp_aretomo3
#SBATCH --error=warp_aretomo3.err
#SBATCH --output=warp_aretomo3.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-mixed

echo "SLURM: ${SLURM_JOB_ID}"

WarpTools ts_aretomo3 \
--settings warp_tiltseries.settings \
--angpix 10 \
--patches 6,4 \
--output_processing aretomo3 \
--axis 84

In your script:

--angpix

We have input a binned pixel size to speed up computation. Ten should be sensible for most datasets.

--axis

We have input the tilt angle axis for this dataset (occasionally correctly present in the mdoc file but often incorrect, which you can read by typing header $name.mdoc into the terminal. For other projects, if you don’t know the tilt angle axis, you can omit the --axis 84 parameter and input --axis_batch 3 and --axis_iter to ask AreTomo3 to refine the tilt axis angle. AreTomo3 will try to guess by running an automatic tilt-angle search.)

To run:

• sbatch 06_aretomo3_alignment.sh

Time for one tomogram (3 GPUs): ~5 minutes.

You cannot check the aligned mrc files for AreTomo3 because it does not output any. You can check how well it performed at the tomogram stage. After successful completion, the log should show "3/3, 00:00 remaining".

Step 7. Check defocus handedness.

Inside the microscope, “defocus” is the distance between the sample plane and the focal plane of the objective lens. The image can be focused above or below the specimen (positive or negative sign convention). If the wrong handedness is used, CTF oscillations will get flipped and phases will be inverted incorrectly. Warp helps you check the defocus handedness using this script.

#!/bin/bash

WarpTools ts_defocus_hand \
--settings warp_tiltseries.settings \
--check > warp_check_defocus.out 2> warp_check_defocus.err

To run:

• ./07_warp_check_defocus.sh

To inspect the result, run:

• catwarp_check_defocus.out

You will see the following; “The average correlation is negative, which means that the defocus handedness should be set to 'flip'.”

Time for one tomogram (4 GPUs): < 30 seconds.

Step 8. Flip the defocus handedness.

We need to set the defocus handedness by saving the ‘flip’ setting into the settings file.

#!/bin/bash

WarpTools ts_defocus_hand \
--settings warp_tiltseries.settings \
--set_flip > warp_set_defocus.out 2> warp_set_defocus.err

To run:

• ./08_warp_set_defocus.sh

Time for one tomogram (4 GPUs): < 5 seconds. After successful completion, the log should show "Setting defocus handedness to 'flip' for all tilt series… 3/3".

Step 9. Correct the CTF.

Warp performs CTF correction by estimating the defocus and astigmatism for each tilt image (performed earlier) and mathematically compensating for the phase distortions introduced by the microscope’s optics. This improves image contrast and restores high-frequency details and makes downstream reconstruction and averaging more accurate.

#!/bin/bash

WarpTools ts_ctf \
--settings warp_tiltseries.settings \
--defocus_min 3.0 \
--defocus_max 8.0 \
--range_low 30.0 \
--range_high 7.0 \
--voltage 300 \
--cs 2.7 \
--amplitude 0.1 > warp_ctf_correction.out 2> warp_ctf_correction.err

To run:

• ./09_warp_ctf_correction.sh

Time for one tomogram (4 GPUs): ~1 minute. After successful completion, the log should show "3/3, 00:00 remaining".

Step 10. Reconstruct tomograms.

Reconstruct tomograms in .mrc format using the parameters set in the settings file. The time this will take depends on the size of the tomogram dimensions and the --angpix variable.

Reconstruct tomograms from the AreTomo3 alignment.

#!/bin/bash
#SBATCH --job-name=warp_reconstruction
#SBATCH --error=warp_reconstruction.err
#SBATCH --output=warp_reconstruction.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-mixed

WarpTools ts_reconstruct \
--settings warp_tiltseries.settings \
--input_processing aretomo3/ \
--angpix 10.0 \
--dont_invert \
--deconv \
--deconv_strength 0.7 \
--deconv_falloff 0.7 \
--perdevice 1

In your script:

--angpix

We have input the desired tomogram pixel size. In most cases, you will want to put either 10 or close to 10, because post-processing, segmentation and subtomogram averaging softwares (i.e. isonet, tomotwin, pytom, membrain, tardis, etc.) were all developed with tomo pixel references of 10 angstrom. If you put something else here, you will need to rescale the tomogram in Eman2 prior to downstream processing.

To run:

• sbatch 10_warp_reconstruction.sh

Time for one tomogram (4 GPUs): ~5 minutes. After successful completion, the log should show "3/3, 00:00 remaining".

Step 11. Inspect tomograms

To inspect the quality of a tomogram, navigate to the output folder where the tomograms are stored;

• cd aretomo3/reconstruction/
• Open the tomograms in IMOD by typing imod $tomoname.mrc.
• Out of interest, measure the tomogram thickness. You sometimes need to do this so that the full reconstruction pipeline can be adjusted and applied to the remaining unreconstructed tilt-series. You will use roughly the height of the tomogram plus 1000 additional pixels (because this specimen is fib-milled and the tilt axis angle is uncorrected. Therefore, we need a little extra space above and below the tomogram to contain the full biological tomogram without cropping it because it’s on a tilt of 8 degrees. See live demonstration for more info).

Measuring Tomogram Thickness

1. Open a tomogram in IMOD.
2. imod $your_tomo.mrc
3. In the Info window, click the ‘Edit’ tab.
4. Click ‘Image’.
5. Click ‘Flip/Rotate’.
6. In the Zapper window, drag the Z slicer bar roughly to the middle.
7. Look for noise distortion in the tomogram. You may slide the Z slicer bar left and right to create some motion as you slab through the Z, which can help with visualization. You should notice strong noise of the lamella with smooth noise above and below (called “void”, essentially the vacuum above and below the specimen). You may need to click the ‘AUTO’ button on the IMOD Info window to adjust the contrast.
8. Click your left-button mouse at the top boundary of the tomogram. The yellow crosshairs will be placed there.
9. Point your left mouse cursor at the bottom of the boundary. DON’T CLICK. Tap Q on your keyboard.
10. The thickness between the yellow crosshairs and the mouse cursor will be measured and printed to the log.
11. The thickness in pixels should be scaled by the pixel size of the tomogram you requested when you reconstructed.

Step 12. Reconstruct the rest of the dataset

Adjust the master script so that we can reconstruct the remaining tilt-series in the datasets provided using a one-stop-shop Warp script. We need to:

• Skip checking the defocus (we already know now that it needs to be flipped. We do not need to check a second time).
• Set the appropriate tomogram height for reconstruction. We will stick with the same here.
• If necessary, navigate up from the aretomo3/reconstruction/ directory with cd ../../
• Queue the job with sbatch warp_workflow.sh

#!/bin/bash
#SBATCH --job-name=warp_workflow
#SBATCH --error=warp_workflow.err
#SBATCH --output=warp_workflow.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-mixed

WarpTools create_settings \
--output warp_frameseries.settings \
--folder_processing /shared/data/projects/${USER}/1_warp_scripts/warp_frameseries \
--folder_data /shared/data/projects/${USER}/raw_data/subset7/ \
--extension "*.eer" \
--gain_path /shared/data/projects/${USER}/raw_data/subset7/reference.gain \
--eer_ngroups 42 \
--angpix 1.56 \
--bin 0 \
--exposure 2.41 \
--tomo_dimensions 4400x4400x2200

WarpTools fs_motion \
--settings warp_frameseries.settings \
--grid 1x1x5 \
--averages

WarpTools fs_ctf \
--settings warp_frameseries.settings \
--voltage 300 \
--cs 2.7 \
--amplitude 0.1 \
--defocus_min 3.0 \
--defocus_max 8.0 \
--range_min 30.0 \
--range_max 7.0 \
--use_sum \
--grid 2x2x1

WarpTools filter_quality \
--settings warp_frameseries.settings \
--histograms

WarpTools ts_import \
--mdocs /shared/data/projects/${USER}/raw_data/subset7/ \
--pattern "*.mdoc" \
--frameseries warp_frameseries \
--tilt_exposure 2.41 \
--dont_invert \
--override_axis 84 \
--output tomostar \
--min_intensity 0.8

WarpTools create_settings \
--output warp_tiltseries.settings \
--folder_processing warp_tiltseries \
--folder_data tomostar \
--extension "*.tomostar" \
--angpix 1.56 \
--bin 0 \
--exposure 2.41 \
--tomo_dimensions 4400x4400x2200

WarpTools ts_aretomo3 \
--settings warp_tiltseries.settings \
--angpix 10 \
--patches 6,4 \
--output_processing aretomo3 \
--axis 84

WarpTools ts_defocus_hand \
--settings warp_tiltseries.settings \
--set_flip

WarpTools ts_ctf \
--settings warp_tiltseries.settings \
--defocus_min 3.0 \
--defocus_max 8.0 \
--range_low 30.0 \
--range_high 7.0 \
--voltage 300 \
--cs 2.7 \
--amplitude 0.1

WarpTools ts_reconstruct \
--settings warp_tiltseries.settings \
--input_processing aretomo3/ \
--angpix 10.0 \
--dont_invert \
--deconv \
--deconv_strength 0.7 \
--deconv_falloff 0.7 \
--perdevice 1

View the reconstructed tomograms by navigating to ./aretomo3/reconstruction/ and opening the tomograms in IMOD;

• imod *.mrc
• Special note: You may need to convert the tomograms to float32 because some IMOD versions have float16 sensitivity. Use: MrcConverter --reverse in the terminal to convert the tomograms if needed.

---

To denoise tomograms and improve the visual appearance of low-signal features, we can perform CTF deconvolution and missing wedge correction in IsoNet. IsoNet has a graphical user interface (which you can launch by typing isonet.py gui in the command line, but for the purposes of this tutorial, we will run IsoNet in the command line.)

We will run IsoNet on only one tomogram to save time in this workshop using a pre-calculated IsoNet model that we prepared for you. We provide you with all the pre-denoised tomograms later. If you want to know how to create a model yourself from your own data, you can follow our instruction guide here.

Step 0. Folder set-up

We have prepared an IsoNet model for you. You will need to first navigate to the new isonet working directory (/shared/data/projects/${USER}/2_isonet_scripts/).

• Move your favourite tomogram into the new folder.
• cp /shared/data/projects/${USER}/1_warp_scripts/aretomo3/reconstruction/$tomoname.mrc .

Your isonet folder should now have the following items in it:

• One tomogram of your choosing (for example, l40t05_5_10.00Apx.mrc)
• The IsoNet slurm scripts
• A folder called pre_trained_models

Step 1. Prepare a star file

Prepare a tomograms.star file.

#!/bin/bash

isonet.py prepare_star ${PWD} --output_star tomograms.star --pixel_size 10.0 > isonet_prep.out 2> isonet_prep.err

Note: provide only the path to the directory. Don’t include the name of the .mrc file.

To run:

• ./1_isonet_prepare_starfile.sh

Time for one tomogram (CPU only): near instant

Step 2. Tomogram filtering with CTF deconvolution

In real life, this step is optional but for this tutorial, we demonstrate deconvolution in IsoNet.

#!/bin/bash

isonet.py deconv tomograms.star --snrfalloff 0.7 --deconv_folder deconv > isonet_deconv.out 2> isonet_deconv.err

To run:

• ./2_isonet_deconvolute.sh

Time for one tomogram (1 GPUs): 1-2 minutes

• Inspect the deconv directory and open the tomogram in IMOD to inspect how well IsoNet performed the filtering. (For those interested, you could compare the performance of Warps in-built deconvolution and IsoNets deconvolution.)

Step 3. Correcting the missing wedge with a pre-trained model

• In the pre_trained_models folder, select an iteration number. The higher the number, the stronger the filter effect. Below, we have selected 15.

#!/bin/bash
#SBATCH --job-name=isonet_predict
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=12
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-mixed
#SBATCH --error=isonet_predict.err
#SBATCH --output=isonet_predict.out

MDL=/shared/data/projects/et-models/isonet/model_iter15.h5

echo "SLURM: ${SLURM_JOB_ID}"

# help text mentions that gpuID is optional, but fails with an error to detect GPUs if not specified

isonet.py predict tomograms.star ${MDL} --gpuID 0,1,2,3

To run:

• sbatch 3_isonet_predict.sh

Time for one tomogram (GPU only): 15 minutes

• Inspect the corrected_tomos directory and open your tomogram in IMOD to inspect how well IsoNet performed.

---

Segmentation in MemBRAIN-seg

We will cover (very briefly) the simple MemBRAIN-seg program for tomogram segmentation. For this workflow, we will use the denoised tomograms we already produced. MemBRAIN-seg works best on denoised tomograms.

• Navigate to the MemBRAIN-seg directory;
• Open your terminal, or use an existing one.
• cd /shared/data/projects/${USER}/3_membrain_scripts
• Make a copy of your denoised tomogram from IsoNet into this directory
• cp /shared/data/projects/${USER}/2_isonet_scripts/corrected_tomos/*_corrected.mrc .

Example slurm submission script for 1_membrain_segment.sh:

#!/bin/bash
#SBATCH -n 1
#SBATCH -N 1
#SBATCH -c 12
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-mixed
#SBATCH --error=membrain_seg.err
#SBATCH --output=membrain_seg.out
#SBATCH --job-name=membrain_seg

#Tomogram path must be a complete path and needs to include the name of the file.
#ckpt-path must be an absolute path to the ckpt model.

TOMO_NAME=l40t05_5_10.00Apx_corrected.mrc
#Make sure to alter the tomoname above to match your tomogram

MDL=/shared/data/projects/et-models/membrain/MemBrain_seg_v10_beta.ckpt

echo "SLURM: ${SLURM_JOB_ID}"

membrain segment --tomogram-path ${PWD}/${TOMO_NAME} \
--ckpt-path ${MDL} \
--out-folder .

IMPORTANT: --tomogram-path needs a full path and must include the name of the tomogram. It cannot parse relative paths.

In your script:

--tomogram-path

Full path to your tomogram. MemBRAIN does not accept relative paths.

--ckpt-path

Full path to the pre-trained model provided.

To run:

• sbatch 1_membrain_segment.sh

Time for one tomogram (GPU only): 10 minutes

Check the segmentation by opening the .mrc output in either UCSF ChimeraX or IMOD.

We can compare the difference between segmentation results from an undenoised tomogram and a tomogram denoised in IsoNet. We have prepared this for you. View the two tomograms in:

/shared/data/projects/cryoet-data/data_for_participants20251022/precalculated_memBRAIN_results/undenoised_vs_denoised_segmentations/

Cluster-Based Particle Picking in TomoTWIN

In this tutorial, we will only be picking particles on a single down-sampled tomogram. We will provide you with pre-calculated results for the rest of the dataset. This is the primary reason why we reconstructed tomograms with a tomo pixel size of 10 angstroms as TomoTWIN is designed to work with tomograms of that size.

To speed up processing, we have provided you with a pre-trained model (cluster_targets.temb) that you will use to pick on your favourite tomogram in the dataset.

• In a terminal window:
• Copy the (original) undenoised and denoised tomograms from your IsoNet folder into the TomoTwin folder. We need them both for TomoTwin. They are both still present in your IsoNet folder and can be moved by:
• cp /shared/data/projects/${USER}/2_isonet_scripts/*.mrc /shared/data/projects/${USER}/4_tomotwin_scripts/pre-trained/
• cp /shared/data/projects/${USER}/2_isonet_scripts/corrected_tomos/*_corrected.mrc /shared/data/projects/${USER}/4_tomotwin_scripts/pre-trained/
• Navigate to your tomotwin working directory
• cd /shared/data/projects/${USER}/4_tomotwin_scripts/pre-trained/
• If you check the contents of your 4_tomotwin_scripts folder, you will see slurm scripts, and a cluster.temb file.

Step 1. Estimate a mask

Estimate a mask that covers the region of interest and excludes the void (empty space above and below the tomogram). This will allow TomoTwin to focus on the areas of the tomogram that are likely to include your specimen.

Adjust script 1_tomotwin_mask_estimate.sh by opening it in geany.

• Adjust script:
  • geany 1_tomotwin_mask_estimate.sh

#!/bin/bash

#SBATCH --job-name=tomotwin_mask
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=12
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-g5
#SBATCH --error=tomotwin_mask.err
#SBATCH --output=tomotwin_mask.out

MDL=/shared/data/projects/et-models/tomotwin/tomotwin_latest.pth
INMRC=l40t05_5_10.00Apx.mrc

echo "SLURM: ${SLURM_JOB_ID}"

tomotwin_tools.py embedding_mask median \
-i ${INMRC} \
-m ${MDL} \
-o mask

IMPORTANT: Supply your undenoised tomogram for this step.

In your script:

-i

We have input the name of the tomogram.

-m

We have input the absolute path to the tomotwin pre-trained model.

-o

We provide a name for the output folder of this task, which is /mask/.

To run:

• sbatch 1_tomotwin_mask_estimate.sh

Time for one tomogram (4 GPUs): 10 minutes.

Step 2. Filter the embeddings

Filter the embeddings (2D vectors that represent signals present in small 3D patches of the tomogram) using the mask you estimated to define the region of interest.

Adjust script 2_tomotwin_filtering_embeddings.sh by opening it in geany.

• Adjust script:
  • geany 2_tomotwin_filtering_embeddings.sh

#!/bin/bash

#SBATCH --job-name=tomotwin_filteringembed
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=48
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-g5
#SBATCH --error=tomotwin_filteringembed.err
#SBATCH --output=tomotwin_filteringembed.out

MDL=/shared/data/projects/et-models/tomotwin/tomotwin_latest.pth
INMRC=l40t05_5_10.00Apx.mrc
INMSK=mask/l40t05_5_10.00Apx_mask.mrc

echo "SLURM: ${SLURM_JOB_ID}"

tomotwin_embed.py tomogram -v ${INMRC} \
-m ${MDL} \
--mask ${INMSK} \
-o out/embed/tomo/ \
-s 2

IMPORTANT: Supply your undenoised tomogram for this step.

In your script:

-v

We have input the name of the volume (i.e. tomogram).

-m

We have input the absolute path to the tomotwin pre-trained model.

--mask

We provide the relative path to the newly created mask.

-o

We provide the path for the outputs of the job.

-s

We provide the stride in voxels for the sliding window.

To run:

• sbatch 2_tomotwin_filter_embeddings.sh

Time for one tomogram (4 GPUs): 30 minutes.

Step 3. Calculate pairwise distances between embeddings

We provide you with a pre-calculated target embedding file to save time. This embedding file represents the expected embeddings from representative ribosomes picked from this dataset. Step 3 will take the embeddings from subvolumes in your tomogram and compare them to the target embeddings, and generate a localization map.

Adjust script 3_tomotwin_map_tomo.sh by opening it in geany.

• Adjust script:
  • geany 3_tomotwin_map_tomo.sh

#!/bin/bash

INEMB=out/embed/tomo/l40t05_5_10.00Apx_embeddings.temb
#Make sure to update the tomoname above to match your tomoname.

tomotwin_map.py distance -r cluster_targets.temb \
-v ${INEMB} \
-o out/map/ > tomotwin_map_tomo.out 2> tomotwin_map_tomo.err

In your script:

-v

We have input the name of the volume (i.e. tomogram).

-r

We have input the temb file.

--mask

We provide the relative path to the newly created mask.

We have provided you with the cluster_targets.temb file for your convenience. It takes about an hour to produce one of these, but the procedure is relatively simple.

To run:

• ./3_tomotwin_map_tomo.sh

Time for one tomogram (CPU only): A few seconds.

Step 4. Locate particles

Locate potential particles within the tomogram.

Adjust script 4_tomotwin_locate_particles.sh by opening it in geany.

• Adjust script:
  • geany 4_tomotwin_locate_particles.sh

#!/bin/bash

#SBATCH --job-name=tomotwin_locate_particles
#SBATCH --ntasks=1
#SBATCH --nodes=1
#SBATCH --cpus-per-task=12
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-g5
#SBATCH --error=tomotwin_locate_particles.err
#SBATCH --output=tomotwin_locate_particles.out

echo "SLURM: ${SLURM_JOB_ID}"

tomotwin_locate.py findmax -m out/map/map.tmap -o out/locate/

In your script:

-m

We provide the path to the tmap file, in this case /out/map/map.tmap.

-o

We provide the path for outputs from this job to be stored.

To run:

• sbatch 4_tomotwin_locate_particles.sh

Time for one tomogram (4 GPUs): 5 minutes.

Step 5. Set an appropriate threshold for particle picks

Visualize and set an appropriate threshold that includes particle picks of your interest and minimal membrane picks.

IMPORTANT: Supply your denoised tomogram for this step. It will help you visualize the contents of the tomogram.

Open the tomogram and the locate.tloc file in napari_boxmanager.

1. In the terminal:
   • napari_boxmanager $tomoname_corrected.mrc out/locate/located.tloc
2. On the right-hand panel, set the size min and metric min to an appropriate threshold. You may need to play around.
3. Move the Z slicer on the bottom horizontal scroll bar from the left to right. As you slab through the Z-height, you should see picks appear and vanish along with the ribosomes.
4. File → Save Selected Layer(s)
   • For “Files of type:” select “Box Manager”
   • Check the location where you are saving files; by default this will be under your home directory. If you save your coordinates there, you'll need to move them back to your project working directory.
   • Name your file coords.tloc.
   • Save
5. Exit and close napari_boxmanager.

Step 6. Convert tloc file to coords

The coordinates are saved in a tloc file. This file needs to be converted into a coordinate file and then converted from there into a STAR file for WarpTools export.

Adjust script 5_tomotwin_convert_tloc.sh by opening it in geany.

• Adjust script:
  • geany 5_tomotwin_convert_tloc.sh

#!/bin/sh

tomotwin_pick.py -l coords.tloc -o coords/ > tomotwin_convert_tloc.out 2> tomotwin_convert_tloc.err

In your script:

-l

We provide the path to the tloc file.

-o

We provide the path for outputs from this job to be stored.

Time for one tomogram (CPU only): instant.

Check the presence of the .coords files in the coords/ folder. Rename the .coords file with tomoname. For example, from cluster_1.coords, type mv cluster_1.coords l40t05_5.coords to rename the file to l40t05_5.coords. Rename it after your tomogram, and leave it in the coords/ subdirectory.

Step 7. Convert coords to star

Convert the coords file to a Warp-compatible STAR file with appropriate formatting using a python script. Make sure the coords are named after the tomogram you picked them on, minus the pixel size suffix. For example, if the tomogram is called l40t05_5_10.00Apx.mrc, then the coordinate file needs to be renamed to l40t05_5.coords. The name of the coords file is used to populate the _rlnMicrographName column of the STAR file with the correct tomoname and a .tomostar suffix. This formatting is required for WarpTools.

Rename the coords file after the tomoname. Then, run the script from the main /4_tomotwin_scripts/ directory.

• ./tomotwin_coords_to_warp.py

Time for one tomogram (in terminal window): instant.

---

End of Day One. Stop here and transfer to the pre-calculated folder for Day Two.

Step 1. Export particles.

Warp will take the particle coordinates and use them to reconstruct 3D or 2D subvolumes de novo from the tiltseries settings and corrections information stored in the Warp working directory.

Navigate back to the working directory.

• cd /shared/data/projects/${USER}/nov6_subset7/

#!/bin/bash

WarpTools ts_export_particles \
--settings warp_tiltseries.settings \
--input_star particle_coordinates.star --normalized_coords \
--output_angpix 4.70 --box 128 --diameter 250 \
--output_processing subtomograms \
--output_star relion/particles_warp.star --2d \
--relative_output_paths > warp_export.out 2> warp_export.err

In your script:

--input_star

The path to the particle coordinates.

--box

The desired box size in pixels.

--diameter

The particle diameter in angstroms.

--output_angpix

The pixel size you want to export particles at. You can input any value up to the unbinned pixel size of the raw data.

--2d

Outputs 2D subtilt particles rather than 3D subvolumes.

--relative_output_paths

The output STAR file that is produced from this job contains paths that are relative to the location the STAR file is stored in. If you don’t intend to move the STAR somewhere else, then you can output relative paths, which are shorter and make for a much cleaner STAR file.

--output_processing

The name of the output file where you would like to have the particle series stored. If you don’t output them into a specific folder, they will go inside the same folder that the alignment files are stored in, which in our case is /aretomo3/ inside a subdirectory called /particlesseries/.

--output_star

The folder to store the STAR files for Relion. This is the folder that you must launch Relion inside of because the paths in the STAR files are relative.

--normalized_coords

Whether the coordinates in the input STAR file are normalized or not. STAR file particle coordinates can come in three different units; angstroms, pixels or normalized coordinates (with a value between 0 and 1).

To run:

• ./1a_WarpTools_export_particles.sh

Time for one tomogram (4 GPUs): ~1m.

This will output 2D subvolumes in a subdirectory called subtomograms. You can check them by viewing the summed average mrc files in IMOD.

---

Subvolumes containing putative particles extracted from the reconstructed tomogram are subjected to iterative alignment procedures to rotate and translate them in 3D space to maximize cross-correlation with a reference. Through successive cycles of alignment and averaging, stochastic noise is reduced while coherent structural features are reinforced. This process is identical to the 3D volume refinement process in single-particle EM.

Step 1. Produce an initial 3D model through particle alignment

At this point in the workshop, to keep jobs within a reasonable runtime, we will continue with a smaller particle subset which will unfortunately limit the resulting resolution. Several steps (Bayesian Polishing, CTF Refinement, Re-extracting refined particles) are not meaningful at these resolutions but may be appropriate for other datasets.

Edit the optimisation_set.star file that WarpTools outputs. There is a missing file name in this star file which must be input manually.

1. Switch to the Relion subdirectory (cd relion or cd /shared/data/projects/${USER}/nov6_subset7/relion)
2. Adjust the star file:
   • geany particles_warp_optimisation_set.star
   • Add particles_warp.star into the _rlnTomoParticlesFile column of the star file.
   • Save

From within the relion/ folder, launch the Relion 5 GUI:

• relion --tomo &

1. Select the "3D Initial Reference" job type.
2. In the "IO" tab:
   • Set "Input optimisation set" to particles_warp_optimisation_set.star.
3. In the "CTF" tab:
   • Set "Do CTF-correction?" to "Yes."
4. In the "Optimization" tab:
   • Set "Mask diameter" to 380 angstroms (1.5 times the size of the particle diameter along the longest axis).
5. In the "Compute" tab:
   • Set "Use parallel disc I/O?" to "Yes".
   • Set "Number of pooled particles" to "30".
   • Set "Pre-read all particles into RAM?" to "No".
   • Set "Use GPU acceleration" to "Yes".
   • Set "Which GPUs to use:" to blank (don’t specify anything).
6. Under the "Running" tab:
   • Set "Number of MPI processes" to "1".
   • Set "Number of threads" to "30".
   • Set "Submit to queue" to "Yes".
   • Set "Queue name" to "relion-gpu-medium-mixed".
   • Set "Number of GPUs requested" to "4".
7. Use the "Run!" button to submit the job.

When the job has run for a while, evaluate your 3D refinement by opening the run_it$$$_class001.mrc volume in UCSF ChimeraX. It should look like a terrible ribosome.

We will abort this job early before it finishes to save time, once we have acquired a reference good enough to progress to Refine3D.

Step 2. Perform 3D auto-refinement

From within the Relion 5 GUI;

1. Select the "3D auto-refine" job type.
2. In the "IO" tab:
   • Set "Input optimisation set" to particles_warp_optimisation_set.star.
   • Set “Reference map” to InitialModel/job001/run_it$$$_class001.mrc.
3. In the "Reference" tab:
   • Set "Ref. map is on absolute greyscale?" to "Yes."
   • Set “Re-size reference if needed?” to “Yes”.
   • Set “Initial low-pass filter (A)” to “40”.
   • Set “Symmetry” to “C1”.
4. In the "CTF" tab:
   • Set "Do CTF correction?" to "Yes."
   • Set “Ignore CTFs until first peak?” to “No”.
5. In the "Optimisation" tab:
   • Set "Mask diameter” to 380 angstroms (1.5 times the size of the particle diameter along the longest axis).
   • Ignore the "Auto-sampling" tab and "Helix" tab.
6. In the "Compute" tab:
   • Set “Use parallel disc I/O?” to “Yes”.
   • Set "Number of pooled particles" to "30".
   • Set “Skip padding?” to “No”.
   • Set “Pre-read all particles into RAM?” to “Yes”.
   • Set “Combine iterations through disc?” to “No”.
   • "Use GPU acceleration" to "Yes".
   • Set “Which GPUs to use:” to “ ” (don’t specify anything).
7. Under the "Running" tab:
   • Set "Number of MPI processes" to "5".
   • Set "Number of threads" to "8".
   • Set "Submit to queue" to "Yes".
   • Set "queue name" to "relion-gpu-medium-mixed".
   • Set "number of GPUs requested" to "4".
8. Use the "Run!" button to submit the job.

When the job has run for a while, evaluate your 3D refinement by opening the run_it$$$_class001.mrc volume in UCSF ChimeraX. It should look like a ribosome!

Moving forward, we will use pre-calculated RELION results to save time (this Refine3D job would otherwise take over an hour to run to completion).

---

2D particles that have been aligned in Relion 5 to a decent volume reconstruction can then be fed into MTools for further refinement. MTools will return to the settings used in WarpTools and use the new particle coordinates and alignment information/poses to calculate per-particle defocus and CTF correction before performing its own refinement procedure. In a high-resolution pipeline, refinements would be made in Relion and MTools iteratively with progressive un-binning until resolution ceases to improve. These steps should be performed in the /shared/data/projects/${USER}/nov6_subset7 precalculated project directory, not the relion subdirectory, so that MTools is able to locate the relative WarpTools files it requires.

Step 1. Create a population

Create a population in MTools with 1_MTools_create_population.sh.

#!/bin/bash

MTools create_population \
--directory m_run01 \
--name population_01 > MTools_create_population.out 2> MTools_create_population.err

In your script:

--directory

the name of the MTools directory for your jobs to store outputs into.

--name

the name of the population you would like to refine.

To run:

• ./1_MTools_create_population.sh

Time for one tomogram (CPU only): A few seconds.

Step 2. Create a source

Create a source in MTools with 2_MTools_create_source.sh.

#!/bin/bash

MTools create_source \
--population m_run01/population_01.population \
--processing_settings warp_tiltseries.settings \
--name source_01 > MTools_create_source.out 2> MTools_create_source.err

In your script:

--population

the name of the population.

--processing_settings

the name of the settings file in the WarpTools processing directory.

--name

the name of the data source.

To run:

• ./2_MTools_create_source.sh

Time for one tomogram (CPU only): A few seconds.

Step 3. Create a species

Create a species in MTools with 3_MTools_create_species.sh.

#!/bin/bash
#SBATCH --job-name=MTools_create_species
#SBATCH --error=MTools_create_species.err
#SBATCH --output=MTools_create_species.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:1
#SBATCH --partition=relion-gpu-medium-g5

echo "SLURM: ${SLURM_JOBID}"

MTools create_species \
--population m_run01/population_01.population \
--name species_01 \
--diameter 250 \
--sym C1 \
--temporal_samples 1 \
--particles_relion relion/Refine3D/job002/run_data.star \
--half1 relion/Refine3D/job002/run_half1_class001_unfil.mrc \
--half2 relion/Refine3D/job002/run_half2_class001_unfil.mrc \
--mask binary_mask_ribosome_box128_apix4.70.mrc \
--angpix 4.70 \
--angpix_resample 4.70

In your script:

--population

the name of the population.

--processing_settings

the name of the settings file in the WarpTools processing directory.

--name

the name of the data source.

To run:

• sbatch 3_MTools_create_species.sh

Time for one tomogram (CPU only): ~40 minutes.

Step 4. MCore without any iterations

Refine in MCore without any iterations with 4_MTools_MCore_1.sh.

#!/bin/bash
#SBATCH --job-name=MTools_mcore_1
#SBATCH --error=MTools_mcore_1.err
#SBATCH --output=MTools_mcore_1.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:4
#SBATCH --partition=relion-gpu-medium-g5

echo "SLURM: ${SLURM_JOBID}"

MCore \
--population m_run01/population_01.population \
--iter 0 \
--perdevice_refine 1 --port -1

In your script:

--population

the name of the population.

--iter

the number of iterations.

--perdevice_refine

whether to run multiple worker processes per GPU.

--port

port to use per REST API calls.

To run:

• sbatch 4_MTools_MCore_1.sh

Time for one tomogram (CPU only): ~10m

Step 5. MCore with iterations

Refine in MCore with iterations with 5_MTools_MCore_2.sh.

#!/bin/bash
#SBATCH --job-name=MTools_mcore_2
#SBATCH --error=MTools_mcore_2.err
#SBATCH --output=MTools_mcore_2.out
#SBATCH --ntasks=5
#SBATCH --nodes=1
#SBATCH --cpus-per-task=6
#SBATCH --gres=gpu:1
#SBATCH --partition=relion-gpu-medium-g5

echo "SLURM: ${SLURM_JOB_ID}"

MCore \
--population m_run01/population_01.population \
--refine_imagewarp 6x4 \
--refine_particles \
--ctf_defocus \
--ctf_defocusexhaustive \
--perdevice_refine 1

In your script:

--population

the name of the population.

--refine_imagewarp

refine image warping with a dimension of XxY dimensions. If left blank, MCore will not refine image warp.

--refine_particles

whether to refine particle poses.

--ctf_defocus

refine CTF defocus using a local search.

--ctf_defocusexhaustive

refine CTF defocus with a more exhaustive grid search in the first sub-iteration prior to local defocus refinement (performed with the --ctf_defocus parameter).

--perdevice_refine

whether to run multiple worker processes per GPU.

To run:

• sbatch 5_MTools_MCore_2.sh

Time for one tomogram (CPU only): ~3m