Containers

WARNING: This page is obsolete.

Please look at https://atlas-software.docs.cern.ch/software-infrastructure/containers/.

Introduction

Containers allows one, for example, to setup a SLC6, centos7 or el9 OS on a host running a different OS with very little overhead. setupATLAS encapsulates the container startup in Linux or MacOSX so that even if, for example, you are running Ubuntu or a Mac you will run in an SLC6, centos7 or el9 ATLAS ready OS environment.

These are the supported containers and OS platforms:

Container Software	Linux	MacOSX
Apptainer	Suported	Unsupported
Docker	Suported	Supported
Podman	Suported	Unsupported
Shifter	Suported	Unsupported

CernVM

Singularity has been deprecated in favor of Apptainer. If you still need Singularity, please contact us.

Important Notes

There are some things users need to be aware of when using containers.

Be realistic with your laptop/desktop expectations. In principle containers should allow you to most efficiently utilize your resources but these also depend on how they are used by your host machine (eg running Mail, web browsers etc); machines where the CPU is dedicated to running jobs see only tiny degradation (< 5%) compared to running on bare metal. It will use cvmfs so if you have a nearby squid, configure that in your host machine's /etc/cvmfs/default.local to speed things along.

• All platforms:
  • Note that your home directory will be mapped to /home/<username>.
  • If you use symbolic links, use relative ones instead of absolute ones (if possible) since the paths in the container can be different for $HOME.

• MacOSX:
  • File names are case insensitive on OSX (default format file system) which is not the case on Linux. If your host is a Mac, then use unique case-insensitive file and dir names.

Installation

Container Software

First, decide on the container software to use. There are the recommended choices:

• Apptainer (default except for MacOSX)
  • Apptainer is now distributed through cvmfs so you no longer need to install it yourself.
  • Note that on RHEL8/9 based OSes, max_user_namespaces is large enough by default, so no configuration change is required.
  • If using a RHEL7-based OS, to enable use of user namespaces:

      echo "user.max_user_namespaces = 15000"  > /etc/sysctl.d/90-max_user_namespaces.conf
      sysctl -p /etc/sysctl.d/90-max_user_namespaces.conf

• Docker (default for MacOSX)
  • Install Docker https://www.docker.com/community-edition
  • (MacOSX) In Docker Preferences, under File Sharing, add these /Users /Volumes /cvmfs /private /tmp.
  • Open a new Terminal and type this and afterwords reboot. defaults write org.xquartz.X11 enable_iglx -bool true. This is needed for vp1.

• Podman: You can install Podman on Linux and use that instead of Docker. You will have to explicitly specify --swtype=podman to use podman since Apptainer is the default for Linux.

cvmfs

• Install cvmfs.
  • You should not let automount unmount the cvmfs repos.
    • eg for Linux, /etc/auto.master should have a no timeout: /cvmfs /etc/auto.cvmfs --timeout 0
  • Relocated cvmfs will also work.

Standalone Script: No-cvmfs (you do not need cvmfs to use setupATLAS -c or containers)

This standalone script is designed to only run containers.

Running it will install a minimal version of ATLASLocalRootBase and, as long as there is a network connection and the last update was longer than 6h ago, it will also update it when invoked. Do one of the following after you are in a local directory:

• cp /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/standalone/setupATLAS ./
• From your browser, save this linked setupATLAS standalone script
• On a command line, use curl -O or wget to download https://atlas-tier3-sw.web.cern.ch/containers/setupATLAS
  • curl -O https://atlas-tier3-sw.web.cern.ch/containers/setupATLAS
  • wget https://atlas-tier3-sw.web.cern.ch/containers/setupATLAS

and then

• chmod +x ./setupATLAS

To run:

• <path to>/setupATLAS -c centos7 # As this is a default ATLAS OS container, /cvmfs needs to be available for any work
• <path to>/setupATLAS -c gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0 # standalone container; will not need cvmfs
• <path to>/setupATLAS -c -h # help
• <path to>/setupATLAS -c --showVersions # show available containers
• <path to>/setupATLAS -c --showVersions analysisbase,24.2,alma9 # show available analysisbase release 24.2, alma 9 containers
• <path to>/setupATLAS -c find=analysisbase,24.2,alma9 # to find and setup an analysisbase release 24.2, alma 9 container

eg to use the cern gitlab registry for an image:

# you need a personal token if on Multi-factor authentication which you enter instead of password.  To get a token, 
# login to https://gitlab.cern.ch, and go to https://gitlab.cern.ch/-/profile/personal_access_tokens
docker login --username desilva gitlab-registry.cern.ch
setupATLAS -c docker://gitlab-registry.cern.ch/atlas/athena/athsimulation:23.0.31.sw21-0 -P linux/amd64 # -P is to specify platform aarch64 or x86_64; see -h

Miscellaneous

• All Platforms
  • Create $HOME/.globus and copy your grid credentials. Make sure that the protections are correct
    • chmod 744 $HOME/.globus
    • chmod 444 $HOME/.globus/usercert.pem
    • chmod 400 $HOME/.globus/userkey.pem
  • Define RUCIO_ACCOUNT either in your $HOME login script or in the $HOME/.*rc.container scripts (see Login Scripts section below)

• MacOSX:
  • Install XQuartz https://www.xquartz.org
    • Launch XQuartz preference, select security and enable "Authenticate connections" and "Allow connects from network clients"
  • In System Preferences, select Sharing, entry a name in "Computer Name". Give it a name without spaces or special characters - only A-Za-z0-9 in the name.

non-cvmfs ATLASLocalRootBase

This is not recommended unless you know what you are doing (eg for HPC resources). You can install a minimal local ATLASLocalRootBase with the --adcRH=7,9 options; see https://gitlab.cern.ch/atlas-tier3sw/manageTier3SW for details.

If you specify the --localalrb option, eg setupATLAS -c <container> --localalrb, or if you start ALRB from a local installation and do not have cvmfs, ATLASLocalRootBase will then be available inside the container from /alrb/ATLASLocalRootBase. This version will be the same as that on the host machine.

Note that you do not need to specify --localalrb or other modifiers. We try to determine the container type and setup accordingly.

Note that for local ATLASLocalRootBase installations, you will be responsible

• for the installation and regular updates (at least once/day) of the host machine's local ATLASLocalRootBase installation,
• ensuring that the tools for the same architecture of the containers you use are installed on the host machine's local ATLASLocalRootBase installation.
• Since grid-middleware CA/CRL updates are only possible on Linux architectures, a MacOSX host machine's local ATLASLocalRootBase will not be fully functional for grid related tools. There is no problem for Linux based host machines.

This table summarizes what is and is not possible for Linux containers:

Host Machine OS	Container ALRB is Host cvmfs	Container ALRB is Host Local Install
	Container /cvmfs is available	Container /cvmfs unavailable
Linux
Mac		(grid tools will not work)

Setup

setupATLAS -c <container> where container can be slc5 (unsupported), slc6, centos7, or el9 or any string that contains these case insensitive keywords to setup the default container.

• To setup from a host machine that has the container runtime software installed, simply do eg setupATLAS -c centos7 or setupATLAS -c el9

• If your machine is part of a cluster and has batch queue access and you want to submit batch jobs from inside the container, do instead eg setupATLAS -c centos7 -b

• If you need GPUs and are on a host machine that provides them, use this option (you can instead use -runtimeOpt; see setupATLAS -c -h for details):
  • setupATLAS -c centos7 -e "--nv" so that the GPUs will be available from inside the container for apptainer
  • setupATLAS -c centos7 -e "--gpus all" so that the GPUs will be available from inside the container for docker/podman

• Apptainer will use the images from cvmfs unless explicit paths are specified; These can be the default OS ones or from the cvmfs unpacked repository. See next sections for details.

• Alternatively, <container> can be a path to a container file (Apptainer) or from the registry (Docker/Podman/Shifter).
  • Apptainer/Docker:
    • setupATLAS -c docker://atlas/analysisbase:21.2.85-centos7
    • setupATLAS -c docker://gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0
  • Podman:
    • setupATLAS -c docker.io/atlas/analysisbase:21.2.85-centos7
    • setupATLAS -c gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0

Note that we recommend defining the environment variable APPTAINER_TMPDIR (export APPTAINER_TMPDIR=<some dir>) so that a sandbox will be built if you download docker images; it will then only update if needed. Note that the container files will be downloaded from the registry the first time that container is run or if it needs refreshing.

Help and Versions

Help is available from

• setupATLAS -c -h

Containers on the cvmfs unpacked repository can be listed by

• setupATLAS -c --showVersions

Default Containers

Apptainer will use the unpacked containers in /cvmfs/atlas.cern.ch/repo/containers/fs/apptainer by default.

If the env $ALRB_CONT_DEFAULTCONTDIR is defined, the default containers will be searched there first. This dir needs to have the same structure as the one on /cvmfs. This feature is meant for HPC sites which have limited bandwidth and prefer to rsync specific cvmfs directories or use atlas-cvmfsexec.

Docker/Podman/Shifter will use the default images from the repository (eg https://hub.docker.com/u/atlasadc for Docker/Podman).

Containers from /cvmfs/unpacked.cern.ch

These will work only with Apptainer. Note that if the env $ALRB_CONT_UNPACKEDDIR is defined, this dir will be searched first before the cvmfs unpacked repository.

showVersions container will produce a list of available containers in the cvmfs unpacked repository and you can use setupATLAS to run them. for example:

$ setupATLAS -c atlas/analysisbase:21.2.108
Info: Migrating env variable SINGULARITY_TMPDIR to APPTAINER_TMPDIR
Info: Migrating env variable SINGULARITY_CACHEDIR to APPTAINER_CACHEDIR
Info: /cvmfs mounted; do 'setupATLAS -d -c ...' to skip default mounts.
------------------------------------------------------------------------------
Apptainer: 1.2.2
Host: Linux, CentOS Linux 7 (Core), x86_64, 3.10.0-1160.102.1.el7.x86_64
From: /cvmfs/atlas.cern.ch/repo/containers/sw/apptainer/x86_64-el7/1.2.2/bin/apptainer
ContainerType: non-atlas-standalone
apptainer  exec -B /afs:/afs -B /eos:/eos -e   -H /afs/cern.ch/user/d/desilva/.alrb/container/apptainer/home.M7kO4A:/alrb -B /cvmfs:/cvmfs -B /afs/cern.ch/user/d/desilva:/srv /cvmfs/unpacked.cern.ch/registry.hub.docker.com/atlas/analysisbase:21.2.108 /bin/bash
------------------------------------------------------------------------------
 setupATLAS is available for this non-atlas-standalone container type.
          
 /afs /eos available in this container
             _ _____ _      _   ___ 
            /_\_   _| |    /_\ / __|
           / _ \| | | |__ / _ \\__ \
          /_/ \_\_| |____/_/ \_\___/

This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:

          source /release_setup.sh

also do afterwords
      source /alrb/postATLASReleaseSetup.sh
Setting COOL_ORA_ENABLE_ADAPTIVE_OPT=Y
Singularity> 

Standalone Containers

Standalone containers (see https://gitlab.cern.ch/atlas/athena/container_registry) are self-contained images which have an ATLAS release included. As such, the setup is different and also submissions to the grid are done from outside the container as detailed in the snapshot below. Note that you may not need to add modifiers as it will try to guess what type of container you are running and adjust accordingly.

eg:

Perfection:~ desilva$ setupATLAS -c docker://gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0
docker  build  --platform linux/arm64 --pull  -t my-gitlab-registry.cern.ch_atlas_athena_analysisbase-24.2.0 /Users/desilva/.alrb/container/docker/files/gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0
[+] Building 75.7s (8/8) FINISHED                          docker:desktop-linux
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load build definition from dockerfile                       0.0s
 => => transferring dockerfile: 560B                                       0.0s
 => [internal] load metadata for gitlab-registry.cern.ch/atlas/athena/ana  3.6s
 => [auth] atlas/athena/analysisbase:pull token for gitlab-registry.cern.  0.0s
 => [1/3] FROM gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0@  71.3s
 => => resolve gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.0@s  0.0s
 => => sha256:2c54ef72daddebad2305a95508adb372bbba70b267c 1.31kB / 1.31kB  1.1s
 => => sha256:edc77131a6f2620af41c186d1fdc92d63d220dc 113.35MB / 113.35MB  8.4s
 => => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d 32B / 32B  0.9s
 => => sha256:5fd9b1b55e4cb59b36a4073b7ef54116f53f23ce2e4 9.39kB / 9.39kB  0.0s
 => => sha256:6ddbf9161c559c174c939cc6dfcab3c5071ff5d923f 4.06kB / 4.06kB  0.0s
 => => sha256:d96861235de41b10ab4e0ac0d44cf4f725fe46e440a76bd 497B / 497B  1.3s
 => => sha256:989cefc75d913b87cb82821c1ce74eeb9906e3 298.98MB / 298.98MB  35.1s
 => => sha256:f9277b307cad05c20e944285815bf89c32dbbb7af33 4.60kB / 4.60kB  1.7s
 => => sha256:bedb029d66e037182750ce60cbd0805938bb9b28ccf4335 201B / 201B  2.2s
 => => sha256:061201b478a4fa581f9527d8293de4d6882e2cd0e493b79 327B / 327B  2.6s
 => => sha256:2d57ef263b4e8ddc9b154a27f1832f0c0fa94e 629.16MB / 629.16MB  72.1s
 => => extracting sha256:edc77131a6f2620af41c186d1fdc92d63d220dc6f5fa10eb  3.1s
 => => sha256:0bcac14b88ff098db98ee1bb2ad09792cfaf3fdec54d081 225B / 225B  9.0s
 => => sha256:1c0182d54b4fe05bb13913eca6ac792cba6671ee5cd5536 529B / 529B  9.7s
 => => sha256:9f4256cb6c6f97d4c8e6871acfb30436b50328877b79b5 310B / 310B  10.5s
 => => sha256:4f1357b11e5f2101f3c1b985bc602b9e8c2a14eada503c 739B / 739B  11.3s
 => => sha256:a12fa18241fb54c76d23a43b755bc0bc67fef78c61637c 163B / 163B  12.4s
 => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6  0.0s
 => => extracting sha256:2c54ef72daddebad2305a95508adb372bbba70b267ced193  0.0s
 => => extracting sha256:d96861235de41b10ab4e0ac0d44cf4f725fe46e440a76bd3  0.0s
 => => sha256:16100be3bd106840f128f78b4b87fffae0f52292df4f5e 858B / 858B  12.9s
 => => sha256:cbcf6e6fc6e0e504aa0dbfd700b7c6e981c519c3 12.12kB / 12.12kB  13.6s
 => => extracting sha256:989cefc75d913b87cb82821c1ce74eeb9906e3608e15e076  5.5s
 => => extracting sha256:f9277b307cad05c20e944285815bf89c32dbbb7af33126e8  0.0s
 => => extracting sha256:bedb029d66e037182750ce60cbd0805938bb9b28ccf4335f  0.0s
 => => extracting sha256:061201b478a4fa581f9527d8293de4d6882e2cd0e493b798  0.0s
 => => extracting sha256:2d57ef263b4e8ddc9b154a27f1832f0c0fa94e94fdace01  13.5s
 => => extracting sha256:0bcac14b88ff098db98ee1bb2ad09792cfaf3fdec54d081a  0.0s
 => => extracting sha256:1c0182d54b4fe05bb13913eca6ac792cba6671ee5cd55364  0.0s
 => => extracting sha256:9f4256cb6c6f97d4c8e6871acfb30436b50328877b79b573  0.0s
 => => extracting sha256:4f1357b11e5f2101f3c1b985bc602b9e8c2a14eada503c5f  0.0s
 => => extracting sha256:a12fa18241fb54c76d23a43b755bc0bc67fef78c61637c96  0.0s
 => => extracting sha256:16100be3bd106840f128f78b4b87fffae0f52292df4f5ef7  0.0s
 => => extracting sha256:cbcf6e6fc6e0e504aa0dbfd700b7c6e981c519c3475cd75f  0.0s
 => [2/3] RUN useradd --no-create-home --home-dir /alrb --uid 501 --shell  0.6s
 => [3/3] RUN mkdir -p /alrb /alrb/ATLASLocalRootBase /srv /scratch /tmp   0.1s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:ea62523b87cdef6886c2dded9d1a0ebd180811931a564  0.0s
 => => naming to docker.io/library/my-gitlab-registry.cern.ch_atlas_athen  0.0s

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
MacOS X: Wait for XQuartz to start
xhost:  unable to open display "localhost:0"
------------------------------------------------------------------------------
Docker: 24.0.7
Host: MacOSX, macOS14, arm64, 23.2.0
From: /usr/local/bin/docker
ContainerType: non-atlas-standalone
docker  run  --platform linux/arm64 -e "ENV=/alrb/.bashrc" --mount "type=bind,source=/Users/desilva/.alrb/container/docker/home.pTJKdI,target=/alrb" --mount "type=bind,source=/Users/desilva/work/ATLASLocalRootBase,target=/alrb/ATLASLocalRootBase" --mount "type=bind,source=/Users/desilva,target=/srv" --mount "type=bind,source=/private/var/folders/41/jp18l62d357g3jz1mnqw6rn00000gn/T,target=/scratch" --mount "type=bind,source=/tmp,target=/tmp" -it --rm --name my-gitlab-registry.cern.ch_atlas_athena_analysisbase-24.2.0-1704390971 my-gitlab-registry.cern.ch_atlas_athena_analysisbase-24.2.0
------------------------------------------------------------------------------
             _ _____ _      _   ___ 
            /_\_   _| |    /_\ / __|
           / _ \| | | |__ / _ \\__ \
          /_/ \_\_| |____/_/ \_\___/

This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:

          source /release_setup.sh

A serious security issue in ROOT's web-based GUI has been identified:
https://root.cern/about/security/#2023-11-26-open-port-for-control-of-web-gui-allows-read-and-write-access-to-file-system

You should not use the web-based browser until further notice and instead
leave the old-style TBrowser as the default, as set in this image.

also do afterwords
      source /alrb/postATLASReleaseSetup.sh
Setting COOL_ORA_ENABLE_ADAPTIVE_OPT=Y
bash-4.2$ 

Login scripts

The container will not use the login scripts in your $HOME dir. If you want to customize your environment, then create these files and they will be used for the container automatically:

• $HOME/.bashrc.container (for bash) or $HOME/.zshrc.container (for zsh)

# example of .bashrc.container to set the prompt to pwd
unset PROMPT_COMMAND
export PS1="`pwd` $ "

# example of .zshrc.container to set the prompt to pwd
export PROMPT="`pwd` $ "

• $HOME/.ssh.container if you need a different configuration (you may not need to; look at: ssh config template)
• $HOME/.globus will be used
• some environment variables will be passed from your host machine to the container; among the important ones, if they exist, are $RUCIO_ACCOUNT, $SITE_NAME and settings for X11 forwarding.

ContainerTools

After the container starts, you will have access to these tools (-h for usage):

eval "echo <path on host> | sed $ALRB_CONT_SED2CONT"

Advanced Options

do setupATLAS -c -h to get the currently available options.

These options affect the behaviour of setupATLAS -c/--container; if set, they will take precedence over any environment variable for the same functionality. The options listed in the table below are incomplete; only those with a matching environment variable are listed. See setupATLAS -c -h for the complete list.

Setting Advanced Options In Site Config File

If $ALRB_localConfigDir is defined, it will look for a file alrb_container.cfg.sh and use those configurations to set up a container. To make it simpler to change the above environment variables without having to check for their existence, or to consider too many back-slashes, you can use the syntax set_ALRB_ContainerEnv inside the alrb_container.cfg.sh to set the variables. For example, see this.

In addition, if run_ALRB_UserContainerConfig is present in the site alrb_container.cfg.sh file, the user's configurations will be setup as well. This will happen only once.

Setting Advanced Options for Users

Note, but you may not want to do this to override site configurations:

• $HOME/alrb_override_container.cfg.sh, if available, will override the settings in $ALRB_localConfigDir/alrb_container.cfg.sh
• $HOME/alrb_override_<runtime>.sh will overrride $ALRB_localConfigDir/alrb_<runtime>.sh initialization script

Users can define a file $HOME/alrb_container.cfg.sh setting up the env variables in the table above to configure the container. The syntax set_ALRB_ContainerEnv can also be used inside this file. This will be run only once, either by the site config file or else when the container sets up.

Batch Jobs

If your cluster allows for batch jobs (eg slurm, condor, lsf or pbs) to start containers, you can submit and access the batch queues from inside your container interactive session. For example, if you type

setupATLAS -c centos7 -b or setupATLAS -c el9 -b

You can next generate a script to submit to the batch system. eg.

batchScript "source <somepath in container>/myJob.sh" -o submitMyJob.sh

(The above will create a file submitMyJob.sh that will run myJob.sh inside the same type of container you are on now but on the batch queue.)

(You can also simply do this: batchScript "\echo \"Starting myJob\"; cd myDirSomewhere; asetup 19.0.3,slc6,here; athena ...." -o submitMyJob.sh)

Finally submit it from inside the container with one of these:

• bsub -L /bin/bash submitMyJob.sh if your site supports lsf
• sbatch --export=NONE submitMyJob.sh if your site supports slurm
• condor_submit <jdl file executing submitMyJob.sh> if your site supports condor
• qsub submitMyJob.sh if your site supports pbs

Educated Guessing

ATLASLocalRootBase will now probe the containers prior to running to determine the type of container and to determine how to set it up. The table below summarizes the logic. Options will still work with this educated guessing. You can suppress guessing by defining export ALRB_CONT_GUESSMOD=NO

FAQ

Running Batch Jobs on lxplus

Generally, you should not run batch jobs on lxbatch; see this note.

However if you need to run a centos7 release on batch where the batch systems are el9 resources, you can run it in a container as described below.

Suppose you have a script you want to run source myJob.sh that needs a centos7 environment and a condor submission script submit.jdl that you will submit with condor_submit.

• Start a centos7 container and allow batch command access (-b option) from inside the container: setupATLAS -c centos7 -b
• Test your code inside that container
• Create the wrapper for myJob.sh as follows: batchScript "source <path>/myJob.sh" -o <path>/myJobWrapper.sh
• In submit.jdl, change the executable to point to <path>/myJobWrapper.sh instead of myJob.sh.
• Submit from inside the container: condor_submiit submit.jdl

Mount point summary

MacOSX: SSH keys in Docker container

There is a known issue with Docker containers and MacOSX that one cannot use the ssh-agent from MacOSX inside the Linux container. If you want to use ssh keys, then do ssh-add (or ssh-add $HOME/.ssh/<keyfile> if the keyfile is not named the default id_dsa or id_rsa) after the container starts to add your key to the container's ssh-agent. Passwordless logins should work again after that.

All platforms: X11 and ssh

If you ssh to a host that has apptainer, you can start it up and portback X11 as follows; eg.

• ssh -AY <hostname>
• setupATLAS -c centos7
• MacOSX local machine, start XQuartz and wait until it starts.
• on your local machine, xhost + localhost

All platforms: Missing favourite software

These containers are lightweight and may be missing software that Tier3 users may take for granted. But there are easy ways around this.

For example: emacs: Keep in mind that your file system is shared with the container and so you can use emacs on your host machine. Similarly for web browsers.

If there is something you think should be made available, and cannot be used from the host machine, do let us know through the DAST eGroup.

All platforms: Why is my dir /srv after startup

If you do setupATLAS -c <container> in a directory, that current directory gets mapped to /srv on the container. (This is intentionally done). So If you are starting from somewhere other than your $HOME directory on the host, you will end up on /srv when the container starts for consistency. Your $HOME dir in the container is still there and unchanged (you can type cd to go to it).

Note that containers running on lxplus will have /afs and /eos mounted and also the env variable $UHOME defined that will point to the user's afs directory.

Docker: Start an Interactive Session in a Running Container

You can choose to automatically start a new container or join an already running container (the latest if you have multiple versions of the same one running). This will save resources and is done automatically if an environment variable is defined to affect the behaviour as shown below. (This is not made default behaviour as users need to be actively aware of doing it because of the caveat below.)

  export ALRB_CONT_CONDUCT="dockerJoin"   # join an already running docker container or start a new one otherwise 
  setupATLAS -c <container>
or
  setupATLAS -c <container> --conduct="dockerJoin" 

Caveat: If you are joining a running container you will see this warning that the dependent session will die once the main container is exited.

$ setupATLAS -c slc6
Docker: Join already running my-atlas-grid-centos6-1519434423
        Remember that if you exit that container, this session will die

Tier3 Sites And Defining setupATLAS

This section is for Tier3 admins who want to define their sites to use containers and setupATLAS

Sites can source a script such as the following. Note the export ALRB_containerSiteOnly=YES which will prevent setupATLAS tools from being available except if it is inside a container (this is to check those users forgetting to start a container). The environment ALRB_containerSiteOnly is for those sites whose host OS is not ATLAS ready and want all users to run ATLAS jobs or tools inside the container.

These are the recommended steps:

• some of the examples below are from ComputeCanada Tier3 site settings which are available in https://gitlab.cern.ch/atlas-computecanada/tier3.

• create a common directory somewhere that all users can access and which only ATLAS administrators at your site can edit. In the example below, I call this directory /atlasadmin.

• create a setup script like the one below that all your uses will source (in ~/.bashrc or ~/.zshrc); for example, I call this /atlasadmin/AtlasUserSiteSetup.sh. Your ATLAS users should add this line in their login scripts source /atlasadmin/AtlasUserSiteSetup.sh.

if [ ! `typeset  -f setupATLAS > /dev/null` ]; then
    unalias setupATLAS  > /dev/null 2>&1
    # eg at Compute Canada sites, $CC_CLUSTER is defined and we use that for local site configurations
    my_AtlasSiteDir="`dirname ${BASH_SOURCE:-$0}`/$CC_CLUSTER"
    function setupATLAS
    {    
        if [ -d  /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase ]; then

            export ALRB_localConfigDir=$my_AtlasSiteDir
            
            # uncomment if all ATLAS users must use containers interactive (eg not ATLAS-ready site)
            #export ALRB_containerSiteOnly=YES

            # modify if you have a site name or are co-located with a Tier2
            #export SITE_NAME=

            # modify if, for example, singularity lives elsewhere
            #export PATH="/opt/software/singularity/bin:$PATH"

            # modify if additional local Tier3 directories need to be bind mounted inside the container   
            #\echo $ALRB_CONT_CMDOPTS | \grep -e "/project:" > /dev/null 2>&1
            #if [ $? -ne 0 ]; then
            #    export ALRB_CONT_CMDOPTS="$ALRB_CONT_CMDOPTS -B /project:/project"
            #fi
 
            export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
           
            source $ATLAS_LOCAL_ROOT_BASE/user/atlasLocalSetup.sh
            return $?
        else
            \echo "Error: cvmfs atlas repo is unavailable"
            return 64
        fi
    }
fi

• (Optional) If the cluster allows batch jobs, add a file /atlasadmin/siteConfigBatch.txt containing the additional commands needed by the site for the batch job. The following is an example for Compute Canada Tier3s. This file will be included in the executable of the batch job generated by batchScript.

#----------------------------------------------------------
# Compute Canada site setup for ATLAS
#----------------------------------------------------------
module load singularity
source /project/atlas/Tier3/AtlasUserSiteSetup.sh

# Reference: https://docs.computecanada.ca/wiki/Running_jobs#Accounts_and_projects

# If you have _more_ than one allocation, configure your ~/.bashrc on the
#  login machine to have these lines (and login again)
#      export SLURM_ACCOUNT=<def-someuser>
#      export SBATCH_ACCOUNT=$SLURM_ACCOUNT
#      export SALLOC_ACCOUNT=$SLURM_ACCOUNT

# Instead of putting the time limit in your batch file,
#  submit with sbatch --time=<timelimit>
# see sbatch --help for other nice things you can add
# and also read the reference link above.

#----------------------------------------------------------

• (Optional) If you want a motd or do any post-setups, add a file /atlasadmin/localPostUserSetup.sh. For example, Compute Canada Tier3s have this:

#
# This is a local post setup script
#   You can, for example, put in a notice for users such as the example below
#

\echo -e "
\033[1m\033[4m19 Sep 2018\033[0m                         *  *  *  *  * 
  /myScratch and /myProject are no longer available. Access it from \$HOME.
"

Known Issues

Tool	Description of Issue	Resolved ?
atlantis	will not work in Singularity because of a java issue
Java timezone issue affecting voms-proxy-* but only unpacked containers in Singularity	Jira ticket	This should now be resolved through an internal ATLASLocalRootBase setting. If not, use images; eg setupATLAS -c images+slc6. Alternately, you can just define this environment variable: export TZ=<your local timezone> A better option is to avoid voms-proxy-* and use arcproxy : arcproxy -r (delete), arcproxy -I (info), arcproxy -voms atlas (create)
Ganga	Not yet working for backend batch submissions from inside a container	Developers are working on it. But you can submit batch jobs from inside a container