Testing ALRB and manageTier3SW before deployment

WARNING: This page is obsolete.

Please look at https://atlas-computing.docs.cern.ch/operations/cvmfs-software/alrb.html.

This information is not for users or site admins; it is for ATLAS software experts for administration.

This section tells you how to make changes in the ATLASLocalRootBase (ALRB) and the manageTier3SW packages, how to test before deploying the changes.

Please be very careful as all of ATLAS (Tier1/2 grid sites, Tier3 sites, cernVM and lxplus) automatically update based on the latest tagged versions and depend on reliability of this software.

Note About Supported Platforms

Platform	Comments	Login CERN	Login TRIUMF
slc5, slc6 x86_64	Very deprecated. Grid submission must now be done from panda-client instead of inside container. No tests required (asetup is only meaningful tool in older containers or python 2 environment; for slc6 and older code is carved out so it remains unchanged).
centos7 x86_64	Full testing (as python3 works on this) but only on atlascau7; mainly for panda and asetup which are OS-independent. Other tools are frozen.		atlascau7
centos7 aarch64	Not supported. DO NOT TEST THIS COMBINATION.
centos8s x86_64	Run tests anyway: only really needed for pilot required tools; eg emi, xrootd, davix, etc and it knows to skip the others. Err on side of caution and test all.	lxplus8.cern.ch; testALRB
centos8 aarch64	Run tests anyway: only really needed for pilot required tools; eg emi, xrootd, davix, etc and it knows to skip the others. Err on side of caution and test all.	lxplus8-arm.cern.ch; testALRB
almaLinux9 x86_64	Full testing	lxplus; testALRB	atlascau9
almaLinux9 aarch64	Full testing	lxplus9-arm.cern.ch; testALRB
almaLinux10 x86_64	Full testing	lxplus10; testALRB	atlascau10
almaLinux10 aarch64	Full testing (but for now is placeholder for aarch64, waiting for real el10 release - except there is native Root for aarch64); Panda pathena job may fail on grid as backward compatibility removed for el10.	lxplus9-arm.cern.ch; testALRB; setupATLAS -c el10 --localalrb -m /eos

Special note about Panda (2026/6/9): We do not require el10 to be able to run el9 releases anymore so that compatibility is broken. For panda testing, we need an ATLAS release but we have only one nightly for x86_64. For aarch64, we need to run in a container. For el8, the support is mainly for pilot tools like xrootd, emi, etc, so it will skip panda testing on el8. The main goal is to test the grid submission mechanism and so that will work although the grid job for aarch64 will fail. This will change once we have a real aarch64 release for use.

Note About Software in GitLab

The software packages are in the gitlab repository at CERN and can be seen on this link. They are:

• ATLASLocalRootBase: the software UI which the user interacts to do various tasks;
• manageTier3SW: the software the Tier3 manager will use to install software, including ATLASLocalRootBase.
• Tier3SWConfig: the configurations for what to install at sites;

You need to have write access to the git repository to checkout/checkin changes.

Step 1: Testbeds

TRIUMF Testbeds

(Access to TRIUMF machines are under TRIUMF Tier-1 policy which does not allow non-TRIUMF personnel access): On atlascau7 or atlascau9 (centos7 and almaLinux9 machines respectively):

• login as atlasadmin to run updateManageTier3SW with the appropriate test tags.
• login with your own private account to test by first setting

  export ATLAS_LOCAL_ROOT_BASE=/opt/atlas/atlasadmin/ATLASLocalRootBase
  alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'

• the atlasadmin account is to be used only for running updateManageTier3SW. Nothing else should be done there.

CERN Testbed

You also need to test on lxplus in addition to the above

Note updating on aibuild003 (now aibuild-interactive) with tier3update to update x86_64 and aarch64

There is a VM that is accessible for installations by authorized users.

• login to lxplus as yourself and then do
• ssh aibuild003 (now aibuild-interactive)
• sudo -i -u atlast3
• You can run tier3update.sh (eg tier3update.sh -c cand-V01-05-63-v3 instead of updateManageTier3SW.sh -c cand-V01-05-63-v3 which will update the software on /eos/atlas/atlascerngroupdisk/proj-alrb/ATLASLocalRootBase for both x86_64 and aarch64.
• The atlast3 account is meant only for installation purposes. Nothing else should be done as atlast3.

To test your changes, first put this inside your .bashrc and .zshrc files of your own lxplus accounts.

For testing:

export RUCIO_ACCOUNT=<your lxplus name>
alias testALRB='source /eos/atlas/atlascerngroupdisk/proj-alrb/scripts/alrbTest.sh'

• login to lxplus as yourself
• type testALRB so that you switch to using the ALRB installation done by atlast3 as described above. You can now do setupATLAS etc.

Note that lxplus runs Alma Linux 9; login to these machines for other platforms:

• for slc6, login to lxplus and run inside a container

ssh -A lxplus.cern.ch
testALRB
setupATLAS -c slc6 --localalrb

• for centos8s, you can login to lxplus and run inside a container or you can log in to lxplus8, lxplus8-arm

ssh -A lxplus.cern.ch
testALRB
setupATLAS -c centos8s --localalrb

• ssh lxplus9.cern.ch # for alma9 x86_64
• ssh lxplus9-arm.cern.ch # for alma9 aarch64; you can run a centos7 container for centos7 aarch64 testing if needed from there.

ssh -A lxplus9-arm.cern.ch # is a alma9 aarch64 machine
testALRB

# if needed to test alma9 aarch64
setupATLAS

# if needed to test centos7 aarch64
setupATLAS -c centos7 --localalrb

Other Testbeds

If you prefer to use your own machines as testbeds, this is how to do it - for example on SL7:

Creating Your Own Instead of Using the VMs

• It is best to create a separate atlasadmin account for Tier3 software installations so that permission issues of any software can be flushed out when accessed as a regular user (ie from your own account).

• Create a file to override the ALRB definitions; eg. testALRB.sh containing:
  export ATLAS_LOCAL_ROOT_BASE=$HOME/testALRB/ATLASLocalRootBase
  alias setupATLAS='source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh'

• Checkout manageTier3SW and install the software

git clone https://gitlab.cern.ch/atlas-tier3sw/manageTier3SW.git ~/userSupport/manageTier3SW
cd ~/userSupport/manageTier3SW
./updateManageTier3SW.sh -a $HOME/testALRB

• Updating: you will be able to simply type source updateManageTier3SW.sh to update to the production version.

• From this point onwards, you will be able to simply type source ~/testALRB.sh which will then use the testbed version to validate the software (and use the existing Athena kits as well in your tests.)

Step 2: Making Changes

Initial Checkout of Repos

You will need to have git checkin privileges to do this for the ATLAS Tier3 SW repo. Make sure that your git account at gitlab.cern.ch has your ssh keys registered.

On your testbed (with own account), create a directory for code development.

mkdir -p $HOME/Tier3Dev

Next, checkout the code that you need to change (most of you will need only the Tier3SWConfig repo so you can clone only that);

cd $HOME/Tier3Dev
git clone ssh://git@gitlab.cern.ch:7999/atlas-tier3sw/manageTier3SW.git
git clone ssh://git@gitlab.cern.ch:7999/atlas-tier3sw/ATLASLocalRootBase.git
git clone ssh://git@gitlab.cern.ch:7999/atlas-tier3sw/Tier3SWConfig.git

Changing Code in a Repo

In this example, I show how to change code (note you will be working with the master branch):

cd $HOME/Tier3Dev/<repo>
git pull
# (make changes)
# make an entry in RELEASE.NOTES
git status # check what is changed
git commit -am "some comment"
git status # check what is not committed
git push
git tag <tag> 
git push origin <tag>

In the above, example, <tag> should be a "candidate" tag unless the repo is ATLASLocalRootBase in which we provide a production tag. Candidate tags are of the form cand-<tag>-v<integer> where the <tag> is the release which will go into production and <integer> is a incrementing number.

You can see the tags already used by git tag when in the repo dir.

Case 1: Updating Versions of Tools (eg Panda, CMake, etc)

Most of the time, changes will involve bumping up versions of 3rd party software such as Panda, etc. This involves making changes in the appropriate file(s) in the package directory Tier3SWConfig /tools.

Note that the tools are usually from a repository somewhere - either from the developers or saved by us at CERN (http://atlas-tier3-sw.web.cern.ch/atlas-Tier3-SW/repo/). The repository at CERN is accessible to authorized people and accessible through /eos/project/a/atlasweb/www/atlas-Tier3-SW. Note that there is a file /eos/project/a/atlasweb/www/atlas-Tier3-SW/DevelopersLocations.txt detailing where to copy new versions from in case the developers omit this information.

You can test and deploy new versions of 3rd party software without making it the default or testing. This way, the new versions can be published on cvmfs and be tested further by others as well without affecting production. setupATLAS; showVersions tells you how to setup the non-default versions.

Panda-client is typically well tested and new versions are expected to be deployed as Production by default. Most other things get deployed first as Testing and then as Production.

Case 2: Updating ATLASLocalRootBase Code

If you do bug fixes or other changes to ATLASLocalRootBase, you can first commit those changes, give the package a production tag and then make the version change in the file Tier3SWConfig /tools/alrb.sh.

Case 3: Updating manageTier3SW Code

(Note, you will rarely have to modify this.)

If you do bug fixes or changes to manageTier3SW, you can commit those changes (do not change the latestVersion file until you finish testing); give it a private tag and use that private tag to test as directed below.

Step 3: Testing

This should be done on all testing platforms including lxplus.

This is done preferably as 2 different accounts, one for installation and one for testing:

Update the Software on Testbed

Login as

• atlasadmin on the TRIUMF VMs
• atlast3@aibuild for the CERN VM

Now, depending on where the changes were made and what you want to test, updateManageTier3SW [--help] has several options that can be used.

Assuming you have the most common case, where you updated a tool in Tier3SWConfig, you will do something like:

updateManageTier3SW.sh -c <candidate tag>

(on aibuild003; it will be tier3update.sh -c <candidate tag>)

which will pull in the changes of Tier3SWConfig tagged as <candidate tag> and test it with the other software unchanged.

User Testing

Now in a fresh terminal session,

• login to your own account on the VMs at TRIUMF to get the testing ATLASLocalRootBase environment
• login to lxplus and type testALRB to get the testing ATLASLocalRootBase environment

Then test the newly installed software or tools. This is an example on how to use toolTest. For example, to test PandaClient:

setupATLAS  # in your account but accessing the ATLASLocalRootbase installation that you own for testing
diagnostics
toolTest -m test panda

For example, toolTest -h panda will show you what environment variables can be exported prior to running the tool to change the versions tested. v is a verbose option that will show you details of the tests as they are done.

toolTest -m option	Shells	Tool Version Tested	Comments
user	same as user's shell	Production	For users
test	all shells	Testing	For checking testing versions
validate	all shells	Production	For validating production versions or Containers

toolTest an be used to test everything at once (eg for any new containers that are generated). eg toolTest -m validate all.

In case you need to test a version that is not tagged as Testing, export ALRB_pandaVersion=<version in ALRB>

Step 4: Publishing

When you are satisfied with the tests, go back to your development area directory, update latestVersion file and give it a production tag that matches the latestVersion file contents.

cd $HOME/Tier3Dev/<repo>
git diff @{upstream}  # make sure that there are no changes that were not tested since we are dealing with master branches
# edit and put an entry in RELEASE.NOTES if not done previously
# edit latestVersion and ensure that that will have a production tag as described in the RELEASE.NOTES entry
git status # check what has changed
git commit -am "latest version"
git status # check that you have committed everything
git push 
git tag <tag>
git push origin <tag>

Step 5: Final Check and Deployment

As a last sanity check, login to your testbed in a fresh terminal session and run updateManageTier3SW.sh It should pull in the latest tag of manageTier3SW and try to update packages.

The cvmfs server automatically runs manageTier3SW every 6 hours so software gets deployed on its own. However, if you want it faster, please login to the cvmfs server and update/publish the changes.

(from lxplus)
ssh cvmfs-atlas
sudo -i -u cvatlas
cat /var/tmp/.install.lock. date; ls -la /var/tmp/.install.lock  # check for lock files and wait until there are none. 
tier3update
# optional, if you need to let others know:  copy the last 2 lines of tier3update output that give you the timestamp of when the changes will be available.
exit

Step 6: Notify

Mail to "atlas-adc-tier3sw-install (ATLAS Tier3 software installation requests and discussions)" <atlas-adc-tier3sw-install@cern.ch> a note saying what the changes are an paste the last 2 lines of the tier3update output showing when the changes would be on cvmfs.

Appendix

Emergency rollbacks on cvmfs

These are the instructions to roll back to a older version on cvmfs if you do not have access to the Tier3 software git repository or want to do it fast.

# login to cvmfs server
ssh lxplus.cern.ch
ssh cvmfs-atlas
sudo -i -u cvatlas

# check if there is a lock; do not proceed until it is not locked 
ls -la /var/tmp/.install.lock; cat /var/tmp/.install.lock; date
# if there is no lock, then continue


# disable crontab
crontab -e 
# and comment out this line
   00 5,11,17,23 * * * /home/cvatlas/bin/tier3update > /dev/null 2>&1


# get a lock and a cvmfs transation
$HOME/bin/lock-install "manualTier3Work"
cvmfs_server transaction atlas.cern.ch


# make sym link changes to "current" in ALRB 
# cd /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
# and change your sym links in the dirs you want and run this next command
$ATLAS_LOCAL_ROOT_BASE/swConfig/createMaps.sh


# publish and unlock
cvmfs_server publish atlas.cern.ch
$HOME/bin/unlock-install


# and email atlas-adc-tier3sw-install@cern.ch detailing the changes so that we can do the follow up actions. 
# Note cron job in crontab remains commented out, we will unblock it.

Special Deployment Instructions

For tools which are used by the pilots on the grid, there are additional testing protocols to follow. These tools are listed in this file ADCTestVersions.sh in the Tier3SWConfig package. (You should not need to modify this file.)

In addition to the interactive tests that are done in the sections above, prior to making a version the default, they undergo these additional testing:

• for rucio-clients, also send a mail to atlas-computing-testers@cern.ch and ask those subscribed users to test that version.
• ALRB Hammercloud tests
  • this is done automatically for testing versions. The test results should be checked. Click the "compare" link for each test and search for "ISSUE HERE" for sites that have an issue on the right column but not on the left column of tests.
• if the ALRB Hammercloud tests pass, the version may first need to be set for preproduction sites to test. This is done by uncommenting the tool in ADCFrontlineSiteVersions.sh in the Tier3SWConfig package.
  • After deploying this, send a mail to atlas-adc-alrb-preproduction@cern.ch to notify the site about this testing version being activated.
  • The deployment to preproduction should be done on a working day and hours of the site (ie not Friday or holidays / long weekend eves)
  • After 2 or more days, if no-one reports an issue, restore the ADCFrontlineSiteVersions.sh file (ie comment the tool) so that it is no longer used in preproduction. Also inform the sites (atlas-adc-alrb-preproduction@cern.ch)
• rucio-client developers prefer we deploy versions into production at their call. The preference is to deploy it at the end of our day (ie at CERN morning hours) so that they can monitor for issues.
• Be Careful what you deploy as Testing:
  • testing versions that have to work for pilots are listed here.
  • testing versions that have to work for HammerCloud are listed here.