AtlasCanada Web>GridActivities>SiteTests (2026-10-05, AsokaDeSilva)

SiteTests

WARNING: This page is obsolete.

(Last updated: 2026-10-05 13:43:35 UTC)

This package allows anyone to submit Analysis Functional Test (AFT) Hammercloud jobs to the grid using Panda or Ganga. The purpose of this is to provide a tool for testing Panda and Ganga clients prior to deployment in ATLASLocalRootBase. They can also be used by Tier2 admins but this use case is no longer a primary one since there are now regular Hammercloud testing done on queues.

SiteTests

Requirements

ATLASLocalRootBase need to be available as well as cvmfs for releases; It can be run on any ATLAS ready machine. You will need valid grid certificates and the environment RUCIO_ACCOUNT must be set.

This software will run on bash and zsh; tcsh is unsupported.

Template Status

ID	Interactive	Panda AFT	Panda Test	Ganga AFT	Ganga Test
722	OK	OK	OK	OK	OK
775	OK	OK	OK	OK	OK
808	OK	OK	OK	OK	OK

Initial Setup

New Installations

ssh into your machine and in your $HOME dir, checkout the package:
- svn co http://svn.cern.ch/guest/atcansupport/siteTests/trunk siteTests
Next update the software:
- cd ~/siteTests; ./updateSiteTests.sh
Remember to update regularly

Updating

You are recommended to update the software on a regular basis (e.g. a cron job). Simply rerun ~/siteTests/updateSiteTests.sh

Updating (Special instructions for updating from versions prior to Aug 25, 2016)

run cd ~/siteTests; ./updateSiteTests.sh twice.
you can remove the ~/public/tier2Tests dir as well as any residual ~/setup-.sh scripts.

Test Work Area Tools

List Templates

The command source ~/siteTests/do.sh is the tool that will allow you to

source ~/siteTests/do.sh list will show the available templates
- note the letters D (default) and, if applicable, C (created) next to the template id

$ source ~/siteTests/do.sh list
 Template   Description                                                 
 722        AFT RootAnalysis 2.3.14 QuickAna-00-00-60 Analy             
 729        (deprecated, do not use) AFT AtlasDerivation 20.1.5.7       
 775 D      AFT AthAnalysis 2.3.48 QuickAna                             
 808        AFT AtlasDerivation 20.7.6.4

Create Work Area

create a work area for the template:
- source ~/siteTests/do.sh create will create the default template
- source ~/siteTests/do.sh create <id> will create template id
Note that this also fetches a sample dataset (if not already available) and will ask you for grid credentials.
- On occasion, there may be DDM warnings and it may be slow; wait to the end to see if it was successful.
  - If needed, you can restart from a failed step; to do this: ~/siteTests/create.sh --template=<id> --startStep=<N>

$ source ~/siteTests/do.sh create
 
--> Step 1 - Check host
 
--> Step 2 - Check for disk space

--> Step 3 - Check Grid certificates
Looking for ~/.globus/usercert.pem ...                    [  OK  ]
Looking for ~/.globus/userkey.pem ...                     [  OK  ]

--> Step 4 - Create Ganga Configuration

--> Step 5 - Setup release
Using AthAnalysisBase/2.3.48 [cmt] with platform x86_64-slc6-gcc49-opt
   at /cvmfs/atlas.cern.ch/repo/sw/software/AthAnalysisBase/x86_64-slc6-gcc49-opt/2.3.48
Test area: /afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48
asetup ... seems ok ...                                   [  OK  ]

--> Step 6 - Build
Py:WorkAreaMgr       INFO ################################################################################
Py:WorkAreaMgr       INFO Creating a WorkArea CMT package under: [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48] 
Py:WorkAreaMgr       INFO Scanning [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48]
Py:WorkAreaMgr       INFO Found 0 packages in WorkArea
Py:WorkAreaMgr       INFO => 0 package(s) in suppression list
Py:WorkAreaMgr       INFO Generation of WorkArea/cmt/requirements done [OK]
Py:WorkAreaMgr       INFO ################################################################################
setup PyUtils workarea                                    [  OK  ]
Doing cmt config ...
#--------------------------------------------------------------
# WorkArea WorkArea-00-00-00 : Now trying [cmt config] in /afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/WorkArea/cmt (114/114)
#--------------------------------------------------------------
Creating setup scripts.
Creating cleanup scripts.
Installing the run directory
Installing the python directory
cmt config                                                [  OK  ]
Compiling and linking ...
#--------------------------------------------------------------
# WorkArea WorkArea-00-00-00 : Now trying [make] in /afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/WorkArea/cmt (114/114)
#--------------------------------------------------------------
#CMT---> (Makefile.header) Rebuilding ../x86_64-slc6-gcc49-opt/x86_64-slc6-gcc49-opt.make
#CMT---> (constituents.make) Rebuilding library links
#CMT---> (constituents.make) Building post_merge_rootmap.make
#CMT---> (constituents.make) Building post_merge_genconfdb.make
#CMT---> (constituents.make) Building checkreq.make
#CMT---> (constituents.make) Building post_build_tpcnvdb.make
#CMT---> (constituents.make) Building check_install_python_modules.make
#CMT---> (constituents.make) Building install_root_include_path.make
#CMT---> (constituents.make) Building install_includes.make
#CMT---> Info: Document post_merge_rootmap
#CMT---> Info: Document install_root_include_path
#CMT---> Info: Document post_merge_genconfdb
#CMT---> Info: Document check_install_python_modules
#CMT---> Info: Document install_includes
#CMT---> Info: Document checkreq
#CMT---> Info: Document post_build_tpcnvdb
#CMT---> (constituents.make) Starting post_merge_rootmap
#CMT---> (constituents.make) Starting install_root_include_path
#CMT---> (constituents.make) Starting post_merge_genconfdb
#CMT---> (constituents.make) Starting check_install_python_modules
#CMT---> (constituents.make) Starting install_includes
#CMT---> (constituents.make) Starting checkreq
No standard include file area
#CMT---> (constituents.make) Starting post_build_tpcnvdb
#CMT---> (constituents.make) install_root_include_path done
No standard include file area
#CMT---> (constituents.make) install_includes done
#CMT---> (constituents.make) check_install_python_modules done
#CMT---> (constituents.make) Building install_python_modules.make
#CMT---> Info: Document install_python_modules
#CMT---> (constituents.make) Starting install_python_modules
make[2]: `install_python_modules' is up to date.
#CMT---> (constituents.make) install_python_modules done
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::: abuild-merge-rootmap
::: collecting dsomap files...
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::: abuild-merge-genconfdb
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::: abuild-merge-genconfdb
::: collecting genconfdb files...
::: collecting genconfdb files...
::: collecting genconfdb files... [done] (nbr=0)
::: collecting dsomap files... [done] (nbr=0)
::: merging files into [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/InstallArea/x86_64-slc6-gcc49-opt/lib/python2.7/SiteTestsWork_merged_confDb.py]...
::: merging files into [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/InstallArea/x86_64-slc6-gcc49-opt/lib/SiteTestsWork.rootmap]...
::: collecting genconfdb files... [done] (nbr=0)
::: merging files into [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/InstallArea/x86_64-slc6-gcc49-opt/lib/python2.7/SiteTestsWork_merged_confDb.py]...
::: merging files into [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/InstallArea/x86_64-slc6-gcc49-opt/lib/python2.7/SiteTestsWork_merged_confDb.py]... [done]
::: merging files into [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/InstallArea/x86_64-slc6-gcc49-opt/lib/SiteTestsWork.rootmap]... [done]
::: bye.
::: bye.
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#CMT---> (constituents.make) post_merge_genconfdb done
#CMT---> (constituents.make) post_merge_rootmap done
::: merging files into [/afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48/InstallArea/x86_64-slc6-gcc49-opt/lib/python2.7/SiteTestsWork_merged_confDb.py]... [done]
::: bye.
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#CMT---> (constituents.make) post_build_tpcnvdb done

CHECKREQ---> checkreq SUCCESS
#CMT---> (constituents.make) checkreq done
#CMT---> all ok.
make                                                      [  OK  ]
Build cmt                                                 [  OK  ]

--> Step 7 - Get Input File

Getting a valid proxy ...
Enter GRID pass phrase for this identity:
Contacting lcg-voms2.cern.ch:15001 [/DC=ch/DC=cern/OU=computers/CN=lcg-voms2.cern.ch] "atlas"...
Remote VOMS server contacted succesfully.


Created proxy in /tmp/x509up_u28311.

Your proxy is valid until Tue Aug 30 06:32:30 CEST 2016
2016-08-26 06:32:32,729 INFO [Starting download for mc15_13TeV:AOD.05536542._000001.pool.root.1 with 1 files]
2016-08-26 06:32:32,885 INFO [Starting the download of mc15_13TeV:AOD.05536542._000001.pool.root.1]
100% |#########################################################################|
File downloaded. Will be validated
File validated
2016-08-26 06:33:38,239 INFO [File mc15_13TeV:AOD.05536542._000001.pool.root.1 successfully downloaded from TAIWAN-LCG2_DATADISK]
2016-08-26 06:33:38,270 INFO [File mc15_13TeV:AOD.05536542._000001.pool.root.1 successfully downloaded. 371.3 MB bytes downloaded in 65.3842301369 seconds]
2016-08-26 06:33:38,953 INFO [Download operation for mc15_13TeV:AOD.05536542._000001.pool.root.1 done]
----------------------------------
Download summary
----------------------------------------
DID mc15_13TeV:AOD.05536542._000001.pool.root.1
Total files :                                 1
Downloaded files :                            1
Files already found locally :                 0
Files that cannot be downloaded :             0

--> Step 8 - Get JobOption File
Warning: get_files without '-jo|-data|-scripts|-xmls' is SLOW !
Willing to acquire file AthExample.py from
/cvmfs/atlas.cern.ch/repo/sw/software/AthAnalysisBase/x86_64-slc6-gcc49-opt/2.3.48/AthAnalysisBase/2.3.48/InstallArea/jobOptions/QuickAna/AthExample.py
copy AthExample.py
fetching job options file                                 [  OK  ]
Get Job Options File                                      [  OK  ]

--> Step 9 - Generate Script
 
  Step Test Description                                       Result
     1 Check host                                                 OK
     2 Check disk space                                           OK
     3 Check host                                                 OK
     4 Create ganga configuration                                 OK
     5 Setup release                                              OK
     6 Build                                                      OK
     7 Get input file                                             OK
     8 Get job options                                            OK
     9 Create setup scipt                                         OK
 


Completed create.sh; you seem to be ready to use the release.
To use this release whenever you login, simply type
  source /afs/cern.ch/user/d/desilva/siteTests/do.sh setup 775

Setup Template

source ~/siteTests/do.sh setup or source ~/siteTests/do.sh setup <id> to setup a template that was created.

source ~/siteTests/do.sh setup 
Using AthAnalysisBase/2.3.48 [cmt] with platform x86_64-slc6-gcc49-opt
   at /cvmfs/atlas.cern.ch/repo/sw/software/AthAnalysisBase/x86_64-slc6-gcc49-opt/2.3.48
Test area: /afs/cern.ch/user/d/desilva/public/SiteTestsWork/AthAnalysisBase-2.3.48
Type:
  gangaJobs        to submit jobs to the grid via Ganga
  pandaJobs        to submit jobs to the grid via Panda
  interactive      to run an interactive test

and you will be able to run interactive, submit to the grid using Panda or Ganga by typing one of the 3 commands above,

Running

Run one at a time; that is, do not open 3 terminals and setup and run interactive, panda and ganga at the same time or you may encounter issues as files can be overwritten in the same dir.

Note that for grid jobs, the output dataset follows the following naming convention: user.name.client.job.template.site.localhost.uuid where

name = RUCIO_ACCOUNT env variable which must be set
client = p : panda, g : ganga client
job = a: AFT, t: test
template = HC template ID
site = panda site associated with queue
localhost = hostname of the machine where job was submitted from
uuid = a random uuid string to make the dataset unique

Interactive

Simply type interactive to run the test job in your terminal session.

Ganga

Type gangaJobs which will start up ipython.

]$ gangaJobs

  gangadir is /afs/cern.ch/user/d/desilva/public/SiteTestsWork/gangadir

  Ganga will start now.  You may be prompted for the grid proxy password.

  (For SW testing - not for site admins to test !  
  Site admins, use Panda directly)
      execfile('submitPanda.py')
  will use the Panda backend from within Ganga.

  Note: if you want to cleanup your old jobs from the ganga listings, type
      execfile('purgeJobs.py')
  You should do this periodically to free up your disk space.
....

execfile('submitPanda.py') will ask you these questions:

Ganga In [1]: execfile('submitPanda.py')
Is this an AFT HC template job [Y] : 
Panda analy queue to run : ANALY_TRIUMF

Depending on the response above, it will either submit the AFT job or a test script (testJob.sh) to the specified queue.

Panda

pandaJobs -q <queuename> will submit the AFT job to the specified queue
- eg. pandaJobs -q ANALY_TRIUMF to submit a AFT job to TRIUMF's analysis queue
pandaJobs -q <queuename> -s will submit testJob,sh to the specified queue
- eg. pandaJobs -q ANALY_TRIUMF -s to submit a testJob.sh job to TRIUMF's analysis queue

Advanced Questions

How modify the jobs

If you modify, remember to remove it when you no longer need the my_* files; otherwise these will be selected for submissions.

Simply setup the template. Then, depending on what you want to do, copy the appropriate file as shown below and modify the copy. It will be automatically picked up when you submit the job to the grid.

cp pandaJob.sh my_pandaJob.sh and edit my_pandaJob.sh to change the AFT panda job
cp pandaTestJob.sh my_pandaTestJob.sh and edit my_pandaTestJob.sh to change the panda test job
cp gangaJob.py my_gangaJob.py and edit my_gangaJob.py to change the AFT ganga job
cp gangaTestJob.py my_gangaTestJob.py and edit my_gangaTestJob.py to change the ganga test job.

Topic revision: r27 - 2026-10-05 - AsokaDeSilva

AtlasCanada

Navigate ATLAS-Canada Wiki:

* Home* * Activities * Analysis
Computing
Datasets
Getting Started
Grid Activities
* Meetings* * People in ATLAS Canada * User Services
Travel To CERN