July 2008 Tutorial

Introduction

These are supplementary instructions to the July 2008 Tutorial. They are meant to be used on the atlas-vm virtual machine (VM) version 1.06. You do not need to login to lxplus to do the tutorial if you use atlas-vm.

Please refer to the Tutorial links and follow or substitute the instructions as listed below. If there are no supplementary instructions for a particular session, it means that the Tutorial instructions work as-is on the atlas-vm.

When following the tutorial instructions when using atlas-vm, please note that you should not source any scripts that have /afs/cern.ch in its path. Nor should your requirements file or input data have any /afs/cern.ch paths in it. (If it does, you will be running code over the network which will be very slow.)

Software

All required software is available locally on atlas-vm:

• You just need to type setupATLAS in your account and follow the instructions to setup DQ2Client, Ganga, etc, when you need it.
• When checking out packages from CVS, remember to get a kerberos ticket first, for example: kinit -5 desilva@CERN.CH.
• To submit jobs to the grid, or to get datasets with DQ2, get a grid proxy first, for example: voms-proxy-init -voms=atlas:/atlas/ca (leave out the :/atlas/ca if you are not a Canadian).

Data

The following datasets will be used for the tutorial; they are large and whenever possible, the jobs should be submitted to the grid. If you want to download a dataset for local testing, I recommend copying only one files from each dataset:

# go to the data directory in atlas-vm
cd /data/$USER

# setup the local DQ2 software
setupATLAS
localSetupDQ2Client

# This will get a 1.6 GB file from the dataset (needed for the DPD Tutorial)
dq2-get -f fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10._0001.1 fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10
Querying DQ2 central catalogues to resolve datasetname fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10

# This will get a 126 MB file from the dataset (needed for the PyAthena Tutorial)
dq2-get -f AOD.022496._00001.pool.root.1 valid1.005200.T1_McAtNlo_Jimmy.recon.AOD.e322_s435_r432_tid022496 

# This will get a 39 MB file which is needed for the EventView Tutorial
cp /afs/cern.ch/atlas/offline/external/FullChainTest/long/14.1.0.4/root_files/AllBasicSamples.AOD.pool.root ./

Releases

atlas-vm contains 14.2.10 but some of the tutorial sections seem to need 14.2.0. In some cases, this may not matter but where it fails (marked below by ), you have to use 14.2.0. You can install it on the VM (it will use up approximately 7 GB more disk space) by logging in as atlasadmin and typing the command below, or, ( easier - you can just download and run a second atlas-vm (1.04) which has 14.2.0 installed - you can run both at the same time !)

# Do this if you want to install 14.2.0 on atlas-vm instead of runnng a second virtual machine (atlas-vm version 1.04)
$ATLAS_LOCAL_ROOT_BASE/utilities/installSW.sh --name=Athena --platform=i686 --version=AtlasProduction_14_2_0_i686_slc4_gcc34_opt  --mirror=http://pacman.atlas-canada.ca/ATLAS.mirror --installPath=Kits_SL4/14.2.0

Tips

Finally, here are some tips to keep in mind when you go through the Tutorial; these should only be done after doing the source ~/cmthome/setup.sh -tag command (see the sections below).

PyUtils

I recommend using the PyUtils package to manage your software compilations. These are the instructions:

       # checkout the PyUtils package  
       cd $TestArea
       cmt co -r PyUtils-00-03-18 Tools/PyUtils

       # You need to do the next 4 steps after you checkout any new packages to $TestArea
       Tools/PyUtils/bin/setupWorkArea.py
       cd $TestArea/WorkArea/cmt
       cmt bro cmt config
       cmt bro gmake

       #  Use this run directory from now on ...       
       cd $TestArea/WorkArea/run

Athena jobs to the Grid

Checkout PandaTools and use pathena to submit athena jobs to the grid (instead of using Ganga to submit to the Panda backend). The instructions to set this up are;

       # checkout PandaTools (the latest version) and build it.
        cd $TestArea
        cmt co PandaTools PhysicsAnalysis/DistributedAnalysis/PandaTools
        cd PhysicsAnalysis/DistributedAnalysis/PandaTools/cmt
        gmake

        # example: go to my run directory and submit the job to the grid
        cd $TestArea/WorkArea/run
        # pathena --help will show you the options (you can, for example, submit to the CA cloud with --cloud=CA).
        pathena --inDS=fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10 --outDS=user.AsokaDeSilva.14.2.10.DPD --nFiles=1 AODtoDPD.py

(16 July, 10:15) Setting up your account

Reference

• In "Setting up CMT", there is already a ~/cmthome/requirements file in your account. (Note that this has 32, opt and setup specified already.) To setup, do the following:

        cd ~/cmthome
        source $ATLAS_LOCAL_ROOT/Athena/Kits_SL4/14.2.10/CMT/v1r20p20080222/mgr/setup.sh
        cmt config
        mkdir -p ~/scratch0/athena/AtlasOffline-14.2.10

• You should first create your $TestArea as described above; create the directory and then redo the source ~/cmthome/setup.sh if you see an error such as:

        AtlasLogin: WARNING - test directory [/home/desilva/scratch0/athena/AtlasOffline-14.2.10] doesn't exist - the runtime environment won't reflect it

(16 July, 10:45) Running Athena

Reference

• To setup the ATLAS software, you can use 14.2.10 instead of 14.2.0; to do this, type the following to setup the runtime environment:

        source ~/cmthome/setup.sh -tag=14.2.10

(16 July, 13:30) The Athena Simulation Chain

Reference

• For testing purposes, you can try to use the 14.2.10 release in atlas-vm; just type source ~/cmthome/setup.sh -tag=14.2.10 to setup the cmtpath.

• Warning - running these examples may be very slow - they are cpu and memory intensive.

• Reconstruction will fail; for example, see this HyperNews thread and also this one for more information. You probably need 14.2.0 for this exercise - you can use atlas-vm 1.04, or install 14.2.0 as described at the top of this page.

(16 July, 15:00) An Introduction to the Grid

Reference

• Do not source any /afs/.../grid_env.sh script; a local version of gLite is available and it is already setup when you log in.

• Canadians, type voms-proxy-init --voms atlas:/atlas/ca.

(16 July, 16:15) Running Athena on the Grid

Reference

• Do not source /afs/.../ganga/...setup-atlas.sh; instead, type setupATLAS followed by localSetupGanga to use the local version of Ganga that is installed on atlas-vm. Note that if you are not a Canadian, you need to edit ~/.gangarc and remove the /atlas/ca role from the voms as outlined on the Initial Setup - user page.

• You don't need to do a "ganga -g" - there is already a ~/.gangarc file availeble on your home directory.

• ("Athena on the Grid using Ganga" section); cmtsetup 14.2.10 from within Ganga and it will work.

(17 July, 10:00) Physics analysis with an Athena algorithm (PyAthena)

PyAthena Reference

• type source ~/cmthome/setup.sh -tag=14.2.10 to setup the cmtpath

• In jobo.py, change the line svcMgr.EventSelector.InputCollections to point to where you have the dataset fetched from DQ2 (in /data/$USER/ - see the top of this page).

(17 July, 10:00) Physics analysis with an Athena algorithm

Reference of second Twiki link

• You can try this in 14.2.10 instead of 14.2.0 (note the different package tag versions below for 14.2.10); type source ~/cmthome/setup.sh -tag=14.2.10 to setup the cmtpath. (You can get the package versions for a particular release, for example, by typing cmt show versions PhysicsAnalysis/BPhys/BPhysAnalysisObjects.)

• Instead of copying all the files from share/*, type get_files AnalyzeJpsiphi.py when in your run directory.

• Running the job will abort because of the following error in 14.2.10 ( You probably need 14.2.0 for this exercise - you can use atlas-vm 1.04, or install 14.2.0 as described at the top of this page.).

...
GeoModelSvc         ERROR *** *** Geometry configured through jobOptions does not match TagInfo tags! *** ***

• Note that you could have submitted the above job to the grid using Panda - see the next section (DPD Making).

(17 July, 11:00) DPD Making

Reference

• One/Every-time setup, do this instead:

        # if you have not already done this before ...
        cd ~/cmthome
        source $ATLAS_LOCAL_ROOT/Athena/Kits_SL4/14.2.10/CMT/v1r20p20080222/mgr/setup.sh
        cmt config
        mkdir -p ~/scratch0/athena/AtlasOffline-14.2.10

        # everytime you login
        source $HOME/cmthome/setup.sh -tag=14.2.10,groupArea
        cd $TestArea

• Copy the file $GroupArea/AODtoDPD.py to your run directory.

• in AODtoDPD.py, change the input file to what you downloaded with DQ2 (the line is svcMgr.EventSelector.InputCollections. This should only be for testing purposes; you should be running the job on the grid instead.)

• For submission of the job to the grid, this is what I did after setting up PandaTools as instructed:

        pathena --inDS=fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10 --outDS=user.AsokaDeSilva.14.2.10.DPD --nFiles=1 AODtoDPD.py

(17 July, 15:30) EventView Tutorial

Reference

• Instead of the "Setup for the Tutorial" section, do as directed in the One/Every-time setup "DPD Making" supplementary information section.

• copy the tutorial1.py file from /afs/cern.ch as instructed in the tutorial instructions. The datafile should be copied to /data/$USER as instructed at the top of this page.

• Instead of a soft link to the input datafile on /afs/cern.ch, do ln -s /data/$USER/AllBasicSamples.AOD.pool.root AOD.pool.root before running the job.

• when using AthenaRootAccess, I see this warning but it did not seem to affect anything.

[desilva@atlasvm-106 run]$ python -i evInARA.py
TClass::TClass:0: RuntimeWarning: no dictionary for class DataLink<EventView,DataProxyStorage<EventView> > is available
TClass::TClass:0: RuntimeWarning: no dictionary for class UserDataBlock is available

• if you submit the job to the grid with Panda, remember to checkout the PandaTools package and compile as outlined in the Tips section.

(17 July, 17:00) EWPA Tutorial

Reference

• Ignore the "Setup the Environment section" and do as directed in the One/Every-time setup "DPD Making" supplementary information section.

• in Skeleton_topOptions.py, comment out the line for the dataset at CERN and include a line for the dataset file you downloaded earlier

# comment this line
#include ("EWExamples/InputCollections_valid1.005145.PythiaZmumu.recon.AOD.e322_s443_r455.0.py" )

# add this line (equivalent to your dataset location)
InputCollections    += [ "/data/desilva/valid1.005200.T1_McAtNlo_Jimmy.recon.AOD.e322_s435_r432_tid022496/AOD.022496._00001.pool.root.1" ]

• When running the "Making a D1PD" job, I see the following types of errors but it still ran successfully to completion and created the D1PD.pool.root file.

DataProxy                               WARNING accessData: conversion failed for data object 239760293/HLTAutoKey__239760293_2
 Returning NULL DataObject pointer  
T_AthenaPoolCustomCnv                     FATAL Failed to convert persistent object to transient: Unsupported persistent version of Data container
AthenaPoolConverter                       ERROR createObj PoolToDataObject() failed, token = [DB=3C27568C-BD2C-DD11-95DF-00A0D1E505C7][CNT=POOLContainer_TrigTauTracksInfo][CLID=630E4944-7EDE-4938-B189-020DBC0436AE][TECH=00000202][OID=00000055-0000001E]
DataProxy         

• When running the "Accessing EWPA derived ... with AthenaRootAccess" section, do not copy the D2PD file from CERN castor; you can use the one you just created in atlas-vm (create a softlink AOD.pool.root to that D2PD file). However, you will need the ewpa.C file which you can simply copy from lxplus to atlas-vm.

(18 July, 09:30) Tag Usage Tutorial - Tag Browser

Reference

• Note the location of the tag files you create on /afs/cern.ch and copy it to your run directory (make the appropriate changes in the instructions in the next section.).

• Otherwise, if you have problems running later with the tag file you created, use the one for the tutorial:

        cd $TestArea/run
        cp /afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/TAG/fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root ./

(18 July, 09:30) Tag Usage Tutorial - Using Tag Files

• Note that if you only downloaded one file out of the dataset fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10 as instructed above, you will not be able to run on all events in the tag root file since the tags are for events in the entire dataset (the correct way is to submit this to the grid). Otherwise you may have to download all the files in the dataset (~9 G in size).

• For Exercise 1.1, do the following to create a pool catalog file (do not copy the /afs/cern.ch/.../PoolFileCatalogRun52280Egamma.xml file).

        # temporarily move any existing PoolFileCatalog.xml (mv PoolFileCatalog.xml PoolFileCatalog.xml.save)

        # create a new PoolFileCatalog.xml file with only the following contents:
        <!-- Edited By POOL -->
        <!DOCTYPE POOLFILECATALOG SYSTEM "InMemory">
        <POOLFILECATALOG>
        </POOLFILECATALOG>
         
        # add the dataset file that you downloaded as instructed at the top of this page
        pool_insertFileToCatalog  /data/$USER/fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10/fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10._0001.1
        # you can repeat the above line to add any number of dataset files.
        # Tip: to add all files in the dataset which you downloaded, just do 
        #    ls -1  /data/$USER/fdr08_run2.0052280.physics_Egamma.merge.AOD.o3_f8_m10/* | awk '{print "pool_insertFileToCatalog "$1""}' | sh

        # rename the file
        mv PoolFileCatalog.xml PoolFileCatalogRun52280Egamma.xml

        # if you had an existing catalog file, restore it
        cp PoolFileCatalog.xml.save PoolFileCatalog.xml

• In exercise 1.1, replace the instructions for svcMgr.PoolSvc.ReadCatalog with your own PoolFileCatalogRun52280Egamma.xml file:

        svcMgr.PoolSvc.ReadCatalog += [ "xmlcatalog_file:PoolFileCatalogRun52280Egamma.xml" ]

• To submit to the grid, look at this Panda submission page for instructions. (Remember, do not source the grid_env.sh file.) You should have checked out PandaTools and compiled it as described in the Tips section. The summary of the instructions to submit the job are:

        cd $TestArea/run
        # this is where your TAG root file should exist (which you created from the web browser ELSSI) 
        #  in the example below, it is fdr08_run2_physics_Egamma_o3_f8_m10_1216853201_user.root

        echo fdr08_run2_physics_Egamma_o3_f8_m10_1216853201_user.root > myinputs.txt

        pathena --shipInput --inputFileList=myinputs.txt --outDS user.AsokaDeSilva.test.tag.v1 AODtoDPD.py

(18 July, 11:00) ATLANTIS

• To install a standalone version of ATLANTIS on the altasvm, please see these installation instructions. You can then type setupATLAS followed by localSetupAtlantis if you followed the instructions in the link.

• Another way if to checkout and compile the code for ATLANTIS in your release; this is outlined in this Vancouver Tutorial page. You will need the corresponding tags for 14.2.10 which you can get with cmt show versions as shown below

        cmt show versions graphics/AtlantisJava
        graphics/AtlantisJava AtlantisJava-09-10-14 /atlas/ATLASLocalRootBase/i686/Athena/Kits_SL4-i686/14.2.10/AtlasAnalysis/14.2.10
#     so to checkout and compile this package 
#     (see appropriate instructions above if you use the PyUtils to manage packages), do 
#         cd $TestArea
#         cmt co -r AtlantisJava-09-10-14  graphics/AtlantisJava
#         cd graphics/AtlantisJava/cmt
#         gmake

• Note that there is another way to view events VP1

(18 July, 11:30) FileStager

Reference

• To setup, source setup.sh -tag=14.2.10,notPedantic.

• Instead of cd FileStager/cmt/ in the instructions, it should be cd Database/FileStager/cmt/

• Do not source /afs/cern.ch/.../DQ2Clients/setup.sh. For grid certificate, type voms-proxy-init -voms=atlas:/atlas/ca and for DQ2, type setupATLAS followed by localSetupDQ2Client.

• Note that the manager stores files in /tmp; you can (and should for atlas-vm) change this to /data/$USER/tmp/ (after creating the directory). To do this, export TMPDIR=/data/$USER/tmp/

• Warning: This is an expensive operation because it involves fetching datafiles and deleting them immediately after use. (I did not complete the examples - only looked to see the file fetching / removal mechanism.) If you do a CRTL-C on the athena job, the datafiles will not be deleted and you should do it manually.

• If you did this section, I recommend deleting your $TestArea contents (rm -rf $TestArea/*) and starting fresh - logout and in and do the source cmthome/setup.sh -tag, checking out whatever you need next, etc when you start on another section.

(18 July, 13:30) Physics Analysis on the Grid

Reference

• You can do this in 14.2.10.

• Skip the "Ganga Setup on lxplus or afs" section.

• For the AthenaRootAccess example, copy the gangaARAExample_502.py file to your run area. Skip the setup instructions preceding this and do instead source cmthome/setup.sh -tag=14.2.10 followed by cd $TestArea.

• setup Ganga by typing setupATLAS followed by localSetupGanga.

• When you submit with ganga, remember to change the --athena_release to 14.2.10 in the tutorial instructions. If you use the iPython submission, change the appropriate line to substitute for the 14.2.10 release.

• For the UserAnalysis example, these are the instructions (replacing the section); you should consider the PyUtils package if you checkout more than one package.

        source cmthome/setup.sh -tag=14.2.10
        cd $TestArea
        cmt co -r UserAnalysis-00-13-03 PhysicsAnalysis/AnalysisCommon/UserAnalysis
        cd PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
        source setup.sh
        cmt broadcast gmake
        cd ../run
        get_files AnalysisSkeleton_topOptions.py
        setupATLAS
        localSetupGanga
        ganga athena --inDS fdr08_run2.0052283.physics_Muon.merge.AOD.o3_f8_m10 \
                          --outputdata histos.root --lcg --site LRZ \
                          AnalysisSkeleton_topOptions.py

• The Panda backend has not been configured in the ~/.gangarc file. If you want to use it, then make the following change and restart Ganga in a new terminal window:

        # edit ~/.gangarc and do replace RUNTIME_PATH as shown here
        #RUNTIME_PATH = GangaAtlas
        RUNTIME_PATH = GangaAtlas:GangaPanda

• Instead of using Ganga to submit to the Panda backend, I find it much easier to submit it directly with pathena; for help, type pathena --help. You should have checked out and compiled PandaTools as instructed in the Tips section. (You do not need to setup Ganga in this case.) Note that you can also submit ARA jobs with pathena. For example, to submit the UserAnalysis job, I typed

        pathena AnalysisSkeleton_topOptions.py --inDS fdr08_run2.0052283.physics_Muon.merge.AOD.o3_f8_m10 --outDS user.AsokaDeSilva.`uuidgen` --split 6

-- AsokaDeSilva - 21 Jul 2008