Setup Information Needed for Tutorial at the University of Toronto

Here are some brief instructions for use of the Toronto Fish Cluster in preparation for the upcoming tutorial.

Dietrich Liko's Wiki page https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorialInToronto

The University of Toronto Physics Desktop Fish Cluster consists of about 30 machines of various vintages running RHL7.3 and SL3 mostly and a few at SL4, linked by a common home area and NIS (yellowpage) passwords. A few of these desktops have been designated for use during the workshop. You are encouraged to log into your account, change your password, install your grid certificate in the .globus directory and test running a few simple grid protocols to ensure there are no issues with your certificate and access. However please be cognisant of the fact that these machines are currently in use by students and others doing active work, so don't run any unnecessary large loads.

If you have not registered for the workshop, please send the following information to Leslie Groer groer_at_physics_dot_utoronto_dot_ca.

---------------------------------------------------------------
First Name:
Last Name:
Email:
Preferred Username:
Cern LXPLUS Username:
Grid Certificate Distinguished Name (DN):
University/Institution:
---------------------------------------------------------------

1. Log into the cluster

There is a gateway machine that is used to access the cluster from outside of the .physics.utoronto.ca domain, with ssh running on a non-standard port (2222 rather than 22).

% ssh -p2222 username@guppy01.physics.utoronto.ca

Once you are logged into this machine, you should ssh to one of the "standard" machines to be used for the workshop. To spread the load, log into the machine as follows:

Your Last Name begins with A-F: antler
Your Last Name begins with G-L: altair
Your Last Name begins with M-S: cherub
Your Last Name begins with T-Z: conan

% ssh -p2222 nodename

2. Change your password

% yppasswd

3. Copy your user certificate and user key files to .globus.

You can use the web registration form at Alberta to apply for a grid certificate. Note that the certificate file should be named usercert.pem and the key should be userkey.pem. These are not the names of the files you will get from the web page registration. You may also want your .p12 file if you want to access some of the CERN and other web pages that require a certificate. The .p12 certificate will need to be uploaded to your browser (see instructions from the LCG).

% mkdir .globus
% scp username@yournode.youruniversity.ca:.globus/usercert.pem .globus/usercert.pem
% scp username@yournode.youruniversity.ca:.globus/userkey.pem .globus/userkey.pem

Your userkey.pem should only be readable by you.

% chmod 400 userkey.pem

Note: If you wish to scp into the Fish cluster, you need to go through the gateway node guppy01.physics.utoronto.ca and you need to specify the port number with a capital P e.g.

% scp -P2222 yourlocalfile username@guppy01.physics.utoronto.ca:.globus/

4. Run the setup to define the environmentals for the User Interface

15-April-2007 Note: There are two new setup scripts that do all the necessary setups for the tutorial (Grid, DQ2, Ganga) except for the Athena setup which needs to be done separately. For csh/tcsh % source /home/atlassoft/setup_tutorial.csh For ksh,bash % . /home/atlassoft/setup_tutorial.sh

5. Get a grid-proxy via the VOMS server (with the Canadian atlas/ca role)

% voms-proxy-init -voms atlas:/atlas/ca

Cannot find file or dir: /home/groer/.glite/vomses
Your identity: /C=CA/O=Grid/OU=physics.utoronto.ca/CN=Leslie Groer
Enter GRID pass phrase:
Creating temporary proxy ................................................... Done
Contacting  lcg-voms.cern.ch:15001 [/C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch] "atlas" Done
Creating proxy ................................................. Done
Your proxy is valid until Thu Mar 22 02:45:33 2007

% voms-proxy-info -all

subject   : /C=CA/O=Grid/OU=physics.utoronto.ca/CN=Leslie Groer/CN=proxy
issuer    : /C=CA/O=Grid/OU=physics.utoronto.ca/CN=Leslie Groer
identity  : /C=CA/O=Grid/OU=physics.utoronto.ca/CN=Leslie Groer
type      : proxy
strength  : 512 bits
path      : /tmp/x509up_u30104
timeleft  : 11:59:05
=== VO atlas extension information ===
VO        : atlas
subject   : /C=CA/O=Grid/OU=physics.utoronto.ca/CN=Leslie Groer
issuer    : /C=CH/O=CERN/OU=GRID/CN=host/lcg-voms.cern.ch
attribute : /atlas/ca/Role=NULL/Capability=NULL
attribute : /atlas/Role=NULL/Capability=NULL
attribute : /atlas/lcg1/Role=NULL/Capability=NULL
timeleft  : 11:59:05

If you got this far, pat yourself on the back - you are about ready to start running Atlas code.

6. Set up Ganga

To run ganga for the first time, you need grid_env set up. Note: If you do not have a valid grid-proxy, Ganga will prompt you to enter one, but this won't get the VOMS extensions for atlas or atlas/ca so you should get out of ganga and run the voms-proxy-init as described above.

Before you start, copy the .gangarc file to your area (note this tells ganga to create a local work area for you on /data0/workshop/).

% cp /home/atlassoft/.gangarc ~/

Run ganga and answer any questions that you are prompted for:

% /home/atlassoft/Ganga/install/slc3_gcc323/4.2.12/bin/ganga

You can use Ctrl-D to log-out. Here is the first Hello World job you can try:

In [1]:j = Job(application=Executable(exe='/bin/echo',args=['Hello World']))

In [2]:j.backend=LCG()

In [3]:j.submit

It takes a few minutes to get some output.

------>j.submit()
Ganga.GPIDev.Lib.Job               : INFO     submitting job 1
Ganga.GPIDev.Adapters              : INFO     submitting job 1 to LCG backend
Ganga.GPIDev.Lib.Job               : INFO     job 1 status changed to "submitted"

You can use

In [4]:j

to get details on the job, or

In [5]:jobs

to see all jobs

7. More Advanced Ganga Tests

In the above test, your job goes to a Resource Broker (RB) at CERN that matches your job to any resource in the LCG. You can specify the TRIUMF or Toronto Compute Element to send your job to before you submit by specifying the appropriate CE:

After In [2]:j.backend=LCG()

enter

In [3]:j.backend.CE='lcgce01.triumf.ca:2119/jobmanager-lcgpbs-atlas'

or

In [3]:j.backend.CE='bigmac-lcg-ce.physics.utoronto.ca:2119/jobmanager-lcgcondor-atlas'

8. Test Running with DQ2 tools

% /home/atlassoft/DQ2/dq2 listDatasetsInSite triumfdisk

trig1_misal1_mc12.006337.PythiaA110mumu_30.recon.AOD.v12000601
trig1_misal1_mc12.005929.McAtNlo0310_JIMMY_WpWm_taunumunu.recon.NTUP.v12000601
trig1_misal1_csc11.005987.WZ_Herwig.recon.NTUP.v12000601_tid006069
etc...

Run this dq2_get command from a directory where you have write permission:

% source /home/atlassoft/DQ2/setup.csh

% /home/atlassoft/DQ2/dq2_get -r trig0_ideal0_mc12.007061.singlepart_e_E100.recon.AOD.v12000601_tid006084 trig0_ideal0_mc12.007061.singlepart_e_E100.recon.AOD.v12000601_tid006084._00130.pool.root.2

Select TORONTO or TRIUMF from the list of SRMs.

9. Setting up Athena for Physics Analysis

This section describes how to set up your environement to run Athena Physics Analysis algorithm. First you need to create a working area which will host all your Athena applications:

   mkdir ${HOME}/workarea

Create cmthome diretory:

   mkdir ${HOME}/cmthome
   cd ${HOME}/cmthome

Copy the requirements file needed for cmt set-up:

   cp /home/atlassoft/cmthome/requirements ${HOME}/cmthome/

and check that the macro ATLAS_TEST_AREA in the requrements file points to your workarea

Initialize cmt for Release 12.0.6:

In csh:  
   source /home/atlassoft/12.0.6/CMT/v1r19/mgr/setup.csh
In bash,sh:  
   . /home/atlassoft/12.0.6/CMT/v1r19/mgr/setup.sh

Configure cmt:

   cmt config

NOTE: The previous steps should be done only once

Now, change to your working directory, and source the setup script to configure Athena. This should be done everytime you login or you open a new shell.

   cd ~/workarea
In csh:  
   source ~/cmthome/setup.csh -tag=12.0.6
In bash, sh:
   . ~/cmthome/setup.sh -tag=12.0.6

Now you are ready to use Athena. A simple test consists on running Athena with the HelloWorldOptions.py jobOption file. It initializes Athena and prints out several messages. You don't need to check out any package to run it. Just do:

   get_files HelloWorldOptions.py
   athena.py HelloWorldOptions.py

You should get something like:

..
HelloWorld           INFO initialize()
HelloWorld           INFO   MyInt =    42
HelloWorld           INFO   MyBool =   1
HelloWorld           INFO   MyDouble = 3.14159
HelloWorld           INFO   MyStringVec[0] = Welcome
HelloWorld           INFO   MyStringVec[1] = to
HelloWorld           INFO   MyStringVec[2] = Athena
HelloWorld           INFO   MyStringVec[3] = Framework
HelloWorld           INFO   MyStringVec[4] = Tutorial
...

To use the Physics Analysis Tools, you need to check out the standard UserAnalysis Algorithm available from the ATLAS software repository. The local repository is already set-up in the setup_tutorial.sh. NOTE: You will have to enter the ssh password for the cvsuser account (to be handed out at the workshop) a number of times (at least 5?) during the checkout process. Then you need to configure cmt for this package and compile it:

   cmt co -r UserAnalysis-00-09-10 PhysicsAnalysis/AnalysisCommon/UserAnalysis
   cd PhysicsAnalysis/AnalysisCommon/UserAnalysis/cmt
   cmt broadcast cmt config
In csh:  
    source setup.csh
In sh:  
    . setup.sh
   gmake

In the above checkout you may safely ignore errors from the CVS server of the sort:

    cvs update: warning: cannot open /expt/1/cern.ch/atlascvs/CVSROOT/val-tags
    read/write: Permission denied

Note the gmake takes a few minutes to compile the necessary packages

10. Running Athena on the Grid

Make sure you have done the setups for Athena and the PhysicsAnalysis package described in the previous steps in your area.

   cd ~/workarea
In csh:  
   source ~/cmthome/setup.csh -tag=12.0.6
   source /home/atlassoft/setup_tutorial.csh
In bash, sh:
   . ~/cmthome/setup.sh -tag=12.0.6
   . /home/atlassoft/setup_tutorial.sh

And you have your atlas-canada voms extension

     voms-proxy-init -voms atlas:/atlas/ca

Get the example submission ipython script (this script defaults to submissions to the TRIUMF CE):

   cp /home/atlassoft/workarea/anal_lcg.sub .

Change to the run area in the PhysicsAnalysis package

   cd PhysicsAnalysis/AnalysisCommon/UserAnalysis/run

If you don't already have the python scripts, copy these in to your run/ directory

    cp /home/atlassoft/workarea/AnalysisSkeleton_topOptions.py .
    cp /home/atlassoft/workarea/StructuredAAN_topOptions.py  .

Fire up ganga

   ganga

and issue the execution file with the full path to your area (i.e. replace USERNAME with your username)

   execfile('/home/<USERNAME>/workarea/anal_lcg.sub')

You should see something like:

   Ganga.GPIDev.Lib.Job               : INFO     submitting job 19
   Ganga.GPIDev.Adapters              : INFO     submitting job 19.0 to LCG backend
   Ganga.GPIDev.Lib.Job               : INFO     job 19.0 status changed to "submitted"

11. Running Ganga with Cronus

An experimental setup has been created for the workshop using the highly efficient submission mechanism that Rod Walker uses for Atlas production. A separate CronusGanga Wiki page describes this mechanism, assuming you have already successfully submitted Ganga and Athena jobs via the standard LCG mechanisms.

12. Existing Datasets

This list of complete datasets that exist at both TRIUMF and Toronto may be of interest if people want to try analyzing different sort of Monte Carlo data.

There are also some very useful automatically updated web pages that list datasets and their various stages of completion at the various Canadian sites, e.g.:

• TORON AODs

• TRIUMFDISK AODs

To browse all the Canadian Tier-1 and Tier-2 sites

• http://gridinfo.triumf.ca/vobox/monitor_files_sites

13. Running Other Ganga Tutorial Examples

There is an extensive ganga tutorial page hosted on the CERN Wiki pages which has many more in depth examples. Note they tend to depend on AFS being set up, but most should easily be tailored to running in the Toronto Tutorial environment now that you have some of the basics in hand if you followed the steps above. https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorial427

A detailed presentation on Ganga was given at the Atlas Software Week in Munich, March 29, 2007, by Karl Harrison which is very educational about what the design philosophy behind Ganga is and some tips on setting it up and running Ganga.

Other Information Presented and Used During the Actual Tutorial

• Slides presented during the tutorial are on the Indico site

• https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorialInToronto - Wiki page at CERN set up by Dietrich Liko

• https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTutorial427 - full Ganga Tutorial on the CERN Wiki

-- LeslieGroer - 19 Mar 2007