How to Apply for Prioritized Access to ATLAS-Canada Only Resources

Please note that the concept of "poweruser" has changed.

Additional computing cpu and disk storage resources at the Canadian Tier-1 and Tier-2s beyond those used for ATLAS production and user grid resources are made available to ATLAS-Canada memebers.

In the past, ATLAS-Canada members accessed extra resources in two ways:

1. Grid jobs with CA role that were brokered to Canadian grid sites were effectively given a priority boost in the panda queue. We will call this skewing of Canadians. If the jobs were brokered to non-Canadian sites, there was no skewing.
2. Canadians could request power-user role to directly use beyond-pledged resources at Canadian grid sites only.

Due to technical features, method 1. above has not actually worked in the last 3-4 years. Method 2. becomes less relevant as the ATLAS computing model evolves. To first order, the Canadian beyond-pledged resources have been going to the benefit of ATLAS as a whole with very little advantage for ATLAS-Canada members. Thus, we are revisiting the usage of our beyond-pledge resources to allow a possibly bigger advantage to ATLAS-Canada members. The need for vetting the requests for these resources by Canada pcom remains unchanged.

The computing use cases roughly divide into two functions: MC production and user grid jobs.

Canadian MC production . In the last 9 years Canadians have requested resources to perform MC production 4 times. This occurred mostly in the earlier years. Nowadays, it is very difficult for a user to execute the MC production chain privately, and the produced samples would not be considered official. In keeping with ATLAS policy, we propose that extra resources request by ATLAS-Canada members for production jobs actually be produced in the ATLAS central production system in the normal way. Once approved by pcom, the request would be vetted by ATLAS PC/PMG as a beyond-pledged request (ATLAS as actually asked that the ATLAS-Canada computing coordinator consult with PC/PMG before Canadian internal review). The procedure would be identical to that implemented in 2017 for USA. The understanding is that these requests would be infrequent – about one a year – and should not require more resources then the Canadian beyond-pledged resources. In discussions with ADC, it would be considered a gentleman’s agreement and accounting would be done privately. In terms of implementation, the request would be submitted by a Canadian MC production manager. The later is simple as Doug Gingrich currently has that role, and Alberta has an institutional responsibility in MC production. In the long-distance future, the ATLAS-Canada computing manager can keep the books. But as mentioned above, these are likely to be once-off cases that happen only about once a year.

User grid jobs . These can be divided into three cases: a need for essentially cpu only, diskspace only, or both. The use case for needing extra cpu is not clear to many computing people. It seems equivalent to the need for a priority boost during times of heavy user grid activity, or tight schedules. The 6 use cases in the last 9 years seem less relevant lately and in some cases could have been accommodated if the user had access to a decent Tier-3, or a Canadian-only batch queue at a Canadian Tier-2 site. We propose to handle these requests on a case-by-case basis. In addition, a batch queue will be implemented at the Tier-2(s) for those cases in which it is useful.

1. If the user needs a short priority boost on standard ATLAS grid resources, we make an argument to ADC similar to the MC production argument, in which our beyond-pledge resources are leveraged into the priority boost. The accounting is managed as a gentleman’s agreement by the Canadian MC production manager (in the future ATLAS-Canada computing manager) so as not to exceed the total Canadian beyond-pledged resources.
2. For the Canadian user that requires extra resources in Canada, these can be handle as power-user role as they have been done for the last 9 years. This is also how diskspace can be handled. When this becomes technically not possible (almost now), the user can use the batch queue at the Tier-2.

To initiate your request, contact the ATLAS-Canada computing coordinator (currently Doug Gingrich <gingrich@ualberta.ca>) . If none of the above fits your situation discuss it with the ATLAS-Canada computing coordinator. One to the roles of the ATLAS-Canada computing coordinator is to enable a computing advantage to ATLAS-Canada members.

The old "poweruser" procedure follows.

Additional computing resources (CPUs and grid storage space) are available at the Canadian Tier-1/2 sites. A fraction of these are made available for "powerusers" (ATLAS Canada users with an atlas:/atlas/ca/Role=poweruser0 VOMS role) who will then get priority access with the intent that these resources be primarily used for short-term special projects.

This page describes the steps a user needs to do in order to become a "poweruser" and gain access to these ATLAS-Canada only computing resources.

Become a poweruser

Submit request

A request should first be submitted to the ATLAS-Canada Prioritization Committee (PCom) which consists of the Physics coordinator and the Computing coordinator; please send an email to atlas-canada-pcom@cern.ch.

This request must specify:

• The physics motivation and justification for the request,
• A list of the members on the project, and which of them require the poweruser role,
• The required cputime for the project with the detail of how you estimated the cputime,
• The required disk space, if storage is required, with the detail of how you estimated it,
• The timeline of the project which estimates when and how long the resources will be needed.

If the requests deal with official datasets and non-Canadian users running jobs, we may recommend you to go through official ADC channels but run the jobs on the CA cloud where we will ensure the jobs have the resources needed. On the other hand, if you are an individual user or a group of CA users who need these resources, we may grant you the poweruser0 role instead so that you can submit the tasks. Note that in no situation will ATLAS Canada user support run jobs for you; we will diagnose and help resolve site issues and ensure you get the required resources like we do for any non-poweruser0 jobs.

After this request has been approved, and if we do not recommend official channels to submit the tasks, please continue with the next two steps below which can be done in parallel. (If you are directed to official ADC channels, please contact your ADC physics production managers instead telling them that the CA cloud will provide resources for the task.)

Request VOMS role

(You do not need to do this step if you are only using Panda)

Using your web browser which should be loaded with your grid certificate, access the LCG ATLAS registration page in this link.

You should see a new page with "+ Members" on the left column; click on the "+" to expand and then click on the "Select Groups & Group Roles" link.

A new page would show you the various roles and check-boxes; select the check-box next to the Group role of "poweruser0" which is under the group "/atlas/ca".

Click on the submit button at the bottom of the page.

You should get an email once the ATLAS-Canada VOMS manager approves your request. You can do the next step in the meantime.

Request Panda workingGroup role

(Reference: puserinfo)

You need to do the following to request a poweruser0 role for panda jobs (you can select for only the sites you plan to run at):

• setup the Panda Client tool at your site / desktop / or on lxplus,
• issue the following commands:
  

  puserinfo request ANALY_TRIUMF 
  puserinfo request ANALY_MCGILL
  puserinfo request ANALY_SFU
  puserinfo request ANALY_SCINET
  puserinfo request ANALY_VICTORIA

You will receive an email when your request is approved and you would should see the following when you issue the puserinfo -l list command:

$ puserinfo -l list


            ANALY_SFU : approved
                   created : 2009-07-17 20:35:38
                   poffset : 0
                    rights : None
             workingGroups : poweruser0

        ANALY_SCINET : approved
                   created : 2009-07-17 20:36:00
                   poffset : 0
                    rights : None
             workingGroups : poweruser0

         ANALY_TRIUMF : approved
                   created : 2009-06-29 19:57:43
                   poffset : 0
                    rights : None
             workingGroups : poweruser0

       ANALY_VICTORIA : approved
                   created : 2009-07-17 20:36:23
                   poffset : 0
                    rights : None
             workingGroups : poweruser0


INFO : Done

Note the workingGroups should have poweruser0 as shown above; otherwise, please contact us.

Running jobs as a poweruser

After you have completed the steps above, and have been approved, you can now submit your jobs using your poweruser role.

Ganga

Edit your ~/.gangarc file and find the line

init_opts = -voms atlas:/atlas/ca

and change this to

init_opts = -voms atlas:/atlas/ca/Role=poweruser0

Next, destroy any proxies you aleady have; do this in a new terminal window (not the same one as you setup Ganga):

setup the grid middleware environment as you would normally do on a new terminal session

and issue

voms-proxy-destroy

Finally, startup Ganga and submit your jobs as usual.

Note that every job you submit with this modified ~/.gangarc file is now going to be run as a poweruser.

Panda

When you submit a panda job, add the following option to the pathena, prun or psequencer commands you use to submit to the grid:

--workingGroup poweruser0

You can setup your voms proxy as normal (eg. voms-proxy-init --voms=atlas:/atlas/ca) since Panda does not look at the proxy attributes to understand that you are a poweruser.

PrunDriver in EventLoop

The driver option to use the poweruser0 working group role should be:

 EL::PrunDriver driver; //  PrunDriver grid driver

driver.options()->setString(EL::Job::optSubmitFlags,"--workingGroup=poweruser0");
 driver.options()->setDouble("nc_showCmd", 1); // print prun command