STEP09 Exercise in Canada

General Information

• ATLAS STEP09 TWiki https://twiki.cern.ch/twiki/bin/view/Atlas/Step09
• STEP09 Logbook (ATLAS-wide) https://twiki.cern.ch/twiki/bin/view/Atlas/Step09Logbook

CA Cloud Preparations

• CA Cloud Operations Planning (20090519)
  • caron-STEP09_20090519.pdf
  • presented at weekly CA Cloud Operations Meeting (20090519)

• CA Cloud Homework for STEP09 Preparations
  • caron-CA-STEP09_20090520.pdf
  • submitted to atlas-project-adc-operations-step (20090520)

• CA Cloud merge.AOD and data distribution shares for STEP09
  • space_needed_MCDISK-DATADISK-20090525-CA_v2.pdf: (PDF)
  • space_needed_MCDISK-DATADISK-20090525-CA_v2.xlsx: (XLSX)
    • submitted to atlas-project-adc-operations (20090526)

Monitoring Links

• Hammercloud Tests http://gangarobot.cern.ch/st/
• STEP09 Analysis Status Table (all clouds and sites, every 30 minutes) http://gangarobot.cern.ch/st/status.html
• Panda Production http://panda.cern.ch:25980/server/pandamon/query?dash=prod
• Panda Analysis http://panda.cern.ch:25980/server/pandamon/query?dash=analysis

• ATLAS Dashboard (CA Cloud) - last 24 hours - http://dashb-atlas-data.cern.ch/dashboard/request.py/site?name=TRIUMF&statsInterval=24
• ATLAS Dashboard (CA Cloud) - last 4 hours - http://dashb-atlas-data.cern.ch/dashboard/request.py/site?name=TRIUMF&statsInterval=4
• ATLAS Dashboard (CA Cloud) - last 1 hour - http://dashb-atlas-data.cern.ch/dashboard/request.py/site?name=TRIUMF&statsInterval=1

• Functional Test Monitoring (T1-T1 Data Distribution) http://atladcops.cern.ch:8000/drmon/ftmon_T1-T1.html
• Functional Test Monitoring (T2 Data Distribution) http://atladcops.cern.ch:8000/drmon/ftmon_tier2s.html
• Functional Test Distribution (T0-T1) http://panda.cern.ch:25980/server/pandamon/query?mode=listFunctionalTests&testType=T0toT1s
• Functional Test Distribution (T1-T1) http://panda.cern.ch:25980/server/pandamon/query?mode=listFunctionalTests&testType=T1toT1s
• Functional Test Distribution (T1-T2) http://panda.cern.ch:25980/server/pandamon/query?mode=listFunctionalTests&testType=T1toT2s

• DQ2 Storage Accounting (CA Cloud) http://atlddm02.cern.ch/dq2/accounting/cloud_view/CANADASITES/30/

• TRIUMF Network Monitoring http://gridinfo.triumf.ca/network/

Daily Notes

Tuesday 2 June

• SE problems at both Victoria and Toronto with pool failures and under high load; other CA T2s show high rate of successful transfers near 100%
  • sites reduced the maximum number of allowed running jobs so as to limit heavy impact upon dCache

Wednesday 3 June

• Victoria and Toronto dCache services stabilized
  • subsequently removed from Hammercloud blacklist (originally added due to SE and SAM test failures)

• SFU site re-enabled within panda setup so as to accept production jobs (offline in panda since May 23)

Thursday 4 June

Production and Analysis Shares - Status Check

Cloud homework today included request for status of job sharing levels between production, analysis (WMS) and analysis (pilots). Here are the results as of 20090604.

 Toronto               Jobs             Jobs     Share
                      Running ( %)     Queued   Target (%)
Production (panda)      201 ( 55%)       20      45
Analysis (WMS)           91 ( 25%)      234      22
Analysis (Panda)         59 ( 16%)       30      22
Local                    16 (  4%)       61      10
                        ----------
TOTAL                   367 (100%)

Comment: number of jobs = 403 ; set maxjobs of each analysis type at
100 and production at 360, and 360 (all types combined) to keep load on
SE managable). These changes were done this morning, so some further time
to let observed shares stabilize to target values.

Alberta               Jobs             Jobs     Share
                      Running ( %)     Queued   Target (%)
Production (panda)       53 ( 50%)       22      50
Analysis (WMS)           23 ( 22%)      218      25
Analysis (Panda)         30 ( 28%)       23      25
                        ----------
TOTAL                   106 (100%)

Comment: max number of running jobs = 115 ; since 20090530 have set soft
limit on maxjobs of each analysis type at 30 (panda) and 25 (wms),
production at 60.  Hard limit values are higher.

Victoria              Jobs             Jobs     Share
                      Running ( %)     Queued   Target (%)*
Production (panda)       65 ( 50%)       35      50
Analysis (WMS)           70 ( 22%)      330      25
Analysis (Panda)         40 ( 28%)       39      25
                        ----------
TOTAL                   175 (100%)

Comment: max number of allowed running jobs decreased from total job
slots to reduce the the frequency of gridftp timeouts.  Site currently does not 
employ maui fairshare settings but rather sets soft and hard limits on maxjobs.

* As of 20090604: maxjobs for production/analysis-wms/analysis-pilots = 100/50/50

SFU                   Jobs             Jobs     Share
                      Running ( %)     Queued   Target (%)
Production (panda)        0 ( 50%)       20      50*
Analysis (WMS)           96 ( 57%)      363      50*
Analysis (Panda)         72 ( 43%)       32      50
                        ----------
TOTAL                   168 (100%)

* see comment below.

Comment: currently production (panda) jobs are aggregated with the
Analysis (WMS) jobs to form the 50% fairshare target.  
Shares subsequently adjusted 20090604 to be the requested values for STEP09 and
will be left as such for a few days to see how they stabilize:
50% atlas production via pilots (atprd), 
25% atlas analysis via wms (atlas), and 
25% atlas analysis via pilots (atcan)

VICTORIA_MCDISK becomes full

The 12.7 TB of MCDISK space became full today. This size was to be sufficient for the 10% share of merge.AOD samples given the centrally defined estimate of merge.AODs totalling approx. 75 TB. At the time of becoming full only 5 of the 259 merge.AOD datasets to go to Victoria were missing. After consultation with ADC Operations there will be a central cleaning of mc08*recon.AOD*tid* in VICTORIA-LCG2_MCDISK (to gain 2.7 TB). Datasets data08_cos*tid* in VICTORIA-LCG2_DATADISK (to gain 1.8 TB for the future) will also be cleaned centrally. Time required for cleaning was estimated at a few hours, after which the FTS channel from TRIUMF can be reopened.

Dataset transfers to ALBERTA - update to FTS Settings

It was observed yesterday that nearly all of the dataset subscriptions to Alberta_DATADISK were remaining in a 'subscribed' state (red) rather than 'complete/done' (green) on the T1-T2 FT status pages for the past 24 hour period. This is although nearly 100% success was observed in dataset file transfers to Alberta from the dashboard. The majority of the datasets 'subscribed' to Alberta are stream datasets. Querying ADC Ops explained the difference between stream and other dataset handling. Further discussion at the morning ADC Ops meeting pointed toward a non-optimal FTS configuration for the TRIUMF-Alberta channel. The rate of transfer was approximately 100 Mbps at the time.

Di changed parameters from streams=2 and files=3 to streams=6 and files=4; the transfer rates increased roughly from 250Mb/s to 350Mb/s on average. Increasing them again to streams=7 and files=8, and transfer rates reached about 500Mb/s and some time even more.

The dCache setup at Alberta is handling the new FTS channel settings nicely, with a per pool node load increase from less than 1 to around 2 - 3, and with +20% cpu utilization. An observed doubling in the sum inbound network traffic to the worker nodes occurred during the same period and is being investigated for any effects on analysis job efficiency. Will keep the latest FTS settings for the next 24 hours or so to see if the backlog of 'subscribed' datasets can be reduced.

Friday 5 June

Status as of this evening:

• T1 reprocessing at TRIUMF has started following completion of higher-priority validation tasks
  • HITS merging defined centrally that will read small HITS from MCDISK and wrirte jumbo HITS to tape

• improved values of completed/done datasets at Alberta with updated FTS settings from previous day

Saturday 6 June and Sunday 7 June

• TRIUMF Tier-1 reprocessing proceeding very well
  • comment from GS: "The STEP09 target for a 5% T1 is about 1150 files a day (this is 5x data taking speed, with 40% machine efficiency) and you've [both] been doing almost three times that (which means you can reprocess at 5x even with 100% machine efficiency). Well done!"

Some snapshots of T1 activities (since June 1 2009):

• Alberta PRODDISK and DATADISK - no space reported [20090607]
  • 4 small pools of the total of 6 on dCache became full ; subsequently marked read-only
  • FTS transfers paused during error reporting period

• Toronto SE errors [20090607]
  • restored following morning with restart of SRM and door services

Some snapshots of Analysis and Data Distribution (last 24 hours)

Monday 8 June

• continued good reprocessing activity at TRIUMF Tier-1
  • HITS merging problems common across clouds ; discussions regarding workflow with developer

Some snapshots of T1 activities (since June 1 2009):

• Alberta and Toronto (weekend SE failures) and SFU (SE errors?) previously resulted in sites being added to Hammercloud blacklist ; requested removal [20090608]

• Alberta fairshares look to be on target
  • in general balance between production (60 jobs), analysis (WMS) (0 jobs) and analysis (pilots) (30 jobs) behaving as desired but will continue to monitor
    • 0 WMS jobs as none are in submitted state; investigating this case

• SFU fairshare between Atlas and Westgrid is 50/50 and the actual usage is also very close to that. The Atlas usage was about 52% over the last week. The production/Atlas WMS/Atlas Canada fairshare is set to 50/25/25 and the actual usage is also very close.

• Toronto fairshares [2009-06-09]
  • increased number of job slots to compensate for blocked ganga, panda and production jobs by large number of locally submitted jobs that share the same atlas queue

• backlog of subscribed datasets observed at SFU (much like what was seen last week for Alberta, but corrected with update to FTS settings) - already achieving ~ 800 Mbps via Westgrid link to SFU

• oversubscription of jobs via glite-wms observed (this is standard mode of operating during STEP09 via hammercloud) [2009-06-08]
  • will purge from queues jobs identified as already having expired proxies

Some snapshots of Analysis and Data Distribution (last 24 hours)

Tuesday 9 June

Some snapshots of T1 activities (since June 1 2009):

• ggus ticket open to investigate timeouts and control channel hanging on FTS transfers to/from CNAF (STORM)

Some snapshots of Analysis and Data Distribution (last 24 hours)

• Alberta and Toronto sites blacklisted in ganga WMS analysis job submissions (see 0 jobs queued or running in Hammercloud snapshot table below).

• problems connecting to Toronto dCache
  • SAM test failures and FTS transfers set to inactive this morning
    • new host certificate required

• Alberta site had job submission problems due to site bdii failure (corrected 2009-06-09)

• permission problem with writes to Alberta DATADISK encountered by FTS transfers - later corrected at site

Wednesday 10 June

Some snapshots of T1 activities (since June 1 2009):

Some snapshots of Analysis and Data Distribution (last 24 hours)

• new host certificate installed for dCache services at Toronto

• Alberta and Toronto requested to be removed from Hammercloud blacklist

Thursday 11 June

Some snapshots of T1 activities (since June 1 2009):

Some snapshots of Analysis and Data Distribution (last 24 hours)

Friday 12 June : STEP down for ATLAS

• Data export load generator will stop. Sites have 48 hours to clear any remaining backlogs.
  • Many T2s still have challenging backlogs to clear (deadline 14 June at 2100 UTC)
• Hammercloud tests will stop submitting new analysis jobs. Existing jobs will drain. *All step09. production tasks will be aborted

Some snapshots of T1 activities (since June 1 2009):

Some snapshots of Analysis and Data Distribution (last 24 hours)

• Central STEP09 news: at 2100 UTC the kill command to be send to all running or submitting HC jobs

• Alberta SE down due to failure with access to LUNs on temporary pools mounted from WestGrid-UofA SAN - under investigation

Saturday 13 June

Some snapshots of T1 activities (since June 1 2009):

Some snapshots of Analysis and Data Distribution (last 24 hours)

• Alberta SE still down due to failure with access to LUNs on temporary pools mounted from WestGrid-UofA SAN - under investigation

Sunday 14 June : Data Export backlog clearing deadline

• backlogs of data distribution to be cleared by 14 June 2100UT
• two CA sites (ALBERTA, SFU) failed to achieve STEP09 data export metric for completion (less than 10 datasets left to be completed). See link for complete list of all sites that fit this category.

• ALBERTA details
  • Completed: 3000, Missing: 394
  • Comments:
    • SAN storage failure 12 June in the afternoon causing failed access to temporary pools on Westgrid-UA T2 storage system; no further writes to pools possible as others were full.
    • FTS settings were updated 4 June and gave significant improvement for transfers compared to those earlier on during STEP09. Can they be further optimized?
      • FTS Settings: concurrent files = 8, streams per file transfer = 7 (for approx. 600 Mbps)

• SFU details
  • Completed: 9483, Missing: 140
  • Comments:
    • data export transfers heavily utilized link from TRIUMF to SFU; brief period with little data export activity it seems (around 6 hours on 11 June) could have made a difference; why the gap?
    • significantly larger share of data transfers (72%) assigned to site, concurrent with large fraction of merge.AOD to go to MCDISK (65%)
    • future improvement via FTS optimization?
      • FTS Settings: concurrent files = 3, streams per file transfer = 1 (for approx. 800 Mbps)

• cleanup of STEP09 datasets to begin 16 June

Some snapshots of T1 activities (since June 1 2009):

Some snapshots of Analysis and Data Distribution (last 24 hours)

• Alberta SE still down due to failure with access to LUNs on temporary pools mounted from WestGrid-UofA SAN - under investigation

Summary Report: 2009-06-16

Post-Mortem Analysis

Feedback has been requested of sites in regards to STEP09. The main information requested is given here. The general idea is to generate reports for each site similar to what was done previously in the example from Graeme for Glasgow.

Requested Baseline Information

To generate the overall CA and individual T2 site feedback for STEP09 sites are requested to provide the following information:

• torque accounting log files for the period of STEP09 operations (June 2 to 12 inclusive)
  • this information will be used to extract some of the metrics (see below)
• network throughput graphs (if available) for both internal and external network access to your local SE nodes and WNs
  • internal - from analysis jobs
  • external - from T1-T2 transfers (We have access to these already from the TRIUMF T1 network monitoring pages here).
• a description of your network layout and disk servers (only if different from what was presented at our CA T1-T2 face-to-face meeting (January 2009); otherwise I will use the information presented at that meeting).

Site Feedback to be provided

The overall set of metrics and requested feedback from the sites for STEP09 are given below.

TRIUMF T1

See requested information here.

T2s

• Data distribution
  • Any problems with getting large volume of data distributed during STEP09?
    • Was your infrastructure adequate?
    • Did you have enough headroom to cope with backlogs quickly?
    • Did other activities, specifically user analysis, interfere?
  • dataset subscription backlog analysis for SFU and Alberta sites
    • SFU: 72% share, requiring 552 Mbps
    • Alberta: 25% share, requiring 192 Mbps
  • overall access rates (internal and external)
  • dataset completion distribution as a function of time for individual T2 sites?

• workload
  • Note: STEP09 Week1 included file stager + local dcap ; STEP09 Week2 was local dcap only (file stager off).
  • number of jobs run each day for production, WMS analysis and Panda pilot analysis
  • job efficiencies (cputime v walltime) in the batch system for each type of job each day
  • throughput (number of running jobs x job efficiency) each day
    • this is a useful metric of the capacity of your site to perform analysis
    • is there an optimal number of jobs to run to maximize analysis throughput?
  • job efficiency as a function of the number of concurrently running jobs
  • job efficiencies (completed v failed) as a function of time
    • current gangarobot summary has the aggregate values over the entire STEP09 period.
    • for the failed jobs, where in the job did the failure occur? Setup? Middle? End?
  • job efficiencies for WMS analysis for filestager and direct dcap access to site storage
    • is there a well defined data access method for which your site performs best?
    • Note: this cannot be distinguished by the site at the LRMS-level for STEP09, so requires input from hammercloud people.
  • do you know what the limitations are in your site infrastructure in delivering successful analysis results?
    • network, i/o on servers or WNs, etc.
    • how can analysis performance be improved before LHC data arrives?

• Fairshares
  • summarize defined fairshare settings at the sites (to conform with requested STEP09 settings)
    • any applied limitations?
      • such as caps on maximum number of allowed running jobs, both aggregate as well as a function of job type
      • reason(s) for which such limitations were applied during STEP09.

STEP09 Feedback Reports

-- BryanCaron - 15 Jun 2009