STEP09 Report - SFU Tier-2

Introduction

The performance of the SFU Tier-2 site during the STEP09 exercise is reviewed for the three Tier-2 functions of data distribution, monte carlo production and user analysis (glite WMS and panda pilot submissions) via HammerCloud.

Data Distribution

During STEP09 the SFU site requested a 72% share of the derived dataset distribution of ATLAS data (written to the ATLASDATADISK space token area). Such a value is larger than is planned for the steady-state Tier-2 operations model in the CA cloud where a smaller fraction would be subscribed to the SFU site. Due to storage space constraints at two of the 4 CA Tier-2s the bulk of the subscriptions were targeted toward SFU (72%) and Alberta (25%). Concurrent with the distribution of ATLAS data to DATADISK was also the distribution of merge AOD samples to the MCDISK space token area. The SFU share of the merge.AOD sample distribution is 65%.

Figure 1 shows the ATLAS data distribution dashboard summary for the SFU site for the STEP09 period.

Figure 2 shows the daily growth profile for the SFU DATADISK and MCDISK areas covering the STEP09 period as taken from the DQ2 Accounting webpages.

Figure 3 shows the measured network throughput on the TRIUMF-to-SFU 1 Gbps link. Changes to the FTS settings for transfers between TRIUMF and SFU were made during the STEP09 period and significantly improved the dataset transfer completion and network utilization (see below for further discussion). The network throughput corresponding to a 72% share of data is 552 Mbps. Changes to the FTS settings pushed the data transfer rate well above this (> 800 Mbps).

Figure 1: Dataset and file transfer rates and errors during STEP09

Figure 2: DATADISK and MCDISK occupancy during STEP09

Figure 3: TRIUMF-SFU (T1-T2) Network Link Activity

Several aspects of the SFU site performance related to the data distribution exercise of STEP09 should be noted:

backlog of subscribed datasets at SFU (8 June 2009)

• similar to what was seen last week for Alberta, but corrected with update to FTS settings achieving ~ 800 Mbps via Westgrid link to SFU

Alberta and Toronto (weekend SE failures) and SFU (SE errors)

• resulted in sites being added to Hammercloud blacklist
• requested removal (8 June 2009)

Data Export backlog clearing deadline (14 June 2009)

• backlogs of data distribution to be cleared by 14 June 2100UT
• SFU failed to achieve STEP09 data export metric for completion (less than 10 datasets left to be completed). See link for complete list of all sites that fit this category.
• SFU details
  • Completed: 9483, Missing: 140
  • Comments:
    • data export transfers heavily utilized link from TRIUMF to SFU; brief period with little data export activity it seems (around 6 hours on 11 June) could have made a difference; why the gap?
    • significantly larger share of data transfers (72%) assigned to site, concurrent with large fraction of merge.AOD to go to MCDISK (65%)
    • future improvement via FTS optimization?
      • FTS Settings: concurrent files = 3, streams per file transfer = 1 (for approx. 800 Mbps)

Production and Analysis

The SFU site uses torque + moab as the LRMS. The target shares between production, analysis (WMS) and analysis (panda pilots) for STEP09 were applied shortly after the start of the STEP09 exercise and checked daily for any problems, either in the targets not being met or if any negative impact on performance was occurring due to the settings (such as total number of jobs).

Below is a snapshot as taken on June 4 showing the target and observed split between each of the job types at the SFU site. Overall the maximum number of allowed jobs at the site was limited to approximately 120 jobs representing the ATLAS-accessible portion of the cluster (as determined by moab fairshare settings target of 50% for overall ATLAS activity on the cluster).

Batch System Shares

SFU                   Jobs             Jobs     Share
                      Running ( %)     Queued   Target (%)
Production (panda)        0 (  0%)       20      50*
Analysis (WMS)           96 ( 57%)      363      50*
Analysis (Panda)         72 ( 43%)       32      50
                        ----------
TOTAL                   168 (100%)

* see comment below.

Comment: currently production (panda) jobs are aggregated with the Analysis (WMS) jobs to form the 50% fairshare target.  
Shares subsequently adjusted 20090604 to be the requested values for STEP09 and will be left as such for a few days to see how they stabilize:
50% atlas production via pilots (atprd), 25% atlas analysis via wms (atlas), and 25% atlas analysis via pilots (atcan)

Based upon the data collected from the site torque accounting logs the number of running jobs of each type at the site as a function of time during STEP09 are shown in Figure 5. The average values are as expected based upon the torque settings. Figure 6 shows a stacked plot of the running jobs further demonstrating the total number of jobs running during STEP09.

Given the distribution of running jobs shown in Figure 5 the jobs of each type appear to have been processed on the cluster in bursts rather than maintaining any approximate steady-state level of activity. However as expected the overall level of ATLAS activity, of all job types combined, is approximately constant (Figure 6).

Production

The SFU site was not very efficient in terms of overall average successful job completion (62.2%) and average cputime to walltime efficiency (84.5%) as indicated in the production dashboard snapshot shown in Figure 4. This requires further investigation. It is however not of significant concern given that once the site was re-activated in panda to accept production jobs the job efficiency was very high as seen in Figure 7.

It should be noted though that the SFU site was offline for production since 23 May 2009 due to a reported SE problem at that time, and was re-enabled within the Panda setup so as to accept production jobs once again on 3 June 2009.

Figure 4

The job efficiency (cputime to walltime ratio) as a function of time during the STEP09 exercise is shown in Figure 7.

Analysis

The final STEP09 Summary Results for the analysis jobs submitted to the SFU site are given below. Overall the SFU site was more efficient for WMS analysis job submissions in terms of success rate, but had higher per job cputime/walltime efficiency for pilots (see Figure 7). The SFU site performed very well for both analysis job types, showing 94.6% success for pilots and 99.3% success for WMS analysis jobs.

Figures 8 and 9 demonstrate the measured job throughput (= number of running jobs x the efficiency of those jobs) for each analysis job type. Higher throughput is observed for the Panda pilot analysis jobs.

Several aspects related to the user analysis component of STEP09 should be noted for the SFU site:

Over-subscription of jobs via glite-wms (8 June 2009)

• this is standard mode of operating during STEP09 via hammercloud
• the SFU admins purged from queues jobs identified as already having expired proxies

WMS Analysis Jobs

On average the job efficiency (cputime/walltime) for WMS analysis jobs was approximately 10-15%. Job efficiencies as a function of data access method (dcap versus FileStager) cannot be determined from the local site logging information alone but would of course be worthwhile to know for the future Tier-2 deployments in Canada.

The type and frequency of the very few failures encountered for the WMS analysis job submissions are linked in the table below.

Errors at SFU-LCG2_MCDISK

Site	Exit Code (or Pilot Code)	Failure Type	#	Ex. Job
SFU-LCG2_MCDISK	0	Job terminated succe	3	429 44.63
SFU-LCG2_MCDISK	0	Warning: job exit co	4	426 433.30

Panda Analysis Jobs

On average the job efficiency (cputime/walltime) for Panda pilot analysis jobs was approximately 25% (with some occasions being > 50%), higher than that for the glite WMS submissions.

The type and frequency of the failures encountered for the Panda pilot analysis job submissions are linked in the table below.

Errors at ANALY_SFU

Site	Exit Code (or Pilot Code)	Failure Type	#	Ex. Job
ANALY_SFU	0		39	1008773537
ANALY_SFU	0	Adder._updateOutputs	49	1009040148
ANALY_SFU	0	Setupper._setupSourc	1	1009882038
ANALY_SFU	1097	Get function can not	81	1008784885
ANALY_SFU	1099	Get error: get_data	4	1009169398
ANALY_SFU	1099	Get error: Unexpecte	22	1008773570
ANALY_SFU	1100	Get error: Replica w	333	1009650049
ANALY_SFU	1114	Put error: lfc_statg	1	1009916307
ANALY_SFU	1170	Get error: Adler32 f	1	1008806174
ANALY_SFU	1179	Get error: Failed to	1	1009271804

Conclusions

• data distribution proceeded well following improvements in the FTS settings for the T1-T2 transfers. The site only slightly missed completing the backlog prior to the 14 June end-date. The lack of network of activity on 11 June for a few hours could have likely allowed the site to process the backlog and needs further investigation (no service outages were occurring at the site during that time).

• The production at the SFU site resumed 3 June but proceeded less overall job success and efficiency compared to what would have been expected. This requires further investigation but is not of real concern given the job efficiency observed once production jobs resumed coming to SFU.

• For user analysis both the Panda pilot and WMS submissions were very efficient, well over 90% success rate. The WMS jobs had a higher success rate (+ 5%) but the pilot jobs were more cpu efficient per job. This is even while receiving 72% of the ATLAS data distribution into the site dcache setup.

• improvements are foreseen overall for the Alberta Tier-2 capability as we are transitioning to additional WN hardware in the next 1 to 2 months. The new Tier-2 cluster WNs will also present additional challenges in terms of job scheduling as it will be very much a shared environment with non-ATLAS users.

• A repeat of similar tests with the new Tier-2 hardware over the summer would be beneficial to further tune the site analysis and data distribution capabilities, and decide on the most appropriate data access protocols (dcap versus FileStager for example). The additional number of WNs to access the site storage will also present an opportunity to stress the dcache services at scale.

Site Tasks for Follow-up

• WMS failed job follow-up and details
• Storage Services
  • reason for low T1 transfer traffic for approx 6 hours? (Bryan will investigate central transfer scheduling which is outside scope of site operations)
  • investigate dq2 v SRM discrepacy on MCDISK
    • recommendation from ATLAS DDM:
      • site to compute from the files/pnfs the actual size of all files in /.../atlasmcdisk/.... ?
      • get the list of files which are associated to the space token (if you are motivated to find the missing ones to understand the problem). This was already done in Lyon after some lost files last year.
  • dark data cleanup on PRODDISK (see https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsScripts#Detection_of_LFC_files_not_regis)
  • stream v closed datasets and fnc of time (Bryan)
  • scrutinize new T2 dcache design and setup

Figures

Figure 5: Running jobs

Figure 6: Running jobs (cumulative)

Figure 7: Job Efficiency (cputime/walltime)

Figure 8: Job Throughput

Figure 9: Job Throughput (cumulative)