STEP09 Report - Toronto Tier-2

Introduction

The performance of the Toronto Tier-2 site during the STEP09 exercise is reviewed for the three Tier-2 functions of data distribution, monte carlo production and user analysis (glite WMS and panda pilot submissions) via HammerCloud.

Data Distribution

During STEP09 the Toronto site requested a 1% share of the derived dataset distribution of ATLAS data (written to the ATLASDATADISK space token area). Such a value is far less than is planned for the steady-state Tier-2 operations model in the CA cloud where a larger fraction would be subscribed to the Toronto site. Due to storage space constraints at two of the 4 CA Tier-2s the bulk of the subscriptions were targeted toward SFU (72%) and Alberta (25%). Concurrent with the distribution of ATLAS data to DATADISK was also the distribution of merge AOD samples to the MCDISK space token area. The Toronto share of the merge.AOD sample distribution is 10%.

Figure 1 shows the ATLAS data distribution dashboard summary for the Toronto site for the STEP09 period. In general the transfers to DATADISK were problematic and showed many errors with 54% efficiency. High load on the SE from too large a number of analysis jobs early on in STEP09 likely impacted the WAN transfers.

Figure 2 shows the daily growth profile for the Toronto DATADISK and MCDISK areas covering the STEP09 period as taken from the DQ2 Accounting webpages.

Figure 3 shows the measured network throughput on the TRIUMF-to-Toronto 1 Gbps lightpath link.

Figure 1: Dataset and file transfer rates and errors during STEP09

Figure 2: DATADISK and MCDISK occupancy during STEP09

Figure 3: TRIUMF-Toronto (T1-T2) Network Link Activity

Several aspects of the Toronto site performance related to the data distribution exercise of STEP09 should be noted:

SE problems at both Victoria and Toronto with pool failures and under high load (2 June 2009)

• other CA T2s show high rate of successful transfers near 100%
• sites reduced the maximum number of allowed running jobs so as to limit heavy impact upon dcache
• See Analysis section below for further discussion

Victoria and Toronto dCache services stabilized (3 June 2009)

• subsequently removed from Hammercloud blacklist (originally added due to SE and SAM test failures)

Alberta and Toronto (weekend SE failures) and SFU (SE errors)

• resulted in sites being added to Hammercloud blacklist
• requested removal (8 June 2009)

Toronto dcache access problems (9 June 2009)

• SAM test failures and FTS transfers set to inactive this morning
  • new host certificate required (installed 10 June 2009)

Production and Analysis

The Toronto site uses torque + maui as the LRMS. The target shares between production, analysis (WMS) and analysis (panda pilots) for STEP09 were applied in advance of the exercise and checked daily for any problems, either in the targets not being met or if any negative impact on performance was occurring due to the settings (such as total number of jobs).

Below is a snapshot as taken on June 4 showing the target and observed split between each of the job types at the Toronto site.

Batch System Shares

 
Toronto               Jobs             Jobs     Share
                      Running ( %)     Queued   Target (%)
Production (panda)      201 ( 55%)       20      45
Analysis (WMS)           91 ( 25%)      234      22
Analysis (Panda)         59 ( 16%)       30      22
Local                    16 (  4%)       61      10
                        ----------
TOTAL                   367 (100%)

Comment: number of jobs = 403 ; set maxjobs of each analysis type at
100 and production at 360, and 360 (all types combined) to keep load on
SE managable). These changes were done this morning, so some further time
to let observed shares stabilize to target values.

Production

The Toronto site was not very efficient both in terms of overall average successful job completion (72.6%) and average cputime to walltime efficiency (82.9%) as indicated in the production dashboard snapshot shown in Figure 4.

Figure 4

The job efficiency (cputime to walltime ratio) as a function of time during the STEP09 exercise is shown in Figure 7 and shows typically job efficiencies of around 90% for production activities.

Analysis

The final STEP09 Summary Results for the analysis jobs submitted to the Toronto site are given below. Overall the Toronto site was more efficient for analysis pilot job submissions, both in terms of success rate (64.9% for pilots and 55.5% for WMS jobs) and individual job cputime/walltime efficiency (see Figure 7). However in both cases the efficiences for success are well below the average of most sites including the other CA Tier-2s.

Figures 8 and 9 demonstrate the measured job throughput (= number of running jobs x the efficiency of those jobs) for each analysis job type. Higher throughput is observed for the Panda pilot analysis jobs.

Several aspects of the Toronto site performance related to the user analysis component of STEP09 should be noted:

SE problems at both Victoria and Toronto with pool failures and under high load (2 June 2009)

• other CA T2s show high rate of successful transfers near 100%
• sites reduced the maximum number of allowed running jobs so as to limit heavy impact upon dcache
• Toronto dCache configuration notes
  • Figure 10 shows the measured network utilization of each of the site dCache pool nodes during STEP09 (Weeks 22 and 23)
    • possible hot pool visible (storage18) compared to other pools

Victoria and Toronto dCache services stabilized (3 June 2009)

• subsequently removed from Hammercloud blacklist (originally added due to SE and SAM test failures)

Toronto fairshares update (9 June 2009)

• increased number of job slots to compensate for blocked ganga, panda and production jobs by large number of locally submitted jobs that share the same atlas queue

Toronto site blacklisted (9 June 2009)

• blacklisted in ganga WMS analysis job submissions (see 0 jobs queued or running in Hammercloud snapshot table of 9 June 2009)
• Alberta and Toronto sites blacklisted in ganga WMS analysis job submissions (see 0 jobs queued or running in Hammercloud snapshot table below). [20090609]

Toronto dcache access problems (9 June 2009)

• SAM test failures and FTS transfers set to inactive this morning
  • new host certificate required (installed 10 June 2009)

WMS Analysis Jobs

On average the job efficiency (cputime/walltime) for WMS analysis jobs was approximately 10%. Job efficiencies as a function of data access method (dcap versus FileStager) cannot be determined from the local site logging information alone but would of course be worthwhile to know for the future Tier-2 deployments in Canada.

The type and frequency of the failures encountered for the WMS analysis job submissions are linked in the table below. A detailed understanding of the reasons associated with the lower success rate compared to that seen with the pilot submissions is still under investigation.

Errors at TORONTO-LCG2_MCDISK

Site	Exit Code (or Pilot Code)	Failure Type	#	Ex. Job
TORONTO-LCG2_MCDISK	0	Job terminated succe	55	427 11.0
TORONTO-LCG2_MCDISK	0	non-zero app. exit c	1735	427 11.18
TORONTO-LCG2_MCDISK	0	request expired	37	428 161.2
TORONTO-LCG2_MCDISK	33	non-zero app. exit c	50	427 88.45

Panda Analysis Jobs

On average the job efficiency (cputime/walltime) for Panda pilot analysis jobs was approximately 40%, higher than that for the glite WMS submissions.

The type and frequency of the failures encountered for the Panda pilot analysis job submissions are linked in the table below.

From Figures 5 and 6 it is clearly seen that three times there are periods during which no pilot analysis jobs are being run at the Toronto site (two of the periods correspond to known times of blacklisting).

Errors at ANALY_TORONTO

Site	Exit Code (or Pilot Code)	Failure Type	#	Ex. Job
ANALY_TORONTO	0		10	1009234404
ANALY_TORONTO	1098	Too little space lef	6	1010441073
ANALY_TORONTO	1099	Get error: Copy comm	286	1008771550
ANALY_TORONTO	1137	Copy command returne	2	1008771418
ANALY_TORONTO	1137	Put error: Copy comm	49	1010439344
ANALY_TORONTO	1151	Get error: Copy comm	1	1009882887
ANALY_TORONTO	1151	Get error: lcg-cp ge	4038	1008771395
ANALY_TORONTO	1152	lcg-cr get was timed	1	1009303956
ANALY_TORONTO	1152	Put error: lcg-cr ge	1	1008771460
ANALY_TORONTO	1180	Get error: Globus sy	288	1008771392
ANALY_TORONTO	1181	Globus system error:	2	1010161186
ANALY_TORONTO	1181	Put error: Globus sy	69	1008771414

Conclusions

• data distribution was at the level of 1% for the Toronto site (share of overall ATLAS data distributed). In general various dcache errors observed early-on in STEP09 due to overloaded dcache services from too large a number of WNs actively processing analysis jobs.

• Overall the production proceeded with less overall job success and efficiency compared to what would have been expected. The Toronto site was not very efficient in terms of overall average successful job completion (72.6%) and average cputime to walltime efficiency (82.9%) as indicated in the production dashboard snapshot shown in Figure 4. This requires further investigation.

• For user analysis the Panda pilots and WMS job submissions had lower success rates compared to those at other T2s (in Canada and elsewhere). Heavy load on dcache at the beginning in particular necessitated capping the maxmum number of allowed jobs of each type to run. As seen elsewhere the pilot analysis jobs were more efficient compared to those for WMS submissions.

• improvements are foreseen overall for the Toronto Tier-2 capability as it is transitioning to new hardware in the next 2 to 3 months. This will give better network access to each of the dcache pool nodes from the cluster WNs (which there will be more of as well). The new Tier-2 cluster will also present additional challenges in terms of job scheduling as it will be very much a shared environment with non-ATLAS users (much like is and will be the case with the SFU Tier-2 site).

• A repeat of similar tests with the new Tier-2 hardware over the summer would be beneficial to further tune the site analysis and data distribution capabilities, and decide on the most appropriate data access protocols (dcap versus FileStager for example) and to test the dcache setup at scale.

Site Tasks for Follow-up

• WMS failed job follow-up and details
• Pilot failed jobs - why so high?
• Storage Services
  • improvement to services needed
  • distribute across more nodes, pool nodes
  • scrutinize new T2 dcache design and setup

Figures

Figure 5: Running jobs

Figure 6: Running jobs (cumulative)

Figure 7: Job Efficiency (cputime/walltime)

Figure 8: Job Throughput

Figure 9: Job Throughput (cumulative)

Figure 10: Network utilzation for each of the four dCache pool nodes

Extra Figures

Figure 11: Cumulative Network Utilization

Figure 12: dCache SRM Door Network Utilization