TWiki> AtlasCanada Web>T2WeeklyReport20120124 (2012-01-24, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-01-24

General News

  • ADC and ATLAS Computing News [20120124]
    • ADC Weekly meeting
    • T1 computing shares
      • Pressure to complete the ongoing MC production well in advance before the winter conferences.
      • CREM decision: /atlas/Role=production (95) : /atlas/Role=pilot (5) instead of historical 80 (prod):20 (group+user)
      • Half T1's implemented this by Jan 13. TRIUMF done Jan 20.
    • ADCR+ATLR downtime Tue Jan 17 for Oracle upgrade to 11g
      • https://twiki.cern.ch/twiki/bin/viewauth/Atlas/ADCDowntime2012Jan17
      • Draining only over 4 hours. Equip pilots with retry in 4hrs option when server is down.
      • Did not shutdown panda servers (with retries to the DB). Jobs started before the downtime and ending after should have survived (otherwise would have 'lost heartbeat').
      • Successful DB interventions with a minimal possible database downtime
      • ADCR database cluster increased from 3 to 4 nodes
      • Visible decrease in the load on the database machines. In particular on the ADCR1 (DQ2) and ADCR2 (PanDA) instances (not much user workload on the ATLR yet).
      • The new HW with 3 times larger data pool cache and storage with SSD cache brought down the IO requests and thus the CPU usage as well.
      • Few small issues fixed and tweaks still going on. Another 15min outage needed to change a 'compatible' parameter from 10.2.0.5 to 11.2.0.3 in the next few weeks. Probably will be transparent to end users (most clients should retry)
      • No issues with DDM services
      • Panda monitor need update of older client
    • T2D Sites - change in criteria going forward - defined by bandwidth to 10 T1's + CERN + NIKHEF
      • All transfers from the candidate T2D to 10/12 T1s for big files must be above 5 MB/s during the last week and during 3 out of the 4 last weeks.
      • All transfers from 10/12 T1s to the candidate T2D for big files must be above 5 MB/s during the last week and during 3 out of the 4 last weeks.
      • List of T2Ds will be reevaluated every month as ADCD. Help from squad is needed. They need to list the sites that pass the metric.
      • The sites that succeed will be promoted as T2Ds.
      • Having the site in the list of T2Ds in the FTS configuration is not a pre-requisit anymore to become a T2Ds.
      • The new T2Ds will be added to the FTS configuration each 3-4 months to lower the load on T1s.
    • Disk Cleaning thresholds in Victor
      • DATADISK - increase usage
        • Current thresholds are 15% free for cleaning, down to at least 20% free. Change to 10% and 15%.
      • SCRATCHDISK - Cleanup more aggresively to prevent user jobs failing
        • Current thresholds are 20% free for cleaning, down to at least 25% free. Change to 50% and 55%.
    • FTS 2.2.8
      • New version containing new features : Overwrite option, checksum validation for transfers with gridftp, new monitoring : http://dashb-wlcg-transfers.cern.ch/ui/
      • Tested for about a month at CERN for FT machines and no problems observed. Will be tested at RAL then once validated, to be deployed on T1s.
    • LFC Consolidation
      • So far CERN, NL, TW, DE, IT hosted on CERN LFC. Next cloud is UK.
      • Consolidations delayed due to Oracle 11g migration and need to move to LFC registration on the Panda server
      • Asking for voulnteers for next consolidation
    • Site Services
      • US sites moved to a VObox at CERN.
      • Lifetime of intermediate copies on T1 SCRATCHDISK has been decreased to 2 days (but they are pinned by the Site Services till the transfer to the destination is done). This should relax the problems with full SCRATCHDISK.
    • ATLAS and SLC6
      • New hardware arriving with SLC6 or variants. Should be able to run slc5 builds on slc6
      • Initial tests look good on lxplus https://twiki.cern.ch/twiki/bin/viewauth/Atlas/PlatformsAndCompilers
      • Aim at validating 17.2.X slc5 build on slc6 as well (17.2.X will be the major production release for nearly everything this year) - Spring 2012?
      • Native slc6 builds probably not readybefore next year (need e.g. lcg to migrate)
        • Will be using gcc46 on both slc5 and slc6 (current gcc43 not available on slc6), to decouple compiler effort from the rest
        • One cannot run slc6 build on slc5, so slc5 build to be maintained as long as slc5 resources are available
      • Some claims of 5% improvement but not proven yet
      • Want to minimize parallel work on different builds (this was painful going from slc4 to slc5)
      • Sites will be encouraged to migrate once the build is available. Expect slc5 support dropped only end of 2013.
      • Looking for volunteer sites (BNL already helping)
      • Question about EMI and other middleware and ADC clients * Tier-0 deployment will be largely driven by offline needs and readiness with the caveat that facilities are shared with LCG so some coordination necessary
    • LCGR upgrade slated for 31 Jan.
  • CA Cloud Squad Actions [20120124]
    • DaTRI requests approved / processed (last week)
SITE NumRequests NumDatasets NumSubscribed SizeofAllDatasets (GB)
AUSTRALIA-ATLAS-LOCALGROUPDISK 0 0 0 0
AUSTRALIA-ATLAS_SCRATCHDISK 4 85 85 1631
CA-ALBERTA-WESTGRID-T2_LOCALGROUPDISK 0 0 0 0
CA-MCGILL-CLUMEQ-T2_SCRATCHDISK 0 0 0 0
CA-MCGILL-CLUMEQ-T2_LOCALGROUPDISK 0 0 0 0
CA-SCINET-T2_LOCALGROUPDISK 1 5 0 0
CA-SCINET-T2_PHYS-TOP 0 0 0 0
CA-SCINET-T2_SCRATCHDISK 3 451 385 193
CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK 1 1 0 0
CA-VICTORIA-WESTGRID-T2_PHYS_EXOTICS 3 36 36 0.9
CA-VICTORIA-WESTGRID-T2_SCRATCHDISK 0 0 0 0
SFU-LCG2_LOCALGROUPDISK 0 0 0 0
SFU-LCG2_PHYS-SM 0 0 0 0
SFU-LCG2_SCRATCHDISK 0 0 0 0
TRIUMF-LCG2_LOCALGROUPDISK 47 290 289 6069
TRIUMF-LCG2_SCRATCHDISK 0 0 0 0
TRIUMF-LCG2_PERF-TAU 1 1 1 328
    • Declared Downtimes
      • CA-VICTORIA-WESTGRID-T2/SCHEDULED downtime (OUTAGE) [16-01-2012 17:00 to 17-01-2012 03:00 UTC] - dcache upgrade to 1.9.12
      • CA-MCGILL-CLUMEQ-T2/SCHEDULED downtime (WARNING) [17-01-2012 10:30 to 18-01-2012 13:00 UTC] - Upgrading storm SE from storm1.7 to 1.8, Automatic provisioning via xcat for this node will be put in place, hence a new install.
      • CA-SCINET-T2 SCHEDULED downtime (OUTAGE) [17-01-2012 13:00 to 18-01-2012 23:59 UTC] - Annual chiller maintenance at the datacentre. All services off for at least a day.
      • TRIUMF-LCG2 SCHEDULED downtime (OUTAGE) [17-01-2012 18:30 to 17-01-2012 22:30 UTC] - Restart SRM for new parameters, upgrade ce1 and ce2 memory, upgrade NFS server OS, Core switch recabling, switch WNs to use CVMFS, etc
      • TRIUMF-LCG2 UNSCHEDULED downtime (WARNING) [17-01-2012 22:57 to 18-01-2012 01:00 UTC ] - This is an at risk warning, due to some network issues for switch reconfigurations.
      • CA-MCGILL-CLUMEQ-T2 SCHEDULED downtime (OUTAGE) [18-01-2012 13:00 to 19-01-2012 13:00 UTC] - Continue Upgrading storm SE from storm1.7 to 1.8.
      • CA-SCINET-T2 UNSCHEDULED downtime (WARNING) [17-01-2012 23:59 to 18-01-2012 02:00 UTC ] - Extending at risk period as datacentre is up but services only coming online now
      • CA-VICTORIA-WESTGRID-T2 UNSCHEDULED downtime (WARNING) [23-01-2012 23:30 to 24-01-2012 00:15 UTC ] - Victoria-Vancouver network disruption during transponder replacement
      • CA-ALBERTA-WESTGRID-T2 SCHEDULED downtime (OUTAGE) [25-01-2012 18:00 to 26-01-2012 15:00 UTC ] - Scheduled outage for fixing network switch
    • ATLAS Site Exclusion Savannah entries [20120124]
      • HammerCloud autoexclusions
        • ANALY_TRIUMF_CVMFS reset online at Wed Jan 18 13:23:05 2012 CET
        • ANALY_TRIUMF reset online at Wed Jan 18 13:30:16 2012 CET
        • ANALY_SCINET reset online at Thu Jan 19 08:30:12 2012 CET
        • ANALY_VICTORIA-WG1 reset online at Thu Jan 19 10:00:22 2012 CET
        • ANALY_SCINET Auto-Excluded at Thu Jan 19 16:31:29 2012 CET
        • ANALY_SCINET reset online at Thu Jan 19 22:00:10 2012 CET
        • ANALY_MCGILL reset online at Fri Jan 20 19:30:17 2012 CET
        • ANALY_AUSTRALIA Auto-Excluded at Sun Jan 22 12:01:42 2012 CET
        • ANALY_AUSTRALIA reset online at Sun Jan 22 15:00:15 2012 CET
    • other cloud squad related savannah and / or ROC ggus tickets [20110705]
      • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
      • p2p_20120124.png
  • Open Issues and Ongoing Discussions

  • Security News [20120110]

  • gLite News [20120110]

  • ROC_Canada News [20120110]

  • WLCG and General Computing News

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

panda_reliability_20120124.png

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View):

-- LeslieGroer - 2012-01-24

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng p2p_20120124.png r1 manage 32.5 K 2012-01-24 - 19:14 LeslieGroer  
PNGpng panda_reliability_20120124.png r1 manage 154.7 K 2012-01-24 - 19:14 LeslieGroer  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2012-01-24 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback