TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20140128 (2014-01-28, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2014-01-28

General News

  • ADC and ATLAS Computing News [20140128]
    • ADC Weekly meeting
    • Switcher 2 Commissioning
      • Controls panda queues in response to downtimes
      • Summary of differences from Switcher1
        • Split between long and short downtimes
        • Treatment of analysis queues for CE downtime
        • Longer offline time for analysis queues for long SE downtime (to avoid angry users)
        • Resources are automatically picked up from AGIS (not conf file)
      • Some recent discussion led to following limits
      • Switcher2 only acts on online queues of "type" anaylsis or production
        • queues that it has set to another state (checking comment)
        • e.g. setting brokeroff with comment leave.brokeroff means Switcher2 won.t touch it
      • Switcher2 has been running in dry-run mode since November for all resources
      • Publishing to SSB: http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistory?columnid=10100
      • Running in real mode for DE cloud for ~2 weeks with no problems observed
      • Plan to switch all queues to Switcher2 by end of this week
      • Announcement to atlas-adc-cloud-all list
      • If (for a good reason) you don't want Switcher2 to treat your queues, let David Cameron know
      • More info: https://twiki.cern.ch/twiki/bin/view/AtlasComputing/SiteStatusBoardInstructionForShifter#Instructions_for_Queue_Autoexclu
    • AMOD/ADCOS Report
      • RAL FTS successfully upgraded to FTS3
      • RAL T1 and NDGF T1 switched to RAL FTS3
      • Some SE instabilities in the TW renaming campaign due to load induced by renaming campaign - should be finished end of the week
    • MC production status
      • Standard MC production
        • Large single-core backlog from mc12/mc11/mc10 production (Christmas production) is shrinking
          • Expected to ramp down in the next month(s); resources will be taken by DC14 simulation
          • To Do: ~1 month work, mc10: 4M full sim events, mc11: 22M full sim events, 88M fast sim events, mc12: 172M full sim events, 67M fast sim events, 707M b-phy fast sim events
      • High-memory production
        • Very good progress - thanks to all participating sites
        • To Do: ~3.5M events, 70k jobs, ~6h/job, 5k running jobs -> 3 days
      • DC14/MC14 Progress
        • Since Jan 1: 51M events, 1.9/2.2M events/day (last month/last week), ramping up
        • Runs in multi core mode on multi core resources
        • Plans for near future discussed with PC and DP
          • Increase MC14 throughput to use 50% of the simulation resources
            • Increase multi core resources to 30% of the simulation resources
            • Run part of the DC14 samples in single core mode
            • This split will leave some flexibility for possible urgent mc12 production (needs single core slots)
          • Still need to fix
            • Running time per event in mc14 is twice that of mc12 (no frozen shower in mc14 yet (~40%))
            • Memory leak (problematic for single core jobs)
          • Preparation of resourcse should start now
          • mc14 digi+reco configuration is expected for March, will run multi core
    • Rucio Migration News
      • Two steps
        • LFC -> Rucio (Feb)
        • DQ2 -> Rucio (March-June)
      • Renaming prerequisite for LFC migration
        • Almost finished for all clouds (CERN 90% done)
        • Few remaining sites that do not provide WebDAV are being contacted
      • DQ2 Integration
        • DQ2 can use Rucio as a file catalog
          • Transfer/deletion/dq2-put/dq2-get/... validated
        • New DQ2 clients testing release: 2.5.0_rc4
          • Transparent Rucio file catalog support
          • Dependency on rucio clients (0.1.8)
        • Both latest clients (Rucio & DQ2) are available on CVMFS
          • localSetupDQ2Client 2.5.0_rc4
      • PanDA -Rucio Integration
        • All direct LFC API calls have been replaced by DQ2 FileCatalogInterface API calls in PanDA server and in the pilot
        • Reminder: All external applications should do the same. Old clients won’t work after the next stable release.
      • Testing and Commissioning
        • Broker test jobs against DDM end-points served by Rucio file catalog
          • Validation of the full chain and components (from server to pilot)
          • Test end-points (*_DATADISK_RUCIOTEST) available for testing
        • The latest DQ2 + rucio clients have been deployed on the PanDA server test instance
        • Few PanDA resources are being setup for test to have inputs from Rucio test endpoints with HC data for APF and PFT (Prague for now)
        • Rucio catalog endpoints are in AGIS/ToA
        • Job testing should occur in next days
      • Once validated, Panda Server can upgrade all production instances
      • Move sites to Rucio file catalog (February)
      • Gradually move to Rucio calls instead of DQ2 ones
        • Some calls can be now replaced by Rucio, e.g., dq2-finger by rucio list-account
      • DQ2 “rucio aware” clients in March-April
    • Shortage of DAST and ADCOS shifters
      • DAST: 4 in US (including Yun-Ha), < 2 in EU
        • Used to have 7-8 in EU
        • Thanks to Asoka and Nils
          • Want to avoid having developers having to answer trouble tickets (reduces their productivity)
        • Desparate cry for more help. Maybe give shifters higher job priorities as an incentive???
      • ADC Shifters
        • Expect 21 shifts to be taken by shifter per year
        • About 18 (=365/21) shifters per timezone needed
        • Need total 18 Experts and 54 Senior shifters.
        • Level 2 shifts: 8-hour for Senior, 9-hour for Expert.
        • OTP: 1 point for Expert, 1.55/0.78 for weekend/day Senior
        • About to require every shifter to take at least 6 shifts every 4 months (to evaluate effective number of shifters)
        • ADCOS Expert
          • In 2012 : missing 24 OTP points (=shifts), average 2 per month
          • In first half of 2013 : missing 10 , avarage 1.6 per month
          • In second half of 2013 : missing 33 , avarage 5.5 per month
          • First 28 days of 2014 : missing 7
        • ADCoS Senior
          • In 2012 : missing 76.3 OTP points, average 6.4 per month
          • In first half of 2013 : missing 87.9 , avarage 14.6 per month
          • In second half of 2013 : missing 122.97 , avarage 21.6 per month
          • First 28 days of 2014 : missing 31
    • T1 DATADISK
      • Still too many primaries in total
        • Decreased BNL share
      • Two T1 DATADISK had less that 2% free space this week requiring immediate action
        • ESDs removed 270TB: data12_8TeV physics_jetcalibdelayed
        • Replicas unused for 6 months removed (AODs, NTUP_COMMONs – if they have tape copy)
          • 160TB removed one week earlier than usual
        • Still have four T1 DATADISKS with less than 5% free
        • DDM ops is analyzing the replication policies and occupied space
          • Can we decrease the valid1 dataset lifetimes to less than 6 months?
          • How popular are mc12_14TeV ESDs (can we remove them)?
          • Reconsider how store Heavy Ion production
          • With upcoming production of xAOD, can we be more aggressive in removing unused replicas? (<6 months?)

  • Security News [20120801]

  • ROC_Canada News [20130625]
    • Current shift - Erming Pei June 24-28, 2013
    • Next Week - Leslie Groer

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide ( Panglia View):

-- Leslie Groer - 2014-01-28

Comments

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng switcher2_actions.png r1 manage 84.6 K 2014-01-28 - 17:50 LeslieGroer  
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2014-01-28 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback