TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130702 (2013-07-02, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-07-02

General News

  • ADC and ATLAS Computing News [20130702]
    • ADC Weekly meeting
    • BNL Data Loss
      • Storage Controller in front of 64 TB disk array became unresponsive June 20
      • Backend hardware RAID controller suffered a hardware problem leading to memory cache corruption which resulted in corruption of the ZFS filesystem metadata
      • After various recovery attempts concluded not possible
      • 618,915 files lost (T0D1) affecting all spacetokens (0.6% of ATLAS-BNL data)
      • Reexamining risk mitigation and storage policies for future
      • Meeting June 27 to understand damage and prioritize recovery
        • List of affected datasets, tasks: https://indico.cern.ch/conferenceDisplay.py?confId=260197
        • Tau-embedding 1st priority, the rest can wait
        • Automated recovery service suspended for BNL till July 1 as LFC had some instabilities
          • LFC lookup was bypassed by manual insertion of data to recovery service
        • Deletion on US PRODDISKs suspended to possibly recover files from there
        • Urgent datasets identified
          • Copying of missing files to BNL PRODDISK, for those with replicas elsewhere
          • Some additional datasets for validation handled as well
        • Tau-embedding DESD->AOD* almost finished (June 27) but some long tails
        • Various other recovery tasks ongoing now for period A reprocessing, MC prod, Group production https://indico.cern.ch/getFile.py/access?contribId=7&sessionId=0&resId=0&materialId=slides&confId=259107
        • Discussions about improving recovery procedures and automation
        • Delay deletions of intermediate steps for some time during production process
    • ATLAS Computing Twiki
    • Change Default Time Limits for User Analysis
      • Jobs killed automatically if do not update output files within 2-3 hours but does not catch all looping jobs which can waste lots of CPU
      • The maximum values required to sites for the Atlas jobs are defined in the Atlas VO Card: http://operations-portal.egi.eu/vo/view/voname/atlas
        • Max used physical non-swap i386 memory size: 2 GB
        • Max used physical non-swap x86_64 memory size: 2 GB
        • Maximum virtual memory: 4 GB
        • Max size of scratch space used by jobs: 20 GB
        • Max time of job execution: 3120 minutes / 52 hours
        • Job wall clock time limit: 5760 minutes / 96 hours
      • Analyzed Pandamon stats for user jobs from January prun (6.4 Mjobs), pathena (3.4 Mjobs) and ganga (161.7 kjobs).
        • A 99% CL limit for all types of jobs to complete is 11.754 hours
        • A 99.9% CL limit for all types of jobs to complete is 15.22 hours
      • Conclusion: set default at 15 hours. Users can modify this on job submission up to limit of VO card.
      • Discussion on limiting job memory not deemed fruitful due to "spikes" in memory usage at end of jobs sometimes
      • Need to notify users
    • AMOD Report
      • Smooth incident-free interventions on Castor, Oracle production DBs, Bourricot, Tracer/Consistency Service
      • Few T1s (and T2s) DATADISK filling up (FZK in particular)
      • Clouds were running out of assigned tasks during the week - need deeper queues
      • New manual whitelisting policy for SAAB
        • “on” (whitelisting = ignore auto-exclusions) added as savannah site exclusion ticket option
        • dq2-set-location-status documentation for the “on” case added to the CentralizedSiteExclusion twiki
  • CA Cloud Squad Actions [20120626]

  • Security News [20120801]

  • ROC_Canada News [20130625]
    • Current shift - Erming Pei June 24-28, 2013
    • Next Week - Leslie Groer

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View):

-- LeslieGroer - 2013-07-02

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2013-07-02 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback