TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20130723 (2013-07-23, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2013-07-23

General News

  • ADC and ATLAS Computing News [20130723]
    • ADC Weekly meeting
    • DDM Operations
      • Few T1's filling up DATADISK (FZK, IN2P3, INFN, SARA)
      • CREM Decisions
        • Reduce the number of primary AOD replicas in Tier-1s from 2 to 1
        • Manual migration of AODs from SARA to NIKHEF
        • DE cloudconfig.mcshare is reduced to 1
        • Reduce the (automatic) lifetime of group intermediate datasets from 2 months to 1 month.
      • Near Future
        • NTUP_COMMON is being produced - estimated size: 900TB
        • Currently: 300TB of primaries and 100TB of default replicas
        • Overall capacity is available but not evenly spread over all T1s
        • Available actions: rebalance AODs/ESDs, stop production at T1s
      • Lost files recovery
        • Done by recovery service - http://bourricot.cern.ch/dq2/recovery/
        • Not optimized for big disasters - took ~month to process all necessary files from 600k BNL file loss
          • 94k not known to DQ2 (dark data)
          • 118k files recovered
          • 13k files lost in transient datasets (removed from system before recovery)
          • 391 files definitely lost and removed from dataset definition
        • The pressure showed some weak points in the recovery service
        • No significant development effort planned for current system
        • Rucio will contain different recovery service to address all these issues
      • Unfrozen datasets
        • From time to time DDM ops is asked to unfreeze a dataset (e.g. lost/corrupted file in the dataset, unfreezing a dataset results in new version of the dataset)
        • Problems
          • Records in AMI need a manual fix
          • Subscriptions need to be reset
          • Existing replicas stay complete but with the old version of the dataset
            • breaks the replication policy
          • Datasets are not re-frozen again
            • currently 23 such datasets (the reason is being investigated)
    • ADC Operations discussion on large scale lost files
      • Requests for reproducing lost files
      • 65 AOD/DESD and 48 NTUP files requested to be reproduced.
      • Discussed 5 different ways to do this
        • Reproduce only the files
          • a) the "old" way with the original tasks
          • b) the "new" way with new tasks
          • c) the "near-future" way with JEDI-alpha
        • Reproduce the whole dataset
          • d) redefine the task with the same configuration
          • e) redefine the task with a new tag
        • Details in report about issues and concerns with each approach
        • How to proceed?
          • JEDI-alpha (c) looks promising, but not in production yet
            • Assumes we can wait
          • AOD/DESD reproduction
            • Reproducing the whole data (d) or (e) may be too much, esp. for period-A
            • new task (b) would not work for subsequent tasks
            • period-A (5 files) with the “old” way (a)?
            • HadDelayed with a new tag (e)? or is it feasible with (a) the “old” way?
          • NTUP reproduction
            • reproducing the whole data (e) would not take too long
            • would replacing the old (incomplete) datasets with the new (complete) datasets which could confuse the analysis?
            • or (b) the “new” way, if the files with “new look” acceptable?
    • AMOD Report
      • Massive PFT (among other production) job failures running at SL6 sites in the US
        • Also in FR, UK, ES
        • Production sites were auto excluded
        • CMTUSERCONTEXT set by Alessandro De Salvo to have all sl6 releases compatible.
        • For 2 releases variables problem -> rolled back
      • US T2s PRODDISK deletion reenabled Friday
    • FTS3 Deployment Statuts and Plans
      • RAL FTS3 manage transfers (i.e. not only Functional Tests) to eight volunteer sites (UK, IL) since June 10
      • Been using RAL FTS3 for UK Functional Tests for several months
      • CERN FTS3 pilot instance used for proto-T1 RRC-KI-T1, now also IN2P3-LAPP
      • http://atlas-agis.cern.ch/agis/ddm_endpoint/table_view/?&state=ACTIVE&fts=FTS3
      • Internal FTS3 monitoring
      • WLCG Transfer Dashboard - http://dashb-wlcg-transfers.cern.ch/ui/ (very similar to the DDM Dashboard)
      • Currently using FTS3 in FTS2-like way, but:
        • FTS3 is channelless -> slow links should not affect now the other transfers!
        • FTS3 has an optimizer to push files as much as possible or to step back in case of issues
        • Feedback from sites can be very useful!
      • Some patches going in for FullSURL
      • Discussing the possibility of having FTS3 clients set links/storage configuration in case of need (would switch off the optimizer)
      • Propose to keep on adding sites to the RAL FTS3 instance and monitor - can we add a full cloud?
      • CERN FTS3 production instance could be used soon for “few” DDMEndpoints (except if RAL FTS3 instance has too much load, in this case we may just use it “in parallel” with RAL)
        • This is needed also to test the behavior of the optimizer if sites are served by separate instances:
        • Possible test would be to have part of a site (e.g. CERN-PROD_DATADISK) served by this instance, and the rest served by RAL FTS3
      • Propose to keep CERN FTS3 pilot instance as flexible as possible
        • Useful to quickly study peculiar behaviors if needed, test updates
        • Move the sites presently served by this instance to RAL FTS3
  • CA Cloud Squad Actions [20120626]

  • Security News [20120801]

  • ROC_Canada News [20130625]
    • Current shift - Erming Pei June 24-28, 2013
    • Next Week - Leslie Groer

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View):
-- LeslieGroer - 2013-07-23
Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r2 - 2013-07-23 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback