TWiki> AtlasCanada Web>T2WeeklyReport20110322 (2011-03-22, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2011-03-22

General News

  • ADC and ATLAS Computing News [20110321]
    • ADC Weekly http://indico.cern.ch/conferenceDisplay.py?confId=131784 [20110321]
    • Request to T1's to implement FTS channels discussed at T1SCM
      • Create a "cloud" T1S containing all the T1s+CERN
      • Create a "cloud" T2DS containing all the T2s in the list (3:IT, 4:UK, 3:FR, 4:DE, 3:ES, 5:US)
      • Expect the list to stay stable for 3 months
    • Implement multi-hop transfers in DDM rather than in DaTRI
    • HI reprocessing (250TB, 44 runs, 85% done, merging underway) - awaiting validation of first run before starting derived dataset replication
    • Muon reprocessing ready for validation (0.2 PB) and started at some sites (note to T1's: one reco job per file)
    • CERN LSF overwhelmed by single non-ATLAS user submitting 180k jobs
    • CERN deletion too slow (1 Hz) due to new SRM - fixed
    • Slow AFS response believed to be due to a single server - fixed
    • Tier-3 technical and policy document to be released this week.
      https://twiki.cern.ch/twiki/bin/view/Atlas/AtlasTier3
      • min requirements (recommendation really): 25-50 cores, 10-50 TB, 0.1-1 Gbps, technical/physicist support
      • No signed agreements/MOUs (unless in LHCONE). Site is officially an ATLAS T3 when ADC confirms the status, and ICB is informed on inclusion
      • T3MON - [http://cdsweb.cern.ch/record/1333701][ATL-COM-SOFT-2011-005]]
      • Currently 26 sites in 7 clouds (none in CA, mostly Europe and USA), 18 with Panda queues
      • Comment: ATLAS computing model is quite rigid and oriented toward static T1/T2 sites. T3 configuration is non trivial.
  • CA Cloud Squad Actions [20110322]
    • DaTRI requests approved / processed (last week)
      • TRIUMF-LCG2_LOCALGROUPDISK: 3 / 1 GB / 1 GB
      • TRIUMF-LCG2-SCRATCHDISK: 0 / 0 GB / 0 GB
      • Australia-ATLAS_LOCALGROUPDISK: 407 / 2600 / 2652 GB
      • CA-ALBERTA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 / 0 GB
      • CA-SCINET-T2_LOCALGROUPDISK: 0 / 0 / 0 GB
      • CA-SCINET-T2_SCRATCHDISK: 0 / 0 / 0 GB
      • CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK: 5 / 125 GB / 125 GB
      • SFU-LCG2_LOCALGROUPDISK: 4 / 350 GB / 350 GB
    • Declared Downtimes
      • TRIUMF-LCG2/ SCHEDULED downtime (OUTAGE) [19-03-2011 15:00 to 20-03-2011 03:00 UTC ] - Power outage in the building of TRIUMF Tier 1 center
    • ATLAS Site Exclusion Savannah entries [20110322]
      • HammerCloud autoexclusions
        • NONE * other cloud squad related savannah and / or ROC ggus tickets [20110322]
      • !PD2P (dynamic data placement) in CA cloud [20110322]
      • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
pd2p_ca_20110322.png

  • Security News [20110322]
    • Nothing new

  • gLite News [20110322]
    • [20110315] UPDATE 25 for gLite 3.2 - Priority HIGH Broadcast 334
      • Highlights of the update are new versions of glite-BDII_site and glite-BDII_top

  • ROC_Canada News [20110322]
    • Current shift - Leslie Groer Mar 21-25
    • Next Week - Neil Knecht Mar 28 - Apr 1

  • Squid Deployment [20110322]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [20110322]
      • CA-ALBERTA-WESTGRID-T2 OK (OK for local squid; OK backup to SFU)
      • CA-SCINET-T2: (ce1) WARN (ce2) OK (both for local squid and backup to SFU)
      • CA-VICTORIA-WESTGRID-T2 OK (OK for local squid; OK for backup to CA-ALBERTA-WESTGRID-T2)
      • SFU-LCG2 OK (both CEs; both for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • ATLAS_AUSTRALIA - OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)

  • General Computing News

SAM Results

  • site service status via Gridview - Please check your site

  • March 16-22 2011

    sam_stats_au_20110322.png
    sam_stats_alberta_20110322.png
    sam_stats_mcgill_20110322.png
    sam_stats_scinet_20110322.png
    sam_stats_sfu_20110322.png
    sam_stats_vic_20110322.png
    sam_stats_triumf_20110322.png

APEL Accounting

APEL Publishing Status


apel_publish_au_20110322.png
apel_publish_ca_20110322.png

APEL Accounting Results


apel_stats_au_20110322.png
apel_stats_ca_20110322.png

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • Occasionally some jobs fail to stage-in .gpg files via scp and have to be terminated. (20100928)
    • 201102 - It seems that these jobs are meant to be cancelled, but the lcg-CE, as well as the Cream CE, fail to cancel them.
    • 20110308 - The reason that the jobs were never cancelled successfully is a Torque incompatibility with the High Availability feature of Moab. qdel would fail whenever moab02 is the active server, if the full job id is used instead of the short job id. It's because the full job id ends with the address of the moab01 server and torque thinks it should therefore try to contact that server instead of the active one.

Toronto

Recent Issues

  • [20110321] Main 10G SciNet to campus fibre route changed to new campus Cisco ASR9K gateway. No issues seen with routing on campus or to R&E or regular internet
  • [20110322] BGP routing changed at TRIUMF to accept UofT Autonomous System number AS 239
  • VLAN tagging at datacentre still needs to be reconfigured

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • [20110307] APEL publishing failing to update information. Auto-yum update of lcg-dgas-tools-1.0.0-2.sl4.i386 to lcg-dgas-tools.i386 1.0.0-4.slc4 on lcg-ce1 on Feb 17 broke the creation of grid-jobmap_YYYYMMDD records.
    • Still issues on CE1 - APEL records updated by hand using gatekeeper information [20110315]
    • [20110322] lcg-ce1 records are being scraped manually from /var/log/messages and gatekeeper information for the moment. lcg-ce1 needs to be reconfigured (YAIM) but will likely break things.

McGill

Recent Issues

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View):
panglia_world_prod_20110322.pngpanglia_world_ana_20110322.png
panglia_ca_prod_20110322.pngpanglia_ca_ana_20110322.png 


Time interval:   2011-03-15 18:00:00   to   2011-03-22 00:00:00
          (finished/failed)

Worldwide Production by Cloud:
   US (673292/45662)
   DE (269590/22173)
   FR (254053/30243)
   UK (179144/22282)
   ES (110590/10557)
   CERN (109413/10393)
   NL (102049/6521)
   CA (72221/24633)
   IT (66051/11811)
   ND (68313/6584)
   TW (58872/5503)

Production:
   TRIUMF (14391/19607)
   CA-SCINET-T2 (9649/131)
   CA-VICTORIA-WESTGRID-T2 (3286/148)
   SFU-LCG2 (2947/23)
   CA-ALBERTA-WESTGRID-T2 (1949/5)
   Australia-ATLAS (629/31)

Analysis:
   ANALY_TRIUMF (12131/900)
   ANALY_SCINET (11335/1005)
   ANALY_ALBERTA-WG1 (4521/326)
   ANALY_SFU (4456/312)
   ANALY_SFU_bugaboo (2539/1726)
   ANALY_AUSTRALIA (2499/273)
   ANALY_VICTORIA-WG1 (1889/146)

Failed Jobs

  • From Panda Monitor (last day)
You can find the meanings of the transform error codes on this link.

  • My Notes:

  • Highlight's of Yuri's notes: (Mar 15-22, 2011) 

A)  During the past week (Mar.15-21,2011) Panda production service
   - completed 1,169,831 managed group, MC production, validation and reprocessing jobs
   - average ~167,119 jobs per day
   - failed 85,033 jobs
   - average efficiency:
      -- jobs     - 93.2%
      -- walltime - 96.4%
   - active tasks from Panda-monitor: 836
      -- distribution by cloud:
     CA:33 CERN:14 DE:152 ES:34 FR:261 IT:23 ND:37 NL:30 TW:6 UK:96 US:150

B) - CA/TRIUMF downtime (CE,SE,SRM,BDII,FTS,Oracle DBS OUTAGE),Mar.19,20. 
    - new DB Release 14.5.1.3,14.7.1 subscription done, Mar.15,18.
    - data11 T1-T1 DS transfer started, Mar.15.
    - IT: T3s (3 sites) were added to the FT, Mar.15.
    - CERN: migration of SLS monitoring for Site Services done, Mar.15.
    - CERN: short DT for adding new DB schemes to ATONR,ATLR, Mar.15.
    - new LCG 60b has been released, Mar.15.
    - HI reprocessing campaign in progress:
     -- reconstruction is ~85% done,
     -- merging is ongoing,
     -- more details in Sasha's talk at ADC weekly:
     https://indico.cern.ch/conferenceDisplay.py?confId=131784

Sat Mar.19 
- CA/TRIUMF-LCG2_MCTAPE huge backlog of staging requests.Elog 23369,23383. 

Sun Mar. 20
+ CA/TRIUMF-LCG2 All FT f./DEST.:CONN.:SRM.After 1h of SD ended.GGUS 68797
  verified:power outage was much longer than expected,services recovered.

-- LeslieGroer - 2011-03-22
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel_publish_au_20110322.png r1 manage 10.2 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng apel_publish_ca_20110322.png r1 manage 20.2 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng apel_stats_au_20110322.png r1 manage 55.2 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng apel_stats_ca_20110322.png r1 manage 89.0 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng panglia_ca_ana_20110322.png r1 manage 42.1 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng panglia_ca_prod_20110322.png r1 manage 36.0 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng panglia_world_ana_20110322.png r1 manage 48.4 K 2011-03-22 - 17:44 LeslieGroer  
PNGpng panglia_world_prod_20110322.png r1 manage 45.1 K 2011-03-22 - 17:17 LeslieGroer  
PNGpng pd2p_ca_20110322.png r1 manage 34.2 K 2011-03-22 - 17:19 LeslieGroer  
JPEGjpg reprocessing_20110321.pdf_-_Adobe_Acrobat_Pro.jpg r1 manage 23.8 K 2011-03-22 - 03:36 LeslieGroer  
PNGpng sam_stats_alberta_20110322.png r1 manage 7.0 K 2011-03-22 - 17:19 LeslieGroer  
PNGpng sam_stats_au_20110322.png r1 manage 6.9 K 2011-03-22 - 17:19 LeslieGroer  
PNGpng sam_stats_mcgill_20110322.png r1 manage 6.9 K 2011-03-22 - 17:19 LeslieGroer  
PNGpng sam_stats_scinet_20110322.png r1 manage 6.8 K 2011-03-22 - 17:19 LeslieGroer  
PNGpng sam_stats_sfu_20110322.png r1 manage 6.9 K 2011-03-22 - 17:19 LeslieGroer  
PNGpng sam_stats_triumf_20110322.png r1 manage 6.8 K 2011-03-22 - 17:19 LeslieGroer  
PNGpng sam_stats_vic_20110322.png r1 manage 6.9 K 2011-03-22 - 17:19 LeslieGroer  
Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2011-03-22 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback