TWiki> AtlasCanada Web>T2WeeklyReport20110301 (2011-03-01, RyanTaylor) EditAttach

Tier1 and Tier-2 Weekly Report 2011-03-01

General News

  • ADC News [20110301]
  • CA Cloud Squad Actions [20110301]
    • DaTRI requests approved / processed (last week)
      • TRIUMF-LCG2_LOCALGROUPDISK: 0 / 0 GB
      • Australia-ATLAS_LOCALGROUPDISK: 0 / 0 GB
      • CA-ALBERTA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 GB
      • CA-SCINET-T2_LOCALGROUPDISK: 0 / 0 GB
      • CA-VICTORIA-WESTGRID-T2_LOCALGROUPDISK: 0 / 0 GB
      • SFU-LCG2_LOCALGROUPDISK: 0 / 0GB
    • Declared Downtimes
      • CA-VICTORIA-WESTGRID-T2/ SCHEDULED downtime (OUTAGE) [22-02-2011 16:00 to 23-02-2011 23:59 UTC] - cluster-wide downtime for maintenance, firmware updated and testing
      • CA-SCINET-T2/ SCHEDULED downtime (OUTAGE) [24-02-2011 13:30 to 25-02-2011 02:30 UTC]- cooling maintenance. Production queues online Sat Feb 26 06:38 UTC
    • ATLAS Site Exclusion Savannah entries [20110301]
      • HammerCloud autoexclusions
        • ANALY-ALBERTA-WG1 analysis queue set to brokeroff Wed Feb 23 20:00:02 2011 CET - Thu Feb 24 10:00:21 2011 CET- SE disk full
        • ANALY-ALBERTA-WG1 analysis queue set to brokeroff Fri Feb 25 09:00:20 2011 CET- dccp failure. Set online by shifters at Fri Feb 25 16:08 UTC
    • other cloud squad related savannah and / or ROC ggus tickets [20110201]
    • PD2P (dynamic data placement) in CA cloud (20110228)
      • panda monitor for PD2P activity within clouds (default view (last 7 days) units = datasets)
      • note that only DATADISK is now a destination for PD2P [since 20110112]

  • Open Issues and Ongoing Discussions [NOTE: Not up to date but a place holder for this meeting]
    • published site installed resource capacity information to be investigated and confirmed against WLCG information [20100928]
      • http://gstat-wlcg.cern.ch/apps/capacities/sites/
      • UPDATE [20110121] - total published resources at each CA Tier2 is correct; further adjustments needed to publish available (pledged) WLCG resources - info to come
      • UPDATE [20110201] - thank you to sites for updating published values of 'Shares'
        • outstanding items to be corrected:
          • SFU-LCG2 - "total online storage" for both all VOs and ATLAS VO incorrectly set at zero
          • CA-ALBERTA-WESTGRID-T2 - "total online storage" for ATLAS VO incorrectly set at zero
          • All sites to update published cpu resource capacities (PhysicalCPUs and LogicalCPUs) to 2011 NRAC levels in April 2011 to coincide with the next WLCG Pledge period
    • CREAM CE deployment preparations
      • new TWiki page https://twiki.atlas-canada.ca/bin/view/AtlasCanada/CreamCE [20101019]
      • [20110112] the plan is to first do the switch at TRIUMF, sometime after the scheduled downtime that takes place early next week coincident with the ADC outage, followed by a scale-up of the number of WNs at the site that receive jobs from the CREAM CE
    • HammerCloud
    • issue of sites within Cloud None within the production dashboard [20100829]
      • issue raised to ADC operations email list given that there was no ADC Weekly meeting yesterday
      • reply from ADC monitoring coordinator (A. Read) is that this long-standing problem is known and being worked on, but is difficult to fix
      • this problem will be on the agenda for the next ADC monitoring meeting scheduled for next week
    • CA-VICTORIA-WESTGRID-T2 site missing from the ATLAS dashboard SAM Visualization page [20100824]

  • Security News [20110125]
    • REMINDER: High Risk Vulnerability CVE-2010-3859 kernel: heap overflow in tipc_msg_build() [EGI-ADV-20110701]

  • Squid Deployment [20110301]
    • CA cloud Frontier and Squid deployment information (versions and network information)
    • SAM tests: (CA Link), (Aus Link): currently status as of this morning (WARN is ok) [20110301]
      • CA-ALBERTA-WESTGRID-T2 OK (OK for local squid; OK backup to SFU)
      • CA-SCINET-T2: (ce1) WARN (ce2) OK (both for local squid and backup to SFU)
      • CA-VICTORIA-WESTGRID-T2 OK (OK for local squid; OK for backup to CA-ALBERTA-WESTGRID-T2)
      • SFU-LCG2 OK (both CEs; both for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • ATLAS_AUSTRALIA - OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
      • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
      • VICTORIA-LCG2 OK (OK for local squid and backup to CA-VICTORIA-WESTGRID-T2)
  • General Computing News
    • CERN will be blocking accounts that have not passed the online computing security course

SAM Results

  • site service status via Gridview - Please check your site

  • January 2011 (valid until 20110228)
triumf.png austrlia.png alberta.png scinet.png sfu.png vic.png

APEL Accounting

APEL Publishing Status

apel_publishing_ca.png

apel_accounting_au.png

APEL Accounting Results

apel_report_ca.png

apel_report_au.png

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • dCache door network tuning
  • cream CE testing
  • dCache testbed - futher studies regarding dCache upgrade path
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3 - targetting mid-September for upgrade - delayed
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
    • network cabling maintenance planned for lighpath connection [postponed from 20100319]

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • see section above for SAM Results regarding regular ATLAS VO test failures which occur at regular intervals over the month [20110125]
    • ended 20110125

  • published online storage in tables (see above) showing as 0 TBs both for all VOs and ATLAS VO [20110201]
  • remove blackhole.westgrid.ca from nagios monitoring [20110301]

Outstanding Issues

  • cream CE installation and testing started [20100929]
    • test installation (blackhole.westgrid.ca) at first which will subsequently be applied to update the lcg-CEs (snowpatch then bugaboo)

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • CREAM CE debugging ongoing

Outstanding Issues

  • CREAM CE installation to be performed in next few weeks [20101005]
  • Occasionally some jobs fail to stage-in .gpg files via scp and have to be terminated. (20100928)
    • 20101214 - This is believed to be fixed now. The probable cause was LDAP connections being sporadically broken, due to the ldap user account on the LDAP server occasionally hitting the ulimit of 1024 open files. We have not observed the problem for awhile now.
    • 201102 - Actually it seems that these jobs are meant to be cancelled, but the lcg-CE fails to cancel them for some reason.

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • tuning applied to dcap service node to overcome access issues (see above tickets) [20110201]

Outstanding Issues

  • SIR to be completed since service errors occurred in December 2010 for a continuous period that lasted greater than 24 hours [See 20101223 and 20101224]

  • gpfs problems required scheduler shutdown and gpfs restart causing all running jobs to be lost [20101225]
    • SIR requested [20110104]

  • tcp tuning exercise [20101116]
    • corrected routing issue between doors and pools which previously were transferring data to pools over external nics rather than internal network [20101112]
    • improvements in communication between LFC server and site WNs observed
    • retry ENV variables to be added still

  • dCache error investigations
    • Occassional "There is no file with pnfsid=.." errors (20100430)
      • under discussion with dCache experts at TRIUMF and elsewhere
    • dCache PutUseSpaceCallbacks errors (20100504)
    • Q: Are these still an issue and under investigation? (BC) [20101026]

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned? A: Not yet.

TRIUMF

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

Outstanding Issues

  • configured CREAM CE [20100929]
    • under local test and debugging of ENV settings

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud (Panglia View):
week-summary_ca_prod.png

week-summary_ca_ana.png

  • How we fared to other ATLAS sites during the past week:
worldwide_prod.bmp

worldwide_ana.bmp

  • Alternate Views (Source link for last 7 days): 

Time interval:   2011-02-22 00:00:00   to   2011-03-01 00:00:00

cloud:  [FINISHED/FAILED]

   US (841990/116902)
   DE (440293/48533)
   FR (328646/85428)
   UK (269349/49343)
   CERN (189318/49664)
   ND (169436/18865)
   IT (168900/16789)
   NL (168906/14046)
   ES (162472/15556)
   CA (101075/13173)
   TW (86432/11154)


cloud:     CA (101075/13173)

Production site:
       TRIUMF (17881/3340)
        CA-ALBERTA-WESTGRID-T2 (4225/107)
        SFU-LCG2 (4189/17)
        CA-VICTORIA-WESTGRID-T2 (2781/44)
        Australia-ATLAS (2123/44)
        CA-SCINET-T2 (1809/89)

Analysis site:
     ANALY_TRIUMF (30518/4084)
     ANALY_ALBERTA-WG1 (3165/936)
     ANALY_AUSTRALIA (10437/593)
     ANALY_SFU (7602/635)
     ANALY_SCINET (5453/1531)
     ANALY_SFU_bugaboo (4705/1315)
     ANALY_VICTORIA-WG1 (786/255)

Failed Jobs

  • From Panda Monitor (last day)
You can find the meanings of the transform error codes on this link.

Panda_joberrors_all.png

Panda_joberrors_au.png

Panda_joberrors_ca.png

Panda_joberrors_sfu.png

Panda_joberrors_triumf.png

  • My Notes:

  • Highlight's of Yuri's notes: (Feb 22-28, 2011) 
A)  During the past week (Feb.22-28,2011) Panda production service 
   - completed 1,374,503 managed group, MC production, validation and reprocessing jobs 
   - average ~196,358 jobs per day 
   - failed 112,725  jobs 
   - average efficiency: 
      -- jobs     - 92.4% 
      -- walltime - 94.3% 
   - active tasks from Panda-monitor: 238 
      -- distribution by cloud: 
      CA:25 CERN:33 DE:28 ES:13 FR:24 IT:12 ND:22 NL:24 TW:11 UK:18 US:28

B) - new release of the GGUS portal online, Feb.23. 
   - new DQ2 clients testing Release 0.1.36_rc6 announced, Feb.25. 
   - new LCG 60a has been released, Feb.25. 
   - new pilot version SULU 46a released, Feb.28. 
   - new DB Release 14.3.1 subscriptions done: Feb.28.

-- LeslieGroer - 2011-02-25
Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng Panda_joberrors_all.png r1 manage 16.4 K 2011-03-01 - 18:45 LeslieGroer  
PNGpng Panda_joberrors_au.png r1 manage 19.7 K 2011-03-01 - 18:40 LeslieGroer  
PNGpng Panda_joberrors_ca.png r1 manage 77.2 K 2011-03-01 - 18:40 LeslieGroer  
PNGpng Panda_joberrors_sfu.png r1 manage 8.7 K 2011-03-01 - 18:40 LeslieGroer  
PNGpng Panda_joberrors_triumf.png r1 manage 40.2 K 2011-03-01 - 18:40 LeslieGroer  
PNGpng alberta.png r1 manage 15.5 K 2011-03-01 - 00:13 LeslieGroer  
PNGpng apel_accounting_au.png r1 manage 15.6 K 2011-03-01 - 00:36 LeslieGroer  
PNGpng apel_publishing_ca.png r2 r1 manage 33.6 K 2011-03-01 - 22:25 LeslieGroer  
PNGpng apel_report_au.png r1 manage 33.6 K 2011-03-01 - 00:36 LeslieGroer  
PNGpng apel_report_ca.png r2 r1 manage 91.5 K 2011-03-01 - 22:21 LeslieGroer  
PNGpng australia.png r1 manage 15.4 K 2011-03-01 - 00:13 LeslieGroer  
PNGpng pd2pd.png r1 manage 27.4 K 2011-02-28 - 23:51 LeslieGroer PD2P week
PNGpng scinet.png r1 manage 15.5 K 2011-03-01 - 00:13 LeslieGroer  
PNGpng sfu.png r1 manage 15.8 K 2011-03-01 - 00:13 LeslieGroer  
PNGpng triumf.png r1 manage 15.3 K 2011-03-01 - 00:13 LeslieGroer  
PNGpng vic.png r1 manage 15.5 K 2011-03-01 - 00:13 LeslieGroer  
PNGpng week-summary_ca_ana.png r2 r1 manage 43.3 K 2011-03-01 - 03:36 LeslieGroer  
PNGpng week-summary_ca_prod.png r2 r1 manage 36.8 K 2011-03-01 - 03:36 LeslieGroer  
Bitmapbmp worldwide_ana.bmp r1 manage 442.4 K 2011-03-01 - 03:29 LeslieGroer  
Bitmapbmp worldwide_prod.bmp r1 manage 442.4 K 2011-03-01 - 03:29 LeslieGroer  
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r8 - 2011-03-01 - RyanTaylor
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback