Tier1 and Tier-2 Weekly Report 2010-08-03

General News

• ADC News [20100803]
  • This Week
    • ADC Weekly [20100802]
      • http://indico.cern.ch/conferenceDisplay.py?confId=102919
    • New AGIS downtime calendar
      • http://atlas-agis.cern.ch/agis/calendar/?print
  • Last Week
    • WLCG Management Board [20100727]
      • http://indico.cern.ch/conferenceDisplay.py?confId=82000

• CA Cloud Squad Actions [20100803]
  • DaTRI requests for approval
    • AUSTRALIA-ATLAS_LOCALGROUPDISK (1.8 TB) http://panda.cern.ch:25980/server/pandamon/query?mode=ddm_req&reqid=13762&action=Info [ACTION =approved] [2010090802]
  • updated values for maxinputsize for CA-ALBERTA-WESTGRID-T2, CA-VICTORIA-WESTGRID-T2 and TRIUMF-LCG2 within panda schedconfig [20100730]
  • ADC e-log entry regarding TRIUMF Frontier server downtime for 2 hours [20100729]
  • initiated query to ADC Operations regarding observed low-level of production activity in the CA cloud [20100727]
    • detailed response and information here.
    • main conclusions/comments from production system experts:
      • "production is tailing down in the [previous] week and perhaps the load assigned to CA was few this time and was quickly processed."
      • "the main reason for empty clouds during the summer has been lack of work (requested tasks)."
  • HammerCloud tests of CA clouds sites is planned to be resumed soon to help measure site performance and identify any site setup bottlenecks
  • Open Issues and Ongoing Discussions:
    • discussion with DDM operations regarding centrally performed space cleaning policy and procedures [20100721]
      • context is the SFU MCDISK blacklisting as discussed in http://savannah.cern.ch/support/?115809
        • new savannah entry about cleaning policy and procedures: http://savannah.cern.ch/bugs/?70746
        • topic is to be discussed at the next DDM meeting (TBA)
    • submitted feature request to DaTRI to allow for disabling of auto-approval for LOCALGROUPDISK transfer requests [20100720]
    • CA-VICTORIA-WESTGRID-T2 site analysis queues marked online by Cloud Squad (20100709)
      • production queue kept in test state pending investigation of fortran runtime error "permission denied" within software release area (20100708) [SOLVED]
        • resolved by changing WN client NFS mounts to v3 from original NFSv4 (root cause within NFSv4 still unknown) [OPEN]
    • ROOTSYS truncated at 127 characters issue [20100608]
      • http://savannah.cern.ch/bugs/?68546 (open)
      • currently affecting Victoria (Tier3) and CA-SCINET-T2
      • varying character lengths below and above 127 characters show variable success and failure
      • related issue: user build jobs failing at CA-SCINET-T2 [20100511]
        • follow-up with site in regards to missing rpm installation; missing rpms added but problem is still occurring [20100518]
    • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • reminder message posted to the Dashboard Savannah entry for this problem https://savannah.cern.ch/bugs/index.php?64507 [20100518] (updated 20100713)
    • reviewing draft Squad How-To and providing feedback [20100407]
      • also contains useful information for sites
    • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
    • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
      • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
    • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • Savannah http://savannah.cern.ch/bugs/?62993 (ticket closed 20100301 but discussion ongoing within Panda team regarding which BDII variables to query)
  • Outstanding Actions to Do:
    • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
      • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.

• ROC_Canada News [20100803]
  • current shift schedule (until Sept 24): https://twiki.atlas-canada.ca/bin/view/AtlasCanada/ROCShifts

• Squid Deployment [20100803]
  • CA cloud Frontier and Squid deployment information (versions and network information)
  • SAM tests: (CA Link), (lAus Link): currently status as of this morning (WARN is ok) [20100720]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108; new installation required; backup to SFU squid passes)
    • CA-SCINET-T2: OK (local squid server online and active for both lcg-ce1 and lcg-ce2; backup to SFU squid)
    • CA-VICTORIA-WESTGRID-T2 OK (both for local at VICTORIA-LCG2 and backup to SFU squid)
    • SFU-LCG2 OK (both CEs; local squid is OK; backup squid at VICTORIA-LCG2 OK)
    • VICTORIA-LCG2 OK (OK for local squid and connection to SFU squid)
    • ATLAS_AUSTRALIA - OK (local squid OK; backup to VICTORIA-LCG2 OK)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)

• Site Action Items:
  • CA-ALBERTA-WESTGRID-T2
    • reinstall local squid server

SAM Results

• ACTION ITEM FOR CA-ALBERTA-WESTGRID-T2: investigate cause of lower measure of availability and reliability for May 2010

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGEE Accounting Portal (Canada)
  • See EGEE Accounting Portal (Australia)

APEL Publishing Status

APEL Accounting Results

Site Reports

Alberta

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • http://savannah.cern.ch/bugs/?68656 DDM Operations: SCRATCHDISK shows DQ2 larger than srm_total on DQ2 accounting (20100610) (in progress)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=59747 gstat 'check-se' problem detected at CA-ALBERTA-WESTGRID-T2 (20100705) (on hold)

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• CA-ALBERTA-WESTGRID-T2 site status [20100720]
  • scheduled downtime completed 2010-07-16 to 2010-07-19 registered in GOCDB for scheduled power interruption to main campus server room which hosts the Tier-2 resources

Outstanding Issues

• CA-ALBERTA-WESTGRID-T2 site status [20100511]
  • DDM: discrepancy between SRM and DQ2 usage (see above savannah)
    • tests on postgres dump of dCache databases pending
  • additional open items being followed-up:
    • need to update to dCache golden release cycle from currently running 1.9.3-3
    • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs
  • network cabling maintenance planned for lighpath connection [postponed from 20100319]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]

SFU

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

Outstanding Issues

Victoria

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• westgrid site router ACLs updated to enable outbound connectivity to SFU and local squids [20100714]
• site squid ACLs update to accept connections from Australia-ATLAS, and corrected for SFU-LCG2 [20100715]

Outstanding Issues

• VICTORIA-LCG2 SCRATCHDISK became full [20091112]
  • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=59744 gstat 'check-ce' and 'check-se' problem detected at CA-SCINET-T2 (20100706) (solved 20100730)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=58997 Many jobs Get error: Failed to get LFC replicas at CA/CA-SCINET-T2 (2010-06-13) (solved 2010-06-13)
    • re-opened and marked unsolved (20100709)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=58116 LFC error at CA/CA-SCINET-T2 (20100510) (unsolved 20100709)
  • http://savannah.cern.ch/bugs/?65337 CA-SCINET-T2 pilot: Job killed by signal 15 [20100405](open)
• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• TORONTO-LCG2 site
  • site offline to ATLAS; currently suspended by CA_ROC in GOCDB
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53830 TORONTO-LCG2 update to lcg-CE (20091203) [on hold]
  • https://gus.fzk.de/ws/ticket_info.php?ticket=53605 TORONTO-LCG2 lcg_cp timeout (20091124) [on hold]

• scheduled downtime for CA-SCINET-T2 [20100723] for 24 hours
  • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19367&grid_id=0

Outstanding Issues

• pilot issues [20100511]
  • significant number of immediately failing pilots for both TRIUMF and LMU factories
  • need to logon to WN directly and run the wrapper script by hand

• dCache error investigations
  • Occassional "There is no file with pnfsid=.." errors (20100430)
    • under discussion with dCache experts at TRIUMF and elsewhere
  • dCache PutUseSpaceCallbacks errors (20100504)

• large number of LFC related errors (20100404)
  • 'Put error: lfc_setfsizeg failed' and 'lfc_statg failed' and 'errstr = Bad magic number'

• SciNet-TRIUMF lightpath network use and optimization [20100413]
  • BGP established on Force 10 router for link to TRIUMF [20100407]
    • Information Request: is a failover test to the ORAN planned? A: Not yet.

• observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
  • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

• discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60675 TRIUMF - job failures indicate a problem with the local pool file catalog (20100729) (verified 20100731)
    • http://savannah.cern.ch/bugs/?70825
  • https://gus.fzk.de/ws/ticket_info.php?ticket=59748 gstat 'check-se' problem detected at TRIUMF-LCG2 (20100705) (on hold 20100706)

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • see "Notes" section below

• scheduled downtimes last week
  • Frontier service [20100729] for 2 hours

• PFC creation time and gaps under investigation following production errors [20100729]
  • solved by subsequent PFC update but larger issue of PFC update timing needs attention
    • https://gus.fzk.de/ws/ticket_info.php?ticket=60675
    • http://savannah.cern.ch/bugs/?70825

Outstanding Issues

Australia-ATLAS

Recent Issues

• summary of this weeks Savannah bugs and GGUS tickets for site:
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60172 No Nagios test results for agh2.atlas.unimelb.edu.au (20100601) (solved 20100730)
  • https://gus.fzk.de/ws/ticket_info.php?ticket=60783 org.sam.CREAMCE-JobSubmit-/ops/Role=lcgadmin is failing on : agh5.atlas.unimelb.edu.au (20100803) (solved 20100803)

• production job issues
  • See "Production" Activities section below

• analysis job issues
  • See "Notes" section below

• scheduled downtimes this last week for e1000 driver fix and other firmware upgrades [12 hours]
  • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19370&grid_id=0
  • https://next.gocdb.eu/portal/index.php?Page_Type=View_Object&object_id=19371&grid_id=0

Outstanding Issues

• identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault
  • causing issues with regards to release 15.5.1 testing. A modified setup for pilot users added to profile.d to get around this in the interim
  • bugzilla report: https://bugzilla.redhat.com/show_bug.cgi?id=563759

McGill

Recent Issues

• the technical information and requirements in the issued RFP are available upon request; please contact Bryan who has a copy forwarded from Simon [20100720]

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

• Within the CA Cloud:
  

• Within the None Cloud (see https://savannah.cern.ch/bugs/index.php?64507)
  

• Within the CA Cloud (Panglia View):
  

• How we fared to other ATLAS sites during the past week:
  

• Alternate Views (See link for last 3 days only):
  

• CA Cloud (last 3 days):
  

Failed Jobs

• From Panda Monitor (last week):
  

• From Dashboard (last week):

• • TRIUMF-LCG2

name,rank,value
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,1,3580
TRF_UNKNOWN ,2,284
EXEPANDA_DQ2_STAGEIN,3,16
TRFERROR,4,14
TRF_SEGFAULT ,5,5
EXEPANDA_GET_FAILEDTOGETLFCREPLICAS,6,1
EXEPANDA_DQ2PUT_FILECOPYERROR,7,1

• • CA-ALBERTA-WESTGRID-T2

name,rank,value
TRFERROR,1,4
TRF_SEGFAULT ,2,3
EXEPANDA_GET_SIZEMISMATCH,3,2

*

• • Australia-ATLAS

name,rank,value
EXEPANDA_GET_GLOBUSSYSTEMERROR,1,11
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,2,6
EXEPANDA_DQ2PUT_FILECOPYERROR,3,5

• • SFU-LCG2

name,rank,value
TRFERROR,1,2
TRF_SEGFAULT ,2,1

You can find the meanings of the transform error codes on this link.

• My Notes:
  • information missing in dashboard (production) still for CA-SCINET-T2.
    • 3rd reminder sent to Dashboard Savannah (20100615) and dashboard-support@cern.ch
    • previous reminder: sent to Dashboard Savannah (20100518)
      • https://savannah.cern.ch/bugs/index.php?64507

• Yuri's notes: week (July 27 - Aug 2)

   - active tasks from PandaDB: 427
      -- distribution by cloud:
         CA:8 CERN:2 DE:68 ES:9 FR:41 IT:5 ND:26 NL:18 TW:7 UK:33 US:210

1) Tue Jul.27 
   -> Cloud/site DDM  efficiency <50% : CA

2) Wed Jul.28
   -> Cloud/site PROD efficiency ever<50% :CA,TW

3) Thu Jul.29
   -> Cloud/site PROD efficiency ever<50% : DE/FZK,CA/TRIUMF,US/BU,FR,UK
 + CA/TRIUMF t.153976 job f./poolToObject:caught er.GGUS 60675 verified:
   PFC update glitch,refreshed.Savannah 70825.

4) Fri Jul.30
   -> Cloud/site PROD efficiency ever<50% : DE,CA,FR,UK,ES

5) Sat Jul.31
   -> Cloud/site PROD efficiency ever<50% : DE,FR,UK,TW,CA

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
    (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
---------------------------------------------------------------------- 

++ 15.6.9.10.1 mc09_7TeV reco merge task 152856 failures:Too many/too
   large input files.Savannah 70725.Jobs failed repeatedly 5 attempts in
   CA due to large size of input ESD files exceeding the max-allowed
   limit.The same problem with t.152867/TW,153711/FR,153947,153901,
   153828,153824,153699,153811,153818,153705,1536961,53700.Tasks ABORTED
   and re-submitted with a different pcache.

• Stats:
  • See Key for what the headers mean

-- BryanCaron - 28 Jul 2010