TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20100427 (2010-04-27, BryanCaron) EditAttach

Tier1 and Tier-2 Weekly Report 2010-04-27 [In Progress]

General News

  • CA Cloud Squad Actions [20100427]
    • CA-SCINET-T2 marked online in Panda [20100413]
      • follow-up action to initially failing PoolCondPFC installations [20100410] (see SciNet site report below)
    • CA-ALBERTA-WESTGRID-T2 marked online in Panda [20100415]
      • manual blacklist in gLite WMS removed [20100415]
    • DaTRI dataset replication request approvals [20100420] http://panda.cern.ch:25980/server/pandamon/query?mode=ddm_req&filter=on&cloud=CA&site=ALL&dpat=*&status=ANY&period=30&reqid=&action=List
      • CA-ALBERTA-WESTGRID-T2_MCDISK - 7.4 GB (for GangaRobot input)
      • CA-SCINET-T2_LOCALGROUPDISK - approx. 21 TB (for JES systematic error studies)
    • updated ADC Site Exclusion Savannah with cloud site status changes
      • announced SFU-LCG2 downtimes [20100418]
      • ALBERTA-LCG2 suspended by CA_ROC until further notice pending migration to T3 [20100415]
    • schedconfig updates
      • submitted request to update maxinputsize for TRIUMF-LCG2 following the site update of Dell chassis WNs with additional drives [20100408]
    • reviewing draft Squad How-To and providing feedback [20100407]
      • also contains useful information for sites
    • Open Issues and Ongoing Discussions:
      • http://savannah.cern.ch/bugs/?65355 TRIUMF-LCG2 Transfer failures; source file does not exist [20100406] (see TRIUMF site report below and ggus 56849)
      • ongoing discussions with ADC experts regarding regional production activity and how to accommodate it cleanly within both the production system and the CA cloud and site operations [20100316]
        • dedicated technical meeting during S&C week scheduled [20100421]
      • running out of jobs at TRIUMF (and other CA cloud sites other than CA-SCINET-T2) [20100402]
        • "The problem may be due to the jet/group production task. Since these jobs have high priority, they are being sent to CA cloud, to the exclusion of lower prio jobs. Of course, they can only run at one site. So, other sites are running empty. Unanticipated side effect of running the task only at CA-SCINET-T2! Let me think how to solve it." K. De (ADC)
      • need to follow-up with the schedconfig team as to why the default values of 14 GB were restored in the schedconfig settings given that the updated values were requested and implemented on Feb19 at the request of the cloud contacts
      • escalate query to ATLAS dashboard support regarding missing CA-SCINET-T2 site in the ARDA dashboard production activity display [20100216]
      • request submitted to Panda team for functionality to set site Panda queues offline automatically based upon scheduled outages/availability [20100216]
      • follow-up discussion ongoing with panda team after the SL4 to SL5 queue migration for TRIUMF [20100125]
    • Outstanding Actions to Do:
      • follow-up information system publishing by dCache sites and use of FileStager within GangaRobot [20100111]
        • ie. dcap missing from TRIUMF and CA-ALBERTA-WESTGRID-T2 and so uses lcg-cp.
      • site information sanity checks
        • sites will be contacted in regards to their published values of logicalCPU, physicalCPU, HEP-SPEC and kSI2K from their site BDIIs show abnormalities in terms of the HEPSPEC number advertisement, sites that haven't run the benchmark yet should do so for their resources and add it to the site BDII.
        • As of today this has only been done for TRIUMF, SFU, CA-ALBERTA-WESTGRID-T2 and AUSTRALIA-ATLAS.

  • Squid Deployment [20100427]
    • new squid rpm release announced [20100426] - See link
    • SAM test: https://lcg-sam.cern.ch:8443/sam/sam.py?sensors=CE&regions=ROC_Canada&vo=atlas&order=SiteName&funct=ShowSensorTests, also within GangaRobot
      • currently status as of this morning (WARN is ok) [20100427]
    • CA-ALBERTA-WESTGRID-T2 WARN (local squid server offline since 20091108)
    • CA-SCINET-T2: OK (local squid server online and active)
    • SFU-LCG2 WARN (for both CEs; local squid is OK; backup squid at Victoria fails)
    • TRIUMF-LCG2 OK (ce1) OK (ce2) (local Frontier launchpad and BNL as backup)
    • VICTORIA-LCG2 WARN (new PoolCondPFC installed to pickup the Frontier configuration; local Squid installation still shows errors but backup is OK)
    • ATLAS_AUSTRALIA - WARN (local squid ok; backup access to VICTORIA error)

SAM Results

APEL Accounting

APEL Publishing Status

  • apel-status-ca-20100427.png:
    apel-status-ca-20100427.png

  • apel-status-aus-20100427.png:
    apel-status-aus-20100427.png

APEL Accounting Results

  • apel-statistics-ca-20100427.png:
    apel-statistics-ca-20100427.png

  • apel-statistics-aus-20100427.png:
    apel-statistics-aus-20100427.png

Site Reports

Alberta

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • ALBERTA-LCG2 site
    • site offline to ATLAS since early December 2009
    • currently in state of scheduled downtime in GOCDB until 20100517 and subsequently suspended by CA_ROC until site re-certification occurs

  • CA-ALBERTA-WESTGRID-T2 site status [20100420]
    • update of site node host certificates during unscheduled downtime of a few hours [20100414]
      • queried GridCanada CA in advance in order to coordinate release of CRLs but received no reply, and so simply proceeded with new certificate requests and hoped for the best (!)
    • ATLAS software release installation and validation [started 20100404] completed on 20100411
      • to overcome failing PoolCondDB installations had to remove stale file accesses from within dCache before the PoolCondPFC install could succeed
    • open items being followed-up:
      • need to update to dCache golden release cycle from currently running 1.9.3-3
      • performance of fairshare trees configuration to be studied with WestGrid site admins in regards to priorities, non-NRAC allocations and effect upon priority of ATLAS jobs

Outstanding Issues

  • daily production and analysis report job error reports to be caught-up with [starting 20100302]
  • network cabling maintenance planned for lighpath connection [postponed from 20100319]
  • squid server (vm) offline since restart to CA-ALBERTA-WESTGRID-T2 services following scheduled downtime for power maintenance [20091108]
    • restart errors under investigation; automatic fail-over to secondary squid at SFU functioning as expected
    • ACTION ITEM: confirm Squid ACLs are configured to accept CA-SCINET-T2 network

SFU

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • PoolCondPFC failing at snowpatch [20100416]
    • resolved by central facility and not a site problem
    • comment from Sergey: "after this change in the central facility DQ2 siteTests jobs started succeeding on snowpatch for the first time after the upgrade on
March 8."

Outstanding Issues

  • identified one chassis of nodes where IB performance was degraded [20100105]
    • nodes in chassis marked offline for repair by vendor - status?

Victoria

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • software tags removed for releases pre-15.6.3 at the site due to incompatibilities with SL5 32-bit WNs [20100416]
    • software release cleanup to have started 20100419

  • ADCoS shifter Savannah bug ticket 65596 issued [20100410]
    • Too little space left on local disk to run job
    • investigating to restrict the maximum numer of production and/or analysis jobs on the 4 cpu blades which share 29 GB of local disk space
    • solution being implemented and tested by D. Leske [20100416]

Outstanding Issues

  • SCRATCHDISK became full [20091112]
    • action to Bryan to write short srm script to generate warning notifications to sites when token areas approach becoming full

Toronto

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • TORONTO-LCG2 site
    • site offline to ATLAS; currently suspended by CA_ROC in GOCDB

  • spacetokens update request [20100330] and updated [20100409]
      • DATADISK 115 TB, MCDISK 115 TB, PRODDISK 15 TB, SCRATCHDISK 15 TB, GROUPDISK 25 TB, LOCALGROUPDISK 50 TB, HOTDISK 1 TB

  • SciNet-TRIUMF lightpath network use and optimization [20100413]
    • BGP established on Force 10 router for link to TRIUMF [20100407]
      • Information Request: is a failover test to the ORAN planned?
    • Canarie update to SciNet-TRIUMF lightpath link to remove cap at 600 Mbps [20100409]
    • network utilization increased above previously seen limit of 800 Mbps following TRIUMF FTS setting increase to number of concurrent files to 25 on channel
    • SciNet network monitoring:TRIUMF%20connection)(64-bit)" target="_blank">(link)

  • installed 2nd CE (lcg-ce2) sharing software area with lcg-ce1 [20100419]
    • configured and published new CE information into site-BDII which was automatically pickedup by the ATLAS software installation system
    • initial incorrect link to shared software area meant that software installation attempts began
    • temporarily disabled lcg-ce2 in installation system (J. Steele)
    • node currently taken offline by site

Outstanding Issues

  • observed high rate of job failures for pilots submitted via CERN factory compared to those from LMU and TRIUMF [20100322]
    • different network path observed from CERN to Scinet compared to from LMU. Pilots from CERN halted pending investigation

  • discussion with pilot factory experts ongoing regarding interplay between condorg and globus gma, and local adjustments to correct the 33% of all pilots which fail on WNs * "submit-helper script running on host gpc-f108n039 gave error: cache_export_dir (/localstorage/lcg/home/prdatl83/.lcgjm/globus-cache-export.W29972) on gatekeeper did not contain a cache_export_dir.tar archive"

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • see "Notes" section below

  • see tickets above for details

Outstanding Issues

Australia-ATLAS

Recent Issues

  • production job issues
    • See "Production" Activities section below

  • analysis job issues
    • See "Notes" section below

  • agh2 (lcg-CE) versus agh5 (cream CE) [20100419]
    • differences observed in the published release tags on ahg2 compared to agh5 even though tags are replicated and all datasets copied locally to disk.
    • under continuing investigation

Outstanding Issues

  • identified bug in SL53/54 x86_64 where a combination of specific LD_PRELOAD and LD_LIBRARY_PATH can cause core binaries (ls, rm etc.) to segfault

McGill

Recent Issues

  • status?

Production

Summary

CA Cloud Statistics Reference Link (Last Week) Overall Statistics Reference Link (Last Week)

  • Within the CA Cloud:
    dashboard-std-summary-all-20100427.png

  • How we fared to other ATLAS sites during the past week:
    dashboard-std-summary-ca-20100427.png

  • Alternate Views (See link for last 3 days only):
    dashboard-summary-all-20100427.png
  • CA Cloud (last 3 days):
    dashboard-summary-ca-20100427.png

Failed Jobs

  • From Panda Monitor: TBA

  • From Dashboard: 

name,rank,value
EXEPANDA_Release_MISSING,1,249
EXEPANDA_DQ2GET_FILE-SIZE-EXCEEDS-LIMIT,2,229
TRF_UNKNOWN ,3,210
EXEPANDA_DQ2PUT_FILECOPYERROR,4,155
TRFERROR,5,135
TRF_SEGFAULT ,6,121
EXEPANDA_DQ2_STAGEIN,7,100
EXEPANDA_DQ2PUT_FILECOPYTIMEOUT,8,79
EXEPANDA_DQ2_SERVERERROR,9,19
EXEPANDA_JOBKILL_BYPILOT,10,9
EXEPANDA_JOBDISPATCHER_HEARTBEAT,11,6
EXEPANDA_GET_SIZEMISMATCH,12,5
EXEPANDA_UNKNOWN_CATCHALL,13,4
EXEPANDA_RUNJOBEXCEPTION,14,2


name,rank,value
EXEPANDA_DQ2PUT_FILECOPYERROR,1,42
TRF_SEGFAULT ,2,37
TRFERROR,3,22
EXEPANDA_DQ2GET_INFILE,4,10
TRF_UNKNOWN ,5,6
EXEPANDA_PUT_GLOBUSSYSTEMERROR,6,1

    • VICTORIA-LCG2 
name,rank,value
EXEPANDA_PILOTEXCEPTION,1,190
EXEPANDA_JOBDISPATCHER_HEARTBEAT,2,13
TRF_SEGFAULT ,3,5
EXEPANDA_DQ2GET_CANTCALL,4,4
EXEPANDA_JOBKILL_NOLOCALSPACE,5,3
TRFERROR,6,2

    • CA-ALBERTA-WESTGRID-T2 
name,rank,value
TRF_SEGFAULT ,1,64
TRFERROR,2,23
TRF_UNKNOWN ,3,17
EXEPANDA_GET_SIZEMISMATCH,4,10
EXEPANDA_JOBKILL_BYPILOT,5,7
EXEPANDA_DQ2GET_INFILE,6,4
EXEPANDA_JOBDISPATCHER_HEARTBEAT,7,4
EXEPANDA_DQ2PUT_FILECOPYERROR,8,3
EXEPANDA_PILOTEXCEPTION,9,1

    • Australia-ATLAS 
name,rank,value
TRFERROR,1,29
EXEPANDA_DQ2PUT_FILECOPYERROR,2,2

You can find the meanings of the transform error codes on this link.

  • My Notes:
    • information missing in dashboard (production) still for CA-SCINET-T2.

  • Yuri's notes:
    • week April 20-26, 2010
      • active tasks from PandaDB: 849
        • distribution by cloud: CA: 53 CERN: 3 DE: 148 ES: 17 FR: 249 IT: 13 ND: 58 NL: 88 TW: 5 UK: 28 US: 187 


1) Tue Apr.20
   -> Cloud/site PROD efficiency < 50% : CERN,CA,NL
   -> Cloud/site DDM  efficiency < 50% : -
     ........................................
 - CA/VICTORIA-LCG2 many job f./pilot:No module named zlib.GGUS 57458 
   waiting for reply:set offline.Change setokens=ATLASPRODDISK done in 
   schedconfig.Test jobs submitted Apr.26. 
 - CA/TRIUMF still FT f./USER_ER./source file doesn't exist.GGUS 57455,
   56849 in progress:more missing files,deletion performed via external 
   srmRm call.List of missing files in Savannah 65122

2) Wed Apr.21
 + CA/TRIUMF-LCG2 FT f./HTTP_TIMEOUT/SRM.GGUS 57507 verified:corrected 
   disk space issue on srm head node which cascaded to problems elsewhere

3) Thu Apr.22
   -> Cloud/site DDM  efficiency < 50% :IT/INFN-T1,FR/LAPP,CA/VICTORIA,
                                        UK/UKI-LT2-QMUL/UKI-LT2-UCL-HEP
     ...................................................................
 + CA/Victoria-LCG2 FT f./SECURITY_ER./Permission denied.GGUS 57528 
   solved:permission is fixed,transfers are OK. 
 + CA/SFU-LCG2 ~2800 jobs stuck in "transferring" >36h.GGUS 57559,57436
   solved(?):not a site problem,frozen DS(?)

4) Fri Apr.23
 - CA/VICTORIA-LCG2 still job f./No module named zlib.GGUS 57583,57458
   assigned.Set offline.Schedconfig change done Apr.26.Test jobs fired.

-- BryanCaron - 27 Apr 2010

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng apel-statistics-aus-20100427.png r1 manage 78.2 K 2010-04-27 - 11:04 BryanCaron  
PNGpng apel-statistics-ca-20100427.png r1 manage 150.5 K 2010-04-27 - 11:03 BryanCaron  
PNGpng apel-status-aus-20100427.png r1 manage 15.1 K 2010-04-27 - 11:05 BryanCaron  
PNGpng apel-status-ca-20100427.png r1 manage 24.7 K 2010-04-27 - 11:05 BryanCaron  
PNGpng dashboard-std-summary-all-20100427.png r1 manage 89.1 K 2010-04-27 - 10:36 BryanCaron  
PNGpng dashboard-std-summary-ca-20100427.png r1 manage 50.9 K 2010-04-27 - 10:37 BryanCaron  
PNGpng dashboard-summary-all-20100427.png r1 manage 125.9 K 2010-04-27 - 09:56 BryanCaron  
PNGpng dashboard-summary-ca-20100427.png r1 manage 19.0 K 2010-04-27 - 10:20 BryanCaron  
Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2010-04-27 - BryanCaron
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback