TWiki> AtlasCanada Web>GridActivities>ATLASCanadaTier-2Operations>T2WeeklyReport20120925 (2012-09-25, LeslieGroer) EditAttach

Tier1 and Tier-2 Weekly Report 2012-09-25

General News

  • ADC and ATLAS Computing News [20120925]
    • 1M XP_Higgs events - all tasks completed Task Ids
    • CA-MCGILL-CLUMEQ-T2 set as delta site for October data placement - still gets PD2P data plus DATADISK is reasonably full.
    • New EGI Trust anchor certificates released (1.50) - please update sites
    • See EGI statement about glite 3.2 retirement https://operations-portal.egi.eu/broadcast/archive/id/739
    • ADC Weekly meeting
    • Technical stop completed, but various issues starting up again (cryogenics)
    • Agreement to increase luminosity to 8.5x10^33cm^-2s^-1.
    • AMOD Report
      • Tuesday - CASTOR nameserver db migration, outage 0800-1100hrs, which was not communicated to the Tzero.
      • INFN and PIC DATADISK cleaned/increased and back online
      • Panda server proxy expiration issue over weekend. Some miscommunication/weekend delayed getting this renewed.
      • ToA switched to AGIS last week - no ill effects
      • No mention of Frontier issues from last week
    • Discussion about reducing the number of DNs with atlas/Role=production
      • Some are needed (robots, AMOD etc) but there are 67 distinct DNs for "cloud support"
      • General agreement this can be reduced, but not clear every use case understood yet (e.g. not needed for panda queue control, approving LOCALGROUPDISK requests)
      • Is currently necessary for bad file declaration, but this may be changed to /atlas/team
      • Robot certificates are recommended rather than personal ones for local pilot factories
      • Cleaning and testing still need production role in some cases, but should not be too many people doing this
      • Ueda would like justification for all the DNs maintaining the role (cleanup already occurring based on the e-mail last week)
  • CA Cloud Squad Actions [20120626] 
TRIUMF Auto-Excluded at Wed Sep 19 18:32:35 2012 CET
ANALY_TRIUMF Auto-Excluded at Wed Sep 19 19:31:38 2012 CET
ANALY_TRIUMF_CVMFS Auto-Excluded at Wed Sep 19 20:01:39 2012 CET
ANALY_TRIUMF reset online at Wed Sep 19 21:31:07 2012 CET
TRIUMF reset online at Wed Sep 19 21:33:09 2012 CET
ANALY_TRIUMF_CVMFS reset online at Wed Sep 19 22:01:01 2012 CET
TRIUMF Auto-Excluded at Wed Sep 19 23:03:17 2012 CET
TRIUMF reset online at Thu Sep 20 00:32:19 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Thu Sep 20 06:01:50 2012 CET
ANALY_AUSTRALIA reset online at Thu Sep 20 07:00:48 2012 CET
ANALY_ALBERTA-WG1 Auto-Excluded at Thu Sep 20 16:02:24 2012 CET
ANALY_AUSTRALIA Auto-Excluded at Thu Sep 20 19:06:23 2012 CET
ANALY_AUSTRALIA reset online at Thu Sep 20 20:30:12 2012 CET
ANALY_ALBERTA-WG1 reset online at Fri Sep 21 04:30:43 2012 CET
IAAS Auto-Excluded at Fri Sep 21 18:35:58 2012 CET
TRIUMF Auto-Excluded at Sat Sep 22 09:12:53 2012 CET
ANALY_TRIUMF Auto-Excluded at Sat Sep 22 10:04:26 2012 CET
CA-VICTORIA-WESTGRID-T2 Auto-Excluded at Sat Sep 22 10:35:29 2012 CET
Australia-ATLAS Auto-Excluded at Sat Sep 22 11:07:35 2012 CET
TRIUMF Auto-Excluded at Sat Sep 22 12:42:28 2012 CET
Australia-ATLAS Auto-Excluded at Sat Sep 22 16:52:33 2012 CET
CA-SCINET-T2 Auto-Excluded at Sat Sep 22 16:52:57 2012 CET
CA-VICTORIA-WESTGRID-T2 Auto-Excluded at Sat Sep 22 16:52:59 2012 CET
CA-SCINET-T2 reset online at Sat Sep 22 17:50:58 2012 CET
CA-VICTORIA-WESTGRID-T2 reset online at Sun Sep 23 02:31:34 2012 CET
Australia-ATLAS reset online at Sun Sep 23 03:01:10 2012 CET
IAAS reset online at Tue Sep 25 03:33:58 2012 CET
CA-SCINET-T2 Auto-Excluded at Tue Sep 25 07:14:10 2012 CET
ANALY_SCINET Auto-Excluded at Tue Sep 25 07:37:54 2012 CET
ANALY_SCINET reset online at Tue Sep 25 17:31:00 2012 CET
CA-SCINET-T2 reset online at Tue Sep 25 17:32:18 2012 CET

  • Security News [20120801] 

Unsupported gLite 3.2 products
====================
The gLite 3.2 components currently out of security support are: APEL, ARGUS, 
BDII, Cluster, CREAM, dCache, LB, LSF utils, MPI utils, SCAS, SGE utils, 
Torque client/server/utils, VOMS [1].

Decommissioning by 30-09-2012
=====================
gLite 3.1 products and *unsupported* gLite 3.2 software components have to be 
retired by 30-09-2012. Site managers can choose between the option of 
upgrading to a supported UMD release of the product [2] in consultation with 
the supported VOs, or the decommissioning of the service following the 
related procedure [3].

Note well: this retirement calendar does not apply to gLite 3.2 products that 
are still supported. As already announced in [4], the support of glite 3.2 
glite-UI, glite-WN, glite-GLEXEC_wn, glite-LFC_mysql/glite-LFC_oracle, 
glite-SE_dpm_disk/glite-SE_dpm_mysql was recently extended to 30/11/2012.

Escalation
=======
Starting from 01-10-2012 site managers of Resource Centres found to be 
hosting unsupported gLite 3.1/3.2 services, will be contacted through GGUS by 
the Central Grid Oversight team to request the retirement of the affected 
products. 
Resource Centres that will fail to retire unsupported gLite software by 
01-11-2012, will be eligible for suspension and the problem will be escalated 
to EGI CSIRT for the enforcement of this suspension policy.

  • ROC_Canada News [20120911]
    • Current shift - Ashok Argawal Sep 17-21, 2012
    • Next Week - Ryan Taylor

  • WLCG and General Computing News [20120911]

SSB Results

SSB PanDA Test Queue Availability (month) - this metric, measured for the analysis queues, defines the site status for data preplacement as alpha (T2D and avail > 90%), bravo (T2 and avail > 90%) or charlie (< 90%).

SAM Results

  • site service status via Gridview - Please check your site

APEL Accounting

APEL Publishing Status

APEL Accounting Results

APEL Synchronization Test Links

Site Reports

Alberta

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

SFU

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Victoria

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Toronto

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

McGill

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

TRIUMF

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Australia-ATLAS

Recent Issues

  • summary of this weeks Savannah bugs and GGUS tickets for site:

Outstanding Issues

Production

Summary

  • NOTE: ARDA ProdSys Dashboard has stopped being used by ADCoS team due to information updating issues with the interface. The historical job usage information is to appear separately within ARDA and is being validated. This will be used in the future for information presented here.

  • Within the CA Cloud and Worldwide (Panglia View): 


      World-wide weekly ATLAS ADCoS summary (Sep.18-24,2012)
________________________________________________________________________

I. General summary:
    ---------------

A)  During the past week (Sep.18-24,2012) Panda production service
      - completed successfully 1,289,728 managed group, MC production,
        validation and reprocessing jobs
      - average ~184,247 jobs per day
      - failed 74,870 jobs
      - average efficiency:
         -- jobs ~94,5%
      - active tasks: 1194
         -- distribution by cloud: CA:101 CERN:81 DE:59 ES:86 FR:177 IT:70 ND:54 NL:140 TW:117 UK:149 US:160

B)   - PIC DT(SE,SRM,CE OUTAGE:storage pools upgrade,network test),Sep.18
     - IN2P3-CC DT (CE|SE,SRM OUTAGE:general+mass storage), Sep.17-18|18.
     - CERN-PROD DT (SRM OUTAGE:CASTOR nameserver DB migration), Sep.18.
     - CERN-PROD DT (ce203 OUTAGE:s/w upgrade+reconfig.), Sep.19-Oct.1.
     - RAL-LCG2 DT (SE,SRM OUTAGE:SRM failure,DB CASTOR issue),Sep.20-21.
     - IT/INFN-T1 (SRM OUTAGE:configuration of the end-point), Sep.24.
     - CERN: Switch ToA->AGIS, Sep.24.
     - New ATLAS releases and TRF caches were distributed:
       -- AytlasOffline 17.2.6,(32+64b),17.3.6(32b)
       -- AtlasProduction 16.6.7.33(32+64b),17.0.6.20(32+64b),
                          17.2.6.1(32+64b),16.6.9.14(32+64b)

II. Site and FT/DDM related interruptions/issues/news (+fixed, -not yet)
     -------------------------------------------------------------------

1) Tue Sep.18
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        ........................................
+ DE/GoeGrid FT f./DEST.:SRM_ABORTED.GGUS 86174 solved:SRM restarted.
- DE/FZK 1000s staging f./SRMV2STAGER:StatusOfBringOnlineRequest:SRM_
  FAILURE.GGUS 85955 waiting for reply:4.6K unique MC HITS files affected
  no task needing it and no subscription,asked ddm ops for help.
+ US/MWT2_UC massive FT f./SOURCE:AsyncWait.GGUS 86173 verified:dCache
  storage node went offline,fixed the same day.
- ES/PIC many job f./transfer timeout.Savannah 132089:increased DATADISK
  by 40TB on Monday,no err. for several days,Sep.23.
+ DE/CSCS-LCG2_DATADISK 1K+ FT f./DEST.:CONN.:SRM.GGUS 86181 verified:DT
+ UK/UKI-SCOTGRID-GLASGOW 134/1h deletion err./SRM_INTERNAL_ERROR.GGUS
  86200 solved:no deletion errors for the last day,corresponding to the
  analysis jobs stopping killing disk servers (due to high load),Sep.20.
+ TW/TAIWAN-LCG2_DATADISK again FT f./GRIDFTP_ER.:severe err.500.GGUS
  86127 reopened,solved:seem to have slow transfer from CA-MCGILL area
  and it is most likely not a problem of the destination.

2) Wed Sep.19
      -> Cloud/site PROD efficiency ever<50%  -
      -> Cloud/site DDM  efficiency <50% : CA
        ................................................................
- US/BU_ATLAS_Tier2 700+ deletion errors/Er.reading token data:Connection
  reset by peer.GGUS 84189 in progress updated:still 788 err.,Sep.22.
- CA/TRIUMF-LCG2 880 FT f./SOURCE:SECURITY_ERR.:proxy expired.GGUS 86216,
  81844,86230 in progress:FTS-VOMS interaction issue,should be solved on
  the next weeks when FTS Support provides a patch in order to fix it.
+ ES/IFIC-LCG2->TRIUMF-LCG2 FT f./SOURCE:SECURITY_ERR.:proxy expired.GGUS
  86230 solved:FTS bug to be fixed(see ticket above).Restart affected FTS
  /FTA services in order to temporary fix the delegation error.Needs a
  final fix,all T1s are expecting for a final patch from the FTS Support.
- CA/CA-TRIUMF-LCG2_DATADISK many FT f./DEST.:SECURITY_ERR.:user has no
  permission to create file.GGUS 86231 assigned:also ~11300 deletion err.
  Had severe issues with user mapping and authentication when refreshing
  dCache authentication mapping file.Fixed.
+ CA/SFU_LCG2 many t.954059,954313 job f./killed by signal 15.GGUS 86225
  solved:jobs were using >2.5GB of memory(not virtual),restriction is
  consistentwith Atlas requirements of 2GB of mem + 1 GB of vmem per job.
- IT/INFN-ROMA2 squid is dead since Sep.18.GGUS 85198 in progress updated

3) Thu Sep.20
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency <50%: NL
        ............................................
+ ES/PIC ~3K FT f./SOURCE:GENERAL_FAILURE:ThePinCallbacks Timeout.GGUS
  86235 verified:lost connectivity with one pool,restarted the same day.
+ DE/PSNC man job+FT f./DBRel.v200301 not transferred/GRIDFTP_ER.:globus_
  gass_copy_register_url_to_url transfer timeout.GGUS 86239 solved:also
  FT f./DEST.:CONN.:SRM.DBrel.was transferred the same day,jobs succeed.
- IT/NFN-MILANO-ATLASC still FT f./DEST.checksum mismatch.GGUS 86129
  assigned,updated:due to a severe h/w failure f/s got corrupted.17K
  files are likely to have been involved,processing the FS repair tool's
  logs to get a list of the files and verify their checksum.
+ NL/SARA-MATRIX FT f./SOURCE:GENERAL_FAILURE:RQueued.GGUS 86259 solved:
  Looking at Ganglia graph, there were a lot of SrmGet requests,so seems
  a limit has been reached(set to protect dCache instance from overloads)
  .There was another user that had put 2K requests in dCache queue.

4) Fri Sep.21
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM efficiency  <50% : -
        ............................................
+ UK/RAL-LCG2 FT f./SOURCE:SRM_ABORTED.GGUS 82262 solved:also SRMV2 test
  f.A huge number of blocking sessions present on the ATLAS SRM,killed
  the offending sessions.Resolved.
+ UK/UKI-LT2-RHUL ~50% FT f./SOURCE:INVALID_PATH:Unknown Reason.GGUS
  86248 solved:1 storage node lost contact to SE.
- UK/UKI-SCOTGRID-GLASGOW t.928152 job f./Staging input file failed.
  Savannah 132315,97674:many job f.,some 4 attempts in a row.Task is done
  as of Sep.24,the ticket can be closed.
- NL/AM-04-YERPHI FT f./DEST.:CONN.:SRM.GGUS 86265,86246 assigned:800|4h
  deletion err./could not open conn.to se.yerphi-cluster.grid.am,Sep.22.
  Problem on SE,waiting for some help.
+ NL/ru-Moscow-FIAN-LCG2 FT f./SOURCE:CONN.:SRM.GGUS 86280 solved:problem
  with one of DPM servers fixed,Sep.23.
+ NL/TR-10-ULAKBIM 40% eff. job f./too little space left on local disk to
  run job.GGUS 86278 solved:jobs needed more storage space than was
  available.Modified ATLAS limits to allow 4GB max input size leaving 1GB
  for output/log files at this site.
- US/AGLT2 many FT f./FIRST_MARKER_TIMEOUT/SOURCE:ThePinCallbacks Timeout
  .GGUS 86268 in progress:2 pool nodes stopped responding.Power-cycled,
  checked and put back into service.3rd server crashed,fixed.Hit
  extremely hard by Analysis skim jobs,that have very low cpu efficiency,
  and are very hard on pool servers.100s FT f./SOURCE:LOCALITY is
  unavailable,Sep.24.Set an emergency OIM outage for the SE to upgrade to
  dCache version 2.2.
+ DE/FZK-LCG2 FT f./DEST.:FILE_EXIST,overwrite is not allowed.GGUS 86302
  solved:errors have stopped,an ATLAS problem, rather than a site problem
  .Savannah 131665(racing condition of deletion service and subscription).
- FR/GRIF-IRFU ~1.3K FT f./failed to contact on remote SRM httpg://node12
  .datagrid.cea.fr.GGUS 84626 in progress since Jul.27 updated:trancient
  issues,can be closed?

5) Sat Sep.22
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : TW
        ..............................
- CERN/HC test f.,many sites auto set to "test".Auto-exclusion disabled
  temporarily.Elog.39520-39527.
- TW/Taiwan-LCG2 MCTAPE 1052 FT f./SRMV2STAGER:StatusOfBringOnlineRequest
  :SRM_FILE_UNAVAILABLE.GGUS 86303 in progress.
+ US/UPENN<-MILANO FT f./DEST.:INTERNAL_ERR.:Checksum mismatch.GGUS 86307
  solved:not related to UPENN,but on MILANO side (due to a severe h/w
  failure the f/s got corrupted).Savannah 97682,97512,132331.

6) Sun Sep.23
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency < 50% : -
        ........................................
+ CERN/CERN-PROD_DATAPREP<-CERN-PROD_TZERO FT f./DEST.:CONN.:SRM.GGUS
  86310 verified:XRootD GSI authentication crashed on the headnode.
  Automatic restart didn't succeed because of overload.The service was
  started manually after namespace compaction the same day.
- NL/SARA-MATRIX 6K/4h FT f./SOURCE:GENERAL_FAILURE:RQueued.GGUS 86314,
  86259 waiting for reply:the high load seems to be caused by jobs
  running at University of Groningen,blocked the user and his last
  remaining activity has died out,Sep.24.
+ NL/IL-TAU-HEP 1366 FT f./DEST.:CONN.:SRM.GGUS 86308 verified:site in DT
+ IT/INFN-MILANO-ATLASC FT f./GRIDFTP_ERROR:an end-of-file was reached.
  GGUS 86309 solved:gridftp server restarted.Transfers OK,Sep.24.

7) Mon Sep.24
      -> Cloud/site PROD efficiency ever<50% : -
      -> Cloud/site DDM  efficiency <50% : -
        .......................................................
+ DE/LRZ-LMU 1000s FT f./DEST.:CONN.:SRM.GGUS 86318 solved:No space left
  on LRZ-LMU_ATLASCALIBDISK.Savannah 132370:blacklisted.There was a major
  power cut at the computing centre.Recovered the same day.
- DE/DESY-HH_LOCALGROUPDISK no space.Savannah 131724:+30 TB,but space is
  about to run out again.
- UK/UKI-SCOTGRID-ECDF_DATADISK no space.Savannah 132366.
+ US/SLACXRD FT f./SOURCE:CONN.:SRM.GGUS 86331 solved: a temporary
  problem with SRM,resolved the same day.
- US/NET2_USERDISK 600+|4h deletion errors.GGUS 85951 reopened,in
  progress.

III. ATLAS Validation,Repro,DDM,ADC Operation Savannah bug-reports
       (++ bug understood/fixed, -+ fixed in part, -- not fixed yet)
----------------------------------------------------------------------

-+ 17.2.4.8.1 data12_8TeV Reco merge task 975051 failures:TRF_UNKNOWN |
 Standard std::exception is caught handling incident0xffa798d4|'Variable
 "Valid" does not exist in the AttributeList (CORAL:"AttributeList" from
 "CoralBase").Savannah 132207,97567:group production DP.All jobs failed
 11 attempts at TAIWAN-LCG2.Task ABORTED,under investigation:no problem
 with DB access,it's a gap in database.

-- 17.2.4.10.1 valid1 DigiMReco task 978200-978205 failures:TRF_UNKNOWN |
 "GeoModelSvc.PixelDetectorTool SZ= 47268Kb Time = 0.34S"|"Problems
 calling GeoModelSvc[0xe18a8d4]+89"|"Problems preloading IOVRanges".
 Savannah 97646:SampleT production - r3950.All jobs have failed up to 11
 attempts at various sites rinning for US,DE,UK clouds.Tasks are either
 still "submitting" or "failed".No comments published so far.

-+ 17.2.4.8.1 data11_7TeV Reco merge task 977554 failures:GUID for ESD.
 495466._000384.pool.root not found in DQ2.Savannah 132292:group
 production DP.4 remaining jobs weren't assigned to any site in DE cloud
 and failed 7 attempts in a row.Task set to "finished".

-+ 17.2.4.6.2 data12_8TeV Reco merge task 976809 failures:TRF_SEGFAULT |
 "Did not find Input Collection CaloTopoCluster" | "Caught signal 11
 (Segmentation fault).Savannah 132323,34523:group production EGAMMA.All
 jobs have failed 25(!) times in a row at NIKHEF-ELPROD.Task is still in
 "submitted" state,owner was notified.

-+ 17.2.4.8.1 data11_7TeV Reco merge task 977697 failures:ERROR Bunch
 CrossingToolBase:../Root/BunchCrossingToolBase.cxx:1050(bool Trig::Bunch
 CrossingToolBase::loadBunchTrains.Savannah 132332,34593:all jobs have
 failed multiple attempts at NL/NL.NIKHEF-ELPROD.Task ABORTED,owner was
 notified.

-+ 17.2.4.8.1 data11_7TeV Reco merge task 977663 failures:"Variable"Valid
 " does not exist in the AttributeList".Savannah 132340:group production
 DP.All jobs have failed up to 24(!) attempts at various sites running
 for FR cloud.Task ABORTED,s/w problem(?) is under investigation.

-+ 17.2.4.10 mc12_8TeV evgen task 979031 failures:TRF_SVRINIT | "Standard
 std::exception is caught"|"basic_string::replace"|"Unable to initialize
 Algorithm Pythia8".Savannah 132348:MC12 production - EXOTICS.All jobs
 have failed multiple attempts at CERN and ND cloud.Task ABORTED,no
 comments added.

++ 17.2.5.2 mc12_8TeV evgen tasks 979782-980039 failures:"Auxiliary file
 susy_RPV_*_simplified.slha not found in DATAPATH".Savannah 132364:MC12
 production - SUSY.All jobs have failed multiple attempts in various
 clouds.Tasks are in "submitted" state.The needed files have been added
 to the JOs tarball by hand - so this issue should now be fixed.

-- 17.2.5.2 mc12_8TeV evgen tasks 980050-980058 failures:"Fortran runtime
 error: Bad value during integer read".Savannah 132365:MC12 production -
 SUSY.All jobs have failed multiple attempts in different clouds. Tasks
 are still in "submitted" state, no comment added so far.

-- LeslieGroer - 2012-09-25

Edit | Attach | Watch | Print version | History: r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r1 - 2012-09-25 - LeslieGroer
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2026 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback