Tier1 and Tier-2 Weekly Report 2016-01-05

General News

• ADC and ATLAS Computing News [20160105]
  • Compute Canada Usage Stats - 2015 (preliminary and unverified)
  • 
  • 
  • ADC Weekly meeting
  • MC Production
    • Grid kept reasonably busy over Xmas break
    • ~600M events still in queue
    • CERN P1 power work removed ~40k job slots for next couple weeks
    • Some tasks still stuck in 'submitting' or 'register'
    • WORLD cloud tasks seen to run slower
    • Don't expect bulk requests from PMG with MC15b (for Moriond) but will have requests from CP/Physics WGs
    • Start MC15c from the 1st Feb. This is rerun of pile/recon with rel 20.7 by using the existing HITS files. The target conference is ICHEP
  • Derivation Production
    • Reprocessing cache, AtlasDerivation -20.7.3.5, was tested fine on the express stream before the break
      • Wait for green light from DataPrep to start derivation production
    • Collecting group requests as to which derivations will run on the reprocessed samples: HIGG1, HIGG2, HIGG4, HIGG5, HIGG6, SUSY, EGAM, TOP, JETM
    • Further discussion with the groups in tomorrow's train coordination meeting
  • Bulk reprocessing
    • Lots of work last minute to get validation to start reprocessing before break but made it
    • First bulk reprocessing tasks sent at 18:00 on December 16th
      • Further tasks and streams launched in stages over the Christmas break
      • Reco of 104/106 runs for physics_Main is now completed
      • Second express stream: 5/112 reco jobs still running, 2 to resubmit (trigger)
      • PixelBeam /VdM streams: some issues with missing beamspot info. New jobs being submitted now
      • Still to come: ZeroBias, physics_MinBias (6 runs comprising period B), and some DESD production, debug_Late stream
    • Issues with error reporting make it hard to debug. Also frequently missing log files.
      • Actually very few software related crashes
    • Some issues with T1 weights and brokering for merges
    • Various sites issues that were observed over the break were also discussed
  • CRC/ADCoS report
    • 3/4 of shifts covered during the break
    • Bunch of minor service or site issues reported from last 2 weeks
    • Open issues
      • CERN-PROD source transfer failures:No such file or directory - few lost files (~50)
      • Few other T1 site issues with transfer failures
  • Network overload for US North Eastern T2 - see slides for details
    • Uploaded 350TB RAW data15_13TeV on Dec 15
    • Large MCORE reprocessing started filling all ~5000 slots
    • Saturated 10G WAN with access from around the world from Dec 18-28
    • Caused large backlog in output transfers plus job timeout failures as could not compete
    • 100G being commissioned now for NET2
• CA Cloud Squad Actions
  • User DaTRI requests approved / processed (last week)
• Open Issues and Ongoing Discussions
  • Availability & Reliability Report [20151001]
    • September 2015 - All Sites Report ATLAS
  • Sites should be providing the 2015 storage pledge.
    • REBUS Site Capacities
    • REBUS Federation Resource Pledges
  • ATLAS Canada HammerCloud Tests
• WLCG and General Computing News
  • Security News
  • gLite/EMI News
• ROC_Canada News
  • Current shift

ABCD Determination and Monitoring

• SSB Qualification and ABCD Determination Wiki
• CA Cloud - Panda Resource Status
• CA Cloud - Switcher Actions
• CA Cloud - ABCD Determination

SSB Results

• Critical tests for each site in the Site Status dashBoard are available at the links below
  • Australia-ATLAS
  • CA-MCGILL-CLUMEQ-T2
  • CA-SCINET-T2
  • CA-ALBERTA-WESTGRID-T2
  • CA-VICTORIA-WESTGRID-T2
  • SFU-LCG2
  • TRIUMF-LCG2

SAM Results (based on VO tests)

• ROC_Canada
• Australia_ATLAS

APEL Accounting

• ACTION ITEM FOR SITES: Sites please regularly check and confirm that your historical APEL data has been published and looks correct when comparing with batch system utilization
  • See example steps circulated by D. Qing (20100406)
    • account09.c
  • See EGI Accounting Portal (Canada)
  • See EGI Accounting Portal (Australia)
  • Links
    • [1] http://wiki.hellasgrid.gr/wiki/bin/view/HellasGrid/GOC/MONToAPEL
    • [2] http://goc.grid.sinica.edu.tw/gocwiki/glite-APEL

APEL Synchronization Test Links

• TRIUMF-LCG2
• Australia-ATLAS
• CA-VICTORIA-WESTGRID-T2
• CA-MCGILL-CLUMEQ-T2
• CA-SCINET-T2
• SFU-LCG2

Production and Usage Status

• ADC Job Summaries Dashboard
• Within the CA Cloud and Worldwide ( Panglia View):

-- Leslie Groer - 2016-01-05

Comments