Tier1 and Tier-2 Weekly Report 2010-11-02

Daily Notes

• Tue Oct 26
  • SFU-LCG2_DATADISK downtime for writes within DDM - autoclean active
  • scheduled downtime at Australia-ATLAS for DPM server upgrade (4 hours) - queues set to brokeroff by ADCoS
  • GGUS ticket: FTS failures for STAR channels to CA Tier-2s
    • https://gus.fzk.de/ws/ticket_info.php?ticket=63470

• Wed Oct 27
  • release 8.0 of GGUS portal is online
  • Australia-ATLAS tested and set online by ADCoS shifters
  • unscheduled downtime to SFU-LCG2 due to host certificate issues and failing startup of dCache
    • site panda production and analysis queues set to test
    • ADC Site Exclusion Savannah http://savannah.cern.ch/support/?117544
  • dataset subscriptions created to CA-ALBERTA-WESTGRID-T2 in preparation for upcoming HammerCloud

• Thur Oct 28
  • start of bulk reprocessing (group 1) at T1s postponed until Friday pending software validation
  • new host certificate (with correct keyfile) received for SE and installed at SFU-LCG2
    • site downtime ended in gocdb and unblacklisted within DDM (except for DATADISK for writes which is still being cleaned)
    • site analysis queues validated and set online (cloud squad)
  • software release installation at TRIUMF:
    • restart installation for 15.6.13.7 and 16.0.2.2 (ce1)
    • submitted cleanup for 16.0.2, 16.0.2.1 and 16.0.2.2 (ce1)

• Fri Oct 29
  • SFU-LCG2 DATADISK unblacklisted (for writes) following completion of DDM cleanup
  • SFU-LCG2 analysis queues validated and set online by cloud squad
    • test jobs submitted to production queues by ADCoS
  • GGUS ticket: CA-SCINET-T2_PRODDISK and CA-SCINET-T2_DATADISK: Protocol(s) specified not supported: [ gsiftp ]
    • https://gus.fzk.de/ws/ticket_info.php?ticket=63581
      • crashed dCache gridftp doors restarted
      • LFC communication errors also observed (slave ticket created ; see GGUS 63636 below)
    • site queues set to brokeroff in Panda by ADCoS
  • Autumn 2010 reprocessing to begin Friday night with 16.0.2.3 (release cache only issued today)
  • FTS settings tuned for TRIUMF-ALBERTA2 channel (CA-ALBERTA-WESTGRID-T2) - improvement from 300 to 800 Mbps peak
  • software release installation at TRIUMF:
    • created step-by-step action plan with Alessandro De Salvo in regards to release 16.0.2.* reinstallations for ce1 and ce2 (returned to separate software areas 20101027)
    • release 16.0.2 and 16.0.2.* cleaned from ce2
    • release 16.0.2 + 16.0.2.1 installed on ce1
    • central ATLAS installation DB problems observed and corrected by Alessandro
  • site production queues temporarily set to test and then online again - pilot factory submissions stopped pending install of 16.0.2.* releases concurrently at ce1 and ce2
  • HammerCloud datasets replicated to final CA T2 site (CA-ALBERTA-WESTGRID-T2)

• Sat Oct 30
  • CA-SCINET-T2 queues set online after validation by cloud squad and ADCoS
  • GGUS ticket: lfc_setfsizeg failed for lfn, jobs failed on CA-SCINET-T2
    • https://gus.fzk.de/ws/ticket_info.php?ticket=63636
    • software release installation at TRIUMF:
      • installation of 16.0.2.3 (for reprocessing) forced ahead of 16.0.2.2 on ce1
        • ce1 production pilots resumed
      • installation of 16.0.2.3 (for reprocessing) forced ahead of 16.0.2.2 on ce2
        • ce2 production pilots resumed
    • HammerCloud run for CA cloud sites including T2s
      • http://hammercloud.cern.ch/atlas/10001648/test/

• Sun Oct 31
  • CA-SCINET-T2 (ggus 63581) - restarted gridftp doors after 2nd failure and limited max jobs to 256

• Mon Nov 1
  • SFU-LCG2 production queues set online by ADCoS shifters
  • GGUS to GOCDB regarding started downtime deletion closed and tracked as feature through Savannah
    • https://gus.fzk.de/ws/ticket_info.php?ticket=63576
    • https://savannah.cern.ch/bugs/index.php?74665 * Australia-ATLAS moving site BDII from 2.3 to 2.4 due to BDII lock-up issue

• Tue Nov 2
  • TRIUMF-LCG2 production queues updated to maxinputsize of 50GB to accommodate reprocessing requirements
  • CA-SCINET-T2 NAGIOS org.sam.SRM-Put-/ops/Role=lcgadmin failed on lcg-se1
    • https://gus.fzk.de/ws/ticket_info.php?ticket=63793
  • CA-ALBERTA-WESTGRID-T2 job failure with error accessing DBRelease-12.2.1.tar.gz
    • https://gus.fzk.de/ws/ticket_info.php?ticket=63784

-- BryanCaron - 2010-10-26