Production Activities

Production Dashboard

• Panda Dashboard
  • http://panda.cern.ch:25980/server/pandamon/query?dash=prod

Cross-Check Information

• First check if site is excluded from production:
  • Shifters take sites offline when massive failure are found
  • Site status can be found here (look for status column offline/online):
  • http://panda.cern.ch:25980/server/pandamon/query?dash=clouds

• Check if there are defined jobs in Panda:
  • No defined jobs means nothing to be run.
  • Check if there are assigned jobs.
  • If so means site should run production unless Input data blocks are not arriving at the site.

• High number of jobs in waiting state:
  • When found, need investigation as usually mean that the input data is not found. Panda brokering is modulated for waiting jobs.

• Check if pilots are running:
  • If there are activated jobs and site is not running production, possibly means that pilots are not arriving, so cannot pull jobs from Panda.

• Check if transferring jobs number is high:
  • Meaning that outputs cannot be aggregated to T1. Panda brokering has a protection against this.

• Check there is enough disk space:
  • Lack of free disk space block brokering

Other Links

• Production Dashboard Summaries
  • Some information about production jobs (walltime,# of jobs,etc. ) is available in proddashboard/summaries. This view is very slow though...And it takes up to 5-10 min. sometimes to receive that weekly summary:
  • http://dashb-atlas-prodsys-test.cern.ch/dashboard/request.py/summary

• Task Brokerage
  • http://panda.cern.ch:25980/server/pandamon/query?mode=mon&hours=12&name=panda.mon.bamboo&type=taskbrokerage

• information about the amount of jobs defined for the cloud and remaining number of jobs to run in "tobedone" state from ProdDB
  • tobedone 0 means that a cloud runs out of jobs in ProdDB
  • http://dashb-atlas-prodsys-test.cern.ch/dashboard/request.py/overview-taskjobs

• Bamboo Server Information
  • provides a description of how the task brokering is performed within Bamboo
  • https://twiki.cern.ch/twiki/bin/view/Atlas/BambooServer

Additional Comments

• on 20100727 I initiated a query to ADC Operations regarding observed low-level of production activity in the CA cloud
  • main conclusions/comments from production system experts:
    • "production is tailing down in the [previous] week and perhaps the load assigned to CA was few this time and was quickly processed."
    • "the main reason for empty clouds during the summer has been lack of work (requested tasks)."

-- BryanCaron - 03 Aug 2010