LfcMonitoringScript < AtlasCanada

AtlasCanada Web>WebTopicList>LfcMonitoringScript (2026-10-05, AsokaDeSilva) (raw view)

---# Scripts to Create and Graph an LFC Snapshot

%BR%%BR%%BR%%BR%%BR%

%X% *WARNING:* This page is obsolete.

%BR%%BR%%BR%%BR%%BR%

At the moment, it is difficult for a Tier 1 or Tier 2 to know what files have been stored at their site by Atlas. The first tools to display that information are just being written. On this page, you will find simple scripts that you can edit, that will report the number and size of files in your site's LFC.

---++ CVS package for the LFC Monitor

The Atlas Canada CVS has a package, tier1/lfc_monitor, that contains all of these scripts, and wrap-around scripts that can be sourced. It resides at:

http://viewcvs.atlas-canada.ca/viewvc.cgi/ATLAS%20Canada/tier1/

---++ Create the LFC Snapshot

This script, [[%ATTACHURL%/script_to_create_lfc_snapshot.sh][script_to_create_lfc_snapshot.sh]], was written to access an SQL database, and create a file (called "CA_sfns.lis") that lists all of the files, with the file size. (There is a command that is commented out, that should access an Oracle database.) At TRIUMF, this script is run by a cron job once a day for the Canadian cloud.

---++ Analyze the LFC Snapshot

These two scripts, [[%ATTACHURL%/lfc_monitor.py.txt][lfc_monitor.py.txt]], and [[%ATTACHURL%/lfc_monitor_ref.py.txt][lfc_monitor_ref.py.txt]], analyze the LFC snapshot. lfc_monitor_ref.py contains the list of sites, the file name strings you want to search for (i.e., AOD, RDO, log, etc.), and the version numbers you're interested in displaying (i.e., v10, v11, etc.). Edit lfc_monitor_ref.py to change those paramenters. You should not need to edit lfc_monitor.py.

The script can be run as a simple python job:

&gt; python lfc_monitor.py

It will make two output files called "files_in_the_lfc" and "lfc_ganglia_input".

---++ Web Display - Numeric

[[%ATTACHURL%/start_lfc_monitor_summary.html][start_lfc_monitor_summary.html]] with [[%ATTACHURL%/lfc_monitor_summary.py.txt][lfc_monitor_summary.py.txt]] will display "files_in_the_lfc" on a simple web page. Edit lfc_monitor_summary.py to set your web directory.

Here are the results of running these scripts on the Canadian cloud. There is a text file with the numbers: http://trshare.triumf.ca/~steele/CA/lfcdump_storage.html

---++ Web Display - Graphical

At TRIUMF, these numbers are input to ganglia. Ganglia is a monitoring system for high-performance computing systems. It runs well on Linux, and is becoming quite common in grid computing. (Most Atlas T1's and T2's probaly run it already.) Details can be found at the ganglia website, http://ganglia.info/ or http://ganglia.wiki.sourceforge.net/. The file "lfc_ganglia_input" is fed into some custom php scripts, which plot the data and choose colors to distinguish different elements. Those php scripts will be available on this TWiki page, as soon as they can be rewritten to make them appropriately generic.

As an example, the ganglia output for the Canadian cloud can be seen here: http://gridinfo.triumf.ca/lfc/

Atlas User Support, atlas-user-support@triumf.ca, is eager to help interested Atlas sites set up and run this program.

---++ Running at TRIUMF

Where and when the scripts are run at TRIUMF are listed [[AtlasCanada/RunningLfcScripts][here]].

-- Main.JosephSteele - 08 May 2008

   * [[%ATTACHURL%/get_the_dump_file.sh][get_the_dump_file.sh]]: Script to copy over the daily LFC dump

   * [[%ATTACHURL%/storage_by_user_and_group.py.txt][storage_by_user_and_group.py.txt]]: Script to display storage by DN or by user's group affiliation

   * [[%ATTACHURL%/storage_by_token.py.txt][storage_by_token.py.txt]]: Script to display storage usage in the LFC by storage token.

   * [[%ATTACHURL%/storage_by_token_bysite.py.txt][storage_by_token_bysite.py.txt]]: Script to display data in the LFC by storage token, for an individual site.

   * [[%ATTACHURL%/dq2_by_site_monitor.py.txt][dq2_by_site_monitor.py.txt]]: Script to call DQ2, and display all datasets listed at a DQ2 endpoint.

   * [[%ATTACHURL%/weekly_flush_of_data.py.txt][weekly_flush_of_data.py.txt]]: Script to delete all cached DQ2 information - necessary for clean-up.

   * [[%ATTACHURL%/lfc_monitor.py.txt][lfc_monitor.py.txt]]: Python script to monitor the LFC

   * [[%ATTACHURL%/lfc_monitor_ref.py.txt][lfc_monitor_ref.py.txt]]: Parameters for the LFC monitoring

   * [[%ATTACHURL%/start_lfc_monitor_summary.html][start_lfc_monitor_summary.html]]: Header file for LFC monitor web page

   * [[%ATTACHURL%/script_to_create_lfc_snapshot.sh][script_to_create_lfc_snapshot.sh]]: Script to list the files in the LFC

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
txt	dq2_by_site_monitor.py.txt	r1	manage	8.5 K	2010-05-28 - 22:38	UnknownUser	Script to call DQ2, and display all datasets listed at a DQ2 endpoint.
sh	get_the_dump_file.sh	r1	manage	1.3 K	2010-05-26 - 21:25	UnknownUser	Script to copy over the daily LFC dump
txt	lfc_monitor.py.txt	r1	manage	8.5 K	2008-05-09 - 21:07	UnknownUser	Python script to monitor the LFC
txt	lfc_monitor_ref.py.txt	r1	manage	1.4 K	2008-05-09 - 20:31	UnknownUser	Parameters for the LFC monitoring
txt	lfc_monitor_summary.py.txt	r1	manage	0.9 K	2008-05-09 - 18:49	UnknownUser	File to create web page for LFC monitor
sh	script_to_create_lfc_snapshot.sh	r1	manage	0.5 K	2008-05-09 - 18:38	UnknownUser	Script to list the files in the LFC
html	start_lfc_monitor_summary.html	r1	manage	0.4 K	2008-05-09 - 18:47	UnknownUser	Header file for LFC monitor web page
txt	storage_by_token.py.txt	r1	manage	14.7 K	2010-05-28 - 22:36	UnknownUser	Script to display storage usage in the LFC by storage token.
txt	storage_by_token_bysite.py.txt	r1	manage	14.7 K	2010-05-28 - 22:37	UnknownUser	Script to display data in the LFC by storage token, for an individual site.
txt	storage_by_user_and_group.py.txt	r1	manage	14.0 K	2010-05-28 - 22:35	UnknownUser	Script to display storage by DN or by user's group affiliation
txt	weekly_flush_of_data.py.txt	r1	manage	1.3 K	2010-05-28 - 22:39	UnknownUser	Script to delete all cached DQ2 information - necessary for clean-up.

Topic revision: r10 - 2026-10-05 - AsokaDeSilva

AtlasCanada

Navigate ATLAS-Canada Wiki:

* Home* * Activities * Analysis
Computing
Datasets
Getting Started
Grid Activities
* Meetings* * People in ATLAS Canada * User Services
Travel To CERN

Navigate ATLAS-Canada Wiki:

Wikis:

ATLAS-Canada Institute Websites: