Scripts to Create and Graph an LFC Snapshot

WARNING: This page is obsolete.

At the moment, it is difficult for a Tier 1 or Tier 2 to know what files have been stored at their site by Atlas. The first tools to display that information are just being written. On this page, you will find simple scripts that you can edit, that will report the number and size of files in your site's LFC.

CVS package for the LFC Monitor

The Atlas Canada CVS has a package, tier1/lfc_monitor, that contains all of these scripts, and wrap-around scripts that can be sourced. It resides at:

http://viewcvs.atlas-canada.ca/viewvc.cgi/ATLAS%20Canada/tier1/

Create the LFC Snapshot

This script, script_to_create_lfc_snapshot.sh, was written to access an SQL database, and create a file (called "CA_sfns.lis") that lists all of the files, with the file size. (There is a command that is commented out, that should access an Oracle database.) At TRIUMF, this script is run by a cron job once a day for the Canadian cloud.

Analyze the LFC Snapshot

These two scripts, lfc_monitor.py.txt, and lfc_monitor_ref.py.txt, analyze the LFC snapshot. lfc_monitor_ref.py contains the list of sites, the file name strings you want to search for (i.e., AOD, RDO, log, etc.), and the version numbers you're interested in displaying (i.e., v10, v11, etc.). Edit lfc_monitor_ref.py to change those paramenters. You should not need to edit lfc_monitor.py.

The script can be run as a simple python job:

> python lfc_monitor.py

It will make two output files called "files_in_the_lfc" and "lfc_ganglia_input".

Web Display - Numeric

start_lfc_monitor_summary.html with lfc_monitor_summary.py.txt will display "files_in_the_lfc" on a simple web page. Edit lfc_monitor_summary.py to set your web directory.

Here are the results of running these scripts on the Canadian cloud. There is a text file with the numbers: http://trshare.triumf.ca/~steele/CA/lfcdump_storage.html

Web Display - Graphical

At TRIUMF, these numbers are input to ganglia. Ganglia is a monitoring system for high-performance computing systems. It runs well on Linux, and is becoming quite common in grid computing. (Most Atlas T1's and T2's probaly run it already.) Details can be found at the ganglia website, http://ganglia.info/ or http://ganglia.wiki.sourceforge.net/. The file "lfc_ganglia_input" is fed into some custom php scripts, which plot the data and choose colors to distinguish different elements. Those php scripts will be available on this TWiki page, as soon as they can be rewritten to make them appropriately generic.

As an example, the ganglia output for the Canadian cloud can be seen here: http://gridinfo.triumf.ca/lfc/

Atlas User Support, atlas-user-support@triumf.ca, is eager to help interested Atlas sites set up and run this program.

Running at TRIUMF

Where and when the scripts are run at TRIUMF are listed here.

-- JosephSteele - 08 May 2008

• get_the_dump_file.sh: Script to copy over the daily LFC dump

• storage_by_user_and_group.py.txt: Script to display storage by DN or by user's group affiliation

• storage_by_token.py.txt: Script to display storage usage in the LFC by storage token.

• storage_by_token_bysite.py.txt: Script to display data in the LFC by storage token, for an individual site.

• dq2_by_site_monitor.py.txt: Script to call DQ2, and display all datasets listed at a DQ2 endpoint.

• weekly_flush_of_data.py.txt: Script to delete all cached DQ2 information - necessary for clean-up.

• lfc_monitor.py.txt: Python script to monitor the LFC

• lfc_monitor_ref.py.txt: Parameters for the LFC monitoring

• start_lfc_monitor_summary.html: Header file for LFC monitor web page

• script_to_create_lfc_snapshot.sh: Script to list the files in the LFC