Running Scripts to Monitor and Display the LFC Data

WARNING: This page is obsolete.

This document describes scripts that display the type of data contained in the TRIUMF LFC. The scripts themselves can be found here.

1) get_the_dump_file.sh

Runs on dcache-acc at /home/steele/monitoring_scripts/current at 13:05 each day. It copies the LFC dump to dcache-acc (/home/steele/monitoring_scripts/current/sfns.lis, group.lis, and users.lis), so that it can be accessed by the other scripts. It also creates a *_sfns.lis file for each site, which lists the information for only that site. (eg. ALBERTA_sfns.lis, AUSTRALIA_sfns.lis, etc.) It takes about 2 minutes.

2) lfc_dump_monitor_by_directories.py

Runs on dcache-acc at /home/steele/monitoring_scripts/lfc_monitoring at 13:15 each day, reads the LFC dump, and creates a web page:

http://gridinfo.triumf.ca/CA/lfcdump_storage.html

This same script (/home/trteam/lfc_monitoring/doit.sh) also runs on gridweb, at 13:01 each day, and creates the inputs for the ganglia plots at:

http://gridinfo.triumf.ca/lfc/

It takes about 20 minutes.

3) storage_by_user_and_group.py and storage_by_token.py

Runs on dcache-acc at /home/steele/monitoring_scripts/current at 14:01 each day, reads the LFC dump, and creates two web pages:

storage_by_user_and_group.py creates a web page with a table that shows storage by each user and a table that shows storage by each group.

http://gridinfo.triumf.ca/CA/storage_tables/storage_by_user_summary.html

and storage_by_token.py creates a table that shows storage in each storage token.

http://gridinfo.triumf.ca/CA/storage_tables/storage_by_token_summary.html

It takes ~ 15 minutes.

4) storage_by_token_bysite.py

Runs on dcache-acc at /home/steele/monitoring_scripts/current at 15:01 each day, reads the LFC dumps for the individual sites (ALBERTA_sfns.lis, etc.), and creates eight web pages at /var/www/html/dcache-acc/CA/storage_tables/sites/. These pages have the same information provided by storage_by_token.py, but they are broken down to individual sites. They are linked to the web page at

http://gridinfo.triumf.ca/CA/storage_tables/storage_by_user_summary.html .

It takes ~ 15 minutes.

5) storage_tokens_by_DN.cgi

Can be run on trshare (which might be moved to dcache-acc in the future) by an individual. It takes a User's Distinguished Name (DN) as input, reads the LFC dump, and prints out the storage used by that User. It can be found at:

http://trshare.triumf.ca/~steele/cgi-bin/storage_tokens_by_DN.cgi

Takes 5 to 10 minutes.

6) storage_tokens_by_username.cgi

Can be run on trshare (which might be moved to dcache-acc in the future) by an individual. It takes a username as input, reads the LFC dump, and prints out the storage used by that User. This information is obtained from dataset names, so does not depend on DN information. It can be found at:

http://trshare.triumf.ca/~steele/cgi-bin/storage_tokens_by_username.cgi

Takes 5 to 10 minutes.

7) dq2_by_site_monitor.py

Runs on dcache-acc at /home/steele/monitoring_scripts/dq2_queries at 08:01 on Mondays, Wednesdays, and Fridays. It makes DQ2 calls, and prints a list of all the datasets at each site, according to the DQ2 DB. The list also includes the number of files per dataset, and the creation date of the dataset. The lists for each site are shown at:

http://gridinfo.triumf.ca/CA/datasets_in_CA/find_Canadian_datasets.html

This script caches information at /home/steele/monitoring_scripts/dq2_queries, so the web page lists are built up over time. (Therefore, deleted datasets are kept in the list until there is some intervention - see script #8.) It takes several hours to run each day.

8) weekly_flush_of_data.py

Runs on dcache-acc at /home/steele/monitoring_scripts/dq2_queries at 07:30 on Fridays. It deletes the cached information about datasets, which causes dq2_by_site_monitor.py to start from scratch. It's necessary to do this occasionally, so that deleted datasets will be removed from the web page lists.

This will cause dq2_by_site_monitor.py to run for most of the day on Friday, if not longer. However, the lists put together on Wednesday will be displayed on the web page until they are overwritten by the Friday job.

Takes < 2 minutes to run.

9) Cron job file

Scripts 1), 2), 3), 4), 7), and 8) are run as cron jobs on dcache-acc, and are listed in the cron job script at /home/steele/cron_jobs/monitoring_cron .

-- JosephSteele - 03 Mar 2009