MTA Limit Trends

Description

This set of scripts extracts msid data and plot their trends.

Location of Scripts and Their Names

Location: /data/mta/Script/MTA_limit_trends/

GitHub Address: https://github.com/chandra-mta/MTA

Script Name Last Updated
mta_limit_data_main_script Feb 04, 2021
mta_limit_data_wrap_script Feb 04, 2021
mta_limit_fetch_main_script Feb 04, 2021
mta_limit_fetch_wrap_script Feb 04, 2021
mta_sun_angle_main_script Feb 04, 2021
mta_sun_angle_wrap_script Feb 04, 2021
mta_tephin_main_script Feb 04, 2021
mta_tephin_wrap_script Feb 04, 2021
mta_limit_dea_main_script Feb 04, 2021
mta_limit_dea_wrap_script Feb 04, 2021
mta_limit_deposit_main_script Feb 04, 2021
mta_limit_deposit_wrap_script Feb 04, 2021
run_trend_wrap_script_sun Feb 04, 2021
run_trend_wrap_script_mon Feb 04, 2021
run_trend_wrap_script_tus Feb 04, 2021
run_trend_wrap_script_wed Feb 04, 2021
run_trend_wrap_script_thu Feb 04, 2021
run_trend_wrap_script_fri Feb 04, 2021
run_trend_wrap_script_sat Feb 04, 2021
run_trend_main_script_sun Feb 16, 2021
run_trend_main_script_mon Feb 16, 2021
run_trend_main_script_tus Feb 16, 2021
run_trend_main_script_wed Feb 16, 2021
run_trend_main_script_thu Feb 16, 2021
run_trend_main_script_fri Feb 16, 2021
run_trend_main_script_sat Feb 16, 2021
run_trend_wrap_script_sun2 Feb 04, 2021
run_trend_wrap_script_mon2 Feb 04, 2021
run_trend_wrap_script_tus2 Feb 04, 2021
run_trend_wrap_script_wed2 Feb 04, 2021
run_trend_wrap_script_thu2 Feb 04, 2021
run_trend_wrap_script_fri2 Feb 04, 2021
run_trend_wrap_script_sat2 Feb 04, 2021
run_trend_main_script_sun2 Feb 04, 2021
run_trend_main_script_mon2 Feb 04, 2021
run_trend_main_script_tus2 Feb 04, 2021
run_trend_main_script_wed2 Feb 04, 2021
run_trend_main_script_thu2 Feb 04, 2021
run_trend_main_script_fri2 Feb 04, 2021
run_trend_main_script_sat2 Feb 04, 2021
run_trend_wrap_script_sun3 Feb 04, 2021
run_trend_wrap_script_mon3 Feb 04, 2021
run_trend_wrap_script_tus3 Feb 04, 2021
run_trend_wrap_script_wed3 Feb 04, 2021
run_trend_wrap_script_thu3 Feb 04, 2021
run_trend_wrap_script_fri3 Feb 04, 2021
run_trend_wrap_script_sat3 Feb 04, 2021
run_trend_main_script_sun3 Feb 04, 2021
run_trend_main_script_mon3 Feb 04, 2021
run_trend_main_script_tus3 Feb 04, 2021
run_trend_main_script_wed3 Feb 04, 2021
run_trend_main_script_thu3 Feb 04, 2021
run_trend_main_script_fri3 Feb 04, 2021
run_trend_main_script_sat3 Feb 04, 2021
check_file_update_date.py May 28, 2019
check_msid_exist.py May 28, 2019
create_msid_plots.py Feb 04, 2021
copy_alias_plot.py Feb 04, 2021
create_derivative_plots.py Feb 04, 2021
create_html_page.py Feb 04, 2021
create_html_suppl.py Feb 04, 2021
create_top_html.py Feb 04, 2021
date_update.py Feb 04, 2021
envelope_common_function.py Feb 04, 2021
glimmon_sql_read.py Feb 04, 2021
read_mta_limits_db.py Feb 04, 2021
remove_html_from_interactive_dir.py May 28, 2019
update_database_suppl.py Feb 04, 2021
update_msid_data.py Feb 04, 2021
update_sub_html_pages.py Feb 04, 2021
violation_estimate_data.py Feb 04, 2021
create_interactive_page.py Jan 07, 2021
run_fetch.py Jan 07, 2021
msid_data_interactive.php Jan 07, 2021
README Feb 04, 2021
Comp:  
compute_acis_power.py Feb 04, 2021
compute_sim_flex.py Feb 04, 2021
compute_sim_flex_save.py Feb 04, 2021
convert_acistemp_into_c.py Feb 04, 2021
update_grad_and_comp_data.py Feb 04, 2021
DEA:  
average1.pl May 16, 2019
average2.pl May 16, 2019
average3.pl May 16, 2019
convert_dea_data_to_fits.py May 16, 2019
deahk.pl Jan 11, 2021
extract_from_acheive.perl May 16, 2019
find_new_dump.py Jul 25, 2019
getnrt Jul 25, 2019
out2in.pl May 16, 2019
prep.perl Jul 24, 2019
lib/acisEng.ttm May 16, 2019
Deposit:  
dea_full_data_update.py Feb 04, 2021
update_acis_ctemp.py Feb 04, 2021
update_acis_power.py Feb 04, 2021
update_compdatabase.py Feb 04, 2021
update_compgradkodak.py Feb 04, 2021
update_eph_l1.py Feb 04, 2021
update_graddata_from_comm.py Feb 04, 2021
update_grad_data.py Feb 04, 2021
update_sim_flex.py Feb 04, 2021
Simdiag:  
update_simdiag_data.py Feb 04, 2021
update_simsuppl_data.py Feb 09, 2021
Recover_fits:  
rerun_long.py Feb 04, 2021
rerun_short.py Feb 04, 2021
rerun_week.py Feb 04, 2021
Sun_angle:  
create_msid_sun_angle_file.py Feb 04, 2021
create_sun_angle_html_page.py Feb 04, 2021
create_top_sun_angle_html.py Feb 04, 2021
sun_angle_plot.py Feb 04, 2021
update_sun_angle_file.py Feb 04, 2021
Retired
EPH:  
update_eph_data_from_comm.py Feb 04, 2019
update_ephkey_data.py Feb 04, 2019
update_ephkey_l1_data.py Feb 04, 2019
Gratgen:  
create_otg_time_list.py Feb 04, 2019
gratgen_categorize_data.py Feb 04, 2019
HRC:  
classify_hrc_data.py Jun 26, 2019
update_hrcelec_data_hrchk.py May 28, 2019
find_hrc_period.py Oct 18, 2018
hrc_condition Sep 18, 2018
hrc_plot.py Oct 17, 2018
update_hrchk_elec_data.py Nov 16, 2018
get_hrcelec_from_archive_recover.py Jan 11, 2018
Hrcveto:  
classify_hrcveto_data.py May 29, 2019
extract_hrcveto_data.py May 28, 2019
refill_hrcveto.py Nov 14, 2018
week_refill_hrcveto.py Nov 14, 2018
Hrcveto_eph:  
create_hrcveto_eph_page.py May 28, 2019
hrcveto_eph_plot.py May 28, 2019
create_top_hrcveto_eph_html.py Feb 02, 2018
extract_simgdiag_data.py Jan 18, 2018
extract_simsapple_data.py Jan 18, 2018
msid_list_simdiag Jan 17, 2018
msid_list_simsupple Jan 17, 2018
expand_range.py Feb 07, 2018
find_dy_range.py Feb 08, 2018
find_range.py Feb 05, 2018
Tephin_eph:  
check_ephtemp_plotting_range.pyMay 28, 2019
create_ephin_leak_html_page.py May 28, 2019
create_top_ephin_leak_html.py May 28, 2019
tephin_leak_data_update.py Jun 03, 2019
tephin_plot.py May 28, 2019

Please read /data/mta/Script/MTA_limit_trends/Scripts/README file for more details.

python3.8 updatefuture warning element rectifiedprevious process checking mech added
Apr 06, 2018Initially logged
Apr 30, 2018update_eph_data_from_comm.py: cosmetic update
Jun 27, 2018classify_hrc_data.py bug fixed
Jul 03, 2018classify_hrc_data.py bug fixed
Jan 09, 2019sun angle missing data hang up bug fixed
Jan 09, 2019sun angle plotting routine hang up bug fixed
Jan 09, 2019dea data extraction separated and run on r2d2-v
Jan 10, 2019classify_hrc_data.py;selection method updated
Jun 26, 2019python3.6 update
Jul 25, 2019DEA memory filling bug fixed
Aug 30, 2019update_grad_and_comp_data.py: removing excessive none essential error notification
May 04, 2020create_msid_plots.py: short termp plottting bug fixed
Sep 03, 2020interactive page creating disabled
Jan 07, 2021year long plotting routine updated
Jan 07, 2021interactive page updated
Jan 07, 2021sun angle data aquisition bug fixed
Jan 07, 2021run_fetch.py added (originally part of create_interactive_page.py)
Jan 11, 2021dea data aquisition bug fixed
Feb 04, 2021
Feb 09, 2021
Feb 16, 2021

Data Needed

Environment Setting

Output Locations and File Names

  • v_table.sqlite3
  • possible_violation
  • tscpos_list
  • imbpast_list
  • hrc_off_list
  • retr_hetg
  • retr_letg
  • insr_hetg
  • insr_letg
  • grat_active
  • grat_inactive

    • How to Run the Script(s)

    1. setenv SKA /proj/sot/ska
      setenv PYTHONPATH "/data/mta/Script/Python3.8/envs/ska3-shiny/lib/python3.8/site-packages:/data/mta/Script/Python3.8/lib/python3.8/site-packages"
    2. go to: cd /data/mta/Script/MTA_limit_trends/Exc
    3. run: /data/mta/Script/MTA_limit_trends/Scripts/run_glimmon_trend_data_update.py msid_list_data week
    4. run: /data/mta/Script/MTA_limit_trends/Scripts/run_glimmon_trend_data_update.py msid_list_data short
    5. run: /data/mta/Script/MTA_limit_trends/Scripts/run_glimmon_trend_data_update.py msid_list_data long
    6. run: /data/mta/Script/MTA_limit_trends/Scripts/create_msid_plots.py msid_list_all week
    7. run: /data/mta/Script/MTA_limit_trends/Scripts/create_msid_plots.py msid_list_ <wday>short
    8. run: /data/mta/Script/MTA_limit_trends/Scripts/create_msid_plots.py msid_list_<wday> long
    9. run: /data/mta/Script/MTA_limit_trends/Scripts/DEA/convert_dea_data_to_fits.py
    10. run: /data/mta/Script/MTA_limit_trends/Scripts/Comp/update_grad_and_comp_data.py
    11. run: /data/mta/Script/MTA_limit_trends/Scripts/HRC/update_hrcelec_data_hrchk.py
    12. run: /data/mta/Script/MTA_limit_trends/Scripts/HRC/classify_hrc_data.py
    13. run: /data/mta/Script/MTA_limit_trends/Scripts/Hrcveto/extract_hrcveto_data.py
    14. run: /data/mta/Script/MTA_limit_trends/Scripts/Hrcveto/classify_hrcveto_data.py
    15. run: /data/mta/Script/MTA_limit_trends/Scripts/EPH/update_eph_data_from_comm.py
    16. run: /data/mta/Script/MTA_limit_trends/Scripts/EPH/update_ephkey_l1_data.py
    17. run: /data/mta/Script/MTA_limit_trends/Scripts/Simdiag/update_simdiag_data.py
    18. run: /data/mta/Script/MTA_limit_trends/Scripts/Simdiag/update_simsuppl_data.py
    19. run: /data/mta/Script/MTA_limit_trends/Scripts/remove_html_from_interactive_dir.py
    20. run: /data/mta/Script/MTA_limit_trends/Scripts/check_file_update_date.py
    21. go to: /data/mta/Script/MTA_limit_trends/Deposit
    22. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/update_compgradkodak.py
    23. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/update_grad_data.py
    24. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/update_acis_power.py
    25. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/update_sim_flex.py
    26. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/update_eph_l1.py
    27. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/update_acis_ctemp.py
    28. run: /data/mta/Script/MTA_limit_trends/Scripts/Deposit/dea_full_data_update.py
    29. go to: /data/mta/Script/MTA_limit_trends/Exc
    30. run: /data/mta/Script/MTA_limit_trends/Scripts/Sun_angle/update_sun_angle_file.py
    31. run: /data/mta/Script/MTA_limit_trends/Scripts/Sun_angle/create_msid_sun_angle_file.py msid_list_sun_angle
    32. run: /data/mta/Script/MTA_limit_trends/Scripts/Sun_angle/sun_angle_plot.py msid_list_sun_angle
    33. run: /data/mta/Script/MTA_limit_trends/Scripts/Sun_angle/create_sun_angle_html_page.py msid_list_sun_angle
    34. /data/mta/Script/MTA_limit_trends/Scripts/create_html_page.py qtype=static msid_list=msid_list ds=week ms=all
    35. /data/mta/Script/MTA_limit_trends/Scripts/create_html_page.py qtype=static msid_list=msid_list_p# ds=short ms=all
    36. /data/mta/Script/MTA_limit_trends/Scripts/create_html_page.py qtype=static msid_list=msid_list_p# ds=long ms=all
    37. /data/mta/Script/MTA_limit_trends/Scripts/update_sub_html_pages.py
    38. /data/mta/Script/MTA_limit_trends/Scripts/create_top_html.py
    39. /data/mta/Script/MTA_limit_trends/Scripts/remove_html_from_interactive_dir.py
    40. /data/mta/Script/MTA_limit_trends/Scripts/check_file_update_date.py

    How to Run Test(s)

    1. No test available for this set of scripts

    Expected result from the test

    test results -- test results -- test results

    Cron Job

    CPU: c3po-v
    User: mta

    16 0 * * * /data/mta/Script/MTA_limit_trends/Scripts/mta_limit_dea_wrap_script > $HOME/mta/Logs/mta_trends_dea_data_retrieve.cron 2>&1

    CPU: boba-v
    User: mta

    4 1 * * * cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/mta_limit_deposit_wrap_script > $HOME/Logs/msid_trend_deposit.cron 2>&1
    34 1 * * * cd /data/mta/Script/MTA_limit_trends/Exc2; /data/mta/Script/MTA_limit_trends/Scripts/mta_limit_simdiag_wrap_script > $HOME/Logs/msid_trend_simdiag.cron 2>&1
    08 2 * * 0 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_sun > $HOME/Logs/msid_trend_sun.cron 2>&1
    08 5 * * 0 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_sun3 > $HOME/Logs/msid_trend_short_sun.cron 2>&1
    08 2 * * 1 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_mon > $HOME/Logs/msid_trend_mon.cron 2>&1
    08 5 * * 1 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_mon3 > $HOME/Logs/msid_trend_short_mon.cron 2>&1
    08 2 * * 2 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_tus > $HOME/Logs/msid_trend_tus.cron 2>&1
    08 5 * * 2 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_tus3 > $HOME/Logs/msid_trend_short_tus.cron 2>&1
    08 2 * * 3 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_wed > $HOME/Logs/msid_trend_wed.cron 2>&1
    08 5 * * 3 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_wed3 > $HOME/Logs/msid_trend_short_wed.cron 2>&1
    08 2 * * 4 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_thu > $HOME/Logs/msid_trend_thu.cron 2>&1
    08 5 * * 4 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_thu3 > $HOME/Logs/msid_trend_short_thu.cron 2>&1
    08 2 * * 5 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_fri > $HOME/Logs/msid_trend_fri.cron 2>&1
    08 5 * * 5 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_fri3 > $HOME/Logs/msid_trend_short_fri.cron 2>&1
    08 2 * * 6 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_sat > $HOME/Logs/msid_trend_sat.cron 2>&1
    08 5 * * 6 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_sat3 > $HOME/Logs/msid_trend_short_sat.cron 2>&1
    24 9 * * 0,2,4 cd /data/mta/Script/MTA_limit_trends/Exc; /data/mta/Script/MTA_limit_trends/Scripts/mta_sun_angle_wrap_script > $HOME/Logs/msid_trend_sun_angle.cron 2>&1

    CPU: vodata
    User: mta

    34 2 * * 0 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_sun2 > $HOME/Logs/mta_trend_long_sun.cron 2>&1
    34 2 * * 1 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_mon2 > $HOME/Logs/mta_trend_long_mon.cron 2>&1
    34 2 * * 2 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_tus2 > $HOME/Logs/mta_trend_long_tus.cron 2>&1
    34 2 * * 3 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_wed2 > $HOME/Logs/mta_trend_long_wed.cron 2>&1
    34 2 * * 4 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_thu2 > $HOME/Logs/mta_trend_long_thu.cron 2>&1
    34 2 * * 5 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_fri2 > $HOME/Logs/mta_trend_long_fri.cron 2>&1
    34 2 * * 6 cd /data/mta/Script/MTA_limit_trends/Exc3; /data/mta/Script/MTA_limit_trends/Scripts/run_trend_wrap_script_sat2 > $HOME/Logs/mta_trend_long_sat.cron 2>&1

    Web Addreess

    https://cxc.cfa.harvard.edu/mta/MSID_Trends/mta_trending_main.html

    Note

    None

    Future Plan/Update Schedule

    None

    Back to Top Page