Folder Structure

.

├── LICENSE

├── README.md

├── back-up_reports_configs

│ ├── pre_prod_reports

│ └── prod_reports

├── cloud_storage

│ ├── __init__.py

│ ├── aws.py

│ ├── cloud.py

│ ├── gcp.py

│ ├── ms_azure.py

│ ├── oracle.py

│ └── py_cloud_migration.py

├── config.ini

├── config.sample

├── druid_data_product_query_config.txt

├── hp_reports_metrics

│ └── hawkeye_report_metric.py

├── manual_report_scripts

│ ├── domain_criteria_report.py

│ ├── question_domain_criteria_report.py

│ ├── question_report.py

│ └── status_report.py

├── migrations

│ ├── lib

│ └── releases

├── observations

│ ├── ml_observation_faust-data

│ ├── py_observation_batch_del_update.py

│ ├── py_observation_evidence_streaming.py

│ ├── py_observation_evidence_streaming_batch_refreshment.py

│ ├── py_observation_streaming.py

│ ├── py_observation_streaming_update_druid_data.py

│ ├── pyspark_obs_domain_batch.py

│ ├── pyspark_obs_status_batch.py

│ ├── pyspark_observation_rollup.py

│ ├── pyspark_observation_status_batch.py

│ ├── pyspark_observation_streaming_refreshment_batch.py

│ ├── pyspark_temp_punjab_entities.py

│ └── pyspark_temp_script_add_cols.py

├── projects

│ ├── __pycache__

│ ├── py_gather_program.py

│ ├── pyspark_project_batch _entire_db.py

│ ├── pyspark_project_batch.py

│ ├── pyspark_project_deletion_batch.py

│ ├── pyspark_project_pagination_batch.py

│ ├── pyspark_project_rollup.py

│ ├── pyspark_project_wo_pagination.py

│ ├── pyspark_started_inProgress_programdashboard.py

│ ├── pyspark_submitted_programdashboard.py

│ └── udf_func.py

├── python3 -> /usr/bin/python3

├── release-notes

│ └── 6.0.0.md

├── requirements.txt

├── run.sh

├── run_program.sh

├── run_weekly.sh

├── shell_script_config

├── survey

│ ├── ml_survey_evidence_faust-data

│ ├── ml_survey_faust-data

│ ├── py_survey_batch_del_update.py

│ ├── py_survey_evidence_streaming.py

│ ├── py_survey_evidence_streaming_batch_refreshment.py

│ ├── py_survey_streaming.py

│ ├── py_survey_streaming_batch_refreshment.py

│ ├── pyspark_survey_rollup.py

│ └── pyspark_survey_status.py

├── trust_review_scripts

│ ├── Trust Review Report.pdf

│ ├── active_users.py

│ ├── config.sample

│ └── ml_submissions_unique_users.py

└── urgent_data_metrics

└── imp_project_metrics.py

Module details

• back-up_reports_configs - In this folder, we store old reports that are no longer in use, solely for reference purposes.

• cloud_storage - Within this directory, we maintain the "Cloud" class. This class is utilized in scripts to facilitate the transmission of data or files to the cloud.

• hp_reports_metrics - Within this folder, we have scripts dedicated to generating CSV files specifically for program, solution, and district metrics.

• lib - Within this folder, we have scripts dedicated to generating and pussing logs to MongoDB

• manual_report_scripts - Within this folder, we have scripts dedicated to generating data related to observations.

• migrations - Within this folder, we have JSON data and script related to report creations.

• observations - Inside this directory, you can find both real-time and batch scripts intended for handling observations.

• projects - Inside this directory, you can find batch scripts intended for handling projects.

• release-notes - In this folder, we store documentation detailing the necessary steps and tasks to be carried out for deployment.

• survey - Inside this directory, you can find both real-time and batch scripts intended for handling Survey.

• trust_review_scripts -Within this folder, you can find scripts designed to provide metrics relevant to Trust review meetings.

• config.sample - This file stores all the essential variables needed for our scripts.

• druid_data_product_query_config.txt -This file contains the necessary URLs (CURLs) responsible for updating the program dashboard's CSV-related data.

• requirements.txt - This file includes a list of necessary packages required for the successful execution of all our scripts.

• run.sh - This script outlines the sequence and instructions for running our batch scripts in the intended order.

• run_program.sh - This script outlines the sequence and instructions for running selected scripts in the intended order.

• shell_script_config.sample - This file stores all the essential variables needed for our .sh files.