Build
Overview
To install SunbirdED release-4.10.0
, we will need to run many Jenkins jobs and a couple of manual steps. Some of the jobs need to be run in a specific order. This page details out the list of jobs, their order and the github release tags to be used for each job.
Create Plugin Containers / Folder Structure
We will create few folders in azure for storing sunbird plugins
Our deployment job always tries to delete the folder first before uploading, so we need to have the placeholders before running the jobs
We will create the folders by uploading random files from a random folder on your PC
Create a public container with the name you have set for the variable
sunbird_content_azure_storage_container
and run the below commands
Upload the default T&C file to
sunbird_content_azure_storage_container
container in a folder named terms-and-conditions. You can get a copy of the sample T&C file from here. You are free to edit the HTML as per your requirementsCreate a private container named
label
in AzureRun the below command in your Jenkins VM where you cloned the sunbird-devops repo
Upload Initial Plugins
Upload the initial set of plugins to
sunbird_content_azure_storage_container
containerDownload the initial Plugins from here
Run the below command from the directory where content-plugins.zip is unzipped
Create a container named
public
in your Azure storage account from the Azure portal with public access levelCreate a container named
artifacts
in your Azure storage account from the Azure portal with private access level
Upload Maxmind Database to Azure Blob
Download the Maxmind city database in zip format from Maxmind website
Upload the zip file to the
artifacts
container in Azure
Code Builds
Note:
Jenkins will have many jobs which may not be documented or is not suitable for adoption yet
Jobs listed on this page is sufficient to setup Sunbird and not every job present in Jenkins is required to be run
Jobs can be run in parallel to speed up execution. The only exception is you should NOT run multiple jobs which clone the code from same repo in parallel
If you get errors in some of the build jobs, rerun the job again
The errors are usually due to missing maven jars or maven repo timeouts
If you get errors even after re-running the job, come back to it later, there are other jobs which would generate dependent jars
JENKINS JOB TO RUN | GITHUB TAG | GITHUB REPO | COMMENTS |
---|---|---|---|
Build/Core/AdminUtils | release-4.5.0_RC3 | Handles mobile device registration, refresh token | |
Build/Core/Analytics | release-4.8.5_RC2 | Location Register, Location Retrieval | |
Build/Core/APIManager | release-4.10.0_RC1 | The API Gateway | |
Build/Core/ApiManagerEcho | master-RC1 | Simply echos the request | |
Build/Core/Assessment | release-4.10.0_RC2 | Handles course assessments | |
Build/Core/Bot | release-3.9.0_RC1 | Frontend for the chatbot | |
Build/Core/Cassandra | release-4.10.0_RC1 | Handles the Cassandra database schema | |
Build/Core/CassandraTrigger | release-4.6.0_RC1 | Creates few Cassandra Triggers | |
Build/Core/Cert | release-3.5.0_RC2 | Handles course certificates | |
Build/Core/CertRegistryService | release-4.9.0_RC1 | Helper for Certificate service | |
Build/Core/Content | release-4.10.0_RC2 | Handles the Contents and Content Metadata | |
Build/Core/Dial | release-4.9.0_RC1 | Handles QR code generation | |
Build/Core/DiscussionsMiddleware | release-4.7.0_RC4 | Middleware for discussion forum | |
Build/Core/EncService | release-3.8.0_RC1 | Encryption service to setup certificate signing keys | |
Build/Core/Groups | release-4.5.0_RC1 | Handles groups functions | |
Build/Core/Keycloak | release-3.8.0_RC1 | User authentication service | |
Build/Core/KnowledgeMW | release-4.8.0_RC2 | Middleware for Content service | |
Build/Core/Learner | release-4.10.0_RC4 | Handles user and organizations | |
Build/Core/Lms | release-4.9.0_RC4 | Handles courses | |
Build/Core/Nodebb | github_release_tag: release-4.7.0_RC1, nodebb_branch: v1.18.6 | Handles the discussion forum | |
Build/Core/Notification | release-4.6.0_RC2 | Handles notifications | |
Build/Core/Player | release-4.10.2_RC14 | Handles the UI elements | |
Build/Core/Print | release-3.8.0_RC1 | Handles PDF generation of certificates | |
Build/Core/Proxy | release-3.8.0_RC14 | Nginx web server / proxy | |
Build/Core/Report | release-4.10.0_RC6 | Handles some of the reports | |
Build/Core/Router | release-3.9.0_RC1 | Backend for chatbot | |
Build/Core/Search | release-4.10.0_RC2 | Handles contents / course search | |
Build/Core/Taxonomy | release-4.10.0_RC2 | Handles the taxonomies | |
Build/Core/Telemetry | release-3.3.0_RC1 | Handles the telemetry generated | |
Build/Core/Yarn | release-2.6.0 | Handles SSO and account merges | |
Build/KnowledgePlatform/CassandraTrigger | release-3.8.0_RC18 | Creates few Cassandra Triggers | |
Build/KnowledgePlatform/FlinkJobs | release-4.10.0_RC1 | Multiple functions like search indexing, video streaming | |
Build/KnowledgePlatform/Learning | github_release_tag: release-4.8.0_RC7, profile_id: platform-services | Handles frameworks | |
Build/KnowledgePlatform/Neo4j | release-4.5.0_RC12 | Generates neo4j plugins | |
Build/KnowledgePlatform/SyncTool | release-4.5.0_RC12 | Generates a jar to manually sync contents | |
Build/KnowledgePlatform/Yarn | release-4.9.1_RC1 | Multiple functions like publishing contents, issuing certificates | |
Build/DataPipeline/AdhocScripts | release-4.5.1_RC1 | Generates supporting scripts for reporting | |
Build/DataPipeline/AnalyticsCore | release-4.10.5_RC1 | Generates core libraries for reporting | |
Build/DataPipeline/CoreDataProducts | release-4.10.5_RC2 | Generates supporting libraries for reporting | |
Build/DataPipeline/EdDataProducts | release-4.10.5_RC3 | Generates supporting libraries for reporting | |
Build/DataPipeline/ETLJobs | release-4.10.5_RC2 | Extract, Transform, Data Loading jobs | |
Build/DataPipeline/FlinkPipelineJobs | release-4.9.0_RC2 | Handles processing of telemetry data | |
Build/DataPipeline/Secor | secor-0.29_RC2 | Handlles backup of telemetry data to Azure Blob | |
Build/Plugins/CollectionEditor | release-3.6.0_RC1 | A bunch of plugins to handle collection editing | |
Build/Plugins/ContentEditor | release-4.1.0_RC1 | A bunch of plugins to edit certain types of content | |
Build/Plugins/ContentPlayer | release-4.6.0_RC1 | A bunch of plugins to handle content playback | |
Build/Plugins/ContentPlugins | release-4.9.0_RC1 | A bunch of base plugins to support the content editors and content player | |
Build/Plugins/GenericEditor | release-3.6.0_RC1 | A bunch of plugins to edit certain types of content |
Infra Provision
Note:
Jobs can be run in parallel to speed up execution. The only exception is you should NOT run multiple jobs which will provision packages on the same server in parallel
JENKINS JOB TO RUN | GITHUB TAG | GITHUB REPO | COMMENTS |
---|---|---|---|
OpsAdministration/Core/Bootstrap | hosts: env, branch_or_tag: release-4.10.1_RC2, tag: bootstrap_any,node_exporter | Creates the deployer user and installs node exporter and python packages for ansible | |
OpsAdministration/KnowledgePlatform/Bootstrap | hosts: env, branch_or_tag: release-4.10.1_RC2, tag: bootstrap_any,node_exporter | Creates the deployer user and installs node exporter and python packages for ansible | |
OpsAdministration/DataPipeline/Bootstrap | hosts: env, branch_or_tag: release-4.10.1_RC2, tag: bootstrap_any,node_exporter | Creates the deployer user and installs node exporter and python packages for ansible | |
Provision/Core/ApplicationElasticSearch | release-4.10.1_RC2 | Installs Elasticsearch used by the applications | |
Provision/Core/Cassandra | release-4.10.1_RC2 | Installs Cassandra database | |
Provision/Core/CassandraExporter | release-4.10.1_RC2 | Installs Cassandra prometheus exporter | |
Provision/Core/ESMapping | indices_name: all, branch_or_tag: release-4.10.1_RC2 | Creates Elasticsearch indices and mappings | |
Provision/Core/Keycloak | release-4.10.1_RC2 | Installs java and other pre-requisites for Keycloak service | |
Provision/Core/LogES | release-4.10.1_RC2 | Installs Elasticsearch used to store application and VM logs | |
Provision/Core/MongodbCluster | release-4.10.1_RC2 | Installs Mongo database | |
Provision/Core/Postgres | release-4.10.1_RC2 | Installs Postgres database | |
Provision/Core/PostgresDbUpdate | release-4.10.1_RC2 | Creates Postgres tables, schema and users | |
Provision/KnowledgePlatform/Learning | release-4.10.0_RC1 | Install tomcat and other pre-requisites for Learning service | |
Provision/KnowledgePlatform/Neo4j | release-4.10.0_RC1 | Installs Neo4j database | |
Provision/KnowledgePlatform/Yarn | release-4.10.0_RC1 | Install Hadoop and other pre-requisites to run Yarn | |
Provision/DataPipeline/AnalyticsGeoLocationDBSetup | release-4.10.5_RC3, geoip_zip_csv_file_name: Enter the maxmind city database zip file name you uploaded to | Creates tables and schema in Postgres | |
Provision/DataPipeline/AnalyticsSpark | release-4.10.5_RC3 | Install Apache Spark and other pre-requisites for reporting | |
Provision/DataPipeline/Druid | release-4.10.5_RC3, service: Select All, remote: raw | Installs Druid | |
Provision/DataPipeline/InfluxDB | release-4.10.5_RC3 | Install Influx DB used by monitoring system | |
Provision/DataPipeline/Kafka | release-4.10.5_RC3 | Installs Kafka | |
Provision/DataPipeline/KafkaIndexer | release-4.10.5_RC3 | Install Logstash to read telemetry data from Kafka and index into Log ES | |
Provision/DataPipeline/Postgres | release-4.10.5_RC3 | Installs Postgres database and users | |
Provision/DataPipeline/PostgresDbUpdate | release-4.10.5_RC3 | Creates Postgres tables, schema and users | |
Provision/DataPipeline/Redis | release-4.10.5_RC3 | Install Redis | |
Provision/DataPipeline/Zookeeper | release-4.10.5_RC3 | Installs Zookeeper |
ArtifactUpload
Every job in the Build directory has a corresponding job in ArtifactUpload directory with the same name
These jobs are auto triggered and usually run without issues after the corresponding job in the build directory succeeds
The job will fail if your ansible inventory setup is incorrect or incomplete
If the job has failed, fix the ansible variables issue and rerun the job to upload the artifact / docker image
Ensure the artifact upload jobs are successful before proceeding
Code Deploy
Note:
We will run the jobs which are a pre-requisite for other jobs first
The pre-requisite jobs reside in different folders so we will be jumping across folders
Jobs in the Deploy directory can be run in parallel and don’t have any restrictions if the code is from same repo unlike the Build directory
Ensure you don’t run those jobs in parallel which modify the databases (such as cassandra, neo4j etc)
JENKINS JOB TO RUN | GITHUB TAG | GITHUB REPO | COMMENTS |
---|---|---|---|
Deploy/Kubernetes/BootstrapMinimal | release-4.10.1_RC2 | Creates namespaces, configmaps and secrets | |
Deploy/DataPipeline/BootstrapMinimal | release-4.10.1_RC2 | Creates namespaces, configmaps and secrets | |
Deploy/Kubernetes/Monitoring | branch_or_tag: release-4.10.1_RC2, tag: all | Deploys the monitoring stack which can be accessed via DOMAIN/grafana/ post nginx-public-ingress deployment | |
Deploy/Kubernetes/CassandraTrigger | release-4.10.1_RC2 | Installs the Cassandra triggers | |
Deploy/KnowledgePlatform/CassandraTrigger | release-4.10.0_RC1 | Installs the Cassandra triggers | |
Deploy/Kubernetes/CassandraDBUpdate | release-4.10.1_RC2 | Creates / Updates Cassandra DB schema | |
Deploy/KnowledgePlatform/CassandraDbUpdate | release-4.10.0_RC1 | Creates / Updates Cassandra DB schema | |
Deploy/Kubernetes/Cassandra | release-4.10.1_RC2 | Creates / Updates Cassandra DB schema | |
Deploy/Kubernetes/KafkaSetup | release-4.10.1_RC2 | Creates / Updates Kafka Topics | |
Deploy/DataPipeline/KafkaSetup | release-4.10.5_RC3 | Creates / Updates Kafka Topics | |
Deploy/KnowledgePlatform/KafkaSetup | release-4.10.0_RC1 | Creates / Updates Kafka Topics | |
Deploy/KnowledgePlatform/Learning | release-4.10.0_RC1 | Deploys the Learning service on tomcat | |
Deploy/KnowledgePlatform/Neo4j | release-4.10.0_RC1 | Deploys the Neo4j plugins | |
Deploy/KnowledgePlatform/Neo4jDefinitionUpdate | release-4.10.0_RC1 | Creates / Updates the Neo4j database | |
Deploy/KnowledgePlatform/Yarn | release-4.10.0_RC1 | Deploys the Samza jobs | |
Deploy/KnowledgePlatform/FlinkJobs | branch_or_tag: release-4.10.0_RC1, job_names_to_deploy: Select All | Deploys the Flink jobs in Kubernetes cluster | |
Deploy/KnowledgePlatform/LoggingFileBeatsVM | branch_or_tag: release-4.10.1_RC2, hosts: Select All, tags: default | Deploys filebeat in all the selected VMs | |
Deploy/Kubernetes/nginx-private-ingress | release-4.10.1_RC2 | Deploys nginx with a private load balancer | |
Deploy/Kubernetes/APIManager | release-4.10.1_RC2 | Deploys Kong, the API gateway | |
Deploy/Kubernetes/OnboardAPIs | release-4.10.1_RC2 | Creates the APIs in Kong | |
Deploy/Kubernetes/OnboardConsumers | release-4.10.1_RC2 | Creates the consumers in Kong. Now read the comment mentioned in Core/Secrets.yml against the variable | |
Deploy/Kubernetes/Keycloak | release-4.10.1_RC2 | Deploys Keycloak | |
Deploy/Kubernetes/KeycloakRealm | release-4.10.1_RC2 | Creates the Sunbird Realm. Now read the comment mentioned in Core/Secrets.yml against the variables | |
Deploy/Kubernetes/BootstrapMongodb | release-4.10.1_RC2 | Creates the initial token in Mongo database | |
Deploy/Kubernetes/CertTemplate | branch_or_tag: release-4.10.1_RC2, sunbird_util_branch_or_tag: release-4.10.0_RC1 | Uploads the certificate templates into Azure blob | |
Deploy/Kubernetes/UploadChatbotConfig | branch_or_tag: release-4.10.1_RC2, bot_repo_branch: master | Uploads the chatbot configs to Azure blob | |
Deploy/Kubernetes/UploadFAQs | branch_or_tag: release-4.10.1_RC2, source_folder: Select All | Uploads the FAQs to Azure blob | |
Deploy/Kubernetes/UploadSchema | branch_or_tag: release-4.10.1_RC2, kp_branch_or_tag: release-4.9.0_RC8 | Uploads the Content schemas to Azure blob | |
Deploy/Kubernetes/AdminUtils | release-4.10.1_RC2 | Deploys Adminutils service | |
Deploy/Kubernetes/Analytics | release-4.10.1_RC2 | Deploys Analytics service | |
Deploy/Kubernetes/APIManagerEcho | release-4.10.1_RC2 | Deploys Echo service | |
Deploy/Kubernetes/Assessment | release-4.10.1_RC2 | Deploys Assessment service | |
Deploy/Kubernetes/Bot | release-4.10.1_RC2 | Deploys Bot service | |
Deploy/Kubernetes/Cert | release-4.10.1_RC2 | Deploys Cert service | |
Deploy/Kubernetes/CertRegistry | release-4.10.1_RC2 | Deploys CertRegistry service | |
Deploy/Kubernetes/Content | release-4.10.1_RC2 | Deploys Content service | |
Deploy/Kubernetes/Dial | release-4.10.1_RC2 | Deploys Dial service | |
Deploy/Kubernetes/Enc | release-4.10.1_RC2 | Deploys Enc service | |
Deploy/Kubernetes/Groups | release-4.10.1_RC2 | Deploys Groups service | |
Deploy/Kubernetes/KnowledgeMW | release-4.10.1_RC2 | Deploys KnowledgeMW service | |
Deploy/Kubernetes/Learner | release-4.10.1_RC2 | Deploys Learner service | |
Deploy/Kubernetes/Lms | release-4.10.1_RC2 | Deploys Lms service | |
Deploy/Kubernetes/Notification | release-4.10.1_RC2 | Deploys Notification service | |
Deploy/Kubernetes/Player | release-4.10.1_RC2 | Deploys Player service | |
Deploy/Kubernetes/Print | release-4.10.1_RC2 | Deploys Print service | |
Deploy/Kubernetes/Report | release-4.10.1_RC2 | Deploys Report service | |
Deploy/Kubernetes/Router | release-4.10.1_RC2 | Deploys Router service | |
Deploy/Kubernetes/Search | release-4.10.1_RC2 | Deploys Search service | |
Deploy/Kubernetes/Taxonomy | release-4.10.1_RC2 | Deploys Taxonomy service | |
Deploy/Kubernetes/Telemetry | release-4.10.1_RC2 | Deploys Telemetry service | |
Deploy/Kubernetes/Yarn | release-4.10.1_RC2 | Deploys Samza jobs | |
Deploy/Kubernetes/LoggingFileBeatsVM | branch_or_tag: release-4.10.1_RC1, hosts: Select All, tags: default | Deploys filebeat in all the selected VMs | |
Deploy/Kubernetes/Logging | release-4.10.1_RC2 | Choose one chartname each time - | |
Deploy/Kubernetes/nginx-public-ingress | release-4.10.1_RC2 | Deploys the nginx web server. Now you can open visit your domain | |
Deploy/Kubernetes/Nodebb | release-4.10.1_RC2 | Deploys NodeBB service. Now read the comment mentioned in Core/Secrets.yml against the variable | |
Deploy/Kubernetes/DiscussionsMW | release-4.10.1_RC2 | Deploys DiscussionsMW service | |
Deploy/Plugins/CollectionEditor | release-4.10.1_RC2 | Deploys CollectionEditor files to Azure blob | |
Deploy/Plugins/ContentEditor | release-4.10.1_RC2 | Deploys ContentEditor files to Azure blob | |
Deploy/Plugins/ContentPlayer | release-4.10.1_RC2 | Deploys ContentPlayer files to Azure blob | |
Deploy/Plugins/ContentPlugins | release-4.10.1_RC2 | Deploys ContentPlugins files to Azure blob | |
Deploy/Plugins/GenericEditor | release-4.10.1_RC2 | Deploys GenericEditor files to Azure blob | |
Deploy/DataPipeline/AdhocScripts | release-4.10.5_RC3 | Deploys AdHoc scripts in Spark VM | |
Deploy/DataPipeline/AnalyticsCore | release-4.10.5_RC3 | Deploys jars in Spark VM | |
Deploy/DataPipeline/CoreDataProducts | release-4.10.5_RC3 | Deploys jars in Spark VM | |
Deploy/DataPipeline/DruidIngestion | branch_or_tag: release-4.10.5_RC3, ingestion_task_names: Select All | Deploys and starts Druid ingestion tasks. This job will be updated in a couple of days. Please wait until then | |
Deploy/DataPipeline/EdDataProducts | release-4.10.5_RC3 | Deploys jars in Spark VM | |
Deploy/DataPipeline/ETLJobs | release-4.10.5_RC3 | Deploys ETL scripts in Spark VM | |
Deploy/DataPipeline/ETLDruidContentIndexer | release-4.10.5_RC3 | Deploys ETLDruidContentIndexer | |
Deploy/DataPipeline/ETLUserCacheIndexer | release-4.10.5_RC3 | Deploys ETLUserCacheIndexer | |
Deploy/DataPipeline/Secor | release-4.10.5_RC3 | Deploys Secor Statefulsets in Kubernetes cluster | |
Deploy/DataPipeline/FlinkPipelineJobs | release-4.10.5_RC3 | Deploys Flink jobs Kubernetes cluster | |
Deploy/DataPipeline/GraphitePrometheusExporter | release-4.10.5_RC3 | Deploys graphite prometheus exporter in Druid VM | |
Deploy/Kubernetes/LoggingFileBeatsVM | branch_or_tag: release-4.10.1_RC2, hosts: Select All, tags: default | Deploys filebeat in all the selected VMs |
\
Last updated