Introduction

Welcome to the Samagra DevOps guide!

Whether you're a startup or a large enterprise, your ability to deliver software quickly, reliably, and securely is critical to your success. That's where DevOps comes in.

DevOps is a set of practices that combines development and operations teams to work together in a more agile, collaborative, and automated way.

At Samagra, we're passionate about adopting DevOps practices and improve the software delivery capabilities. In this guide, we'll explore the key concepts, tools, and processes that make up a successful DevOps strategy. Whether you're just getting started with DevOps or looking to improve your existing practices, this guide will provide practical tips and best practices to help you achieve your goals.

Let's get started with the guide.

Infrastructure Overview

Development : Developers work on code changes locally and push their changes to a GitHub repository.
GitHub Actions : When a code change is pushed to GitHub, a GitHub Action is triggered to run automated tests on the code. This ensures that any issues are caught early and can be fixed before they are deployed.
GitHub Webhooks : Once the code is merged, GitHub webhooks trigger a Jenkins job to build and deploy the code changes to a staging environment.
Jenkins : Jenkins runs a build pipeline to verify the build.
Deploy : Jenkins has another pipeline (deploy-staging) which deploy the code changes to the production environment using Ansible playbooks.
Hashicorp Vault: It can be used to manage secrets that are required by the various tools and processes involved in the pipeline. Ansible fetches these secrets from here.

Github

Repositories

samagra-devops : It is the main repo that contains Jenkins job and Ansible roles for services that are reproducible in other projects at Samagra.
Each project then contains a (project-name)-DevOps repo. For example ULP-Devops etc.

Project Structure

For each project which we have as pipeline, In their github repository we follow a common structure for getting the Dockerfile and Jenkinsfile.

root-project-repo/
    ├── Dockerfile
    ├── build/       
            ├── Jenkinsfile

The Dockerfile needs to be tested by developers for any errors and once it is ready, we can use it to deploy the project in private docker registry on docker swarm.

The Jenkinsfile contains all the necessary steps for several stages in a pipeline which eventually builds the project on server and deploys it.

Webhook URLs

In each project, we have the webhook which triggers the events occuring on github to the jenkins which helps in detecting the changes going in a repository.

Jenkins View

Dashboard

It is the main Jenkins screen which you would be able to see just after login.

Project Space

In project space there are directories for specific projects or job.

Pipelines

For each job we have two pipelines.

build
deploy-staging
deploy-prod (third pipeline yet to come)

Build Pipeline

This is a multibranch pipeline and it can make Docker image builds of all branches and new PRs that are created. Images are tagged by branch name or PR-(Number) and pushed to a private docker registry.

Deploy-staging

Whenever commits are made to main/master branch, this job automatically deploys the code to the server and pushes a notification on the discord channel regarding the success and failure of the jobs.

Deploy-prod

This pipeline will be used to deploy to the prod server and will be manually used to deploy the changes

Jenkins Configuration

As we have the build pipeline running the stages as defined in the Jenkinsfile for any project, we have a lot more going once the build is succedded. Many things are configured into the deploy staging pipeline.

Running the ansible playbook for testing.
Sending the build status notification to Samagra discord server.

Jenkins allow us to configure all such tasks.

Creating build tags as string parameters.

Running Shell commands after each build runs.

Here as you can see, the tag parameter which we created in step 1 gets used with ansible-playbook. You can create several parameters if needed.

Setting up discord notifier using webhook URL.

If need arises, you can manually deploy the projects using the build with parameters option.

Jenkins Build Runs

You would wish to see the build history and the logs in case if the build fails. This section is all about knowing about where the builds history would be visible and where you can see the output logs.

Now, If you click on any build ( Eg #45 ) then you get more clear overview about the job.

Setup of DevOps pipeline

DevOps pipeline has the following main components.

Jenkins Server
Hashicorp Vault
Docker Swarm (This could be your localhost as well)
Database Server (This could be your localhost as well, same as Docker Swarm)
Private Docker Registry
Ansible
Preferred OS environment is Linux, Setup is not tested on windows

Setup

Install Jenkins
Setup Hashicorp Vault
- Generate Unseal keys and vault token
Install Docker
Initialize Docker Swarm
- Docker swarm needs an external overlay network to be created. Create a network named application_default in Docker
Install ansible on Docker Swarm and Database servers

Setup Docker Registry

Use the following compose for private registry setup.

version: '3'

services:
  registry:
    image: registry:2
    ports:
    - "[Replace this with private_ip or nothing]:5000:5000"
    environment:
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data
    volumes:
      - ./data:/data

Configure

Configure Jenkins
- Setup Pipelines
  - Go to /home location and clone the <project>-devops pipelines. For example, go to /home and clone https://github.com/Unified-Learner-Passbook/ULP-devops
  - Create symbolic links to Jenkins jobs ln -s /home/ulp-devops/jobs/ULP /var/lib/jenkins/jobs/
  - Set permissions chown -R jenkins:jenkins jobs && chown -R jenkins:jenkins /var/lib/jenkins
  - Restart Jenkins - systemctl restart jenkins. You should be able to see all the Jobs on the Jenkins Dashboard.
- Setup environment variables in Jenkins
  - Go to Dashboard -> Manage Jenkins -> Configure System
  - Select environment variables checkbox and add VAULT_ADDR_DEV and VAULT_TOKEN_DEV with corresponding values you got while setting up Hashicorp Vault
  - These variables are needed in the deploy jobs in the pipeline.
Configure Ansible
- jenkins user should be able to ssh on to the Docker Swarm and Database Servers
  - Switch to jenkins user and run ssh-keygen. Select all default options.
  - Copy public key from ~/.ssh/id_rsa.pub and paste it under ~/.ssh/authorized_keys on the Docker Swarm and Database Server
- Configure Ansible Hosts
  - Go to https://github.com/Unified-Learner-Passbook/ULP-devops/blob/master/ansible_workspace_dir/inventory/hosts
  - Uncomment the localhost lines or add localhost under dev group. You can read more about ansible hosts here

Samagra-Devops-Guide