Elastic Search Cluster on Ubuntu 14.04

Elastic Search is a popular open source search server that is used for real-time distributed search and analysis of data. When used for anything other than development, Elastic Search should be deployed across multiple servers as a cluster, for the best performance, stability, and scalability.

Demonstration:

OMegha Platform.

Image – Ubuntu-14.04

Prerequisites:

You must have at least three Ubuntu 14.04 servers to complete this, because an Elastic Search cluster should have a minimum of 3 master-eligible nodes. If you want to have dedicated master and data nodes, you will need at least 3 servers for your master nodes plus additional servers for your data nodes.

Install Java 8:

Elastic Search requires Java, so we will install that now. We will install a recent version of Oracle Java 8 because that is what Elastic Search recommends. It should, however, work fine with OpenJDK, if you decide to go that route.

Complete this step on all of your Elastic Search servers.

Add the Oracle Java PPA to apt:

$ sudo add-apt-repository -y ppa:webupd8team/java

Update your apt package database:

$ sudo apt-get update

Install the latest stable version of Oracle Java 8 with this command (and accept the license agreement that pops up):

$ sudo apt-get -y install oracle-java8-installer

Be sure to repeat this step on all of your Elastic Search servers.

Now that Java 8 is installed, let’s install Elastic Search.

Install Elastic Search:

Elastic Search can be downloaded directly from elastic.co in zip, tar.gz, deb, or rpm packages. For Ubuntu, it’s best to use the deb (Debian) package which will install everything you need to run Elastic Search.

$ wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb

Then install it in the usual Ubuntu way with the dpkg command like this:

$ sudo dpkg -i elasticsearch-1.7.2.deb

This results in Elastic Search being installed in /usr/share/elastic Search/ with its configuration files placed in /etc/elastic Search and its init script added in /etc/init.d/elastic search.

$ sudo update-rc.d elasticsearch defaults

Be sure to repeat these steps on all of your Elastic Search servers.

Elastic Search is now installed but it needs to be configured before you can use it.

Configure Elastic search Cluster

Now it’s time to edit the Elastic search configuration. Complete these steps on all of your Elastic search servers.

Open the Elastic search configuration file for editing:

$ sudo vi /etc/elasticsearch/elasticsearch.yml

Set Cluster Name:

Next, set the name of your cluster, which will allow your Elastic search nodes to join and form the cluster. You will want to use a descriptive name that is unique (within your network).

Find the line that specifies cluster.name, uncomment it, and replace its value with the your desired cluster name. In this tutorial, we will name our cluster “elastic search_cluster”:

Set Node Name:

Next, we will set the name of each node. This should be a descriptive name that is unique within the cluster.

Find the line that specifies node.name, uncomment it, and replace its value with your desired node name. In this tutorial, we will set each node name to the host name of server by using the ${HOSTNAME}environment variable:

For Master Node:

For Master set the node.master as True and for node.data as False

For Data Node:

For Data set the node.master as False and for node.data as True

Network Host:

Set the network host as 0.0.0.0

Set Discovery Hosts:

Next, you will need to configure an initial list of nodes that will be contacted to discover and form a cluster. This is necessary in a unicast network.

Find the line that specifies discovery.zen.ping.unicast.hosts and uncomment it.

For example, if you have three servers node01, node02, and node03 with respective VPN IP addresses of 10.0.0.1, 10.0.0.2, and 10.0.0.3, you could use this line:

Save and Exit.

Your servers are now configured to form a basic Elastic search cluster. There are more settings that you will want to update, but we’ll get to those after we verify that the cluster is working.

Save and exit elasticsearch.yml.

Start Elastic search:

Now start Elastic search:

$ sudo service elasticsearch restart

Then run this command to start Elastic search on boot up:

$ sudo update-rc.d elasticsearch defaults 95 10

Be sure to repeat these steps (Configure Elastic search) on all of your Elastic search servers.

Testing:

By now, Elastic search should be running on port 9200. You can test it with curl, the command line client-side URL transfers tool and a simple GET request like this:

$ curl -X GET 'http://localhost:9200'

You should see the following response:

If you see a response similar to the one above, Elastic search is working properly. If not, make sure that you have followed correctly the installation instructions and you have allowed some time for Elastic search to fully start.

Check Cluster State:

If everything was configured correctly, your Elastic search cluster should be up and running. Before moving on, let’s verify that it’s working properly. You can do so by querying Elastic search from any of the Elastic search nodes.

From any of your Elastic search servers, run this command to print the state of the cluster:

$ curl -XGET 'http://localhost:9200/_cluster/state?pretty'

If you see output that is similar to this, your Elastic search cluster is running! If any of your nodes are missing, review the configuration for the node(s) in question before moving on.

Next, we’ll go over some configuration settings that you should consider for your Elastic search cluster.

Enable Memory Locking:

Elastic recommends to avoid swapping the Elastic search process at all costs, due to its negative effects on performance and stability. One way avoid excessive swapping is to configure Elastic search to lock the memory that it needs.

Complete this step on all of your Elastic search servers.

Edit the Elastic search configuration:

$ sudo vi /etc/elasticsearch/elasticsearch.yml

Find the line that specifies bootstrap.mlockall and uncomment it:

Save and exit.

Now restart Elastic search to put the changes into place:

$ sudo service elasticsearch restart

Cluster Health:

This API can be used to see general info on the cluster and gauge its health:

$ curl -XGET 'localhost:9200/_cluster/health?pretty'

Cluster State:

This API can be sued to see a detailed status report on your entire cluster. You can filter results by specifying parameters in the call URL.

$ curl -XGET 'localhost:9200/_cluster/state?pretty'

Conclusion:

Your Elastic search cluster should be running in a healthy state, and configured with some basic optimizations.

 

Node.js Installation

Node.js is a cross-platform environment and library for running JavaScript applications which are used to create networking and server-side applications.

It is used to develop I/O intensive web applications like video streaming sites, single-page applications, and other web applications.

Prerequisites

To follow this blog, you will need:

• One Ubuntu 16.04 server set up by following this initial server setup, including a sudo non-root user.

Node.js Installation In Ubuntu 16.04

Step-1 Update the Package List

Before installing the node.js on to the Ubuntu system, update all available repositories.

sudo apt-get update

Step-2  Install Node.js

Run below command to install standard package of node.js

sudo apt-get install nodejs

Step-3 Install NPM

For installing Node.js the npm package is also required. By using below command we install the npm package.

sudo apt-get install npm

In order for some npm packages to work (those that require compiling code from source, for example), you will need to install the build-essential package:

sudo apt-get install build-essential

Installation Check

After Installing the node.js and npm we check that installation is correct or not by typing the following commands:

Node.js Installation Check

nodejs --version

NPM Installation Check

npm --version

Remove Node.js

For removing the node.js from ubuntu system completely we type following commands:

Remove Package without Configuration Files

This command removes node.js but configuration files remain so next time we install node.js than it will be used.

sudo apt-get remove nodejs

Remove Package with Configuration Files

If you don’t want to kept the configuration files then use following command.

sudo apt-get purge nodejs

Finally, Remove Unused Packages

For removing the unused packages that are installed with node.js run the following command:

sudo apt-get autoremove 

Installing Open Source Hosting Control Panel (ZesleCP) on Ubuntu 16.04

Zesle Control Panel

Secure Web Control Panel for your needs…

Zesle is one of the popular open source control panel that any own can download and install. This is very simple and can be installed in just one command.

System Requirements:

• Ubuntu 14/16
• 1Ghz CPU
• 512MB RAM
• 10+GB DISK

Zesle is simpler and very User friendly.  Using Zesle you’ll be able to do the below tasks…

• Add multiple domains without hassle;
• Add multiple sub domains;
• Install WordPress easily with one-click app;
• Install free Let’s Encrypt SSL certificates with ease;
• Configure basic PHP settings;
• Manage Email accounts;
• Access phpMyAdmin.

and much more. Let’s see how to install Zesle in your hosting.

Step 1: It’s super-easy to install Zesle. Run the following command with Root privilege.

$ cd /home && curl -o latest -L http://zeslecp.com/release/latest && sh latest

Step 2: The installation will begin and in between It will ask the admin’s email address. Please provide your email id and click enter.

Step 3: You will see the below screen at the end of the installation.

Step 4: This is how Zesle will looks like. Once the installation is completed, it’ll show you the temporary password and the login URL.

Step 5: The login URL will be your IP address followed by the port number(2087 will be the default one). For example: https://11.11.11.111:2087 is a sample URL.

Step 6: Just enter this in your browser and you’ll get the login screen.

Step 7: Use root as your username

Step 8: Copy paste the temporary root password provided. Once you entered the correct password, the control panel will open and here is how it looks like. All the options mentioned above will be available in the left side of the UI.

Step 9: In the Dashboard, you can create your account, and install WordPress on your domain using “Once Click Apps“

Step 10: There is installation steps end for free Linux Web Hosting control panel called ZesleCP.

 

Installation of Open Project Management System on Ubuntu 16.04

Open Project is a web-based management system for location-independent team
collaboration, released under GNU GPL 3 License. It’s a project management software
that provides task-management, team collaboration, scrum etc. Open Project is written
in Ruby on Rails and AngularJS. In this tutorial, I will show you how to install and
configure the Open Project management system using Ubuntu 16.04. The tool can be
installed manually or by using packages from the repository. For t his guide, we will
install Open Project from repository.

Prerequisites

•  Ubuntu 16.04.
•  Good Internet Connectivity.
•  Root Privileges.

What we will do

• Update and Upgrade System.
• Install Open Project Management System.
• Configure the Open Project System.
• Testing.

Step 1: Update and Upgrade System

Before installing the Open Project on to the Ubuntu system, update all available repositories and upgrade the Ubuntu system.

Run the following commands.

$ sudo apt update
$ sudo apt upgrade -y

Step 2: Install Open Project Management System

Download the open project key and add it to the system.

$ sudo wget -qO- https://dl.packager.io/srv/opf/openproject-ce/key | sudo apt-key add

And download the open project repository for Ubuntu 16.04 in the ‘/etc/apt/sources.list.d’ directory.

$ sudo wget -O /etc/apt/sources.list.d/openproject-ce.list \
  https://dl.packager.io/srv/opf/openproject-ce/stable/7/installer/ubuntu/16.04.repo

Now update the Ubuntu repository and install open project using the apt command as shown below.

$ sudo apt update
$ sudo apt-get install openproject -y

Step 3: Configure the Open Project System

Run the Open Project configuration command. A Graphical based UI screen will appear.

$  sudo openproject configure

Select ‘Install and configure MySQL server locally’ and click ‘OK’. It will automatically install MySQL server on the system, and automatically create the database for openproject installation.

For the web server configuration, choose the ‘Install apache2 server’ and click ‘OK’. it will automatically install the apache2 web server and configure the virtual host for Open Project application.

Now type the domain name for your Open project application, and choose ‘OK’.

Next, for the SSL configuration. If you have purchased SSL certificates, choose ‘yes’, and ‘no’ if you don’t have SSL certificates.

Skip the subversion support, GitHub support and SMTP configuration. (if not needed).

And for memcached installation choose ‘Install’ and select ‘OK’ for better performance of Open Project.

Finally, installation and configuration of all the packages required for Open Project installation should happen automatically.

Step 4: Testing

Check whether the Open Project service is up and running.

$  sudo service openproject status

Now run the openproject web service using the following command.

$  sudo openproject run web

Now open your web browser and type on the address bar your floating IP to access the system

Now click the ‘Sign in’ button to log in to the admin dashboard initially using ‘admin’ as user and ‘admin’ as password and later you can change it.

Finally, the installation and configuration for Open Project on Ubuntu 16.04 has been completed successfully.

 

 

 

Apache Virtual Hosts on Ubuntu 14.04

The Apache web server is the most popular way of serving web content on the internet. It accounts for more than half of all active websites on the internet and is extremely powerful and flexible.

Apache breaks its functionality and components into individual units that can be customized and configured independently. The basic unit that describes an individual site or domain is called a Virtual Host.

These designations allow the administrator to use one server to host multiple domains or sites off of a single interface or IP by using a matching mechanism. This is relevant to anyone looking to host more than one site off of a single VPS.

In this document, we will walk you through how to set up Apache virtual hosts on an Ubuntu 14.04 VPS. During this process, you’ll learn how to serve different content to different visitors depending on which domains they are requesting.

Prerequisites

• Before you begin this tutorial, you should create a non root user.
• You will also need to have Apache installed in order to work through these steps.

Demonstration:

OMegha platform.

Image – Ubuntu-14.04

Lets get Started,

At first we need to update the packages list.

$ sudo apt-get update

Install Apache

$ sudo apt-get install apache2

For the purposes of this document, my configuration will make a virtual host for infra.com and another for infra1.com

Step 1: Create the Directory Structure

Our document root will be set to individual directories under the /var/www directory. We will create a directory here for both of the virtual hosts we plan on making.

Within each of these directories, we will create a public_html folder that will hold our actual files. This gives us some flexibility in our hosting.

$ sudo mkdir -p /var/www/infra.com/public_html

$ sudo mkdir -p /var/www/infra1.com/public_html

The portions in red represent the domain names that we are wanting to serve from our VPS.

Step 2: Grant Permissions

Changing the Ownership

$ sudo chown -R $USER:$USER /var/www/infra.com/public_html

$ sudo chown -R $USER:$USER /var/www/infra1.com/public_html

We should also modify our permissions a little bit to ensure that read access is permitted to the general web directory and all of the files and folders

$ sudo chmod -R 755 /var/www

Step 3: Create Demo Pages for Each Virtual Host

We have to create index.html file for each site.

Let’s start with infra.com. We can open up an index.html file in our editor by typing

$ sudo vi /var/www/infra.com/public_html/index.html

In this file, create a simple HTML document that indicates the site it is connected to and My file looks like this

<html>

  <head>

    <title>Welcome to infra.com!</title>

  </head>

  <body>

    <h1>Success!  The infra.com virtual host is working!</h1>

  </body>

</html>

Save and close the file when you are finished.

We can copy this file to use as the basis for our second site by typing

cp /var/www/infra.com/public_html/index.html /var/www/infra1.com/public_html/index.html

Then we can open the file and modify the relevant pieces of information

$ sudo vi /var/www/infra1.com/public_html/index.html

<html>

  <head>

    <title>Welcome to infra1.com!</title>

  </head>

  <body>

    <h1>Success!  The infra1.com virtual host is working!</h1>

  </body>

</html>

Save and close the file.

Step 4: Create New Virtual Host Files

Virtual host files are the files that specify the actual configuration of our virtual hosts and dictate how the Apache web server will respond to various domain requests.

Apache comes with a default virtual host file called 000-default.conf and we can
copy that to our first domain of the virtual host file.

Creating First Virtual Host File

Start by copying the file for the first domain

$ sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/infra.com.conf

Open the new file in your editor with root privileges

$ sudo vi /etc/apache2/sites-available/infra.com.conf

our virtual host file should look like this

<VirtualHost *:80>

    ServerAdmin admin@infra.com

    ServerName infra.com

    ServerAlias www.infra.com

    DocumentRoot /var/www/infra.com/public_html

    ErrorLog ${APACHE_LOG_DIR}/error.log

    CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

Save and close the file.

Copy first Virtual Host and Customize for Second Domain

Now that we have our first virtual host file established, we can create our second one by copying that file and adjusting it as needed.

Start by copying

$ sudo cp /etc/apache2/sites-available/infra.com.conf /etc/apache2/sites-available/infra1.com.conf

Open the new file with root privileges

$ sudo vi /etc/apache2/sites-available/infra1.com.conf

You now need to modify all of the pieces of information to reference your second domain. When you are finished, it may look something like this

<VirtualHost *:80>

    ServerAdmin admin@infra1.com

    ServerName infra1.com

    ServerAlias www.infra1.com

    DocumentRoot /var/www/infra1.com/public_html

    ErrorLog ${APACHE_LOG_DIR}/error.log

    CustomLog ${APACHE_LOG_DIR}/access.log combined

</VirtualHost>

Save and close the file.

Step 5: Enable the New Virtual Host Files

Created Virtual host files needs to be enabled.

We can use the a2ensite tool to enable each of our sites

$ sudo a2ensite infra.com.conf

$ sudo a2ensite infra1.com.conf

Restart the apache server.

$ sudo service apache2 restart

Step 6: Setup Local Hosts File

$ sudo vi /etc/hosts

The details that you need to add are the public IP address of your VPS server followed by the domain you want to use to reach that VPS.

127.0.0.1 localhost

***.***.***.*** infra.com

***.***.***.*** infra1.com

Save and close the file.

This will direct any requests for infra.com and infra1.com on our computer and send them to our server at ***.***.***

Step 7: Test Your Results

Now that you have your virtual hosts configured, you can test your setup easily by going to the domains that you configured in your web browser

http://infra.com

You should see a page that looks like this

Likewise, if you can visit your second page

http://infra1.com

You will see the file you created for your second site

Step 8: Conclusion

If both of these sites work well, you’ve successfully configured two virtual hosts on the same server.

If you need to access this long term, consider purchasing a domain name for each site you need and setting it up to point to your VPS server.

Centralize Logs from Node.js Applications

Prerequisites

• Installation of Node.js and NPM
• Installation of Fluentd

Modifying the Config File

Next, please configure Fluentd to use the forward Input plugin as its data source.

$ sudo vi /etc/td-agent/td-agent.conf

Fluent daemon should listen on TCP port.

Simple configuration is following:

Restart your agent once these lines are in place.

$ sudo service td-agent restart

fluent-logger-node

Install fluent-logger-node

$ npm install fluent-logger

Now use npm to install your dependencies locally:

$ npm install

Send an event record to Fluentd

index.js

This is the simplest web app.

Run the app and go to http://localhost:4000 in your browser. This will send the logs to Fluentd.

$ node index.js

The logs should be output to /var/log/td-agent/td-agent.log  

Store Logs into MongoDB

Fluentd does 3 things:

1. It continuously “tails” the access log file.
2. It parses the incoming log entries into meaningful fields (such as ip,path, etc.) and buffers them.
3. It writes the buffered data to MongoDB periodically.

Configuration         

Fluentd’s config file

$ sudo vi /etc/td-agent/td-agent.conf

 The Fluentd configuration file should look like this:

Restart your agent once these lines are in place.

$ sudo service td-agent restart

Then, access MongoDB and see the stored data.

$ mongo

Fluentd + MongoDB makes real-time log collection simple, easy, and robust.

Installation of MongoDB on Ubuntu 16.04

MongoDB is a free and open-source NoSQL document database used commonly in modern web applications.

MongoDB is one of several database types to arise in the mid-2000s under the NoSQL banner. Instead of using tables and rows as in relational databases, MongoDB is built on an architecture of collections and documents. Documents comprise sets of key-value pairs and are the basic unit of data in MongoDB. Collections contain sets of documents and function as the equivalent of relational database tables.

Like other NoSQL databases, MongoDB supports dynamic schema design, allowing the documents in a collection to have different fields and structures. The database uses a document storage and data interchange format called BSON, which provides a binary representation of JSON-like documents. Automatic sharding enables data in a collection to be distributed across multiple systems for horizontal scalability as data volumes increase.

This blog will help you set up MongoDB on your server for a production application environment.

Prerequisites

To follow this blog, you will need:

• One Ubuntu 16.04 server set up by following this initial server setup, including a sudo non-root user.

Adding the MongoDB Repository

MongoDB is already included in Ubuntu package repositories, but the official MongoDB repository provides most up-to-date version and is the recommended way of installing the software. In this step, we will add this official repository to our server.

Ubuntu ensures the authenticity of software packages by verifying that they are signed with GPG keys, so we first have to import they key for the official MongoDB repository.

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA31292

After successfully importing the key, you will see:

gpg: Total number processed: 1
gpg:        imported: 1    (RSA:  1)

Next, we have to add the MongoDB repository details so apt will know where to download the packages from.

Issue the following command to create a list file for MongoDB.

$ echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list

After adding the repository details, we need to update the packages list.

$ sudo apt-get update

Installing and Verifying MongoDB

Now we can install the MongoDB package itself.

$ sudo apt-get install -y mongodb-org

This command will install several packages containing latest stable version of MongoDB along with helpful management tools for the MongoDB server.

Next, start MongoDB with systemctl.

$ sudo systemctl start mongod

You can also use systemctl to check that the service has started properly.

$ sudo systemctl status mongod

$ mongo

mongodb.service - High-performance, schema-free document-oriented database
Loaded: loaded (/etc/systemd/system/mongodb.service; enabled; vendor preset: enabled)
Main PID: 4093 (mongod)
Tasks: 16 (limit: 512)
Memory: 47.1M
CPU: 1.224s
CGroup: /system.slice/mongodb.service
└─4093 /usr/bin/mongod --quiet --config /etc/mongod.conf

The last step is to enable automatically starting MongoDB when the system starts.

$ sudo systemctl enable mongod

The MongoDB server is now configured and running, and you can manage the MongoDB service using the systemctl command (e.g. sudo systemctl stop mongod, sudo systemctl start mongod).

Installing Asterisk on Ubuntu 16.04

Tags

Asterisk, PBX15

Asterisk is a software implementation of a telephone private branch exchange (PBX). It allows telephones interfaced with a variety of hardware technologies to make calls to one another, and to connect to telephony services, such as the public switched telephone network (PSTN) and voice over Internet Protocol (VoIP) services. Its name comes from the asterisk symbol “*”.

Some of the many features of Asterisk include:

• The Asterisk software includes many features available in commercial and proprietary PBX systems: voice mail, conference calling, interactive voice response and automatic call distribution.
• Users can create new functionality by writing dial plan scripts in several of Asterisk’s own extensionslanguages, by adding custom loadable modules written in C, or by implementing Asterisk Gateway Interface (AGI) programs using any programming language capable of communicating via the standard streams system (stdin and stdout) or by network TCP sockets.
• Asterisk supports several standard voice over IPprotocols, including the Session Initiation Protocol (SIP), the Media Gateway Control Protocol (MGCP), and 323.
• Asterisk supports most SIP telephones, acting both as registrar and back-to-back user agent.
• By supporting a variety of traditional and VoIP telephony services, Asterisk allows deployers to build telephone systems, or migrate existing systems to new technologies.

 Install Asterisk from Source

After logging in to your Ubuntu Server as an user, issue the following command to switch to root user.

$ sudo su 

Now you are root, but you need to set the password with the following command.

# passwd

Next step would be to install initial dependencies for asterisk.

# apt-get install build-essential wget libssl-dev libncurses5-dev libnewt-dev libxml2-dev linux-headers-$(uname -r) libsqlite3-dev uuid-dev git subversion

Installing Asterisk

Now when we are root and dependencies are satisfied, we can move to /usr/src/ directory and download latest asterisk version there.

# cd /usr/src
# wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-15-current.tar.gz

Next we unpack it.

# tar zxvf asterisk-15-current.tar.gz

Now we need to enter into the newly unpacked directory,

# cd asterisk-15*

Before we actually compile the asterisk code, we need ‘pjproject’ as asterisk-15 introduces the support for pjsip. So we will compile it first:

# git clone git://github.com/asterisk/pjproject pjproject
# cd pjproject
# ./configure --prefix=/usr --enable-shared --disable-sound --disable-resample --disable-video --disable-opencore-amr CFLAGS='-O2 -DNDEBUG'
# make dep
# make && make install
# ldconfig
# ldconfig -p |grep pj

Configuring Asterisk

And now we commence to configuring and compiling the Asterisk code.

# cd ..
# contrib/scripts/get_mp3_source.sh
# contrib/scripts/install_prereq install

This will install mp3 tones and satisfy additional dependencies which might take some time and ask you for your country code. Following command will compile and install asterisk.

# ./configure && make menuselect && make && make install

When that is finished, to avoid making hundred of config files yourself, after install you normally want to run this command, which will make initial config for you:

# make samples

And for having the start up script installed and enabled to start asterisk on every boot, we run make config, followed by ldconfig:

# make config
# ldconfig

Now we can start asterisk for the first time and see if it actually works.

# /etc/init.d/asterisk start

and then we can enter asterisk console with command.

# asterisk -rvvv

Now we need to do additional steps to make it run as asterisk user. First we need to stop asterisk.

# systemctl stop asterisk

Then we need to add group and user named asterisk.

# groupadd asterisk
# useradd -d /var/lib/asterisk -g asterisk asterisk

Asterisk needs to be configured to start as the user we just created, we can edit /etc/default/asterisk by hand but it is more efficient to use following two sed commands.

# sed -i 's/#AST_USER="asterisk"/AST_USER="asterisk"/g' /etc/default/asterisk
# sed -i 's/#AST_GROUP="asterisk"/AST_GROUP="asterisk"/g' /etc/default/asterisk

To run properly, asterisk user needs to be assigned ownership to all essential asterisk directories.

# chown -R asterisk:asterisk /var/spool/asterisk /var/run/asterisk /etc/asterisk /var/{lib,log,spool}/asterisk /usr/lib/asterisk

The asterisk.conf also needs to be edited to uncoment lines for runuser and rungroup:

# sed -i 's/;runuser = asterisk/runuser = asterisk/g' /etc/asterisk/asterisk.conf
# sed -i 's/;rungroup = asterisk/rungroup = asterisk/g' /etc/asterisk/asterisk.conf

when this is done, reboot the server so Asterisk brings up automatically by systemd, and then type asterisk -rvvv to enter the asterisk console.

# asterisk -rvvv

 

The Evolution of Cloud Computing: Embracing Serverless Web Applications

Tags

Cloud Computing, DevOps, Serverless, SRE

The landscape of cloud computing has undergone significant transformations over the past decade, evolving from traditional server-based architectures to virtualized environments, and now, to the revolutionary paradigm of serverless computing. Serverless web applications represent the pinnacle of this evolution, offering unparalleled scalability, cost efficiency, and simplicity. In this blog, we’ll delve into what serverless computing is, its advantages, and why it is poised to be the future of cloud computing. We’ll also walk through an example, complete with commands, to illustrate how serverless computing works in practice.

Understanding Serverless Computing

At its core, serverless computing is a cloud computing execution model where the cloud provider dynamically manages the allocation and provisioning of servers. Contrary to its name, serverless does involve servers, but the key difference is that developers do not have to manage or maintain these servers. Instead, they focus on writing code and defining the business logic, while the cloud provider handles the operational aspects such as scaling, patching, and monitoring.
Advantages of Serverless Web Apps

1. Cost Efficiency: One of the most significant benefits of serverless architecture is the pay-as-you-go pricing model. You only pay for the compute time your code actually uses. This is a stark contrast to traditional models where you need to pay for server uptime, regardless of actual usage.
2. Scalability: Serverless platforms automatically scale your applications in response to incoming traffic. This means your web app can handle a spike in users without any manual intervention, ensuring seamless performance.
3. Reduced Operational Overhead: With serverless, you don’t need to worry about server management tasks like patching, updating, and maintaining the underlying infrastructure. This allows developers to focus on writing code and deploying features.
4. Rapid Deployment: Serverless computing enables faster time-to-market by simplifying the deployment process. Developers can push new features and updates without the need to manage the deployment pipeline manually.
5. Enhanced Developer Productivity: By abstracting server management, developers can concentrate on creating functionality rather than dealing with infrastructure issues. This leads to increased productivity and faster development cycles.

Example: Building a Serverless Web App

Let’s illustrate the power of serverless computing with a practical example. Suppose we want to create a simple serverless web application that allows users to upload images, which are then processed to generate thumbnails.

Step 1: Set Up the Frontend

We’ll start with a basic HTML form that allows users to upload images.

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <title>Image Upload</title>

</head>

<body>

    <h1>Upload an Image</h1>

    <form id="uploadForm" enctype="multipart/form-data">

        <input type="file" id="image" name="image" accept="image/*">

        <button type="submit">Upload</button>

    </form>

    <script>

        document.getElementById('uploadForm').addEventListener('submit', async (event) => {

            event.preventDefault();

            const fileInput = document.getElementById('image');

            const file = fileInput.files[0];

            const formData = new FormData();

            formData.append('image', file);

            const response = await fetch('https://your-api-gateway-endpoint', {

                method: 'POST',

                body: formData

            });

            const result = await response.json();

            console.log(result);

        });

    </script>

</body>

</html>

Step 2: Set Up AWS Lambda and API Gateway

Next, we’ll create an AWS Lambda function that handles image uploads and processing.

AWS CLI Commands:

1. Create a Lambda function:

aws lambda create-function --function-name ImageProcessor \

  --runtime python3.8 --role arn:aws:iam::your-account-id:role/your-lambda-role \

  --handler lambda_function.lambda_handler --zip-file fileb://function.zip

2. Create an S3 bucket:

aws s3 mb s3://your-bucket-name

3. Set up S3 trigger for Lambda:

aws s3api put-bucket-notification-configuration --bucket your-bucket-name --notification-configuration file://notification.json

notification.json should contain:

{

  "LambdaFunctionConfigurations": [

    {

      "LambdaFunctionArn": "arn:aws:lambda:your-region:your-account-id:function:ImageProcessor",

      "Events": ["s3:ObjectCreated:*"]

    }

  ]

}

Lambda Function Code:

import boto3

import os

from PIL import Image

from io import BytesIO

s3_client = boto3.client('s3')

def lambda_handler(event, context):

    bucket_name = 'your-bucket-name'

    image_key = event['Records'][0]['s3']['object']['key']

    # Download the image from S3

    response = s3_client.get_object(Bucket=bucket_name, Key=image_key)

    image = Image.open(response['Body'])

    # Create a thumbnail

    thumbnail = image.copy()

    thumbnail.thumbnail((128, 128))

    # Save the thumbnail to a BytesIO object

    thumbnail_io = BytesIO()

    thumbnail.save(thumbnail_io, format=image.format)

    thumbnail_io.seek(0)

    # Upload the thumbnail to S3

    thumbnail_key = f'thumbnails/{os.path.basename(image_key)}'

    s3_client.put_object(Bucket=bucket_name, Key=thumbnail_key, Body=thumbnail_io, ContentType='image/jpeg')

    return {

        'statusCode': 200,

        'body': f'Thumbnail created at {thumbnail_key}'

    }

Deploy the Lambda Function:

zip function.zip lambda_function.py

aws lambda update-function-code --function-name ImageProcessor --zip-file fileb://function.zip

Set Up API Gateway:

1. Create a REST API:

aws apigateway create-rest-api --name 'ImageUploadAPI'

2. Get the API ID and create a resource:

aws apigateway get-rest-apis

aws apigateway create-resource --rest-api-id your-api-id --parent-id your-root-id --path-part upload

3. Create a POST method for the resource:

aws apigateway put-method --rest-api-id your-api-id --resource-id your-resource-id --http-method POST --authorization-type NONE

4. Integrate the method with Lambda:

aws apigateway put-integration --rest-api-id your-api-id --resource-id your-resource-id --http-method POST --type AWS_PROXY --integration-http-method POST --uri arn:aws:apigateway:your-region:lambda:path/2015-03-31/functions/arn:aws:lambda:your-region:your-account-id:function:ImageProcessor/invocations

5. Deploy the API:

aws apigateway create-deployment --rest-api-id your-api-id --stage-name prod

Why Serverless is the Future of Cloud Computing

1. Aligns with Modern Development Practices: The rise of microservices and event-driven architectures fits perfectly with serverless computing. Serverless allows for building modular, loosely-coupled services that can be independently developed, deployed, and scaled.
2. Fosters Innovation: With the operational burden lifted, developers and organizations can allocate more resources to innovation and improving user experiences. This accelerates the pace at which new, impactful features can be delivered.
3. Environmental Impact: Serverless computing promotes more efficient resource utilization. Since you only use resources when necessary, this can lead to lower energy consumption compared to traditional, always-on servers.
4. Community and Ecosystem Growth: Major cloud providers are heavily investing in serverless technologies, expanding their features, and fostering a growing ecosystem of tools and frameworks. This continuous investment is driving the maturity and adoption of serverless computing.
5. Future-Proofing: As businesses grow, their infrastructure needs can change rapidly. Serverless computing offers the flexibility to adapt to these changes without extensive reengineering, making it a future-proof solution for dynamic business environments.

Challenges and Considerations

While serverless computing offers numerous benefits, it is essential to consider potential challenges:

• Cold Starts: Serverless functions may experience latency when scaling up from zero, known as a cold start.
• Vendor Lock-In: Relying heavily on a specific cloud provider’s serverless offerings can make it challenging to switch providers.
• Complexity in Monitoring and Debugging: With the abstraction of infrastructure, traditional monitoring and debugging tools may not be as effective.

Conclusion

Serverless web apps represent a significant leap forward in the evolution of cloud computing. They provide a compelling combination of cost efficiency, scalability, and operational simplicity. As the technology matures and more businesses recognize the benefits, serverless computing is set to become the backbone of modern web applications. Embracing serverless today could give your business the agility and innovation edge needed to thrive in the competitive digital landscape of tomorrow.

Streamlining Development with Docker: Application Development Through Standardization and Automation

Tags

Automation, CI/CD Integration, Cloud Computing, DevOps, Docker, Version Control

Docker has revolutionized the way developers build, ship, and run applications by providing a consistent, portable, and efficient environment. Here’s a breakdown of its key benefits in these areas:

Consistent Development Environments
Docker ensures that the development environment is identical across all stages of the application lifecycle. This eliminates the “works on my machine” problem, allowing developers to work with confidence.

Easy Integration with CI/CD Pipelines
Docker integrates seamlessly with continuous integration and deployment pipelines, automating the testing, building, and deployment of applications, which speeds up the overall development cycle.

Version Control for Environment
With Docker, you can version control the environment setup just like you do with code. This means any developer can recreate a specific setup from any point in time, aiding in debugging and historical comp

Microservices-Friendly Architecture
Docker’s lightweight containers make it easy to break down applications into microservices, allowing teams to develop, test, and deploy services independently, improving parallel development and scalability. This level of flexibility enables organizations to adapt to changing demands more efficiently, as each microservice can be modified and redeployed without affecting the rest of the application. Furthermore, because microservices can be written in different programming languages and use different data storage technologies, Docker’s containerization approach minimizes conflicts and dependencies, fostering a more modular and agile development environment.

Dockerfile Components

  1. FROM

•The FROM instruction sets the base image for subsequent instructions in the Dockerfile. It is typically the first instruction in a Dockerfile.
eg:-httpd

 2. LABEL

•The LABEL instruction adds metadata to an image. It can include information like the maintainer, version, description, etc

3. RUN
•The RUN instruction executes commands in a new layer on top of the current image and commits the results. It is used to install software packages.

4. COPY
The COPY instruction copies files and directories from the host machine to the Docker image.

5. ADD
The ADD instruction is similar to COPY, but it can also fetch files from a URL and extract TAR files.

6. WORKDIR
The WORKDIR instruction sets the working directory for any subsequent instructions in the Dockerfile.

 7. CMD

•The CMD instruction provides the default command to run when a container is started. It can be overridden by providing a command at the end of docker run.

8. ENTRYPOINT

•The ENTRYPOINT instruction sets the command and parameters that will be executed when a container starts. It is similar to CMD, but it will not be overridden by arguments provided in docker run.

 9. EXPOSE

•The EXPOSE instruction informs Docker that the container listens on the specified network ports at runtime. It doesn’t actually publish the ports

10. ENV

The ENV instruction sets environment variables in the container

Why Docker
•Using Docker lets you ship code faster, standardize application operations, seamlessly move
code, and save money by improving resource utilization. With Docker,
you get a single object that can reliably run anywhere. Docker’s
simple and straightforward syntax gives you full control. Wide adoption means
there’s a robust ecosystem of tools and off-the-shelf applications that are
ready to use with Docker.

Architecture and components of Docker

•Docker Engine

•Docker Daemon (dockerd)

•Manages Docker images, containers, networks, and storage volumes

•Listens for Docker API requests and interacts with the underlying operating system

•Docker Client (docker)

•Primary user interface for Docker

•Accepts commands from users and communicates with the Docker Daemon via the Docker API

•REST API

•Interface for applications to communicate with the Docker Daemon

•Exposes Docker’s functionality for external tools and services

How Docker Components Interact

Containers created by the Docker Daemon run isolated applications with the help of Docker Images, and they utilize Docker Networks and Volumes as required.

The user issues commands using the Docker Client, which communicates with the Docker Daemon through REST APIs.
The Docker Daemon manages containers and interacts with the Docker Registry to pull or push images.

DOCKER COMPONENTS

1. Docker Client

The Docker Client is the primary user interface for Docker, allowing users to interact with the Docker Daemon via command-line commands. The client is an essential component for developers and administrators who manage Docker objects like images and containers.

• Functionality:
  • Acts as the main interface for the user to communicate with Docker.
  • Converts high-level commands like docker run and docker build into API requests that are processed by the Docker Daemon.
• Key Commands:
  • docker build: Builds an image from a Dockerfile.
  • docker pull: Downloads an image from a Docker registry.
  • docker run: Creates and starts a container from an image.
  • docker ps: Lists running containers.
• Architecture:
  • The client communicates with the Docker Daemon either locally on the same host or remotely over a network using REST APIs.

2. Docker Daemon (dockerd)

The Docker Daemon is the background service running on the host machine. It is responsible for managing Docker objects such as images, containers, networks, and volumes.

• Functionality:
  • Listens for Docker API requests from clients and processes them.
  • Handles container orchestration, including creating, starting, stopping, and deleting containers.
  • Manages Docker images, networks, and storage volumes.
• Communication:
  • Communicates with the Docker Client over Unix sockets or a network interface.
  • Acts as the central control for all Docker activities on the host machine.

3. Docker Engine

Docker Engine is the core part of Docker, comprising the Docker Daemon, REST API, and Docker CLI. It provides the environment required to create, run, and manage containers.

• Components:
  • Docker Daemon: As described, it manages Docker operations.
  • REST API: An interface that programs and tools use to interact with the Docker Daemon.
  • Docker CLI: A command-line tool that uses the REST API to control Docker.
• Functionality:
  • The Docker Engine streamlines the workflow of containerizing applications by integrating the client and daemon functions.

4. Docker Registry

A Docker Registry is a repository for storing and distributing Docker images. The most well-known public registry is Docker Hub, but private registries can also be used to store proprietary images.

• Functionality:
  • Stores Docker images for public or private access.
  • Supports versioning of images, allowing multiple versions of an application to be stored and managed.
• Types of Registries:
  • Public Registry: Open to the public, like Docker Hub.
  • Private Registry: Configurable by organizations for internal use, providing secure and controlled access to images.
• Key Commands:
  • docker push: Uploads an image to a registry.
  • docker pull: Downloads an image from a registry.

5. Docker Images

Docker Images are read-only templates containing the instructions to create a Docker container. They include the application code, libraries, dependencies, tools, and any other files required to run an application.

• Functionality:
  • Acts as the blueprint for containers.
  • Images are layered; each layer represents a change or addition to the image, such as adding a file or installing a package.
  • They can be shared, versioned, and reused.
• Layering and Caching:
  • Docker images use a layered filesystem, which allows each layer to be cached, speeding up the build process by reusing layers that have not changed.
  • The top layer is writable, allowing containers to make changes independently of the base image.

6. Docker Containers

Containers are the runnable instances of Docker images. They are lightweight, isolated environments that package an application and all its dependencies to ensure consistency across multiple platforms.

• Functionality:
  • Runs applications in a consistent and isolated environment.
  • Each container operates independently with its own filesystem, network, and process space.
  • Containers are portable across any system running Docker, ensuring the application behaves the same in development, testing, and production.
• Lifecycle:
  • Containers can be started, stopped, moved, or deleted, allowing for dynamic and flexible application deployment.

7. Docker Networks

Docker Networks provide a mechanism for Docker containers to communicate with each other and with external systems. Networks are critical for building microservices architectures, where multiple containers need to interact.

• Types of Networks:
  • Bridge: The default network type; containers on the same bridge can communicate with each other.
  • Host: Removes network isolation between the container and the Docker host.
  • Overlay: Enables multi-host communication, often used in swarm mode.
  • None: Disables networking, isolating the container entirely.
• Functionality:
  • Offers DNS capabilities, allowing containers to resolve each other by name.
  • Facilitates secure communication between containers.

8. Docker Volumes

Docker Volumes are used for data persistence, enabling containers to store data that can outlast the container’s lifecycle. Volumes are independent of the container filesystem, allowing data to be shared between containers.

• Functionality:
  • Stores data outside the container’s writable layer, preventing data loss when a container is removed.
  • Provides better performance and manageability compared to mounting host directories.
• Types:
  • Named Volumes: Managed by Docker, easily shared and reused between containers.
  • Bind Mounts: Direct access to files on the host filesystem, more flexible but less isolated.
• Key Commands:
  • docker volume create: Creates a new volume.
  • docker volume ls: Lists volumes.
  • docker volume rm: Removes a volume.

Docker Networking

Docker networking is a fundamental part of Docker that enables containers to communicate with each other, with the Docker host, and with external networks.
Types of Docker Networks

Docker provides several built-in network drivers, each with its own set of capabilities and use cases:

1. Bridge Network (Default)
   • Description: The Bridge network is the default network driver used when a container is started without specifying a network. It acts as a private internal network on the Docker host.
   • Use Case: Best for standalone containers that need to communicate with other containers on the same host.
   • Functionality:
     • Containers within the same bridge network can communicate with each other using IP addresses or container names.
     • Provides isolation from external networks unless explicitly exposed via port mapping.
   • Commands:
     • Create a custom bridge network: docker network create my_bridge
     • Run a container on a bridge network: docker run --network my_bridge my_container
2. Host Network
   • Description: The Host network removes the isolation between the Docker container and the Docker host, allowing the container to use the host’s network stack directly.
   • Use Case: Useful for applications that require high network performance and do not need network isolation, like monitoring tools or services that need direct access to the host network.
   • Functionality:
     • Containers use the host’s IP address and network interfaces.
     • No need for port mapping; all host ports are available to the container.
   • Commands:
     • Run a container on the host network: docker run --network host my_container
3. Overlay Network
   • Description: Overlay networks enable communication between Docker containers across multiple Docker hosts. They are often used in Docker Swarm or Kubernetes environments.
   • Use Case: Ideal for distributed applications and microservices that require communication between containers on different hosts.
   • Functionality:
     • Provides multi-host networking by creating a distributed network across the nodes in a cluster.
     • Secure communication through encryption (optionally).
   • Commands:
     • Create an overlay network: docker network create -d overlay my_overlay
     • Run a container on an overlay network: docker run --network my_overlay my_container
4. None Network
   • Description: The None network driver disables networking for the container entirely.
   • Use Case: Useful when a container should not be able to communicate over a network at all, such as in highly secure or isolated environments.
   • Functionality:
     • Containers have no network interfaces other than the loopback interface.
   • Commands:
     • Run a container with no network: docker run --network none my_container
5. Macvlan Network
   • Description: Macvlan networks allow you to assign a MAC address to each container, making them appear as physical devices on your network.
   • Use Case: Useful when you need containers to appear as full-fledged devices on your physical network with their own IP addresses, suitable for legacy applications.
   • Functionality:
     • Bypasses Docker’s bridge network and connects directly to the host’s network adapter.
   • Commands:
     • Create a macvlan network: docker network create -d macvlan --subnet=192.168.1.0/24 --gateway=192.168.1.1 -o parent=eth0 my_macvlan
     • Run a container on a macvlan network: docker run --network my_macvlan my_container

Key Features of Docker Networking

1. DNS-Based Service Discovery:
   • Docker networks provide built-in DNS capabilities that allow containers to discover each other using container names instead of IP addresses.
   • This makes managing and scaling microservices easier since you don’t need to hardcode IP addresses.
2. Port Mapping:
   • Port mapping enables external access to containers by exposing specific ports on the Docker host.
   • Example: docker run -p 8080:80 nginx exposes port 80 in the container as port 8080 on the host.
3. Isolation and Security:
   • By default, Docker containers on different networks cannot communicate with each other, providing a level of security.
   • Custom networks can be created to segment and isolate services, enhancing security and traffic control.
4. Network Policies:
   • Docker supports network policies (primarily in Kubernetes environments) to control traffic flow between containers based on rules and policies, providing granular control over network security.
5. Multi-Host Networking:
   • Overlay networks enable containers to communicate across different Docker hosts, essential for distributed applications in a clustered environment.

Managing Docker Networks

Remove Network: docker network rm <network_name> – Deletes a specified network.

List Networks: docker network ls – Lists all available networks on the host.

Inspect Network: docker network inspect <network_name> – Provides detailed information about the specified network, including connected containers and settings.

Building a Scalable Voting Application on AWS.

•The application architecture consists of a Python-based Voting App, a Node.js-based Result App, Redis for in-memory storage, PostgreSQL for persistent storage, and a .NET Worker service. We’ll map each component to AWS services and guide you through the setup process.

•Architecture Overview

•The application architecture is designed to handle high traffic and process voting results efficiently. The architecture includes the following components:

•Voting App (Python): The frontend service where users cast their votes.

•In-Memory Database (Redis): A caching layer that temporarily stores votes before they are processed.

•Worker (.NET): A background service that processes the votes and writes them to the database.

•Database (PostgreSQL): A persistent storage layer that stores the final vote counts.

•Result App (Node.js): The frontend service that displays the voting results to users.

Step 1: Setting Up the Voting App (Python) on AWS
Launching an EC2 Instance

Connecting to Your EC2 Instance via SSH

Installing Docker on the EC2 Instance

installed Docker on your EC2 instance. Docker is a platform that enables developers to automate
the deployment of applications inside lightweight, portable container

apt install docker.io -y

Git clone
Clone git to GitHub, so we want to install git
•apt install git -y

•git clone repository link

Building and Running Docker Image Checking Docker Images

Running the containers
1:Running the Redis Container
docker run -d –name redis -h redis redis
2: Listing Running Containers
docker ps
 3: Running the Postgres Container
docker run -d –name postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres postgres
4: Verifying Running Containers Again
docker ps

Mapping to each containers

•This is the user interface page
Here are a few sentence options you can use for the user interface:
“Cats vs Dogs: Cast Your Vote
Now!”
“Vote for Your Favorite: Cats or
Dogs?”
“Make Your Choice: Are You Team Cat or
Team Dog?”
“Which Do You Prefer? Cast Your Vote
Below!”
“Click to Vote: Cats or Dogs?“

Final Output get this like —–

Monitoring OTP Requests: A Comprehensive Guide to Visualizing Logs with AWS RDS, Python, Fluentd, and Kibana

Tags

aws, Cloud Computing, DevOps, kibana, Monitoring OTP, RDS, Visualizing Logs

In today’s data-driven world, monitoring and visualizing logs is crucial for identifying potential security threats and understanding system usage. In this article, I’ll walk you through how to set up a logging and visualization system for OTP (One-Time Password) requests using AWS RDS, Python, Fluentd (TD Agent), and Kibana. By the end, you’ll be able to track and analyze OTP requests, spotting any anomalies or potential attacks.

1. Overview

Our goal is to:

1. Fetch OTP request data from AWS RDS.
2. Log the data to a file on a server.
3. Send the logs to Elasticsearch using Fluentd.
4. Visualize the data in Kibana.

2. Fetch OTP Requests from AWS RDS

We’ll start by writing a Python script to fetch OTP request data from AWS RDS. This script will run as a cron job to regularly query the RDS database and log the results.

Python Script: query_and_log.py

import pymysql
import time

# Database connection details
connection = pymysql.connect(
    host='your-rds-endpoint',
    user='your-username',
    password='your-password',
    db='your-database-name'
)

def fetch_data():
    with connection.cursor() as cursor:
        cursor.execute("SELECT email, primary_number FROM table name ORDER BY created_at DESC LIMIT 200;")
        result = cursor.fetchall()
        return result

if __name__ == "__main__":
    data = fetch_data()
    timestamp = time.strftime("%Y-%m-%d %H:%M:%S")
    
    with open("/var/log/otp_requests.log", "a") as log_file:
        for row in data:
            log_file.write(f"{timestamp},{row[0]},{row[1]}\n")

Setting Up Cron Job (optional)

Add the following line to your crontab (crontab -e) to run the script every 5 minutes:

*/5 * * * * /usr/bin/python3 /path/to/query_and_log.py

3. Send Logs to Elasticsearch Using Fluentd

We’ll use Fluentd (TD Agent) to ship logs from our server to Elasticsearch. Here’s how you can configure Fluentd:

Fluentd Configuration: /etc/td-agent/td-agent.conf

<source>
  @type tail
  path /var/log/otp_requests.log
  pos_file /var/log/td-agent/otp_requests.pos
  tag otp.production
  format /(?<timestamp>[^,]+),(?<email>[^,]+),(?<primary_number>[^\n]+)/
</source>

<match otp.production>
  @type elasticsearch
  host yourELK---endpoint
  port 443
  scheme https
  logstash_format true
  logstash_prefix otp-production
  type_name fluentd
  log_es_400_reason true
  <buffer>
    flush_interval 10s
    retry_forever true
    retry_type exponential_backoff
  </buffer>
</match>

4. Visualize Data in Kibana

With your data now in Elasticsearch, you can visualize it using Kibana.

Steps to Visualize Data:

1. Create an Index Pattern:
   • Go to Management > Index Patterns in Kibana.
   • Create a new index pattern with otp-production*.
   • 
2. Create a Visualization:
   • Go to Visualize > Create Visualization.
   • Choose Bar Chart or Data Table.
3. Configure the Visualization:
   • Bar Chart:
     • X-Axis: Add a Terms aggregation on the email field.
     • Y-Axis: Use Count to show the number of requests per email.
   • Data Table:
     • Buckets: Add a Terms aggregation on the email field.
     • Metrics: Use Count.
4. Save and Add to Dashboard:
   • Save your visualization.
   • Add it to a Kibana dashboard to monitor OTP request trends.

Conclusion

By setting up this system, you can effectively monitor OTP requests and detect any unusual patterns or potential security threats. The combination of Python for data extraction, Fluentd for log shipping, and Kibana for visualization provides a powerful toolkit for comprehensive log analysis.

Feel free to adjust the script and configurations based on your specific requirements and environment. Happy monitoring!

A detailed guide on deploying a highly available PHP application on Elastic Beanstalk, using an external Amazon RDS database.

Tags

aws, Cloud Computing, Database, DevOps, Elastic Beanstalk, RDS

A sample application will be used that stores user-provided text data in a MySQL database. Now, let us check the steps we’re going to perform.
Step1: Launch a DB Instance in Amazon RDS 

Step2: Create an Elastic Beanstalk Environment 

Step3: Configure Security Groups, Environment Properties, and Scaling 

Step4: Deploy the Sample Application 

1. Launch a DB Instance in Amazon RDS 

To use an external database with an application running on Elastic Beanstalk, you’ll need to launch a DB instance using Amazon RDS. This RDS instance is entirely separate from Elastic Beanstalk and its environments, meaning it won’t be managed, monitored, or terminated by Elastic Beanstalk.

We can use the Amazon RDS console to launch a Multi-AZ MySQL DB instance. Opting for a Multi-AZ deployment ensures high availability, as your database will automatically fail over to a standby instance if the primary DB instance experiences downtime.

For launching an RDS DB instance in a default VPC:

1. Open the RDS console.

2. Choose Instances in the navigation pane. 

3. Choose Launch DB instance. 

4. Choose a database engine. Choose Next. 

5. Choose a use case, if prompted. 

6. Under Specify DB details, review the default settings and adjust as necessary. 

Pay attention to the following options:  

• DB instance class – Choose the instance size that has an appropriate amount of memory and CPU power for your workload. 
• Multi-AZ deployment – For high availability, set to Create replicas in different zones. 

7. Choose Next. 

8. Under Database options, for Database name, type ebdb. Make a note of the Database port value for use later. 

9. Verify default settings for the remaining options and choose Launch DB instance.

Next, modify the security group attached to the DB instance to allow inbound traffic on the appropriate port. This be the same security group that you will attach to your Elastic Beanstalk environment later, so the rule that you add will grant ingress permission to other resources in the same security group

To modify ingress rules on RDS instance’s security group:

1. Open the Amazon RDS console. 

2. Choose Instances. 

3. Choose the name of your DB instance to view the details. 

4. Under the Details section, note the Subnets, Security groups, and Endpoints shown on this page, so you can use this information later. 

5. Under Security and network, you can see the security group associated with the DB instance.

6. In the security group details, choose Inbound. 

7. Choose Edit. 

8. Choose Add Rule. 

9. For Type, choose the DB engine that your application uses. 

10. For Source, type sg- to view a list of available security groups. Choose the current security group to allow resources in the security group to receive traffic on the database port from other resources in the same group. 

11. Choose Save. 

Creating a DB instance will takes about 10 minutes. 

In the meantime, we can create Elastic Beanstalk Environment

2. Create an Elastic Beanstalk Environment

Use AWS Management Console to create Elastic Beanstalk environment. Choose PHP platform and accept the default settings and sample code. After you launch environment, you can configure the environment to connect to the database, then deploy the sample application.

To launch an environment (console) 

1. Open the Elastic Beanstalk console. 

2. For Platform, choose the platform that matches the language used by your application. 

3. For Application code, choose Sample application. 

4. Choose Review and launch. 

5. Review the available options. When you’re satisfied with them, choose Create app

Environment creation will takes about 5 minutes and creates the following resources:

EC2 instance – An Amazon Elastic Compute Cloud virtual machine configured to run web apps on the platform that you choose. Each platform runs the specific set of software, configuration files, and scripts to support specific language version, framework, web container, or combination thereof. Most platforms use either Apache or nginx as reverse proxy that sits in front of your web app, forwards requests to it, serves static assets, and generates access and error logs. 

Instance security group – An Amazon EC2 security group configured to allow ingress on port 80. This resource lets HTTP traffic from the load balancer reach EC2 instance running the web app. By default, traffic will not allowed on other ports.  

Load balancer – An Elastic Load Balancing load balancer configured to distribute the requests to instances running your application. A load balancer also eliminates the need to expose instances directly to the internet.

Load balancer security group – An Amazon EC2 security group is configured to allow ingress on port 80. This resource lets HTTP traffic from internet to reach the load balancer. By default, the traffic is not allowed on other ports.  

Auto Scaling group – An Auto Scaling group configured to replace the instance if it is terminated or becomes unavailable.  

Amazon S3 bucket – A storage location for the source code, logs, and other artifacts that are created when you use Elastic Beanstalk. 

Amazon CloudWatch alarms – Two CloudWatch alarms that monitor load on the instances in your environment and are triggered if the load is too high or too low. When an alarm is triggered, the Auto Scaling group scales up or down in response. 

AWS CloudFormation stack – Elastic Beanstalk uses AWS CloudFormation to launch the resources in the environment and propagate configuration changes. The resources are defined in a template that can view in the AWS CloudFormation console.  

Domain name – A domain name that routes to web app in the form subdomain.region.elasticbeanstalk.com.

All these resources are managed by the Elastic Beanstalk. When we terminate the environment, Elastic Beanstalk terminates all the resources that it contains. The RDS DB instance that we launched is outside of the environment, so we are responsible for managing its lifecycle. 

Note: The Amazon S3 bucket that Elastic Beanstalk creates is shared between the environments and is not deleted during environment termination. 

3. Configure Security Groups, Environment Properties, and Scaling

Add the security group of DB instance to the running environment. This procedure causes Elastic Beanstalk to re-provision all instances in the environment with the additional security group attached.

To add a security group to your environment:

To add the security group using the Elastic Beanstalk console: 

1. Open the Elastic Beanstalk console.

2. Navigate to the management page for your environment. 

3. Choose Configuration. 

4. On the Instances configuration card, choose Modify. 

5. Under EC2 security groups, choose a security group to attach to the instances, in addition to the instance security group that Elastic Beanstalk creates. 

6. Choose Apply. 

7. Read the warnings given, and then choose Confirm.

Next, use the environment properties to pass the connection information to your environment. The sample application uses the default set of properties that match the ones that Elastic Beanstalk configures when you provision a database within the environment.  

To configure the environment properties for an Amazon RDS DB instance: 

1. Open the Elastic Beanstalk console. 

2. Navigate to the management page for your environment. 

3. Choose Configuration. 

4. On the Software configuration card, choose Modify. 

5. In the Environment properties section, define the variables that your application reads to construct a connection string. For compatibility with the environments that have an integrated RDS DB instance, use the following. 

• RDS_HOSTNAME – The hostname of the DB instance. Amazon RDS console label – Endpoint (this is hostname)  
• RDS_PORT – The port on which DB instance accepts connections. The default value varies among DB engines. Amazon RDS console label – Port  
• RDS_DB_NAME – The database name, ebdb. Amazon RDS console label – The DB Name  
• RDS_USERNAME – The user name that was configured for the database. Amazon RDS console label – Username  
• RDS_PASSWORD – The password that was configured for the database.

6. Choose Apply. 

Finally, configure the environment’s Auto Scaling group with a higher minimum instance count. Always run at least two instances to prevent the web servers in your environment from being a single point of failure, and to allow you to deploy changes without taking your site out of service.

To configure the environment’s Auto Scaling group for high availability: 

1. Open the Elastic Beanstalk console. 

2. Navigate to the management page for your environment. 

3. Choose Configuration. 

4. On the Capacity configuration card, choose Modify. 

5. In the Auto Scaling Group section, set Min instances to 2. 

6. Choose Apply. 

4. Deploy the Sample Application 

Now the environment is ready to run the sample application and connect to Amazon RDS. Deploy the sample application to the environment.

To deploy a source bundle:

1. Open the Elastic Beanstalk console. 

2. Navigate to the management page for your environment. 

3. Choose Upload and Deploy. 

4. Choose File and use the dialog box to select the source bundle. 

5. Choose Deploy. 

6. When the deployment completes, choose the site URL to open your website in a new tab. 

The site collects user comments and uses a MySQL database to store data. To add a comment, choose Share Your Thought, enter a comment, and then choose Submit Your Thought. The web app writes the comment to database so that any instance in environment can read it, and it won’t be lost if instances go out of service

What is ELK and Installing ELK stack (elasticsearch, logstash, kibana) in Ubuntu

Tags

Cloud Computing, elasticsearch, ELK, Installing ELK stack, Linux, Logstash, Open Source, Ubuntu

What is ELK Stack

The ELK stack comprises three open-source tools—Elasticsearch, Logstash, and Kibana—which collaborate to gather, store, and analyze data. Here is how each tool works:

1. Elasticsearch: Elasticsearch is a search and analytics engine that offers a distributed, real-time search and analytics platform. It stores data in a distributed index, enabling rapid, real-time search and analysis. Highly scalable, Elasticsearch can effortlessly manage large volumes of data.
2. Logstash: Logstash is a data pipeline that gathers, filters, and transforms data from various sources before sending it to Elasticsearch. It can collect data from diverse sources like log files, databases, and message queues. Additionally, Logstash performs data transformation and filtering to ensure that only relevant data is sent to Elasticsearch.
3. Kibana: Kibana is a data visualization tool that enables users to create dashboards, visualizations, and reports based on data stored in Elasticsearch. It offers a web interface for users to interact with the data and visualize it in various formats, including tables, graphs, and maps.

When used together, Elasticsearch, Logstash, and Kibana create a comprehensive data analysis solution. Logstash collects data from various sources, filters and transforms it, and then sends it to Elasticsearch for storage.

Elasticsearch stores the data and offers fast, real-time search and analysis capabilities. Kibana provides a user-friendly interface for visualizing and analyzing the data stored in Elasticsearch.

Overall, the ELK stack is a powerful solution for collecting, storing, and analyzing data in real-time. It can be applied to various use cases, including log analysis, security analysis, and business intelligence.

Installation and Configuration of ELK Stack on Ubuntu

Note: During the installation process, you can use the IP address of your Ubuntu machine instead of localhost, or you can also use localhost.

Install Java environment packages by using the below command

sudo su
apt install default-jdk default-jre -y

Add the elasticsearch APT repository key by using the below command

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -

Add the elastic to the APT source list by using the below command

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list

Update the APT source list by using the below command

apt update

Install the Elastic Search by using the below command

apt install elasticsearch -y

Configure the elasticsearch by using the below command

nano /etc/elasticsearch/elasticsearch.yml

Change the network.host and http.port as per the screenshot(network.host is the IP of the Ubuntu machine)

Add “discovery.type: single-node”

Configure the JVM heap memory by using the below command

vim /etc/elasticsearch/jvm.options

Restart and enable elasticsearch.

systemctl restart elasticsearch
systemctl enable elasticsearch

To verify the working of elasticsearch use curl command as given below

curl -X GET "<IP>:9200"

Install the Logstash using apt

apt install logstash

Start and enable Logstash services

systemctl start logstash
systemctl enable logstash

Check the status of the Logstash Service

systemctl status logstash

Now install Kibana

apt install kibana

Configure kibana.yml file in /etc/kibana

nano /etc/kibana/kibana.yml

Start and enable kibana service

systemctl start kibana
systemctl enable kibana

Check the status of the kibana service

systemctl status kibana

Ping the http://<IP>:5601 or http://localhost:5601 in browser to view the Dashboard of the kibana as show in the below image.

Creating a Thread Dump Script for Java Applications: A DevOps Guide

Tags

DevOps, java, Linux, Open Source, programming, software, software-development, technology

In the dynamic world of software development, maintaining the health and performance of applications is crucial. One essential aspect of this is diagnosing issues that arise during runtime, especially in Java applications. A thread dump can be an invaluable tool in such scenarios, providing a snapshot of all the threads running in a Java Virtual Machine (JVM). This blog post will explore a simple script for generating thread dumps, its utility in DevOps, its origins, and how thread dumps can be analyzed using online tools.

Understanding the Script

Let’s break down the provided script and understand its functionality.

#!/bin/bash

# Check if Java is installed and get the PID of the Java process
JAVA_PROCESS=$(ps aux | grep java | grep -v grep | awk '{print $2}')
if [ -z "$JAVA_PROCESS" ]; then
    echo "Java process not found."
    exit 1
fi

# Generate a thread dump using jstack
TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
THREAD_DUMP_FILE="thread_dump_$TIMESTAMP.txt"
jstack $JAVA_PROCESS > $THREAD_DUMP_FILE

# Check if the thread dump file was created successfully
if [ -f "$THREAD_DUMP_FILE" ]; then
    echo "Thread dump saved to $THREAD_DUMP_FILE"
else
    echo "Failed to generate thread dump."
fi

Script Breakdown

1. Checking for Java Process: The script begins by checking if a Java process is running. It does this by using the ps aux command to list all running processes and then filtering for the java process using grep. The process ID (PID) is extracted using awk.
2. Generating the Thread Dump: If a Java process is found, the script proceeds to generate a thread dump. The jstack utility is used for this purpose. jstack prints Java stack traces of Java threads, and these traces are saved to a file named with a timestamp to ensure uniqueness.
3. Verifying the Dump Creation: Finally, the script checks if the thread dump file was successfully created and provides appropriate feedback.

Importance of Thread Dumps

Thread dumps are crucial for diagnosing and troubleshooting issues in Java applications. They provide a snapshot of the JVM’s thread state, which can help identify deadlocks, high CPU usage, and other performance-related issues. By analyzing thread dumps, developers and DevOps engineers can pinpoint the root cause of problems and implement solutions effectively.

Using the Script in DevOps

In a DevOps environment, automation and monitoring are key to maintaining application health. This script can be integrated into automated monitoring systems to periodically generate thread dumps or trigger them in response to specific alerts, such as high CPU usage or memory leaks. Here are some practical applications:

• Automated Diagnostics: Integrate the script with monitoring tools like Nagios or Prometheus to automatically generate thread dumps when performance anomalies are detected.
• Scheduled Dumps: Use cron jobs to schedule regular thread dumps for periodic health checks.
• Incident Response: Include the script in incident response procedures to quickly gather diagnostic information when issues are reported.

Analyzing Thread Dumps with Online Tools

After generating a thread dump, the next step is to analyze it to understand the state of the JVM. Various online tools can assist in this analysis by providing a user-friendly interface and advanced insights:

• FastThread.io: This tool helps in quickly analyzing thread dumps by providing visualizations and identifying common issues like deadlocks and thread contention.
• ThreadDumpAnalyzer: A part of the Java Mission Control suite, this tool offers detailed analysis and insights into the state of your JVM.
• IBM Thread and Monitor Dump Analyzer: This tool provides comprehensive analysis and reporting on thread dumps, helping to identify performance bottlenecks and threading issues.

How to Use Online Tools

1. Generate the Thread Dump: Use the script provided to generate a thread dump file.
2. Upload the Thread Dump: Go to an online thread dump analysis tool, such as FastThread.io.
3. Analyze the Results: Upload your thread dump file, and the tool will process and analyze it, providing detailed insights and visualizations.
4. Review and Act: Review the findings to identify any issues, such as deadlocks or thread contention, and take appropriate actions to resolve them.

Origins of Thread Dumps and jstack

The concept of thread dumps originates from the need to analyze the state of a program during its execution, particularly in multi-threaded environments like Java applications. The jstack utility, part of the JDK (Java Development Kit), was introduced by Sun Microsystems (now Oracle) to aid developers in debugging and optimizing their Java applications.

Conclusion

The provided script is a simple yet powerful tool for generating thread dumps in Java applications. By integrating it into your DevOps practices and leveraging online tools for analysis, you can enhance your application’s reliability and performance. Thread dumps, facilitated by tools like jstack, are indispensable for diagnosing and troubleshooting complex issues, ensuring that your applications run smoothly in production environments.

Feel free to adapt and extend this script to suit your specific needs, and always ensure that your diagnostic processes are robust and automated to the extent possible.

Monitoring Linux server using Grafana

Tags

Cloud Computing, Database, DevOps, Grafana, Linux, Linux server, Monitoring, Open Source, Ubuntu

What is Grafana

Grafana is an open-source analytics and monitoring platform that allows you to query, visualize, alert on, and understand your metrics data. It specializes in time-series data visualization but also supports other data sources such as logs, application metrics, and databases.

Requirements : 

1. Launch an Ubuntu instance

Install Grafana in ubuntu instance : 

First, we have to create the directory for the apt keyrings : 

sudo mkdir -p /etc/apt/keyrings/

Download and import the Grafana GPG key : 

wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

Then, we have to add the repository : 

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

Update the repositories : 

sudo apt-get update

Now, install Grafana Agent with the command : 

sudo apt-get install grafana-agent

Enable Grafana : 

sudo systemctl enable grafana-agent 

Start Grafana : 

sudo systemctl start grafana-agent

Check weather grafana is running or not : 

sudo systemctl status grafana-agent

After installing Grafana Agent, we have to go the web browser and go to grafana official website and we have to create an account in it.

Then, we have to add a new connection to it.

There will be different options, as we are using a linux instance we have to select  the Linux server option from there.

We have to create a token : 

After creating API token, we will get a command run that command : 

Copy the command and run it in the server : 

Then, in the section of linux server integration, we will get a snippet we have to place that snippet in the configuration file.

Edit the grafana-agent.yaml file : 

sudo nano /etc/alloy/config.alloy

In this configuration file, we have to add the snippet provided by grafana to integrate.

Restart grafana agent : 

sudo systemctl restart alloy.service

After that, we have to check weather that the agent is collecting data from our machine or not.

We will get to see this : 

In the Grafana, we can see that the dashboard : 

From here, we can see different types of data is available for monitoring.

Here, we can see that the CPU, memory, hostname etc..

From network dashboard, which shows the network status of the linux system.

Logs of the system : 

`

We have a dashboard that will collect the logs of the server.

Configuration Management tool Puppet : Installation and Configuration

Tags

Cloud Computing, Configuration Management tool, DevOps, Infrastructure as code, Installation and Configuration, Open Source, Public Cloud, Puppet

Puppet is an open-source configuration management and automation tool that allows sysadmins to define infrastructure as code. In this blog post, we will explore the fundamentals of Puppet, its architecture, and how it can empower organizations to achieve scalable and consistent infrastructure management.

Why Puppet ?

Puppet is chosen for its ability to automate and maintain consistent configurations across diverse IT environments using infrastructure as code, ensuring efficiency, scalability, and centralized management.

Puppet Architecture : 

Requirements

Launch two ubuntu instances(master and slave).

Connect to the Master Node:

Download the Puppet and install the Master

wget https://apt.puppetlabs.com/puppet-release-bionic.deb

Unpack the package :

sudo dpkg -i puppet-release-bionic.deb

Install Puppet Master :

sudo apt-get install puppet-master

Enable it : 

sudo systemctl enable puppet-master

Start Puppet : 

sudo systemctl start puppet-master

To communicate with slave nodes, enable the firewall : 

sudo ufw allow 8140/tcp

Give the host address : 

sudo nano /etc/hosts

Connect to the Slave node and download the puppet package and install puppet agent:

sudo apt-get install puppet

Give the host address : 

sudo nano /etc/hosts

Give the ip of the master node.

Enable it : 

sudo systemctl enable puppet

Start Puppet : 

sudo systemctl start puppet

After this, go to Master and check if any certificates has been sent to the master.

sudo puppet cert sign –-all

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

Warning: `puppet cert` is deprecated and will be removed in a future release.

   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')

Sign the Certificate : 

sudo puppet cert sign –-all

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:315: warning: deprecated Object#=~ is called on Puppet::Transaction::Report; it always returns nil

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

/usr/lib/ruby/vendor_ruby/puppet/util.rb:461: warning: URI.escape is obsolete

Warning: `puppet cert` is deprecated and will be removed in a future release.

   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')

After signing this, both the master and slave nodes can interact.

Manifest – It is the collection of resource declaration using .pp extension

Create a manifest file by going to the directory : 

cd /etc/puppet/code/environments/production/manifests

Create site.pp file 

nano site.pp 

file {'/tmp/test.txt':

 ensure => present,

 mode => '0644',

 content => "it works on ${ipaddress_eth0}!\n",

}

Restart puppet : 

sudo systemctl restart puppet-master

Go to Slave node : 

Check for the changes

sudo puppet agent --test

Read the file : 

root@puppet-2:~#cat test.txt

It works on 192.168.101.66!

Go to the directory : 

Now, we wil install Nginx in the slave node by modifying the manifest file.

cd /etc/puppet/code/environments/production/manifests

Create site.pp file 

nano site.pp 

Create file : 

node default{

package {'nginx':

ensure => installed,

}

file {'/tmp/status.txt':

content => 'Nginx Installed',

mode => '0644',

}

}

Save and Exit.

Check for the changes in the slave node : 

sudo puppet agent --test

Go to web browser and type the ip address : 

As you can see, Nginx has been successfully installed on the slave node.

Minikube install with Horizontal Pod Autoscaling

Tags

Cloud Computing, Deployment, Kubernetes, management, Minikube, ontainer orchestration, Open Source, scaling

what is kubernetes?

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

what is Minikube?

Minikube is an open-source tool for local Kubernetes development and testing. It enables developers to set up a single-node Kubernetes cluster on their machines, providing a convenient and isolated environment to experiment with Kubernetes features before deploying applications in a larger context.

what is Horizontal Pod Autoscaling?

Horizontal scaling means that the response to increased load is to deploy more Pods.

Steps-1

Setup Minikube:

apt -y install docker.io
systemctl status docker
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
apt install conntrack
minikube start –vm-driver=none
minikube start –force
minikube addons enable metrics-server
kubectl top nodes
kubectl get pod,svc -n kube-system
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl top nodes

Step-2

Create kubernetes dashboard for minikube.

enable metrics-server:

minikube addons enable metrics-server

kubectl proxy –address=’0.0.0.0′ –disable-filter=true

Now check minikube dashboard

http://your_ip:8001/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/

step-3

Now deploy an application in minikube

we can see bellow how to deploy a application in kubernetes.
This is the deployment yaml file to deploy a simple HTTPD application.

vi deployment01.yaml

kind: Deployment  #type of manifest file
apiVersion: apps/v1 #version of deployment 
metadata:
   name: mydeploy
spec:
   replicas: 1 #how many replica you want
   selector:
    matchLabels:
     name: deployment
   template:
     metadata:
       name: testpod
       labels:
         name: deployment
     spec:
      containers:
        - name: demo01  #your container name
          image: httpd  #image name
          ports:
          - containerPort: 80  #port to expose container
          resources:
            limits:
              cpu: 500m  # your cpu limit
            requests:
              cpu: 200m   #application cpu limit

apply this deployment01 yaml file

Kubectl apply -f deployment01.yaml

check your deployment

kubectl get deploy

check pods

kubectl get pods

you can check your pod metric

kubectl top pods

step-4

now apply autoscaling

kubectl autoscale deployment mydeploy –cpu-percent=20 –min=1 –max=10

#in this command if cpu utilization reached 20% autoscale will happen max scaleup limit 20 pods and min scale down pod 1

kubectl get all #this is use to see what are the things are running
kubectl get hpa #this use to see your Horizontal Pod Autoscaling
kubectl get pods –watch

step-5

open a new tab and login the pod

kubectl get pods
kubectl exec mydeploy -it — /bin/bash
apt update -y #keep updateing you can see your pods count will increase

Step-6

Rolling Updates

This type of deployment we are using in production setup to avoid downtime

vi rollingupdate.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-rolling-update
  namespace: default
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        role: rolling-update
    spec:
      containers:
      - image: nginx:1.21  #change nginx version
        imagePullPolicy: Always
        name: nginx-container
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 50%
      
---

kind: Service
apiVersion: v1
metadata:
  name: service-rolling-update
  labels:
    app: nginx
    role: rolling-update
    env: prod
spec:
  
  selector:
    app: nginx
    role: rolling-update
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 80
    nodePort: 31110

kubetcl apply -f rollingupdate.yaml
kubectl get deployment
kubectl get pods
minikube service service-rolling-update
kubectl describe deployment deployment-rolling-update

OSSEC for monitoring systems: Installation and configuration

Tags

Cloud, Open Source, ossec, Security

OSSEC is a comprehensive platform that enables you to oversee and manage your computer systems effectively. It integrates various elements of host-based intrusion detection (HIDS), log monitoring, and security information and event management (SIEM) into a straightforward, potent, and open-source solution..

Requirements

1. Launch 2 CentOs instances.

Phase 1 – Server Installation

• Create 2 CentOS instances. Then name the instances as below.

1 – Server-node, 2 – Client-node

• Connect to the Server-node.

sudo yum -y update && \
sudo yum -y upgrade && \
sudo yum install -y make gcc && \
sudo yum install -y libevent-devel openssl-devel zlib-devel pcre2-devel jq

• Fetch and import the corresponding certificate and key file (.asc)

wget http://www.ossec.net/files/OSSEC-ARCHIVE-KEY.asc && \
wget https://github.com/ossec/ossec-hids/releases/download/3.6.0/ossec-hids-3.6.0.tar.gz.asc && \
gpg --import OSSEC-ARCHIVE-KEY.asc

•  Download the latest version of OSSEC (3.7.0) and verify the file integrity.

wget https://github.com/ossec/ossec-hids/archive/3.6.0.tar.gz && \
gpg --verify ossec-hids-3.6.0.tar.gz.asc 3.6.0.tar.gz

• Extract and run the installation script

tar -zxvf 3.6.0.tar.gz && cd ossec-hids-3.6.0/ && \
wget https://ftp.pcre.org/pub/pcre/pcre2-10.32.tar.gz && \
tar -zxvf pcre2-10.32.tar.gz -C src/external/ && \
sudo PCRE2_SYSTEM=yes ./install.sh

• Select preferred language and server installation as below.

Phase 2 – Client Installation

• Connect to the Client-node.

sudo yum -y update && \
sudo yum -y upgrade && \
sudo yum install -y make gcc && \
sudo yum install -y libevent-devel openssl-devel zlib-devel pcre2-devel

• Fetch and import the corresponding certificate and key file (.asc)

 wget http://www.ossec.net/files/OSSEC-ARCHIVE-KEY.asc && \
wget https://github.com/ossec/ossec-hids/releases/download/3.6.0/ossec-hids-3.6.0.tar.gz.asc && \
gpg --import OSSEC-ARCHIVE-KEY.asc

•  Download the latest version of OSSEC (3.7.0) and verify the file integrity.\

wget https://github.com/ossec/ossec-hids/archive/3.6.0.tar.gz && \
gpg --verify ossec-hids-3.6.0.tar.gz.asc 3.6.0.tar.gz

• Extract and run the installation script

tar -zxvf 3.6.0.tar.gz && cd ossec-hids-3.6.0/ && \
wget https://ftp.pcre.org/pub/pcre/pcre2-10.32.tar.gz && \
tar -zxvf pcre2-10.32.tar.gz -C src/external/ && \
sudo PCRE2_SYSTEM=yes ./install.sh

• Select agent installation along with preferred actions and make sure that you define the IP address of your OSSEC server.

Phase 3 – Server Configuration

• Connect to server-node.

• In the global section of the OSSEC configuration file add the IP addresses of the client.

sudo nano /var/ossec/etc/ossec.conf

<global>
  <allow_list>127.0.0.1</allow_list>
  <allow_list>::1</allow_list>
  <allow_list>localhost.localdomain</allow_list>
  <allow_list>127.0.0.53</allow_list>
  <allow_list>192.168.0.2</allow_list> <!-- OSSEC client -->
</global>

• In order to activate the feature for collecting syslogs from our OSSEC agents, we must first ensure that our remote client connection is both secure and permitted. This involves adding the client’s IP address within the designated remote section.

sudo nano /var/ossec/etc/ossec.conf

<remote>
  <connection>secure</connection>
  <allowed-ips>192.168.0.2</allowed-ips> <!-- OSSEC client -->
</remote>

Phase 4 – Client Configuration

• Connect to Client node

• Edit the agent configuration file and verify that the server IP address is correct.

sudo nano /var/ossec/etc/ossec.conf

<ossec_config>
  <client>
    <server-ip>192.168.0.1</server-ip>
    <config-profile></config-profile>
  </client>

• Add the following lines after the rootcheck segment to enable active response.

sudo nano /var/ossec/etc/ossec.conf

<command>
  <name>firewall-drop</name>
  <executable>firewall-drop.sh</executable>
  <expect>srcip</expect>
  <timeout_allowed>yes</timeout_allowed>
</command>

<!-- Active Response Config -->
<active-response>
  <!-- Specify a comma seperated list of timeouts per
    - re-incidence (in minutes).
    -->
  <repeated_offenders>30,60,120,240,480</repeated_offenders>
</active-response>

<active-response>
  <!-- Firewall Drop response. Block the IP for
    - 600 seconds on the firewall (iptables,
    - ipfilter, etc).
    -->
  <disabled>no</disabled>
  <command>firewall-drop</command>
  <agent_id>001</agent_id>
  <location>local</location>
  <rules_id></rules_id>
  <level>6</level>
  <timeout>600</timeout>
</active-response>

Phase 5 – Managing Clients

• Connect to Server

• To add an agent to your OSSEC server, run the command shown in the code segments below and follow the instructions.

sudo /var/ossec/bin/manage_agents

• Once you’ve added the client proceed by extracting its agent key by providing the assigned agent ID and Copy the agent key.

• Connect to client

sudo /var/ossec/bin/manage_agents

• Restart the OSSEC server and the client to enable and activate OSSEC HIDS.