Ubuntu

We have already seen how to create virtual machines with KVM and Qemu, and how to manage them with tools such as virsh and virt-manager. This approach works when you need to work with a handful of machines and manage few hosts. To operate on a larger scale, you need a tool to manage host machines, VM configurations, images, network, and storage, and monitor the entire environment. OpenStack is an open source initiative to create and manage a large pool of virtual machines (or containers). It is a collection of various tools to deploy IaaS clouds. The official site defines OpenStack as an operating system to control a large pool of compute, network, and storage resources, all managed through a dashboard.

OpenStack was primarily developed and open-sourced by Rackspace, a leading cloud service provider. With its thirteenth release, Mitaka, OpenStack provides tons of tools to manage various components of your infrastructure. A few important components of OpenStack are as follows:

Nova: Compute controller

Neutron: OpenStack networking

Keystone: Identity service

Glance: OpenStack image service

Horizon: OpenStack dashboard

Cinder: Block storage service

Swift: Object store

Heat: Orchestration program

OpenStack in itself is quite a big deployment. You need to decide the required components, plan their deployment, and install and configure them to work in sync. The installation itself can be a good topic for a separate book. However, the OpenStack community has developed a set of scripts known as DevStack to support development with faster deployments. In this recipe, we will use the DevStack script to quickly install OpenStack and get an overview of its workings. The official OpenStack documentation provides detailed documents for the Ubuntu based installation and configuration of various components. If you are planning a serious production environment, you should read it thoroughly.

Getting ready

You will need a non-root account with sudo privileges. The default account named ubuntu should work.

The system should have at least two CPU cores with at least 4 GB of RAM and 60 GB of disk space. A static IP address is preferred. If possible, use the minimal installation of Ubuntu.

DevStack scripts are available on GitHub. Clone the repository or download and extract it to your installation server. Use the following command to clone:

$ git clone https://git.openstack.org/openstack-dev/devstack \

-b stable/mitaka --depth 1

$ cd devstack

You can choose to get the latest release by selecting the master branch. Just skip the -b stable/mitaka option from the previous command.

How to do it…

Once you obtain the DevStack source, it's as easy as executing an installation script. Before that, we will create a minimal configuration file for passwords and basic network configuration:

Copy the sample configuration to the root of the devstack directory:

$ cp samples/local.conf

Edit local.conf and update passwords:

ADMIN_PASSWORD=password

DATABASE_PASSWORD=password

RABBIT_PASSWORD=password

SERVICE_PASSWORD=$ADMIN_PASSWORD

Add basic network configuration as follows. Update IP address range as per your local network configuration and set FLAT_INTERFACE to your primary Ethernet interface:

FLOATING_RANGE=192.168.1.224/27

FIXED_RANGE=10.11.12.0/24

FIXED_NETWORK_SIZE=256

FLAT_INTERFACE=eth0

Save the changes to the configuration file.

Now, start the installation with the following command. As the Mitaka stable branch has not been tested with Ubuntu Xenial (16.04), we need to use the FORCE variable. If you are using the master branch of DevStack or an older version of Ubuntu, you can start the installation with the ./stack.sh command:

$ FORCE=yes ./stack.sh

The installation should take some time to complete, mostly depending on your network speed. Once the installation completes, the script should output the dashboard URL, keystone API endpoint, and the admin password:

Now, access the OpenStack dashboard and log in with the given username and password. The admin account will give you an admin interface. The login screen looks like this:

Once you log in, your admin interface should look something like this:

Now, from this screen, you can deploy new virtual instances, set up different cloud images, and configure instance flavors.

How it works…

We used DevStack, an unattended installation script, to install and configure basic OpenStack deployment. This will install OpenStack with the bare minimum components for deploying virtual machines with OpenStack. By default, DevStack installs the identity service, Nova network, compute service, and image service. The installation process creates two user accounts, namely admin and dummy. The admin account gives you administrative access to the OpenStack installation and the dummy account gives you the end user interface. The DevStack installation also adds a Cirros image to the image store. This is a basic lightweight Linux distribution and a good candidate to test OpenStack installation.

The default installation creates a basic flat network. You can also configure DevStack to enable Neutron support, by setting the required options in the configuration. Check out the DevStack documentation for more details.

There's more…

Ubuntu provides its own easy-to-use OpenStack installer. It provides options to install OpenStack, along with LXD support and OpenStack Autopilot, an enterprise offering by Canonical. You can choose to install on your local machine (all-in-one installation) or choose a Metal as a Service (MAAS) setup for a multinode deployment. The single-machine setup will install OpenStack on multiple LXC containers, deployed and managed through Juju. You will need at least 12 GB of main memory and an 8-CPU server. Use the following commands to get started with the Ubuntu OpenStack installer:

$ sudo apt-get update

$ sudo apt-get install conjure-up

$ conjure-up openstack

While DevStack installs a development-focused minimal installation of OpenStack, various other scripts support the automation of the OpenStack installation process. A notable project is OpenStack Ansible. This is an official OpenStack project and provides production-grade deployments. A quick GitHub search should give you a lot more options.

A step-by-step detailed guide to installing various OpenStack components on Ubuntu server: http://docs.openstack.org/mitaka/install-guide-ubuntu/

DevStack Neutron configuration: http://docs.openstack.org/developer/devstack/guides/neutron.html

OpenStack Ansible: https://github.com/openstack/openstack-ansible

A list of OpenStack resources: https://github.com/ramitsurana/awesome-openstack

Ubuntu MaaS: http://www.ubuntu.com/cloud/maas

Ubuntu Juju: http://www.ubuntu.com/cloud/juju

Read more about LXD and LXC in article 8, Working with Containers

In this recipe, we will look at some common problems with MySQL and learn how to solve them.

Getting ready

You will need access to a root account or an account with sudo privileges.

You will need administrative privileges on the MySQL server.

How to do it…

Follow these steps to troubleshoot MySQL:

First, check if the MySQL server is running and listening for connections on the configured port:

$ sudo service mysql status

$ sudo netstat -pltn

Check MySQL logs for any error messages at /var/log/mysql.log and mysql.err.

You can try to start the server in interactive mode with the verbose flag set:

$ which mysqld

/usr/sbin/mysqld

$ sudo /usr/sbin/mysqld --user=mysql --verbose

If you are accessing MySQL from a remote system, make sure that the server is set to listen on a public port. Check for bind-address in my.cnf:

bind-address = 10.0.247.168

For any access denied errors, check if you have a user account in place and if it is allowed to log in from a specific IP address:

mysql> select user, host, password from mysql.user where user = ‘username’;

Check the user has access to specified resources:

mysql > grant all privileges on databasename.* to ‘username’@’%’;

Check your firewall is not blocking connections to MySQL.

If you get an error saying mysql server has gone away, then increase wait_timeout in the configuration file. Alternatively, you can re-initiate a connection on the client side after a specific timeout.

Use a repair table statement to recover the crashed MyISAM table:

$ mysql -u root -p

mysql> repair table databasename.tablename;

Alternatively, you can use the mysqlcheck command to repair tables:

$ mysqlcheck -u root -p --auto-repair \

--check --optimize databasename

See also

InnoDB troubleshooting at https://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting.html

In this recipe, we will learn how to install and set up Dovecot to enable accessing e-mails over IMAP and POP3 protocols. This will enable mail clients such as thunderbird to download e-mails on a user's local system.

Getting ready

You will need access to a root account or an account with sudo privileges

Make sure that you have set up Postfix and are able to send and receive e-mails on your server.

You may need an e-mail client to connect to and test the Dovecot setup.

How to do it…

Follow these steps to enable IMAP and POP3 with Dovecot:

First, install the Dovecot binaries from the Ubuntu main repository:

$ sudo apt-get update

$ sudo apt-get install dovecot-imapd dovecot-pop3d

You will be prompted for a hostname to be used for certificate generation. Type in a full hostname, for example mail.example.com. You can skip this step if you already have certificates.

Next, proceed with configuring Dovecot. Open the file /etc/dovecot/dovecot.conf:

$ sudo nano /etc/dovecot/dovecot.conf

Find the Enable installed protocols section and add a new line to set the protocols that you want Dovecot to support:

protocols = pop3 pop3s imap imaps

Open /etc/dovecot/conf.d/10-mail.conf and set the mailbox to be used. Dovecot supports mbox as well as Maildir. Make sure you set the correct path of your mail directory:

mail_location = mbox:~/mail:INBOX=/var/spool/mail/%u

Open /etc/dovecot/conf.d/10-ssl.conf and uncomment or change the following lines to enable SSL authentication. Here, I have used certificates created by Postfix. You can use your own certificates or use the one generated by Dovecot:

ssl = yes

ssl_cert = /etc/ssl/certs/ssl-cert-snakeoil.pem

ssl_key =

Restart the Dovecot daemon:

$ sudo service dovecot restart

Test Dovecot by creating a telnet connection. You should see an output similar to the following:

$ telnet localhost pop3

How it works…

Dovecot is one of the most popular Mail Delivery Agents (MDA) with support for IMAP and POP3 protocols. It works with both major mailbox formats, namely mbox and Maildir. The installation process is simple, and a minimal configuration can get you started with your own IMAP or POP3 service.

Dovecot developers have tried to simplify the configuration by separating it across various small files for each section. All these configuration files are located under /etc/dovecot/conf.d. If you prefer to use a single configuration file, you can replace the default file with the entire working configuration. To get all enabled configurations, use the doveconf -n command:

# mv /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.old

# doveconf -n > /etc/dovecot/dovecot.conf

In this recipe, we have configured Dovecot to support POP3, POP3 secure, IMAP, and IMAP secure. You can choose a single protocol or any combination of them. After setting protocol support, we have set the mailbox type to mbox. If you are using Maildir as your mailbox format, instead replace the mailbox setting with following line:

mail_location = maildir:~/Maildir

Now, when a user wants to check his e-mails, they need to authenticate with the Dovecot server. At this stage, only users with a user account on the server will be able to access their e-mails with Dovecot. To support users without creating a user account, we will need to set up Virtual Users, which is covered in the next recipes.

If you plan to skip SSL setup, you may need to enable plain text authentication under the configuration file, /etc/dovecot/conf.d/10-auth.conf. Find and uncomment the following line and set it to no:

disable_plaintext_auth = yes

The default setting is to allow plain text authentication over SSL connections only. That means the clients that do not support SSL will not be allowed to log in.

See also

Dovecot wiki Quick-configuration at http://wiki2.dovecot.org/QuickConfiguration

In this recipe, we will install a collaborative document editor, Hackpad. It is a document editor based on an open source editor, EtherPad. Hackpad was acquired by Dropbox, and in early 2015 they open sourced its code.

Getting ready

You will need a system with at least 2 GB of memory.

As always, you will need an account with super user privileges.

How to do it…

Hackpad is a web application based on Java. We will need to install the JDK; Scala, which is another programming language; and MySQL as a data store. We will start by installing dependencies and then cloning the Hackpad repository from GitHub.

Install JDK and Scala. The installation document mentions Sun JDK as a requirement but it works with Open JDK.

$ sudo apt-get update

$ sudo apt-get install openjdk-7-jdk scala -y

Install the MySQL server. You can get more details on MySQL installation in the article handling the database:

$ sudo apt-get install mysql-server-5.6

Next, clone the Hackpad repository. You can choose not to install Git and download the ZIP archive of Hackpad from GitHub:

$ git clone https://github.com/dropbox/hackpad.git

This will create a new directory, hackpad. Before we run the build script, we need to set some configuration parameters to match our environment. Change the directory to hackpad and edit the bin/exports.sh file as follows:

export SCALA_HOME="/usr/share/java"

export SCALA_LIBRARY_JAR="$SCALA_HOME/scala-library.jar"

export JAVA_HOME="/usr/share/java"

Next, create a configuration file as a copy of the default configuration, as follows:

$ cp etherpad/etc/etherpad.localdev-default.properties \ etherpad/etc/etherpad.local.properties

Edit the newly created configuration, get the admin email address, and search for the following line in etherpad/etc/etherpad.local.properties:

etherpad.superUserEmailAddresses = __email_addresses_with_admin_access__

Replace it with:

etherpad.superUserEmailAddresses = admin@yourdomain.tld

Optionally, you can set the project to production mode by setting isProduction to true:

devMode = false

verbose = true

etherpad.fakeProduction = false

etherpad.isProduction = true

If you are using a domain name other than localhost, then configure the same with the following option:

topdomains =yourdomain.tld,localhost

Set your email host settings. You will need an email address to receive your registration confirmation email. However, this is not a hard requirement for initial setup:

smtpServer = Your SMTP server

smtpUser = SMTP user

smtpPass = SMTP password

Next, run a build script from the bin directory:

$ ./bin/build.sh

Once the build completes, set up the MySQL database. The script will create a new database named hackpad and a MySQL user account. You will be asked to enter your MySQL root account password:

$ ./contrib/scripts/setup-mysql-db.sh

Finally, you can start the server by executing run.sh from the bin directory:

$ ./bin/run.sh

This will take a few seconds to start the application. Once you see the HTTP server is listening to the line, you can access Hackpad at http://yourdomain.tld:9000:

Access Hackpad and register with an email address that is used for an admin account. If you have set up an email server, you should receive a confirmation email containing a link to activate your account.

If you have not set up email server access to the MySQL database to get your authentication token, open the MySQL client and use the following queries to get your token. The MySQL password for the Hackpad account is taken from the configuration file:

$ mysql -h localhost -u hackpad -ppassword

mysql> use hackpad;

mysql> select * from email_signup;

Select your token from the row matching your email address and replace it in the following URL. In this case, the auth toke is PgEJoGAiL3E2ZDl2FqMc:

http://yourdomain.com:9000/ep/account/validate- email?email=user@youremail.com&token=your_auth_token_from_db

The full auth URL for my admin account will look like this:

http://localhost.local:9000/ep/account/validate-email?email=admin@localh... PgEJoGAiL3E2ZDl2FqMc

Open this URL in the browser and your account registration will be confirmed. You will be logged in to your Hackpad account.

Once you log in to your new account, Hackpad will start with a welcome screen listing all the default pads that looks something like the following:

You can click any of them and start editing or create a new document. When opened, you will get a full page to add contents, with basic text editing options in the top bar:

The document can be shared using the invite box or simply by sharing the URL.

How it works…

As mentioned before, Hackpad is a collaborative editor based on an open source project, EtherPad. It allows you to create online documents directly in your browser. In the same way as Google Docs, you can use Hackpad to create and store your documents in the cloud. Plus, you can access Hackpad from any device. All your documents will be rendered in a proper format suitable for your device.

When you log in for the first time, the home screen will greet you with stock pads. You can edit existing pads or start a new one from the top bar. An editor will give you a basic text editing setting, plus options to create lists and add comments. You can even add data in a tabular format. Click on the gear icon from the top bar and it will give you options to view document history, get an embedded link, or delete the document.

Every change in the document will be marked with your username, and if two or more people are working with the document at the same time, then the specific line being edited by each user is marked with the user's tag:

On the right-hand side of the document, you can see the options to invite your peers to collaborate on this document. You can invite people using their email address. Make sure that you have configured your email server before using this feature. Alternatively, the invites are also shown in a chat window with clickable links, as shown in the following screenshot:

At the bottom of the document, you can find all activity logs about the new initiation and the editing of this document. There is an option to chat with participating people directly from the same window. It is located at the bottom corner of the right-hand side; it's the small bar with a chat icon named after your domain. This provides one-to-one chat, as well as a group chat:

There's more

Hackpad is a collaborative document editor. You can add snippets of code in a given document but not entire code files. To edit your code, you can use an open source Cloud IDE named Cloud 9 IDE. Check out the GitHub repo at https://github.com/c9/core/ . Alternatively, you can get Docker images set up quickly and play around with the IDE.

Using Hackpad with Docker

The Hackpad setup contains a Docker file as well. If you have Docker installed, you can build a Docker image for Hackpad. Simply change your directory to Hackpad git repo and build a Docker image with the following command:

$ docker build -t hackpad

See also

Read more about Hackpad at the following links:

Hackpad with Docker at https://github.com/dropbox/hackpad/blob/master/DOCKER.md

Hackpad repo at https://github.com/dropbox/hackpad

Etherpad at http://etherpad.org/

Cloud 9 IDE at https://c9.io/

In this recipe, we will look at some best practices to secure Ubuntu systems. Linux is considered to be a well secured operating system. It is quite easy to maintain the security and protect our systems from unauthorized access by following a few simple norms or rules.

Getting ready

You will need access to a root or account with sudo privileges. These steps are intended for a new server setup. You can apply them selectively for the servers already in productions.

How to do it…

Follow these steps to discuss Ubuntu security best practices:

Install updates from the Ubuntu repository. You can install all the available updates or just select security updates, depending on your choice and requirement:

$ sudo apt-get update

$ sudo apt-get upgrade

Change the root password; set a strong and complex root password and note it down somewhere. You are not going to use it every day:

$ sudo passwd

Add a new user account and set a strong password for it. You can skip this step if the server has already set up a non-root account, like Ubuntu:

$ sudo adduser john

$ sudo passwd john

Add a new user to the Sudoers group:

$ sudo adduser john sudo

Enable the public key authentication over SSH and import your public key to new user's authorized_keys file.

Restrict SSH logins:

Change the default SSH port:

port 2222

Disable root login over SSH:

PermitRootLogin no

Disable password authentication:

PasswordAuthentication no

Restrict users and allow IP address:

AllowUsers john@(your-ip) john@(other-ip)

Install fail2ban to protect against brute force attacks and set a new SSH port in the fail2ban configuration:

$ sudo apt-get install fail2ban

Optionally, install UFW and allow your desired ports:

$ sudo ufw allow from to any port 22 proto tcp

$ sudo ufw allow 80/tcp

$ sudo ufw enable

Maintain periodic snapshots (full-disk backups) of your server. Many cloud service providers offer basic snapshot tools.

Keep an eye on application and system logs. You may like to set up log-monitoring scripts that will e-mail any unidentified log entry.

How it works…

The preceding steps are basic and general security measures. They may change according to your server setup, package selection, and the services running on your server. I will try to cover some more details about specific scenarios. Also, I have not mentioned application-specific security practices for web servers and database servers. A separate recipe will be included in the respective articles. Again, these configurations may change with your setup.

The steps listed earlier can be included in a single shell script and executed at first server boot up. Some cloud providers offer an option to add scripts to be executed on the first run of the server. You can also use centralized configuration tools such as Ansible, Chef/Puppet, and some others. Again, these tools come with their own security risks and increase total attack surface. This is a tradeoff between ease of setup and server security. Make sure that you select a well-known tool if you choose this route.

I have also mentioned creating single user account, except root. I am assuming that you are setting up your production server. With production servers, it is always a good idea to restrict access to one or two system administrators. For production servers, I don't believe in setting up multiple user accounts just for accountability or even setting LDAP-like centralized authentication methods to manage user accounts. This is a production environment and not your backyard. Moreover, if you follow the latest trends in immutable infrastructure concepts, then you should not allow even a single user to interfere with your live servers. Again, your mileage may vary.

Another thing that is commonly recommended is to set up automated and unattended security updates. This depends on how trusted your update source is. You live in a world powered by open source tools where things can break. You don't want things to go haywire without even touching the servers. I would recommend setting up unattended updates on your staging or test environment and then periodically installing updates on live servers, manually. Always have a snapshot of the working setup as your plan B.

You may want to skip host-based firewalls such as UFW when you have specialized firewalls protecting your network. As long as the servers are not directly exposed to the Internet, you can skip the local firewalls.

Minimize installed packages and service on single server. Remember the Unix philosophy, do one thing and do it well, and follow it. By minimizing the installed packages, you will effectively reduce the attack surface, and maybe save little on resources too. Think of it as a house with a single door verses a house with multiple doors. Also, running single service from one server provides layered security. This way, if a single server is compromised, the rest of your infrastructure remains in a safe state.

Remember that with all other tradeoffs in place, you cannot design a perfectly secured system, there is always a possibility that someone will break in. Direct your efforts to increase the time required for an attacker to break into your servers.

See also

First 5 Minutes Troubleshooting A Server at http://devo.ps/blog/troubleshooting-5minutes-on-a-yet-unknown-box/

Try to break in your own servers at http://www.backtrack-linux.org/

What Can Be Done To Secure Ubuntu Server? at http://askubuntu.com/questions/146775/what-can-be-done-to-secure-ubuntu-server

In this recipe, we will learn how to set up and use the group chat feature of XMPP. Group chat is also called Multi User Chat (MUC). Ejabberd supports MUC with the help of an extension and is enabled by default.

Getting ready

You will need the Ejabberd server set up and running. Make sure you have enabled MUC with the mod_muc and mod_muc_admin modules.

You will need two users for the group chat. One of them needs to have admin rights to set up MUC and create rooms.

Check your XMPP client for the support of MUC or conference protocol. I will be using PSI as a client for this recipe.

How to do it…

For multi-user chat, we need two or more users logged in on the server at the same time, plus a chat room. Let's first set up our chat client with user accounts and create a chat room.

Follow these steps to enable group chat:

Open PSI and set up two different accounts. Log in to the XMPP server and set the Status to Online. Your PSI window should look something like this:

You can access the MUC statistics on the Ejabberd web panel to check available rooms.

Now we will create our first chat room. In PSI, click the General menu, select Service Discovery, and then select your admin account:

This will open a Service Discovery window with a list of all administrative services on your Ejabberd XMPP server:

Look for the Chatrooms node under the Name column and double-click it to browse its options. A new window will pop up, which should look something like this:

Now type the name of the chat room you want to create under the Room information section. Set your nickname as it should be displayed to other participants and click the Join button.

This will open a new window for your chat room. You will notice the chat room name on the title bar of the window. As the user admin created this room, he is assigned as a moderator:

For now, the admin is the only participant in this room. Repeat the same steps with other user accounts to get them to join this room. Make sure that you use the same room name again. Once a new user joins the room, the admin user will get notified. Both users can see each other in the participants section:

How it works…

A group chat works in a similar way to a one on one chat. In a one-on-one chat, we send a message to the JID of a specific user, while in a multi-user chat we send a message to the JID of a chat room. As the message is received on room ID, XMPP takes care of forwarding it to all participants in that room.

There's more…

By default, XMPP chat rooms are not persistent and will be deleted when all participants leave that room. PSI uses the default configuration to quickly create a new chat room. Once the chat room is created, you can configure it in the same chat room window. Click on the options button, the downward triangle in the upper-right corner of the chat room window, and then select Configure room:

On the first tab, you can set members, administrators, and ban user accounts. On the General tab, you can set other room configurations. You can mark a room as persistent and make it private password-protected. This tab contains a number of other options; check them at your leisure.

You may have noticed we have used an admin account to create a chat room. You can allow non-admin users to act as an MUC admin. Open the Ejabberd configuration and search for muc_admin configuration. Add your desired username below the admin entry and set it to allow.

See also

Candy - JavaScript-based multi-user chat client at https://candy-chat.github.io/candy/

Strophe.js MUC plugin at https://github.com/metajack/strophejs-plugins/tree/master/muc

Ubuntu is a multiuser and multi-process operating system. If a single user or process is consuming too many resources, other processes might not be able to use the system. In this recipe, you will see how to set resource limits to avoid such problems.

Getting ready

User account with root privileges is required.

How to do it...

Following are the steps to set the resource limits:

Check the CPU use limit with $ulimit –t.

To set new limit, open limits.conf with the following command:

$sudo nano /etc/security/limits.conf

Scroll to the end of the file and add following lines:

username soft cpu 0 # max cpu time in minutes

username hard cpu 1000 # max cpu time in minutes

Enter Ctrl + O to save the changes.

Enter Ctrl + X to exit GNU nano editor.

How it works…

PAM stands for pluggable authentication module. The PAM module pam_limits.so provides functionality to set a cap on resource utilization. The command ulimit can be used to view current limits as well as set new limits for a session. The default values used by pam_limits.so can be set in /etc/security/limits.conf.

In this recipe, we are updating limits.conf to set a limit on CPU uses by user username. Limits set by the ulimit command are limited to that session. To set the limits permanently, we need to set them in the limits.conf file.

The syntax of the limits.conf file is as follows:

Here, can be a username, a group name, or a wildcard entry.

denotes the type of the limit and it can have the following values:

soft: This is a soft limit which can be changed by user

hard: This is a cap on soft limit set by super user and enforced by kernel

is the resource to set the limit for. You can get a list of all items with $ulimit –a:

In our example, we have set soft limit on CPU uses to 0 minutes and hard limit to 1000 minutes. You can changes soft limit values with the ulimit command. To view existing limits on open files, use the command $ulimit -n. To change limits on open files, pass the new limit as follows:

$ulimit -n 4096

An unprivileged process can only set its soft limit value between 0 and hard limit, and it can irreversibly lower hard limit. A privileged process can change either limit values.

There's more…

The command ulimit can be used to set limits on per process basis. You can't use the ulimit command to limit resources at the user level. You can use cgroups to set a cap on resource use.

This recipe explores image creation with Dockerfiles. Docker images can be created in multiple ways, which includes using Dockerfiles, using docker commit to save the container state as a new image, or using docker import, which imports chroot directory structure as a Docker image.

In this recipe, we will focus on Dockerfiles and related details. Dockerfiles help in automating identical and repeatable image creation. They contain multiple commands in the form of instructions to build a new image. These instructions are then passed to the Docker daemon through the docker build command. The Docker daemon independently executes these commands one by one. The resulting images are committed as and when necessary, and it is possible that multiple intermediate images are created. The build process will reuse existing images from the image cache to speed up build process.

Getting ready

Make sure that your Docker daemon is installed and working properly.

How to do it…

First, create a new empty directory and enter it. This directory will hold our Dockerfile:

$ mkdir myimage

$ cd myimage

Create a new file called Dockerfile:

$ touch Dockerfile

Now, add the following lines to the newly created file. These lines are the instructions to create an image with the Apache web server. We will look at more details later in this recipe:

FROM ubuntu:trusty

MAINTAINER ubuntu server cookbook

# Install base packages

RUN apt-get update && apt-get -yq install apache2 && \

apt-get clean && \

rm -rf /var/lib/apt/lists/*

RUN echo "ServerName localhost" >> /etc/apache2/apache2.conf

ENV APACHE_RUN_USER www-data

ENV APACHE_RUN_GROUP www-data

ENV APACHE_LOG_DIR /var/log/apache2

ENV APACHE_PID_FILE /var/run/apache2.pid

ENV APACHE_LOCK_DIR /var/www/html

VOLUME ["/var/www/html"]

EXPOSE 80

CMD ["/usr/sbin/apache2", "-D", "FOREGROUND"]

Save the changes and start the docker build process with the following command:

$ docker build.

This will build a new image with Apache server installed on it. The build process will take a little longer to complete and output the final image ID:

Once the image is ready, you can start a new container with it:

$ docker run -p 80:80 -d image_id

Replace image_id with the image ID from the result of the build process.

Now, you can list the running containers with the docker ps command. Notice the ports column of the output:

$ docker ps

Apache server's default page should be accessible at your host domain name or IP address.

How it works…

A Dockerfile is a document that contains several commands to create a new image. Each command in a Dockerfile creates a new container, executes that command on the new container, and then commits the changes to create a new image. This image is then used as a base for executing the next command. Once the final command is executed, Docker returns the ID of the final image as an output of the docker build command.

This recipe demonstrates the use of a Dockerfile to create images with the Apache web server. The Dockerfile uses a few available instructions. As a convention, the instructions file is generally called Dockerfile. Alternatively, you can use the -f flag to pass the instruction file to the Docker daemon. A Dockerfile uses the following format for instructions:

# comment

INSTRUCTION argument

All instructions are executed one by one in a given order. A Dockerfile must start with the FROM instruction, which specifies the base image to be used. We have started our Dockerfile with Ubuntu:trusty as the base image. The next line specifies the maintainer or the author of the Dockerfile, with the MAINTAINER instruction.

Followed by the author definition, we have used the RUN instruction to install Apache on our base image. The RUN instruction will execute a given command on the top read-write layer and then commit the results. The committed image will be used as a starting point for the next instruction. If you've noticed the RUN instruction and the arguments passed to it, you can see that we have passed multiple commands in a chained format. This will execute all commands on a single image and avoid any cache-related problems. The apt-get clean and rm commands are used to remove any unused files and minimize the resulting image size.

After the RUN command, we have set some environment variables with the ENV instruction. When we start a new container from this image, all environment variables are exported to the container environment and will be accessible to processes running inside the container. In this case, the process that will use such a variable is the Apache server.

Next, we have used the VOLUME instruction with the path set to /var/www/html. This instruction creates a directory on the host system, generally under Docker root, and mounts it inside the container on the specified path. Docker uses volumes to decouple containers from the data they create. So even if the container using this volume is removed, the data will persist on the host system. You can specify volumes in a Dockerfile or in the command line while running the container, as follows:

$ docker run -v /var/www/html image_id

You can use docker inspect to get the host path of the volumes attached to container.

Finally, we have used the EXPOSE instruction, which will expose the specified container port to the host. In this case, it's port 80, where the Apache server will be listening for web requests. To use an exposed port on the host system, we need to use either the -p flag to explicitly specify the port mapping or the -P flag, which will dynamically map the container port to the available host port. We have used the -p flag with the argument 80:80, which will map the container port 80 to the host port 80 and make Apache accessible through the host.

The last instruction, CMD, sets the command to be executed when running the image. We are using the executable format of the CMD instruction, which specifies the executable to be run with its command-line arguments. In this case, our executable is the Apache binary with -D FOREGROUND as an argument. By default, the Apache parent process will start, create a child process, and then exit. If the Apache process exits, our container will be turned off as it no longer has a running process. With the -D FOREGROUND argument, we instruct Apache to run in the foreground and keep the parent process active. We can have only one CMD instruction in a Dockerfile.

The instruction set includes some more instructions, such as ADD, COPY, and ENTRYPOINT. I cannot cover them all because it would run into far too many pages. You can always refer to the official Docker site to get more details. Check out the reference URLs in the See also section.

There's more…

Once the image has been created, you can share it on Docker Hub, a central repository of public and private Docker images. You need an account on Docker Hub, which can be created for free. Once you get your Docker Hub credentials, you can use docker login to connect your Docker daemon with Docker Hub and then use docker push to push local images to the Docker Hub repository. You can use the respective help commands or manual pages to get more details about docker login and docker push.

Alternatively, you can also set up your own local image repository. Check out the Docker documents for deploying your own registry at https://docs.docker.com/registry/deploying/ .

We need a base image or any other image as a starting point for the Dockerfile. But how do we create our own base image?

Base images can be created with tools such as debootstrap and supermin. We need to create a distribution-specific directory structure and put all the necessary files inside it. Later, we can create a tarball of this directory structure and import the tarball as a Docker image using the docker import command.

See also

Dockerfile reference: https://docs.docker.com/reference/builder/

Dockerfile best practices: https://docs.docker.com/articles/dockerfile_best-practices

More Dockerfile best practices: http://crosbymichael.com/do ckerfile-best-practices.html

Create a base image: http://docs.docker.com/engine/articles/baseimages/

In the previous recipe, we saw how to start and manage virtual machines with KVM. This recipe covers the use of Virsh and virt-install to create and manage virtual machines. The libvirt Linux library exposes various APIs to manage hypervisors and virtual machines. Virsh is a command-line tool that provides an interface to libvirt APIs.

To create a new machine, Virsh needs the machine definition in XML format. virt-install is a Python script to easily create a new virtual machine without manipulating bits of XML. It provides an easy-to-use interface to define a machine, create an XML definition for it and then load it in Virsh to start it.

In this recipe, we will create a new virtual machine with virt-install and see how it can be managed with various Virsh commands.

Getting ready

You will need access to the root account or an account with sudo privileges.

Install the required packages, as follows:

$ sudo apt-get update

$ sudo apt-get install -y qemu-kvm libvirt-bin virtinst

Install packages to create the cloud init disk:

$ sudo apt-get install genisoimage

Add your user to the libvirtd group and update group membership for the current session:

$ sudo adduser ubuntu libvirtd

$ newgrp libvirtd

How to do it…

We need to create a new virtual machine. This can be done either with an XML definition of the machine or with a tool called virt-install. We will again use the prebuilt Ubuntu Cloud images and initialize them with a secondary disk:

First, download the Ubuntu Cloud image and prepare it for use:

$ mkdir ubuntuvm && cd ubuntuvm

$ wget -O trusty.img.dist \

http://cloud-images.ubuntu.com/releases/trusty/release/ubuntu- 14.04-server-cloudimg-amd64-disk1.img

$ qemu-img convert -O qcow2 trusty.img.dist trusty.img.orig

$ qemu-img create -f qcow2 -b trusty.img.orig trusty.img

Create the initialization disk to initialize your cloud image:

$ sudo vi user-data

#cloud-config

password: password

chpasswd: { expire: False }

ssh_pwauth: True

$ sudo vi meta-data

instance-id: ubuntu01;

local-hostname: ubuntu

$ genisoimage -output cidata.iso -volid cidata -joliet \

-rock user-data meta-data

Now that we have all the necessary data, let's create a new machine, as follows:

$ virt-install --import --name ubuntu01 \

--ram 256 --vcpus 1 --disk trusty.img \

--disk cidata.iso,device=cdrom \

--network bridge=virbr0 \

--graphics vnc,listen=0.0.0.0 --noautoconsole -v

This should create a virtual machine and start it. A display should be opened on the local VNC port 5900. You can access the VNC through other systems available on the local network with a GUI.

You can set up local port forwarding and access VNC from your local system as follows: $ ssh kvm_hostname_or_ip -L 5900:127.0.0.1:5900 $ vncviewer localhost:5900

Once the cloud-init process completes, you can log in with the default user, ubuntu, and the password set in user-data.

Now that the machine is created and running, we can use the virsh command to manage this machine. You may need to connect virsh and qemu before using them:

$ virsh connect qemu:///system

Get a list of running machines with virsh list. The --all parameter will show all available machines, whether they are running or stopped:

$ virsh list --all # or virsh --connect qemu:///system list

You can open a console to a running machine with virsh as follows. This should give you a login prompt inside the virtual machine:

$ virsh console ubuntu01

To close the console, use the Ctrl + ] key combination.

Once you are done with the machine, you can shut it down with virsh shutdown. This will call a shutdown process inside the virtual machine:

$ virsh shutdown ubuntu01

You can also stop the machine without a proper shutdown, as follows:

$ virsh destroy ubuntu01

To completely remove the machine, use virsh undefine. With this command, the machine will be deleted and cannot be used again:

$ virsh destroy ubuntu01

How it works…

Both the virt-install and virsh commands collectively give you an easy-to-use virtualization environment. Additionally, the system does not need to support hardware virtualization. When it's available, the virtual machines will use KVM and hardware acceleration, and when KVM is not supported, Qemu will be used to emulate virtual hardware.

With virt-install, we have easily created a KVM virtual machine. This command abstracts the XML definition required by libvirt. With a list of various parameters, we can easily define all the components with their respective configurations. You can get a full list of virt-install parameters with the --help flag.

The virtinst package, which installs virt-install, also contains some more commands, such as virt-clone, virt-admin, and virt-xml. Use tab completion in your bash shell to get a list of all virt-* commands.

Once the machine is defined and running, it can be managed with virsh subcommands. Virsh provides tons of subcommands to manage virtual machines, or domains as they are called by libvirt. You can start or stop machines, pause and resume them, or stop them entirely. You can even modify the machine configuration to add or remove devices as needed, or create a clone of an existing machine. To get a list of all machine (domain) management commands, use virsh help domain.

Once you have your first virtual machine, it becomes easier to create new machines using the XML definition from it. You can dump the XML definition with virsh dumpxml machine, edit it as required, and then create a new machine using XML configuration with virsh create configuration.xml.

There are a lot more options available for the virsh and virt-install commands; check their respective manual pages for more details.

There's more…

In the previous example, we used cloud images to quickly start a virtual machine. You do not need to use cloud machines, and you can install the operating system on your own using the respective installation media.

See also

Check out the manual pages for virt-install using $ man virt-install

Check out the manual pages for virsh using $ man virsh

The official Libvirt site: http://libvirt.org/

The Libvirt documentation on Ubuntu Server guide: https://help.ubuntu.com/lts/serverguide/libvirt.html

When your application is small, you can use a single MySQL server for all your database needs. As your application becomes popular and you get more and more requests, the database starts becoming a bottleneck for application performance. With thousands of queries per second, the database write queue gets longer and read latency increases. To solve this problem, you can use multiple replicas of the same database and separate read and write queries between them.

In this recipe, we will learn how to set up replication with the MySQL server.

Getting ready

You will need two MySQL servers and access to administrative accounts on both.

Make sure that port 3306 is open and available on both servers.

How to do it…

Follow these steps to create MySQL replicas:

Create the replication user on the Master server:

$ mysql -u root -p

mysql> grant replication slave on *.* TO ‘slave_user’@’10.0.2.62’ identified by ‘password’;

mysql> flush privileges;

mysql> quit

Edit the MySQL configuration on the Master server:

$ sudo nano /etc/mysql/my.cnf

[mysqld]

bind-address = 10.0.2.61 # your master server ip

server-id = 1

log-bin = mysql-bin

binlog-ignore-db = “mysql”

Restart MySQL on the Master server:

$ sudo service mysql restart

Export MySQL databases on the Master server. Open the MySQL connection and lock the database to prevent any updates:

$ mysql -u root -p

mysql> flush tables with read lock;

Read the Master status on the Master server and take a note of it. This will be used shortly to configure the Slave server:

mysql> show master status;

Open a separate terminal window and export the required databases. Add the names of all the databases you want to export:

$ mysqldump -u root -p --databases testdb > master_dump.sql

Now, unlock the tables after the database dump has completed:

mysql> UNLOCK TABLES;

mysql> quit;

Transfer the backup to the Slave server with any secure method:

$ scp master_backup.sql ubuntu@10.0.2.62:/home/ubuntu/master_backup.sql

Next, edit the configuration file on the Slave server:

$ sudo nano /etc/mysql/my.cnf

[mysqld]

bind-address = 10.0.2.62

server-id = 2

relay_log=relay-log

Import the dump from the Master server. You may need to manually create a database before importing dumps:

$ mysqladmin -u admin -p create testdb

$ mysql -u root -p master_dump.sql

Restart the MySQL server:

$ sudo service mysql restart

Now set the Master configuration on the Slave. Use the values we received from show master status command in step 5:

$ mysql -u root -p

mysql > change master to

master_host=’10.0.2.61’, master_user=’slave_user’,

master_password=’password’, master_log_file=’mysql- bin.000010’,

master_log_pos=2214;

Start the Slave:

mysql> start slave;

Check the Slave's status. You should see the message Waiting for master to send event under Slave_IO_state:

mysql> show slave status\G

Now you can test replication. Create a new database with a table and a few sample records on the Master server. You should see the database replicated on the Slave immediately.

How it works…

MySQL replication works with the help of binary logs generated on the Master server. MySQL logs any changes to the database to local binary logs with a lightweight buffered and sequential write process. These logs will then be read by the slave. When the slave connects to the Master, the Master creates a new thread for this replication connection and updates the slave with events in a binary log, notifying the slave about newly written events in binary logs.

On the slave side, two threads are started to handle replication. One is the IO thread, which connects to the Master and copies updates in binary logs to a local log file, relay_log. The other thread, which is known as the SQL thread, reads events stored on relay_log and applies them locally.

In the preceding recipe, we have configured Master-Slave replication. MySQL also supports Master-Master replication. In the case of Master-Slave configuration, the Master works as an active server, handling all writes to database. You can configure slaves to answer read queries, but most of the time, the slave server works as a passive backup server. If the Master fails, you manually need to promote the slave to take over as Master. This process may require downtime.

To overcome problems with Master - Slave replication, MySQL can be configured in Master-Master relation, where all servers act as a Master as well as a slave. Applications can read as well as write to all participating servers, and in case any Master goes down, other servers can still handle all application writes without any downtime. The problem with Master-Master configuration is that it’s quite difficult to set up and deploy. Additionally, maintaining data consistency across all servers is a challenge. This type of configuration is lazy and asynchronous and violates ACID properties.

In the preceding example, we configured the server-id variable in the my.cnf file. This needs to be unique on both servers. MySQL version 5.6 adds another UUID for the server, which is located at data_dir/auto.cnf. If you happen to copy data_dir from Master to host or are using a copy of a Master virtual machine as your starting point for a slave, you may get an error on the slave that reads something like master and slave have equal mysql server UUIDs. In this case, simply remove auto.cnf from the slave and restart the MySQL server.

There’s more…

You can set MySQL load balancing and configure your database for high availability with the help of a simple load balancer in front of MySQL. HAProxy is a well known load balancer that supports TCP load balancing and can be configured in a few steps, as follows:

Set your MySQL servers to Master - Master replication mode.

Log in to mysql and create one user for haproxy health checks and another for remote administration:

mysql> create user ‘haproxy_admin’@’haproxy_ip’;

mysql> grant all privileges on *.* to ‘haproxy_admin’@’haproxy_ip’ identified by ‘password’ with grant option;

mysql> flush privileges;

Next, install the MySQL client on the HAProxy server and try to log into the mysql server with the haproxy_admin account.

Install HAProxy and configure it to connect to mysql on the TCP port:

listen mysql-cluster

bind haproxy_ip:3306

mode tcp

option mysql-check user haproxy_check

balance roundrobin

server mysql-1 mysql_srv_1_ip:3306 check

server mysql-2 mysql_srv_2_ip:3306 check

Finally, start the haproxy service and try to connect to the mysql server with the haproxy_admin account:

$ mysql -h haproxy_ip -u hapoxy_admin -p

See also

MySQL replication configuration at http://dev.mysql.com/doc/refman/5.6/en/replication.html

How MySQL replication works at https://www.percona.com/blog/2013/01/09/how-does-mysql-replication-really-work/

MySQL replication formats at http://dev.mysql.com/doc/refman/5.5/en/replication-formats.html