Ubuntu

In this recipe, we will learn how to use the Rsync utility to synchronize files between two directories or between two servers.

How to do it…

Follow these steps to synchronize files with Rsync:

Set up key-based authentication between source and destination servers. We can use password authentication as well, which is described later in this recipe.

Create a sample directory structure on the source server. You can use existing files as well:

ubuntu@src$ mkdir sampledir

ubuntu@src$ touch sampledir/file{1..10}

Now, use the following command to synchronize the entire directory from the source server to your local system. Note the / after sampledir. This will copy contents of sampledir in the backup. Without /, the entire sampledir will be copied to the backup:

ubuntu@dest$ rsync -azP -e ssh ubuntu@10.0.2.8:/home/ubuntu/sampledir/ backup

As this is the first time, all files from sampledir on the remote server will be downloaded in a backup directory on your local system. The output of the command should look like the following screenshot:

You can check the downloaded files with the ls command:

$ ls -l backup

Add one new file on the remote server under sampledir:

ubuntu@src$ touch sampledir/file22

Now re-execute the rsync command on the destination server. This time, rsync will only download a new file and any other update files. The output should look similar to the following screenshot:

ubuntu@dest$ rsync -azP -e ssh ubuntu@10.0.2.8:/home/ubuntu/sampledir backup

To synchronize two local directories, you can simply specify the source and destination path with rsync, as follows:

$ rsync /var/log/mysql ~/mysql_log_backup

How it works…

Rsync is a well known command line file synchronization utility. With Rsync, you can synchronize files between two local directories, as well as files between two servers. This tool is commonly used as a simple backup utility to copy or move files around systems. The advantage of using Rsync is that file synchronization happens incrementally, that is, only new and modified files will be downloaded. This saves bandwidth as well as time. You can quickly schedule a daily backup with a cron and Rsync. Open a cron jobs file with ctontab-e and add the following line to enable daily backups:

$ crontab -e # open crontab file

@daily rsync -aze ssh ubuntu@10.0.2.50:/home/ubuntu/sampledir /var/backup

In the preceding example, we have used a pull operation, where we are downloading files from the remote server. Rsync can be used to upload files as well. Use the following command to push files to the remote server:

$ rsync -azP -e ssh backup ubuntu@10.0.2.50:/home/ubuntu/sampledir

Rsync provides tons of command line options. Some options that are used in the preceding example are –a, a combination of various other flags and stands for achieve. This option enables recursive synchronization and preserves modification time, symbolic links, users, and group permissions. Option -z is used to enable compression while transferring files, while option -P enables progress reports and the resumption of interrupted downloads by saving partial files.

We have used one more option, -e, which specifies which remote shell to be used while downloading files. In the preceding command, we are using SSH with public key authentication. If you have not set public key authentication between two servers, you will be asked to enter a password for your account on the remote server. You can skip the -e flag and rsync will use a non-encrypted connection to transfer data and login credentials.

Note that the SSH connection is established on the default SSH port, port 22. If your remote SSH server runs on a port other than 22, then you can use a slightly modified version of the preceding command as follows:

rsync -azP -e "ssh -p port_number" source destination

Anther common option is --exclude, which specifies the pattern for file names to be excluded. If you need to specify multiple exclusion patterns, then you can specify all such patterns in a text file and include that file in command with the options --exclude-from=filename. Similarly, if you need to include some specific files only, you can specify the inclusion pattern with options --include=pattern or --include-from=filename.

Exclude a single file or files matching with a single pattern:

$ rsync -azP --exclude 'dir*' source/ destination/

Exclude a list of patterns or file names:

$ rsync -azP --exclude-from 'exclude-list.txt' source/ destination/

By default, Rsync does not delete destination files, even if they are deleted from the source location. You can override this behavior with a --delete flag. You can create a backup of these files before deleting them. Use the --backup and --backup-dir options to enable backups. To delete files from the source directory, you can use the --remove-source-files flag. Another handy option is --dry-run, which simulates a transfer with the given flags and displays the output, but does not modify any files. You should use --dry-run before using any deletion flags.

Use this to remove source files with --dry-run:

$ rsync --dry-run --remove-source-files -azP source/ destination/

There's more…

Rsync is a great tool to quickly synchronize the files between source and destination, but it does not provide bidirectional synchronization. It means the changes are synchronized from source to destination and not vice versa. If you need bi-directional synchronization, you can use another utility, Unison. You can install Unison on Debian systems with the following command:

$ sudo apt-get -y install unison

Once installed, Unison is very similar to Rsync and can be executed as follows:

$ unison /home/ubuntu/documents ssh://10.0.2.56//home/ubuntu/documents

You can get more information about Unison in the manual pages with the following command:

$ man unison

If you wish to have your own Dropbox-like mirroring tool which continuously monitors for local file changes and quickly replicates them to network storage, then you can use Lsyncd. Lsyncd is a live synchronization or mirroring tool, which monitors the local directory tree for any events (with inotify and fsevents), and then after few seconds spawns a synchronization process to mirror all changes to a remote location. By default, Lsyncd uses Rsync for synchronization.

As always, Lsyncd is available in the Ubuntu package repository and can be installed with a single command, as follows:

$ sudo apt-get install lsyncd

To get more information about Lsyncd, check the manual pages with the following command:

$ man lsyncd

See also

Ubuntu Rsync community page at https://help.ubuntu.com/community/rsync

In this recipe, we will learn how to create databases and tables and store data in those tables. We will learn the basic Structured Query Language (SQL) required for working with MySQL. We will focus on using the command-line MySQL client for this tutorial, but you can use the same queries with any client software or code.

Getting ready

Ensure that the MySQL server is installed and running. You will need administrative access to the MySQL server. Alternatively, you can use the root account of MySQL.

How to do it…

Follow these steps to store and retrieve data with MySQL:

First, we will need to connect to the MySQL server. Replace admin with a user account on the MySQL server. You can use root as well but it’s not recommended:

$ mysql -u admin -h localhost -p

When prompted, enter the password for the admin account. If the password is correct, you will see the following MySQL prompt:

Create a database with the following query. Note the semi-colon at the end of query:

mysql > create database myblog;

Check all databases with a show databases query. It should list myblog:

mysql > show databases;

Select a database to work with, in this case myblog:

mysql > use myblog;

Database changed

Now, after the database has changed, we need to create a table to store our data. Use the following query to create a table:

CREATE TABLE `articles` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`title` varchar(255) NOT NULL,

`content` text NOT NULL,

`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,

PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1;

Again, you can check tables with the show tables query:

mysql > show tables;

Now, let’s insert some data in our table. Use the following query to create a new record:

mysql > INSERT INTO `articles` (`id`, `title`, `content`, `created_at`)

VALUES (NULL, ‘My first blog post’, ‘contents of article’, CURRENT_TIMESTAMP);

Retrieve data from the table. The following query will select all records from the articles table:

mysql > Select * from articles;

Retrieve the selected records from the table:

mysql > Select * from articles where id = 1;

Update the selected record:

mysql > update articles set title=”New title” where id=1;

Delete the record from the articles table using the following command:

mysql > delete from articles where id = 2;

How it works…

We have created a relational database to store blog data with one table. Actual blog databases will need additional tables for comments, authors, and various entities. The queries used to create databases and tables are known as Data Definition Language (DDL), and queries that are used to select, insert, and update the actual data are known as Data Manipulation Language (DML).

MySQL offers various data types to be used for columns such as tinyint, int, long, double, varchar, text, blob, and so on. Each data type has its specific use and a proper selection may help to improve the performance of your database.

Storage is one of the slowest components in a server's system, but is still the most important component. Storage is mainly used as a persistence mechanism to store a large amount of processed/unprocessed data. A slow storage device generally results in heavy utilization of read write buffers and higher memory consumption. You will see higher CPU usage, but most of the CPU time is spent waiting for I/O requests.

The recent developments of the flash storage medium have vastly improved storage performance. Still, it's one of the slowest performing components and needs proper planning— I/O planning in the application code, plus enough main memory for read write buffers.

In this recipe, we will learn to monitor storage performance. The main focus will be on local storage devices rather than network storage.

Getting ready

As always, you will need sudo access for some commands.

Some of the commands many not be available by default. Using them will prompt you if the command is not available, along with the process necessary to install the required package.

Install the sysstat package as follows. We have already used it in previous recipes:

$ sudo apt get install sysstat

How to do it…

The first command we will look at is vmstat. Using vmstat without any option displays an io column with two sub entries: bytes in (bi) and bytes out (bo). Bytes in represents the number of bytes read in per second from the disk and bytes out represents the bytes written to the disk:

Vmstat also provides two flags, -d and -D, to get disk statistics. Flag -d displays disk statistics and flag -D displays a summary view of disk activity:

There's one more option, -p, that displays partition-specific disk statistics. Use the command lsblk to get a list of available partitions and then use the vmstat -p partition:

Another command, dstat, is a nice replacement for vmstat, especially for disk statistics reporting. Use it with flag -d to get disk read writes per seconds. If you have multiple disks, you can use dstat to list their stats separately:

$ dstat -d -D total,sda

Next, we will look at the command iostat. When used without any options, this command displays basic CPU utilization, along with read write statistics for each storage device:

The column tps specifies the I/O requests sent to a device per second, and kb_read/s and kb_wrtn/s specifies per second blocks read and blocks written respectively. kb_read and kb_wrtn shows the total number of blocks read and written.

Some common options for iostat include –d, that displays disk only statistics, -g that displays statistics for a group of devices, flag -p to display partition specific stats, and -x to get extended statistics. Do not forget to check the manual entries for iostat to get more details.

You can also use the command iotop, which is very similar to the top command but it displays disk utilization and relevant processes.

The command lsof can display the list of all open files and respective processes using that file. Use lsof with the process name to get files opened by that process:

$ lsof -c sshd

To get a list of files opened by a specific PID, use the following command: $ lsof -p 1134. Or, to get a list of files opened by a specific user, use the $ lsof -u ubuntu command.

All these commands provide details on the read write performance of a storage device. Another important detail to know is the availability of free space. To get details of space utilization, you can use command df -h. This will list a partition-level summary of disk space utilization:

Finally, you can use the sar command to track disk performance over a period of time. To get real-time disk activity, use sar with the -d option, as follows:

$ sar -d 1

Use flag -F to get details on file system utilization and flag -S to display swap utilization. You can also enable sar logging and then extract details from those logs. Check the previous recipes in this article for how to enable sar logging. Also check manual entries for sar to get details of various options.

When an application becomes popular and the number of requests increases beyond the capacity of a single server, we need to scale horizontally. We can always increase the capacity (vertical scaling) of a server by adding more memory and processing power, but a single server cannot scale beyond a certain limit. While adding separate servers or replicas of the application server, we need a mechanism which directs the traffic between these replicas. The hardware or software tool used for this purpose is known as a load balancer. Load balancers work as transparent mechanisms between the application server and client by distributing the requests between available instances. This is a commonly used technique for optimizing resource utilization and ensuring fault tolerant applications.

Nginx can be configured to work as an efficient Layer 7 as well as Layer 4 load balancer. Layer 7 is application layer of HTTP traffic. With Layer 4 support, Nginx can be used to load balance database servers or even XMPP traffic. With version 1.9.0, Nginx has enabled support for Layer 4 load balancing in their open source offerings.

In this recipe, we will learn how to set up Nginx as a load balancer.

Getting ready

You will need access to a root account or an account with sudo privileges.

You will need a minimum of three servers, as follows:

An Nginx server, which will be set as a load balancer

Two or more application servers with a similar code base set up on all

How to do it…

Follow these steps to set load balancing with Nginx:

I assume that you already have Nginx installed. If not, you can refer to the Installing Nginx with PHP_FPM recipe of this article.

Now, create a new configuration file under /etc/nginx/sites-available. Let's call it load_balancer:

$ sudo nano /etc/nginx/sites-available/load_balancer

Add the following lines to this load_balancer file. This is the minimum configuration required to get started with load balancing:

upstream backend {

server srv1.example.com;

server srv2.example.com;

server 192.168.1.12:8080;

# other servers if any

}

server {

listen 80;

location / {

proxy_pass http://backend;

}

}

Enable this configuration by creating a symlink to load_balancer under sites-enabled:

$ sudo ln -s /etc/nginx/sites-available/load_balancer /etc/nginx/sites-enabled/load_balancer

You may want to disable all other sites. Simply remove the respective links under sites-enabled.

Check the configuration for syntax errors:

$ sudo nginx -t

Now, reload Nginx for the changes to take effect:

$ sudo service nginx reload

Yes, you are ready to use a load balancer. Open your favorite browser and point it to the IP of your Nginx server. You should see the contents of example.com or whatever domain you have used.

How it works…

We have created a very basic configuration for a load balancer. With this configuration, Nginx takes the traffic on port 80 and distributes it between srv1.example.com and srv2.example.com. With an upstream directive, we have defined a pool of servers that will actually process the requests. The upstream directive must be defined in a HTTP context. Once the upstream directive is defined, it will be available for all site configurations.

When defining servers under an upstream directive, you can also use the IP address and port of the application server. This is an ideal configuration, especially when both the load balancer and the application servers are on the same private network, and this will help minimize the communication overhead between Nginx and backend servers.

Next, under the server block, we have configured Nginx to proxy_pass all requests to our backend pool.

While setting backend servers, we have not explicitly specified any load balancing algorithm. Nginx provides various load balancing algorithms that define the server that will receive a particular request. By default, Nginx uses a round-robin algorithm and passes requests to each available server in sequential order. Other available options are as follows:

least_connection: This passes the request to the host with the fewest active connections.

least_time: Nginx chooses the host with the lowest latency. This option is available with Nginx plus.

ip_hash: A hash of clients' IP addresses, and is used to determined the host to send the request to. This method guarantees that requests with the same IP address are served by the same host, unless the selected host is down.

Hash uses a user defined key to generate a hash value and then uses the hash to determine the processing host.

There's more…

Nginx provides various other load balancing features, such as weighted load balancing, active and passive health checks, backup servers, and session persistence. With the latest commits to the open source version, it now supports TCP load balancing as well. These settings can be updated at runtime with the help of HTTP APIs. The following are a few examples of different load balancing configurations:

Set server weights:

upstream app-servers {

server srv1.example.com weight 3;

server srv2.example.com;

}

Health checkups and backup servers:

upstream app-servers {

server srv1.example.com max_fails 3 fail_timeout 10;

server srv2.example.com fail_timeout 50;

192.168.1.12:8080 backup;

}

Session persistence with cookies:

upstream app-servers {

server srv1.example.com;

server srv2.example.com;

sticky cookie srv_id expires=1h domain=.example.com path=/;

}

Check the Nginx load balancing guide for various other load balancing options and their respective details.

See also

Nginx admin guide for load balancers at https://www.nginx.com/resources/admin-guide/load-balancer

Git clone allows you to create a copy of your repository in a new directory or location. It can be used to replicate a remote repository on your local system or create a local clone to be shared over an intranet. This recipe covers the git clone command. We will learn to create a clone of a remote repository and then take a look at various transport protocols supported by Git for cloning.

Getting ready

You will need Git binaries installed on your local system, plus a remote repository. Note down the full path (clone URL) of the remote repository.

How to do it…

Create a clone of the repository with the git clone command, as follows:

$ git clone ssh://ubuntu@192.168.0.100:22/home/ubuntu/cookbook.git \ ubuntu_cookbook

You will be asked to enter a password for the user account ubuntu.

This command will create a new directory named ubuntu_cookbook and clone the repository cookbook.git into this directory.

How it works…

As seen in the previous example, the git clone command will create a new copy of an existing repository. The repository can be a local repository or one located on a remote server. Git supports various protocols to transfer the content between two systems. This includes well-known protocols such as SSH, HTTP/S, and rsync. In addition, Git provides a native transport protocol named Git. Note that the Git protocol does not require any authentication and should be used carefully. In the previous example, we have used the SSH protocol. When working with local repositories, you can use file///path/to/repo.git or even an absolute path /path/to/repo.git format.

Cloning requires a single argument, which is the path of the repository to be cloned. You can skip the destination directory and Git will create a clone in a new directory named after the repository name.

You can also create a new bare clone with the --bare option of the git clone command. This is useful for creating a shared central clone that is used by a group of people.

Another important option is the depth clone. When cloning a large repository that contains years of work, and you do not really need the entire history of the repository, the option --depth can be used to copy only a specified number of revisions. This will help you in quickly downloading just the tip of an entire repository, and will save you some bandwidth by avoiding unnecessary downloads. The syntax for the --depth option is as follows:

git clone --depth 1 https://github.com/torvalds/linux.git mylinux

See also

You can read more by following these links:

Git clone: https://git-scm.com/docs/git-clone

In this recipe, we will install and configure the squid proxy and caching server. The term proxy is generally combined with two different terms: one is forward proxy and the other is reverse proxy.

When we say proxy, it generally refers to forward proxy. A forward proxy acts as a gateway between a client's browser and the Internet, requesting the content on behalf of the client. This protects intranet clients by exposing the proxy as the only requester. A proxy can also be used as a filtering agent, imposing organizational policies. As all Internet requests go through the proxy server, the proxy can cache the response and return cached content when a similar request is found, thus saving bandwidth and time.

A reverse proxy is the exact opposite of a forward proxy. It protects internal servers from the outside world. A reverse proxy accepts requests from external clients and routes them to servers behind the proxy. External clients can see a single entity serving requests, but internally, it can be multiple servers working behind the proxy and sharing the load. More details about reverse proxies are covered in article 3, Working with Web Servers.

In this recipe, we will discuss how to install a squid server. Squid is a well-known application in the forward proxy world and works well as a caching proxy. It supports HTTP, HTTPS, FTP, and other popular network protocols.

Getting ready

As always, you will need access to a root account or an account with sudo privileges.

How to do it…

Following are the steps to setup and configure Squid proxy:

Squid is quite an old, mature, and commonly used piece of software. It is generally shipped as a default package with various Linux distributions. The Ubuntu package repository contains the necessary pre-compiled binaries, so the installation is as easy as two commands.

First, update the apt cache and then install squid as follows:

$ sudo apt-get update

$ sudo apt-get install squid3

Edit the /etc/squid3/squid.conf file:

$ sudo nano /etc/squid3/squid.conf

Ensure that the cache_dir directive is not commented out:

cache_dir ufs /var/spool/squid3 100 16 256

Optionally, change the http_port directive to your desired TCP port:

http_port 8080

Optionally, change the squid hostname:

visible_hostname proxy1

Save changes with Ctrl + O and exit with Ctrl + X.

Restart the squid server:

$ sudo service squid3 restart

Make sure that you have allowed the selected http_port on firewall.

Next, configure your browser using the squid server as the http/https proxy.

How it works…

Squid is available as a package in the Ubuntu repository, so you can directly install it with the apt-get install squid command. After installing squid, we need to edit the squid.conf file for some basic settings. The squid.conf file is quite a big file and you can find a large number of directives listed with their explanation. It is recommended to create a copy of the original configuration file as a reference before you do any modifications.

In our example, we are changing the port squid listens on. The default port is 3128. This is just a security precaution and it's fine if you want to run squid on the default port. Secondly, we have changed the hostname for squid.

Other important directive to look at is cache_dir. Make sure that this directive is enabled, and also set the cache size. The following example sets cache_dir to /var/spool/suid3 with the size set to 100MB:

cache_dir ufs /var/spool/squid3 100 16 256

To check the cache utilization, use the following command:

$ sudo du /var/spool/squid3

There's more…

Squid provides lot more features than a simple proxy server. Following is a quick list of some important features:

Access control list

With squid ACLs, you can set the list of IP addresses allowed to use squid. Add the following line at the bottom of the acl section of /etc/squid3/squid.conf:

acl developers src 192.168.2.0/24

Then, add the following line at the top of the http_access section in the same file:

http_access allow developers

Set cache refresh rules

You can change squid's caching behavior depending on the file types. Add the following line to cache all image files to be cached—the minimum time is an hour and the maximum is a day:

refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 3600 90% 86400

This line uses a regular expression to find the file names that end with any of the listed file extensions (gif, png, and etc)

Sarg – tool to analyze squid logs

Squid Analysis Report Generator is an open source tool to monitor the squid server usages. It parses the logs generated by Squid and converts them to easy-to-digest HTML-based reports. You can track various metrics such as bandwidth used per user, top sites, downloads, and so on. Sarg can be quickly installed with the following command:

$ sudo apt-get install sarg

The configuration file for Sarg is located at /etc/squid/sarg.conf. Once installed, set the output_dir path and run sarg. You can also set cron jobs to execute sarg periodically. The generated reports are stored in output_dir and can be accessed with the help of a web server.

Squid guard

Squid guard is another useful plugin for squid server. It is generally used to block a list of websites so that these sites are inaccessible from the internal network. As always, it can also be installed with a single command, as follows:

$ sudo apt-get install squidguard

The configuration file is located at /etc/squid/squidGuard.conf.

See also

Check out the squid manual pages with the man squid command

Check out the Ubuntu community page for squid guard at https://help.ubuntu.com/community/SquidGuard

Transcoding means converting media from one format to another. Suppose your music files are in a format different to MP3 and your media player only understands MP3 format. In that case, you need to convert your music files to MP3. This conversion task is done by transcoder programs. There are various transcoding programs available, such as ffmpeg and avconv. These programs need codec before they can convert media from source format to destination format. We need to separately install and configure these components.

Ampache supports on-the-fly transcoding of media files. That is, your music that is not in an MP3 format can be converted into the MP3 format just before it is delivered to your music player, and your high definition video content can be optimized for mobile consumption to reduce bandwidth use.

In this recipe, we will learn how to install and configure transcoding programs with Ampache.

Getting ready

Make sure you have working a setup of the Ampache server.

You will need access to a root account or an account with root privileges.

How to do it…

Ampache depends on external libraries for transcoding to work. We will first install the dependencies and then configure Ampache to work with them:

First, add the ffmpeg PPA to the Ubuntu installation sources:

$ sudo apt-add-repository ppa:mc3man/trusty-media

$ sudo apt-get update

Now, install ffmpeg and other required codecs:

$ sudo apt-get install flac mp3splt lame faad ffmpeg vorbis- tools

Next, we need to configure Ampache and enable transcoding. Open the configuration file located at /var/www/ampache/config/ampache.cfg.php, find the following lines in the file, and uncomment them:

max_bit_rate = 576

min_bit_rate = 48

transcode_flac = required

transcode_mp3 = allowed

encode_target = mp3

transcode_cmd = "ffmpeg"

Here, we have set ffmpeg for the encoding/decoding of media files. You can choose any encoder of your choice. Change the value of transcode_cmd respectively.

Next, enable debug mode to get details of the transcoding. Find the debug section in the configuration file and set it as follows:

debug = true

Enable log file path which is, by default, set to null

log_path = "/var/log/ampache"

Save the changes to the configuration file and reload the Apache web server:

$ sudo service apache2 reload

Now your transcoding setup should be working. You should be able to upload media in a different format and play it as MP3 or other respective formats.

It often happens that we have content in a format that is not supported by the device we are using for playback. Maybe the device does not have the required codec or the hardware is not capable of playing a high bit rate. We may even need to convert content to a lower bit rate and reduce the bandwidth used to stream. The transcoding feature of Ampache helps us to cover these scenarios.

With transcoding, you can convert the content to the desired device-supported format before actually starting the streaming. This is called on-the-fly transcoding. The contents are encoded in a new format as and when needed. Once the conversion is completed, the new format can be cached for repeat use. In the above example, we set Ampache to convert FLAC files to MP3 using the ffmpeg tool. Now whenever we request a file that is originally available in the FLAC format, Ampache will convert it to MP3 before streaming it to our device.

Ampache uses external, well-established media conversion tools for transcoding. Any tool that works with the Ubuntu command line can be configured to work with Ampache. Refer to the Ampache configuration file to get more details and configuration examples.

In this article, you will see how to add new users to the Ubuntu server, update existing users, and set permissions for users. You will get to know the default setting for new users and how to change them. Also, you will take a look at secure shell (SSH) access and securing user profiles.

LXC is a system built on the modern Linux kernel and enables the creation and management of virtual Linux systems or containers. As discussed earlier, LXC is not a full virtualization system and shares the kernel with the host operating system, providing lightweight containerization. LXC uses Linux namespaces to separate and isolate the processes running inside containers. This provides much better security than simple chroot-based filesystem isolation. These containers are portable and can easily be moved to another system with a similar processor architecture.

Ubuntu 15.04 unveiled a new tool named LXD, which is a wrapper around LXC. The official page calls it a container hypervisor and a new user experience for LXC. Ubuntu 16.04 comes preinstalled with its latest stable release, LXD 2.0. With LXD, you no longer need to work directly with lower-level LXC tools.

LXD adds some important features to LXC containers. First, it runs unprivileged containers by default, resulting in improved security and better isolation for containers. Second, LXD can manage multiple LXC hosts and can be used as an orchestration tool. It also supports the live migration of containers across hosts.

LXD provides a central daemon named LXD and a command-line client named lxc. Containers can be managed with the command-line client or the REST APIs provided by the LXD daemon. It also provides an OpenStack plugin, nova-compute-lxd, to deploy containers on the OpenStack cloud.

In this recipe, we will learn to install and configure the LXD daemon. This will set up a base for the next few recipes in this article.

Getting ready

You will need access to the root account or an account with sudo privileges.

Make sure that you have enough free space available on disk.

How to do it…

Ubuntu 16.04 ships with the latest release of LXD preinstalled. We just need to initialize the LXD daemon to set the basic settings.

First, update the apt cache and try to install LXD. This should install updates to the LXD package, if any:

$ sudo apt-get update

$ sudo apt-get install lxd

Along with LXD, we will need one more package named ZFS—the most important addition to Ubuntu 16.04. We will be using ZFS as a storage backend for LXD:

$ sudo apt-get install zfsutils-linux

Once LXD has been installed, we need to configure the daemon before we start using it. Use lxd init to start the initialization process. This will ask some questions about the LXD configuration:

$ sudo lxd init

Name of the storage backend to use (dir or zfs): zfs

Create a new ZFS pool (yes/no)? yes

Name of the new ZFS pool: lxdpool

Would you like to use an existing block device (yes/no)? no

Size in GB of the new loop device (1GB minimum): 10

Would you like LXD to be available over the network (yes/no)? no

Do you want to configure the LXD bridge (yes/no)? yes

Warning: Stopping lxd.service, but it can still be activated by: lxd.socket

LXD has been successfully configured.

Now, we have our LXD setup configured and ready to use. In the next recipe, we will start our first container with LXD.

How it works…

Ubuntu 16.04 comes preinstalled with LXD and makes it even easier to start with system containers or operating system virtualization. In addition to LXD, Ubuntu now ships with inbuilt support for ZFS (OpenZFS), a filesystem with support for various features that improve the containerization experience. With ZFS, you get faster clones and snapshots with copy-on-write, data compression, disk quotas, and automated filesystem repairs.

LXD is a wrapper around lower-level LXC or Linux containers. It provides the REST API for communicating and managing LXC components. LXD runs as a central daemon and adds some important features, such as dynamic resource restrictions and live migrations between multiple hosts. Containers started with LXD are unprivileged containers by default, resulting in improved security and isolation.

This recipe covers the installation and initial configuration of the LXD daemon. As mentioned previously, LXD comes preinstalled with Ubuntu 16. The installation commands should fetch updates to LXD, if any. We have also installed zfsutils-linux, a user space package to interact with ZFS. After the installation, we initialized the LXD daemon to set basic configuration parameters, such as the default storage backend and network bridge for our containers.

We selected ZFS as the default storage backend and created a new ZFS pool called lxdpool, backed by a simple loopback device. In a production environment, you should opt for a physical device or separate partition. If you have already created a ZFS pool, you can directly name it by choosing no for Create new ZFS pool. To use a separate storage device or partition, choose yes when asked for block storage.

ZFS is the recommended storage backend, but LXD also works with various other options, such as Logical Volume Manager (LVM) and btrfs (pronounced "butter F S"), that offer nearly the same features as ZFS or a simple directory-based storage system.

Next, you can choose to make LXD available on the network. This is necessary if you are planning a multi-host setup and support for migration. The initialization also offers to configure the lxdbr0 bridge interface, which will be used by all containers. By default, this bridge is configured with IPv6 only. Containers created with the default configuration will have their veth0 virtual Ethernet adapter attached to lxdbr0 through a NAT network. This is the gateway for containers to communicate with the outside world. LXD also installs a local DHCP server and the dnsmasq package. DHCP is used to assign IP addresses to containers, and dnsmasq acts as a local name-resolution service.

If you misplace the network bridge configuration or need to update it, you can use the following command to get to the network configuration screen:

$ sudo dpkg-reconfigure -p medium lxd

There's more…

The LXD 2.0 version, which ships with Ubuntu 16, is an LTS version. If you want to get your hands on the latest release, then you can install stable versions from the following repository:

$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable

For development releases, change the PPA to ppa:ubuntu-lxc/lxd-git-master.

For more information, visit the LXC download page at https://linuxcontainers.org/lxc/downloads/ .

If you still want to install LXC, you can. Use the following command:

$ sudo apt-get install lxc

This will install the required user space package and all the commands necessary to work directly with LXC. Note that all LXC commands are prefixed with lxc-, for example, lxc-create and lxc-info. To get a list of all commands, type lxc- in your terminal and press Tab twice.

See also

For more information, check the LXD page of the Ubuntu Server guide: https://help.ubuntu.com/lts/serverguide/lxd.html

The LXC blog post series is at https://www.stgraber.org/2013/12/20/lxc-1-0-blog-post-series/

The LXD 2.0 blog post series is at https://www.stgraber.org/2016/03/11/lxd-2-0-blog-post-series-012/

Ubuntu 16.04 switched to Systemd, which provides its own container framework, systemd-nspawn; read more about systemd containers on its Ubuntu man page at http://manpages.ubuntu.com/manpages/xenial/man1/systemd-nspawn.1.html

See how to get started with systemd containers at https://community.flockport.com/topic/32/systemd-nspawn-containers

In this recipe, we will learn how to install the File Transfer Protocol (FTP) server and configure it to use SSL encryption.

Getting ready

You will need access to a root account or an account with sudo privileges.

How to do it…

Follow these steps to install the secure FTP server:

Install vsftpd with the following command:

$ sudo apt-get update

$ sudo apt-get install vsftpd

After installation, we can configure vsftpd by editing /etc/vsftpd.conf.

First create the SSL certificate for the FTP server:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 - keyout /etc/ssl/private/vsftpd.pem -out /etc/ssl/private/vsftpd.pem

Next, configure Vsftpd. Add or edit the following lines in vsftpd.conf:

anonymous_enable=no

local_enable=yes

write_enable=yes

chroot_local_user=yes

Add the SSL certificate created in the previous step:

rsa_cert_file=/etc/ssl/private/vsftpd.pem

rsa_private_key_file=/etc/ssl/private/vsftpd.pem

ssl_enable=yes

ssl_ciphers=high

force_local_data_ssl=yes

force_local_logins_ssl=yes

Save and exit the configuration file.

Restart the Vsftpd server:

$ sudo service vsftpd restart

Now you can use any FTP client that supports the SFTP protocol to connect to your FTP server. The following is the configuration screen for SFTP client FileZilla:

How it works…

FTP is an insecure protocol and you should avoid using it, especially in a production environment. Limit use of FTP to downloads only and use more secure methods, such as SCP, to upload and transfer files on servers. If you have to use FTP, make sure that you have disabled anonymous access and enable SFTP to secure your data and login credentials.

In this recipe, we have installed Vsftpd, which is a default FTP package in the Ubuntu repository. Vsftpd stands for very secure FTP daemon, and it is designed to protect against possible FTP vulnerabilities. It supports both FTP and SFTP protocols.

As Vsftpd is available in the Ubuntu package repository, installation is very simple, using only a single command. After Vsftpd installed, we created an SSL certificate to be used with an FTP server. With this configuration, we will be using the SFTP protocol, which is more secure than FTP. You can find more details about SSL certificates in article 3, Working with Web Servers.

Under the Vsftpd configuration, we have modified some settings to disable anonymous logins, allowed local users to use FTP, enabled write access, and used chroot for local users. Next, we have set a path for previously generated SSL certificates and enabled the use of SSL. Additionally, you can force the use of TLS over SSL by adding the following lines to the configuration file:

ssl_tlsv1=yes

ssl_sslv2=no

ssl_sslv3=no

There's more…

This recipe covers FTP as a simple and easy-to-use tool for network storage. FTP is inherently insecure and you must avoid its use in a production environment. Server deployments can easily be automated with simple Git hooks or the sophisticated integration of continuous deployment tools such Chef, Puppet, or Ansible.

See also

Ubuntu server FTP guide at https://help.ubuntu.com/lts/serverguide/ftp-server.html