Resources

In this recipe, we will look at Samba configuration parameters in order to get optimum performance out of your Samba installation.

Getting ready

You will need root access or an account with sudo privileges.

It is assumed that you have installed the Samba server and it is properly working.

How to do it…

Open the Samba configuration file located at /etc/samba/smb.conf:

$ sudo vi /etc/samba/smb.conf

Add or edit the following options under the global section of the configuration file:

[global]

log level = 1

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 SO_KEEPALIVE

read raw = Yes

write raw = Yes

strict locking = No

oplocks = yes

max xmit = 65535

dead time = 15

getwd cache = yes

aio read size = 16384

aio write size = 16384

use sendfile = true

Save the configuration file and restart the Samba service:

$ sudo service smbd restart

How it works…

The Samba server provides various configuration parameters. It uses TCP sockets to connect with clients and for data transfer. You should compare Samba's performance with similar TCP services such as FTP.

The preceding example lists some commonly used configuration options for Samba. Some of these options may work for you and some of them may not. The latest Samba version ships with default values for these options that work fairly well for common network conditions. As always, test these options one at a time or in a group, and benchmark each modification to get optimum performance.

The explanation for the preceding is as follows:

log level: The default log level is set to 0. Samba produces a lot of debugging information and writing all this to disk is a slow operation. Increasing the log level results in increased logs and poor performance. Unless you are debugging the server, it is good to have the log level set to the lowest value.

socket options: These are the TCP/IP stack level options.

read raw and write raw: These options enable Samba to use large read and writes to a network up to 64 KB in a single request. Some older clients may have issues with raw reads and writes. Check your setup before using these options.

dead time and so_keepalive: These options set periodic checks for dead connections and close such connections and free unused memory.

oplocks: This allows clients to cache files locally and results in overall performance improvement. The default setting disables oplocks.

aio read size and aio write size: This Asynchronous IO (AIO) allows Samba to read and write asynchronously when a file's size is bigger than the specified size values.

You can find various other options and respective explanations in the Samba manual pages. Use the following command to open the manual pages on your server:

$ man smbd

In this recipe, we will learn how to import and export bulk data with MySQL. Many times it happens that we receive data in CSV or XML format and we need to add this data to the database server for further processing. You can always use tools such as MySQL workbench and phpMyAdmin, but MySQL provides command-line tools for the bulk processing of data that are more efficient and flexible.

How to do it…

Follow these steps to import and export bulk data:

To export a database from the MySQL server, use the following command:

$ mysqldump -u admin -p mytestdb > db_backup.sql

To export specific tables from a database, use the following command:

$ mysqldump -u admin -p mytestdb table1 table2 > table_backup.sql

To compress exported data, use gzip:

$ mysqldump -u admin -p mytestdb | gzip > db_backup.sql.gz

To export selective data to the CSV format, use the following query. Note that this will create articles.csv on the same server as MySQL and not your local server:

SELECT id, title, contents FROM articles

INTO OUTFILE ‘/tmp/articles.csv’

FIELDS TERMINATED BY ‘,’ ENCLOSED BY ‘”’

LINES TERMINATED BY ‘\n’;

To fetch data on your local system, you can use the MySQL client as follows:

Write your query in a file:

$ nano query.sql

select * from articles;

Now pass this query to the mysql client and collect the output in CSV:

$ mysql -h 192.168.2.100 -u admin -p myblog query.sql > output.csv

The resulting file will contain tab separated values.

To import an SQL file to a MySQL database, we need to first create a database:

$ mysqladmin -u admin -p create mytestdb2

Once the database is created, import data with the following command:

$ mysql -u admin -p mytestdb2 db_backup.sql

To import a CSV file in a MySQL table, you can use the Load Data query. The following is the sample CSV file:

Now use the following query from the MySQL console to import data from CSV:

LOAD DATA INFILE ‘c:/tmp/articles.csv’

INTO TABLE articles

FIELDS TERMINATED BY ‘,’ ENCLOSED BY ‘”’

LINES TERMINATED BY \n IGNORE 1 ROWS;

See also

MySQL select-into syntax at https://dev.mysql.com/doc/refman/5.6/en/select-into.html

MySQL load data infile syntax at https://dev.mysql.com/doc/refman/5.6/en/load-data.html

Importing from and exporting to XML files at https://dev.mysql.com/doc/refman/5.6/en/load-xml.html

Until now, in this article we have learned about various performance monitoring tools and commands. This recipe covers a well-known performance benchmarking tool: Sysbench. The purpose of performance benchmarking is to get a sense of system configuration and the resulting performance. Sysbench is generally used to evaluate the performance of heavy load systems. If you read the Sysbench introduction, it says that Sysbench is a benchmarking tool to evaluate a system running database under intensive load. It is also being used as a tool to evaluate the performance of multiple cloud service providers.

The current version of Sysbench supports various benchmark tests including CPU, memory, IO system, and OLTP systems. We will primarily focus on CPU, memory, and IO benchmarks.

Getting ready

Before using Sysbench, we will need to install it. Sysbench is available in the Ubuntu package repository with a little older (0.4.12) version. We will use the latest version (0.5) from Percona Systems, available in their repo.

To install Sysbench from the Percona repo, we need to add the repo to our installation sources. Following are the entries for Ubuntu 14.04 (trusty). Create a new file under /etc/apt/source.list.d and add the following lines to it:

$ sudo vi /etc/apt/sources.list.d/percona.list

deb http://repo.percona.com/apt trusty main

deb-src http://repo.percona.com/apt trusty main

Next, add the PGP key for the preceding repo:

$ sudo apt-key adv --keyserver keys.gnupg.net --recv-keys 1C4CBDCDCD2EFD2A

Now we are ready to install the latest version of Sysbench from the Percona repo. Remember to update the apt cache before installation:

$ sudo apt-get update

$ sudo apt-get install sysbench

Once installed, you can check the installed version with the --version flag to sysbench:

$ sysbench --version

sysbench 0.5

How to do it…

Now that we have Sysbench installed, let's start with performance testing our system:

Sysbench provides a prime number generation test for CPU. You can set the number of primes to be generated with the option --cpu-max-prime. Also set the limit on threads with the --num-threads option. Set the number of threads equal to the amount of CPU cores available:

$ sysbench --test=cpu --num-threads=4 \

--cpu-max-prime=20000 run

Next, we will run a test for main memory. The memory tests provides multiple options, such as block-size, total data transfer, type of memory operations, and access modes. Use the following command to run memory tests:

$ sysbench --test=memory --memory-block-size=1M \

--num-threads=2 \

--memory-total-size=100G --memory-oper=read run

Following is part of the output from the memory test:

If you have enabled huge page support, set the memory test support allocation from the huge page pool with the parameter, --memory-hugetlb. By default, it's set to off.

Next comes the storage performance test. This test also provides you with a number of options to test disk read write speeds. Depending on your requirements, you can set parameters like block-size, random or sequential read writes, synchronous or asynchronous IO operations, and many more.

For the fileio test we need a few sample files to test with. Use the sysbench prepare command to create test files. Make sure to set a total file size greater than the size of memory to avoid caching effects. I am using a small 1GBnode with 20G disk space, so I am using 15 files of 1G each:

$ sysbench --test=fileio --file-total-size=15G \

--file-num=15 prepare

Once the test preparation is complete, you can run the fileio test with different options, depending on what you want to test. The following command will perform random write operations for 60 seconds:

$ sysbench --test=fileio --file-total-size=15G \

--file-test-mode=rndwr --max-time=60 \

--file-block-size=4K --file-num=15 --num-threads=1 run

To perform random read operations, change --file-test-mode to rndrd, or to perform sequential read operations, use seqrd. You can also combine read write operations with rndrw or seqrewr. Check the help menu for more options.

When you are done with the fileio test, execute the cleanup command to delete all sample files:

$ sysbench --test=fileio cleanup

Once you have gathered various performance details, you can try updating various performance tuning parameters to boost performance. Make sure you repeat related tests after each change in parameter. Comparing results from multiple tests will help you to choose the required combination for best performance and a stable system.

There's more…

Sysbench also supports testing MySQL performance with various tests. In the same way as the fileio test, Sysbench takes care of setting a test environment by creating tables with data. When using Sysbench from the Percona repo, all OLTP test scripts are located at /usr/share/doc/sysbench/tests/db/. You will need to specify the full path when using these scripts. For example:

$ sysbench --test=oltp

The preceding command will change to the following:

$ sysbench --test=/usr/share/doc/sysbench/tests/db/ol1tp.lua

Graphing tools

Sysbench output can be hard to analyze and compare, especially with multiple runs. This is where graphs come in handy. You can try to set up your own graphing mechanism, or simply use prebuilt scripts to create graphs for you. A quick Google search gave me two good, looking options:

A Python script to extract data from Sysbench logs: https://github.com/tsuna/sysbench-tools

A shell script to extract Sysbench data to a CSV file, which can be converted to graphs: http://openlife.cc/blogs/2011/august/one-liner-condensing-sysbench-output-csv-file

More options

There are various other performance testing frameworks available. Phoronix Test Suite, Unixbench, and Perfkit by Google are some popular names. Phoronix Test Suite focuses on hardware performance and provides a wide range of performance analysis options, whereas Unixbench provides an option to test various Linux systems. Google open-sourced their performance toolkit with a benchmarker and explorer to evaluate various cloud systems.

See also

Get more details on benchmarking with Sysbench at https://wiki.mikejung.biz/Benchmarking

Sysbench documentation at http://imysql.com/wp-content/uploads/2014/10/sysbench-manual.pdf

A sample script to run batch run multiple Sysbench tests at https://gist.github.com/chetan/712484

Sysbench GitHub repo at https://github.com/akopytov/sysbench

Linux performance analysis in 60 seconds. A good read for what to check when you are debugging a performance issue at http://techblog.netflix.com/2015/11/linux-performance-analysis-in-60s.html

In this recipe, we will learn how to enable HTTPs communication on the Nginx server.

Getting ready

You will need access to a root account or an account with sudo privileges.

How to do it…

Follow these steps to set HTTPs on Nginx:

Obtain a certificate and the related keys from a certification authority or create a self-signed certificate. To create a self-signed certificate, refer to the Securing web traffic with HTTPS recipe in this article.

Create a directory to hold all certificate and keys:

$ sudo mkdir -p /etc/nginx/ssl/example.com

Move the certificate and keys to the preceding directory. Choose any secure method, such as SCP, SFTP, or any other.

Create a virtual host entry or edit it if you already have one:

$ sudo nano /etc/nginx/sites-available/example.com

Match your virtual host configuration with the following:

server {

listen 80;

server_name example.com www.example.com;

return 301 https://$host$request_uri;

}

server {

listen 443 ssl;

server_name example.com www.example.com;

root /var/www/example.com/public_html;

index index.php index.html index.htm;

ssl on;

ssl_certificate /etc/nginx/ssl/example.com/server.crt;

ssl_certificate_key /etc/nginx/ssl/example.com/server.key;

# if you have received ca-certs.pem from Certification Authority

#ssl_trusted_certificate /etc/nginx/ssl/example.com/ca- certs.pem;

ssl_session_cache shared:SSL:10m;

ssl_session_timeout 5m;

keepalive_timeout 70;

ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";

ssl_prefer_server_ciphers on;

ssl_protocols TLSv1.2 TLSv1.1 TLSv1;

add_header Strict-Transport-Security "max-age=31536000";

location / {

try_files $uri $uri/ /index.php;

}

location ~ \.php$ {

include fastcgi_params;

fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;

}

}

Enable this configuration by creating a symbolic link to it under sites-enabled:

$ sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com

Check the configuration for syntax errors:

$ sudo nginx -t

Reload Nginx for the changes to take effect:

$ sudo service nginx reload

Open your browser and access the site with domain or IP with HTTPS.

How it works…

When you know some basic configuration parameters, Nginx is quite simple to set up. Here, we have taken a few SSL settings from the default configuration file and added a simple redirection rule to redirect non-HTTPs traffic on port 80 to port 443. The first server block takes care of the redirection.

In addition to specifying the server certificate and keys, we have enabled session resumption by setting the cache to be shared across the Nginx process. We also have a timeout value of 5 minutes.

All other settings are common to the Nginx setup. We have allowed the virtual host to match with example.com, as well as www.example.com. We have set the index to search index.php, followed by index.html and others. With location directives, we have set Nginx to search for files and directories before forwarding the request to a PHP processor. Note that if you create a self-signed certificate, you will notice your browser complaining about invalid certification authority.

See also

Nginx HTTPs guide at http://nginx.org/en/docs/http/configuring_https_servers.html

Up to now in this article, we have worked with the Git command line interface (CLI). It is a very flexible and powerful interface. This recipe covers the installation of a web interface for Git repositories. We will install GitLab, an open source self-hosted Git server. Through GitLab, you can do most administrative tasks, such as creating new repositories, managing access rights, and monitoring history. You can easily browse your files or code and quickly make small edits. GitLab is also adding support for collaboration tools.

Getting ready

You will need access to a root account or an account with sudo privileges

Make sure you check out the minimum requirements for installation. You can use a single core 1 GB server for an installation with less than 100 users. An server with 2 cores and 2 GB RAM is recommended.

Also check the available disk space. The installer itself takes around 400 MB of disk space.

How to do it…

We will use the recommended Omnibus Package Installer. It provides a .deb package for Debian/Ubuntu systems. Additionally, the omnibus installation takes care of housekeeping tasks such as restarting the worker process to maintain memory use. If you choose to follow the manual installation process, you can get the detailed installation guide from the GitLab documentation:

First, we will need to download the installer package. Download the latest installer package from the GitLab download page at https://packages.gitlab.com/gitlab/gitlab-ce :

$ wget https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/xenial/gitl...

Once download completes, install GitLab using the dpkg command, as follows:

$ sudo dpkg -i gitlab-ce_8.7.1-ce.1_amd64.deb

After installation, use the following command to configure GitLab:

$ sudo gitlab-ctl reconfigure

Optionally, check the system status with the gitlab-ctl status command. It should return a list of processes and their respective PIDs, as follows:

ubuntu@ubuntu:~$ sudo gitlab-ctl status

[sudo] password for ubuntu:

run: gitlab-workhorse: (pid 806) 57803s; run: log: (pid 805) 57803s

run: logrotate: (pid 31438) 202s; run: log: (pid 810) 57803s

run: nginx: (pid 813) 57803s; run: log: (pid 812) 57803s

run: postgresql: (pid 817) 57803s; run: log: (pid 811) 57803s

Then, open your browser and point it to your server IP or hostname. You will be asked to set a new password for the administrator account. Once you set a new password, use root as the username and your password to login.

How it works…

GitLab is a Ruby-based web application that provides centralized hosting for your Git repositories. We have installed an open source community edition of GitLab using their Omnibus installer. It is an integrated installer package that combines all dependencies and default settings. The installer combines Nginx, Redis, Sidekiq, Unicorn, and PostgreSQL. Unfortunately, the community edition with the Omnibus installer does not support switching to the MySQL database server. To use MySQL, you need to follow the manual installation process and compile GitLab from source, along with other various dependencies.

The configuration file is located at /etc/gitlab/gitlab.rb. It is quite a lengthy file and contains numerous parameters, separated by each component. Some important settings to look at include external_url, where you can set your domain name, database settings, if you are planning to use external PostgreSQL setup, and email server settings, to set up your outgoing email server. If you choose to modify any settings, you will need to reconfigure the installation using the gitlab-ctl reconfigure command. You can get a list of enabled configurations using the gitlab-ctl show-config command.

The GitLab Omnibus package ships with some extra components: GitLab CI, a continuous integration service, and GitLab mattermost, an integrated installation of mattermost that provides an internal communication functionality with a chat server and file sharing. GitLab CI is enabled by default and can be accessed at http://ci.your-gitlab-domain.com. You can enable mattermost from the configuration file and then access it at http://mattermost.your-gitlab-domain.com.

There's more…

Git provides an inbuilt web interface to browse your repositories. All you need is a repository, web server, and the following command:

$ git instaweb --httpd apache2 # defaults to lighttpd

You can access the page at http://server-ip:1234

Check the GitWeb documentation for more details at https://git-scm.com/docs/gitweb .

See also

Check out the requirements for GitLab installation: https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/requirements.md .

Network Time Protocol (NTP) is a TCP/IP protocol for synchronizing time over a network. Although Ubuntu has a built-in clock that is helpful for keeping track of local events, it may create issues when the server is connected over a network and provides time-critical services to the clients. This problem can be solved with the help of NTP time synchronization. NTP works by synchronizing time across all servers on the Internet.

NTP uses hierarchies of servers with top-level servers synchronizing time with atomic clocks. This hierarchy levels are known as stratum, and the level can range between 1 and 15, both inclusive. The highest stratum level is 1 and is determined by the accuracy of the clock the server synchronizes with. If a server synchronizes with other NTP server with stratum level 3, then the stratum level for this server is automatically set to 4.

Another time synchronization tool provided by Ubuntu is ntpdate, which comes preinstalled with Ubuntu. It executes once at boot time and synchronizes the local time with Ubuntu's NTP servers. The problem with ntpdate is that it matches server time with central time without considering the big drifts in local time, whereas the NTP daemon ntpd continuously adjusts the server time to match it with the reference clock. As mentioned in the ntpdate manual pages (man ntpdate), you can use ntpdate multiple times throughout a day to keep time drifts low and get more accurate results, but it does not match the accuracy and reliability provided by ntpd.

In this recipe, we will set up a standalone time server for an internal network. Our time server will synchronize its time with public time servers and provide a time service to internal NTP clients.

How to do it…

Following are the steps to install and configure NTP daemon:

First, synchronize the server's time with any Internet time server using the ntpdate command:

$ ntpdate -s ntp.ubuntu.com

To install ntpd, enter the following command in the terminal:

$ sudo apt-get install ntp

Edit the /etc/ntp.conf NTP configuration file to add/remove external NTP servers:

$ sudo nano /etc/ntp.conf

Set a fallback NTP server:

server ntp.ubuntu.com

Block any external access to the server, comment the first restrict line, and add the following command:

restrict default noquery notrust nomodify

Allow the clients on local network to use the NTP service:

restrict 192.168.1.0 mask 255.255.255.0

Save changes with Ctrl + O and exit nano with Ctrl + X.

Reload the NTP daemon with the following command:

$ sudo service ntp restart

How it works…

Sometimes, the NTP daemon refuses to work if the time difference between local time and central time is too big. To avoid this problem, we have synchronized the local time and central time before installing ntpd. As ntpd and ntpdate both use the same UDP port, 123, the ntpdate command will not work when the ntpd service is in use.

After installing the NTP server, you may want to set time servers to be used. The default configuration file contains time servers provided by Ubuntu. You can use the same default servers or simply comment the lines by adding # at the start of each line and add the servers of your choice. You can dig into http://www.pool.ntp.org to find time servers for your specific region. It is a good idea to provide multiple reference servers, as NTP can provide more accurate results after querying each of them.

Additionally, we have set a fallback server that can be used in case of network outage or any other problems when our server cannot communicate with external reference servers. You can also use a system clock as a fallback, which can be accessed at 127.127.1.0. Simply replace the fallback server with the following line to use a system clock as a fallback:

server 127.127.0.1

Lastly, we have set access control parameters to protect our server from external access. The default configuration is to allow anyone to use the time service from this server. By changing the first restrict line, we blocked all external access to the server. The configuration already contains the exception to local NTP service indicated by the following:

restrict 127.0.0.1

We created another exception by adding a separate line to allow access to the clients on local network (remember to replace the IP range with your network details):

restrict 192.168.1.0 mask 255.255.255.0

There's more…

A central DHCP server can be configured to provide NTP settings to all DHCP clients. For this to work, your clients should also be configured to query NTP details from DHCP. A DHCP client configuration on Ubuntu already contains the query for network time servers.

Add the following line to your DHCP configuration to provide NTP details to the clients:

subnet 192.168.1.0 netmask 255.255.255.0 {

...

option ntp-servers your_ntp_host;

}

On the clientside, make sure that your dhclient.conf contains ntp-servers in its default request:

request subnet-mask, broadcast-address, time-offset, routers,

...

rfc3442-classless-static-routes, ntp-servers,

See also

Check the default /etc/ntp.conf configuration file. It contains a short explanation for each setting.

Check the manual pages for ntpd with man ntpd.

A streaming client needs to get the details of the media available on the streaming server. The client needs to authenticate with server access the catalog and list of songs and even request offline access to store media locally. With Ampache, we can use its REST and XML APIs. Through these APIs, clients can communicate with Ampache. You can even write your own client tool using any of the supported APIs.

This recipe covers the setup process for streaming content to remote devices. As of writing this, Ampache allows all users to use all available APIs. We will learn how to modify this setting and configure it to limit access based on user accounts.

Getting ready

Open Ampache in your browser and log in with admin credentials.

How to do it…

We will create a separate user account for remote streaming. From the Ampache homepage, click on the admin icon in the top-left corner and then click on the Add User link from the User Tools section. An add user menu will be shown that looks like this:

Fill in the Username, E-mail, and Password fields for the new user account and set User Access to User.

Click the Add User button to create this user and then click Continue.

We will use this new user account to log in from the remote client.

Next, we need to configure access rights and allow this user to use APIs to stream music.

Click on the admin icon and then click on the Add ACL link under the Access Control section.

Set the name for this access control list.

Set level to Read/Write.

Set the user to the user account created in the previous step.

Set ACL type to Stream Access.

Set the start and end IP addresses to 0.0.0.0 and 255.255.255.255 respectively.

Click Create ACL to save the settings.

Click on the Add ACL link again and repeat the preceding settings, except, for ACL Type that choose API/RPC.

Now you can use Ampache streaming from your mobile client. When asked for your username and password, use our new user account, and for the streaming server URL, use your Ampache FQDN followed by /ampache, for example:

http://myampachehost.com/ampache

If your client needs an API key, you can generate one from the User Tools section.

Click on the Browse Users link and then select the user account in question. Click the edit icon to update user details and then click on the generate API key icon.

Finally, click the Update User button to save your changes.

How it works…

By default, the Ampache server creates an Access Control List that allows all access to all users. It is a good idea to create a separate user and grant only the required permissions. Here, we have created a new user account with access to the REST API and to stream content. This will allow better control over users and content, as well as allow us to set various user-specific default settings, such as default bitrate and encoding formats.

While installing Ubuntu, we add a primary user account on the server; if you are using the cloud image, it comes preinstalled with the default user. This single user is enough to get all tasks done in Ubuntu. There are times when you need to create more restrictive user accounts. This recipe shows how to add a new user to the Ubuntu server.

Getting ready

You will need super user or root privileges to add a new user to the Ubuntu server.

How to do it…

Follow these steps to create the new user account:

1. To add a new user in Ubuntu, enter following command in your shell:

   $ sudo adduser bob

2. Enter your password to complete the command with sudo privileges:
3. Now enter a password for the new user:
4. Confirm the password for the new user:
5. Enter the full name and other information about the new user; you can skip this part by pressing the Enter key.
6. Enter Y to confirm that information is correct:.

This should have added new user to the system. You can confirm this by viewing the file /etc/passwd:

How it works…

In Linux systems, the adduser command is higher level command to quickly add a new user to the system. Since adduser requires root privileges, we need to use sudo along with the command, adduser completes following operations:

1. Adds a new user.
2. Adds a new default group with the same name as the user.
3. Chooses UID (user ID) and GID (group ID) conforming to the Debian policy.
4. Creates a home directory with skeletal configuration (template) from /etc/skel.
5. Creates a password for the new user.
6. Runs the user script, if any.

If you want to skip the password prompt and finger information while adding the new user, use the following command:

$ sudo adduser --disabled-password --gecos "" username

Alternatively, you can use the useradd command as follows:

$ sudo useradd -s -m -d -g UserName

Where:

• -s specifies default login shell for the user
• -d sets the home directory for the user
• -m creates a home directory if one does not already exist
• -g specifies the default group name for the user

Creating a user with the command useradd does not set password for the user account. You can set or change the user password with the following command:

$sudo passwd bob

This will change the password for the user account bob.

There's more…

With adduser, you can do five different tasks:

• Add a normal user
• Add a system user with system option
• Add user group with the--group option and without the--system option
• Add a system group when called with the --system option
• Add an existing user to existing group when called with two non-option arguments

Check out the manual page man adduser to get more details.

You can also configure various default settings for the adduser command. A configuration file /etc/adduser.conf can be used to set the default values to be used by the adduser, addgroup, and deluser commands. A key value pair of configuration can set various default values, including the home directory location, directory structure skel to be used, default groups for new users, and so on. Check the manual page for more details on adduser.conf with following command:

$ man adduser.conf

See also

• Check out the command useradd, a low level command to add new user to system
• Check out the command usermod, a command to modify a user account
• See why every user has his own group at http://unix.stackexchange.com/questions/153390/why-does-every-user-have-his-own-group

In this recipe, we will create our first container with LXD.

Getting ready

You will need access to the root account or an account with sudo privileges.

How to do it…

LXD works on the concept of remote servers and images served by those remote servers. Starting a new container with LXD is as simple as downloading a container image and starting a container out of it, all with a single command. Follow these steps:

To start your first container, use the lxc launch command, as follows:

$ lxc launch ubuntu:14.04/amd64 c1

LXC will download the required image (14.04/amd64) and start the container.

You should see the progress like this:

As you can see in the screenshot, lxc launch downloads the required image, creates a new container, and then starts it as well. You can see your new container in a list of containers with the lxc list command, as follows:

$ lxc list

Optionally, you can get more details about the containers with the lxc info command:

$ lxc info c1

Now that your container is running, you can start working with it. With the lxc exec command, you can execute commands inside a container. Use the following command to obtain the details of Ubuntu running inside a container:

$ lxc exec c1 -- lsb_release -a

You can also open a bash shell inside a container, as follows:

$ lxc exec c1 -- bash

How it works…

Creating images is a time-consuming task. With LXD, the team has solved this problem by downloading the prebuilt images from trusted remote servers. Unlike LXC, where images are built locally, LXD downloads them from the remote servers and keep a local cache of these images for later use. The default installation contains three remote servers:

Ubuntu: This contains all Ubuntu releases

Ubuntu-daily: This contains all Ubuntu daily builds

images: This contains all other Linux distributions

You can get a list of available remote servers with this command:

$ lxc remote list

Similarly, to get a list of available images on a specific remote server, use the following command:

$ lxc image list ubuntu:

In the previous example, we used 64-bit Ubuntu 14.04 from one of the preconfigured remote servers (ubuntu:). When we start a specific container, LXD checks the local cache for the availability of the respective image; if it's not available locally, the required images gets fetched from the remote server and cached locally for later use. These images are kept in sync with remote updates. They also expire if not used for a specific time period, and expired images are automatically removed by LXD. By default, the expiration period is set to 10 days.

The lxc launch command creates a new container and then starts it as well. If you want to just create a container without starting it, you can do that with the lxc init command, as follows:

$ lxc init ubuntu:xenial c2

All containers (or their rootfs) are stored under the /var/lib/lxd/containers directory, and images are stored under the /var/lib/lxd/images directory.

While starting a container, you can specify the set of configuration parameters using the --config flag. LXD also supports configuration profiles. Profiles are a set of configuration parameters that can be applied to a group of containers. Additionally, a container can have multiple profiles. LXD ships with two preconfigured profiles: default and docker.

To get a list of profiles, use the lxc profile list command, and to get the contents of a profile, use the lxc profile show
command.

Sometimes, you may need to start a container to experiment with something—execute a few random commands and then undo all the changes. LXD allows us to create such throwaway or ephemeral containers with the -e flag. By default, all LXD containers are permanent containers. You can start an ephemeral container using the --ephemeral or -e flag. When stopped, an ephemeral container will be deleted automatically.

With LXD, you can start and manage containers on remote servers as well. For this, the LXD daemon needs to be exposed to the network. This can be done at the time of initializing LXD or with the following commands:

$ lxc config set core.https_address "[::]"

$ lxc config set core.trust_password some-password

Next, make sure that you can access the remote server and add it as a remote for LXD with the lxc remote add command:

$ lxc remote add remote01 192.168.0.11 # lxc remote add name server_ip

Now, you can launch containers on the remote server, as follows:

$ lxc launch ubuntu:xenial remote01:c1

There's more…

Unlike LXC, LXD container images do not support password-based SSH logins. The container still has the SSH daemon running, but login is restricted to a public key. You need to add a key to the container before you can log in with SSH. LXD supports file management with the lxc file command; use it as follows to set your public key inside an Ubuntu container:

$ lxc file push ~/.ssh/id_rsa.pub \

c1/home/ubuntu/.ssh/authorized_keys \

--mode=0600 --uid=1000

Once the public key is set, you can use SSH to connect to the container, as follows:

$ ssh ubuntu@container_IP

Alternatively, you can directly open a root session inside a container and get a bash shell with lxc exec, as follows:

$ lxc exec c1 -- bash

See also

The LXD getting started guide: https://linuxcontainers.org/lxd/getting-started-cli/

The Ubuntu Server guide for LXC: https://help.ubuntu.com/lts/serverguide/lxd.html

Container images are created using tools such as debootstrap, which you can read more about at https://wiki.debian.org/Debootstrap

Creating LXC templates from scratch: http://wiki.pcprobleemloos.nl/using_lxc_linux_containers_on_debian_squeeze/creating_a_lxc_virtual_machine_template

In this recipe, we will learn how to use the Rsync utility to synchronize files between two directories or between two servers.

How to do it…

Follow these steps to synchronize files with Rsync:

Set up key-based authentication between source and destination servers. We can use password authentication as well, which is described later in this recipe.

Create a sample directory structure on the source server. You can use existing files as well:

ubuntu@src$ mkdir sampledir

ubuntu@src$ touch sampledir/file{1..10}

Now, use the following command to synchronize the entire directory from the source server to your local system. Note the / after sampledir. This will copy contents of sampledir in the backup. Without /, the entire sampledir will be copied to the backup:

ubuntu@dest$ rsync -azP -e ssh ubuntu@10.0.2.8:/home/ubuntu/sampledir/ backup

As this is the first time, all files from sampledir on the remote server will be downloaded in a backup directory on your local system. The output of the command should look like the following screenshot:

You can check the downloaded files with the ls command:

$ ls -l backup

Add one new file on the remote server under sampledir:

ubuntu@src$ touch sampledir/file22

Now re-execute the rsync command on the destination server. This time, rsync will only download a new file and any other update files. The output should look similar to the following screenshot:

ubuntu@dest$ rsync -azP -e ssh ubuntu@10.0.2.8:/home/ubuntu/sampledir backup

To synchronize two local directories, you can simply specify the source and destination path with rsync, as follows:

$ rsync /var/log/mysql ~/mysql_log_backup

How it works…

Rsync is a well known command line file synchronization utility. With Rsync, you can synchronize files between two local directories, as well as files between two servers. This tool is commonly used as a simple backup utility to copy or move files around systems. The advantage of using Rsync is that file synchronization happens incrementally, that is, only new and modified files will be downloaded. This saves bandwidth as well as time. You can quickly schedule a daily backup with a cron and Rsync. Open a cron jobs file with ctontab-e and add the following line to enable daily backups:

$ crontab -e # open crontab file

@daily rsync -aze ssh ubuntu@10.0.2.50:/home/ubuntu/sampledir /var/backup

In the preceding example, we have used a pull operation, where we are downloading files from the remote server. Rsync can be used to upload files as well. Use the following command to push files to the remote server:

$ rsync -azP -e ssh backup ubuntu@10.0.2.50:/home/ubuntu/sampledir

Rsync provides tons of command line options. Some options that are used in the preceding example are –a, a combination of various other flags and stands for achieve. This option enables recursive synchronization and preserves modification time, symbolic links, users, and group permissions. Option -z is used to enable compression while transferring files, while option -P enables progress reports and the resumption of interrupted downloads by saving partial files.

We have used one more option, -e, which specifies which remote shell to be used while downloading files. In the preceding command, we are using SSH with public key authentication. If you have not set public key authentication between two servers, you will be asked to enter a password for your account on the remote server. You can skip the -e flag and rsync will use a non-encrypted connection to transfer data and login credentials.

Note that the SSH connection is established on the default SSH port, port 22. If your remote SSH server runs on a port other than 22, then you can use a slightly modified version of the preceding command as follows:

rsync -azP -e "ssh -p port_number" source destination

Anther common option is --exclude, which specifies the pattern for file names to be excluded. If you need to specify multiple exclusion patterns, then you can specify all such patterns in a text file and include that file in command with the options --exclude-from=filename. Similarly, if you need to include some specific files only, you can specify the inclusion pattern with options --include=pattern or --include-from=filename.

Exclude a single file or files matching with a single pattern:

$ rsync -azP --exclude 'dir*' source/ destination/

Exclude a list of patterns or file names:

$ rsync -azP --exclude-from 'exclude-list.txt' source/ destination/

By default, Rsync does not delete destination files, even if they are deleted from the source location. You can override this behavior with a --delete flag. You can create a backup of these files before deleting them. Use the --backup and --backup-dir options to enable backups. To delete files from the source directory, you can use the --remove-source-files flag. Another handy option is --dry-run, which simulates a transfer with the given flags and displays the output, but does not modify any files. You should use --dry-run before using any deletion flags.

Use this to remove source files with --dry-run:

$ rsync --dry-run --remove-source-files -azP source/ destination/

There's more…

Rsync is a great tool to quickly synchronize the files between source and destination, but it does not provide bidirectional synchronization. It means the changes are synchronized from source to destination and not vice versa. If you need bi-directional synchronization, you can use another utility, Unison. You can install Unison on Debian systems with the following command:

$ sudo apt-get -y install unison

Once installed, Unison is very similar to Rsync and can be executed as follows:

$ unison /home/ubuntu/documents ssh://10.0.2.56//home/ubuntu/documents

You can get more information about Unison in the manual pages with the following command:

$ man unison

If you wish to have your own Dropbox-like mirroring tool which continuously monitors for local file changes and quickly replicates them to network storage, then you can use Lsyncd. Lsyncd is a live synchronization or mirroring tool, which monitors the local directory tree for any events (with inotify and fsevents), and then after few seconds spawns a synchronization process to mirror all changes to a remote location. By default, Lsyncd uses Rsync for synchronization.

As always, Lsyncd is available in the Ubuntu package repository and can be installed with a single command, as follows:

$ sudo apt-get install lsyncd

To get more information about Lsyncd, check the manual pages with the following command:

$ man lsyncd

See also

Ubuntu Rsync community page at https://help.ubuntu.com/community/rsync

In this recipe, we will learn how to create databases and tables and store data in those tables. We will learn the basic Structured Query Language (SQL) required for working with MySQL. We will focus on using the command-line MySQL client for this tutorial, but you can use the same queries with any client software or code.

Getting ready

Ensure that the MySQL server is installed and running. You will need administrative access to the MySQL server. Alternatively, you can use the root account of MySQL.

How to do it…

Follow these steps to store and retrieve data with MySQL:

First, we will need to connect to the MySQL server. Replace admin with a user account on the MySQL server. You can use root as well but it’s not recommended:

$ mysql -u admin -h localhost -p

When prompted, enter the password for the admin account. If the password is correct, you will see the following MySQL prompt:

Create a database with the following query. Note the semi-colon at the end of query:

mysql > create database myblog;

Check all databases with a show databases query. It should list myblog:

mysql > show databases;

Select a database to work with, in this case myblog:

mysql > use myblog;

Database changed

Now, after the database has changed, we need to create a table to store our data. Use the following query to create a table:

CREATE TABLE `articles` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`title` varchar(255) NOT NULL,

`content` text NOT NULL,

`created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,

PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1;

Again, you can check tables with the show tables query:

mysql > show tables;

Now, let’s insert some data in our table. Use the following query to create a new record:

mysql > INSERT INTO `articles` (`id`, `title`, `content`, `created_at`)

VALUES (NULL, ‘My first blog post’, ‘contents of article’, CURRENT_TIMESTAMP);

Retrieve data from the table. The following query will select all records from the articles table:

mysql > Select * from articles;

Retrieve the selected records from the table:

mysql > Select * from articles where id = 1;

Update the selected record:

mysql > update articles set title=”New title” where id=1;

Delete the record from the articles table using the following command:

mysql > delete from articles where id = 2;

How it works…

We have created a relational database to store blog data with one table. Actual blog databases will need additional tables for comments, authors, and various entities. The queries used to create databases and tables are known as Data Definition Language (DDL), and queries that are used to select, insert, and update the actual data are known as Data Manipulation Language (DML).

MySQL offers various data types to be used for columns such as tinyint, int, long, double, varchar, text, blob, and so on. Each data type has its specific use and a proper selection may help to improve the performance of your database.

Storage is one of the slowest components in a server's system, but is still the most important component. Storage is mainly used as a persistence mechanism to store a large amount of processed/unprocessed data. A slow storage device generally results in heavy utilization of read write buffers and higher memory consumption. You will see higher CPU usage, but most of the CPU time is spent waiting for I/O requests.

The recent developments of the flash storage medium have vastly improved storage performance. Still, it's one of the slowest performing components and needs proper planning— I/O planning in the application code, plus enough main memory for read write buffers.

In this recipe, we will learn to monitor storage performance. The main focus will be on local storage devices rather than network storage.

Getting ready

As always, you will need sudo access for some commands.

Some of the commands many not be available by default. Using them will prompt you if the command is not available, along with the process necessary to install the required package.

Install the sysstat package as follows. We have already used it in previous recipes:

$ sudo apt get install sysstat

How to do it…

The first command we will look at is vmstat. Using vmstat without any option displays an io column with two sub entries: bytes in (bi) and bytes out (bo). Bytes in represents the number of bytes read in per second from the disk and bytes out represents the bytes written to the disk:

Vmstat also provides two flags, -d and -D, to get disk statistics. Flag -d displays disk statistics and flag -D displays a summary view of disk activity:

There's one more option, -p, that displays partition-specific disk statistics. Use the command lsblk to get a list of available partitions and then use the vmstat -p partition:

Another command, dstat, is a nice replacement for vmstat, especially for disk statistics reporting. Use it with flag -d to get disk read writes per seconds. If you have multiple disks, you can use dstat to list their stats separately:

$ dstat -d -D total,sda

Next, we will look at the command iostat. When used without any options, this command displays basic CPU utilization, along with read write statistics for each storage device:

The column tps specifies the I/O requests sent to a device per second, and kb_read/s and kb_wrtn/s specifies per second blocks read and blocks written respectively. kb_read and kb_wrtn shows the total number of blocks read and written.

Some common options for iostat include –d, that displays disk only statistics, -g that displays statistics for a group of devices, flag -p to display partition specific stats, and -x to get extended statistics. Do not forget to check the manual entries for iostat to get more details.

You can also use the command iotop, which is very similar to the top command but it displays disk utilization and relevant processes.

The command lsof can display the list of all open files and respective processes using that file. Use lsof with the process name to get files opened by that process:

$ lsof -c sshd

To get a list of files opened by a specific PID, use the following command: $ lsof -p 1134. Or, to get a list of files opened by a specific user, use the $ lsof -u ubuntu command.

All these commands provide details on the read write performance of a storage device. Another important detail to know is the availability of free space. To get details of space utilization, you can use command df -h. This will list a partition-level summary of disk space utilization:

Finally, you can use the sar command to track disk performance over a period of time. To get real-time disk activity, use sar with the -d option, as follows:

$ sar -d 1

Use flag -F to get details on file system utilization and flag -S to display swap utilization. You can also enable sar logging and then extract details from those logs. Check the previous recipes in this article for how to enable sar logging. Also check manual entries for sar to get details of various options.

When an application becomes popular and the number of requests increases beyond the capacity of a single server, we need to scale horizontally. We can always increase the capacity (vertical scaling) of a server by adding more memory and processing power, but a single server cannot scale beyond a certain limit. While adding separate servers or replicas of the application server, we need a mechanism which directs the traffic between these replicas. The hardware or software tool used for this purpose is known as a load balancer. Load balancers work as transparent mechanisms between the application server and client by distributing the requests between available instances. This is a commonly used technique for optimizing resource utilization and ensuring fault tolerant applications.

Nginx can be configured to work as an efficient Layer 7 as well as Layer 4 load balancer. Layer 7 is application layer of HTTP traffic. With Layer 4 support, Nginx can be used to load balance database servers or even XMPP traffic. With version 1.9.0, Nginx has enabled support for Layer 4 load balancing in their open source offerings.

In this recipe, we will learn how to set up Nginx as a load balancer.

Getting ready

You will need access to a root account or an account with sudo privileges.

You will need a minimum of three servers, as follows:

An Nginx server, which will be set as a load balancer

Two or more application servers with a similar code base set up on all

How to do it…

Follow these steps to set load balancing with Nginx:

I assume that you already have Nginx installed. If not, you can refer to the Installing Nginx with PHP_FPM recipe of this article.

Now, create a new configuration file under /etc/nginx/sites-available. Let's call it load_balancer:

$ sudo nano /etc/nginx/sites-available/load_balancer

Add the following lines to this load_balancer file. This is the minimum configuration required to get started with load balancing:

upstream backend {

server srv1.example.com;

server srv2.example.com;

server 192.168.1.12:8080;

# other servers if any

}

server {

listen 80;

location / {

proxy_pass http://backend;

}

}

Enable this configuration by creating a symlink to load_balancer under sites-enabled:

$ sudo ln -s /etc/nginx/sites-available/load_balancer /etc/nginx/sites-enabled/load_balancer

You may want to disable all other sites. Simply remove the respective links under sites-enabled.

Check the configuration for syntax errors:

$ sudo nginx -t

Now, reload Nginx for the changes to take effect:

$ sudo service nginx reload

Yes, you are ready to use a load balancer. Open your favorite browser and point it to the IP of your Nginx server. You should see the contents of example.com or whatever domain you have used.

How it works…

We have created a very basic configuration for a load balancer. With this configuration, Nginx takes the traffic on port 80 and distributes it between srv1.example.com and srv2.example.com. With an upstream directive, we have defined a pool of servers that will actually process the requests. The upstream directive must be defined in a HTTP context. Once the upstream directive is defined, it will be available for all site configurations.

When defining servers under an upstream directive, you can also use the IP address and port of the application server. This is an ideal configuration, especially when both the load balancer and the application servers are on the same private network, and this will help minimize the communication overhead between Nginx and backend servers.

Next, under the server block, we have configured Nginx to proxy_pass all requests to our backend pool.

While setting backend servers, we have not explicitly specified any load balancing algorithm. Nginx provides various load balancing algorithms that define the server that will receive a particular request. By default, Nginx uses a round-robin algorithm and passes requests to each available server in sequential order. Other available options are as follows:

least_connection: This passes the request to the host with the fewest active connections.

least_time: Nginx chooses the host with the lowest latency. This option is available with Nginx plus.

ip_hash: A hash of clients' IP addresses, and is used to determined the host to send the request to. This method guarantees that requests with the same IP address are served by the same host, unless the selected host is down.

Hash uses a user defined key to generate a hash value and then uses the hash to determine the processing host.

There's more…

Nginx provides various other load balancing features, such as weighted load balancing, active and passive health checks, backup servers, and session persistence. With the latest commits to the open source version, it now supports TCP load balancing as well. These settings can be updated at runtime with the help of HTTP APIs. The following are a few examples of different load balancing configurations:

Set server weights:

upstream app-servers {

server srv1.example.com weight 3;

server srv2.example.com;

}

Health checkups and backup servers:

upstream app-servers {

server srv1.example.com max_fails 3 fail_timeout 10;

server srv2.example.com fail_timeout 50;

192.168.1.12:8080 backup;

}

Session persistence with cookies:

upstream app-servers {

server srv1.example.com;

server srv2.example.com;

sticky cookie srv_id expires=1h domain=.example.com path=/;

}

Check the Nginx load balancing guide for various other load balancing options and their respective details.

See also

Nginx admin guide for load balancers at https://www.nginx.com/resources/admin-guide/load-balancer

Git clone allows you to create a copy of your repository in a new directory or location. It can be used to replicate a remote repository on your local system or create a local clone to be shared over an intranet. This recipe covers the git clone command. We will learn to create a clone of a remote repository and then take a look at various transport protocols supported by Git for cloning.

Getting ready

You will need Git binaries installed on your local system, plus a remote repository. Note down the full path (clone URL) of the remote repository.

How to do it…

Create a clone of the repository with the git clone command, as follows:

$ git clone ssh://ubuntu@192.168.0.100:22/home/ubuntu/cookbook.git \ ubuntu_cookbook

You will be asked to enter a password for the user account ubuntu.

This command will create a new directory named ubuntu_cookbook and clone the repository cookbook.git into this directory.

How it works…

As seen in the previous example, the git clone command will create a new copy of an existing repository. The repository can be a local repository or one located on a remote server. Git supports various protocols to transfer the content between two systems. This includes well-known protocols such as SSH, HTTP/S, and rsync. In addition, Git provides a native transport protocol named Git. Note that the Git protocol does not require any authentication and should be used carefully. In the previous example, we have used the SSH protocol. When working with local repositories, you can use file///path/to/repo.git or even an absolute path /path/to/repo.git format.

Cloning requires a single argument, which is the path of the repository to be cloned. You can skip the destination directory and Git will create a clone in a new directory named after the repository name.

You can also create a new bare clone with the --bare option of the git clone command. This is useful for creating a shared central clone that is used by a group of people.

Another important option is the depth clone. When cloning a large repository that contains years of work, and you do not really need the entire history of the repository, the option --depth can be used to copy only a specified number of revisions. This will help you in quickly downloading just the tip of an entire repository, and will save you some bandwidth by avoiding unnecessary downloads. The syntax for the --depth option is as follows:

git clone --depth 1 https://github.com/torvalds/linux.git mylinux

See also

You can read more by following these links:

Git clone: https://git-scm.com/docs/git-clone

In this recipe, we will install and configure the squid proxy and caching server. The term proxy is generally combined with two different terms: one is forward proxy and the other is reverse proxy.

When we say proxy, it generally refers to forward proxy. A forward proxy acts as a gateway between a client's browser and the Internet, requesting the content on behalf of the client. This protects intranet clients by exposing the proxy as the only requester. A proxy can also be used as a filtering agent, imposing organizational policies. As all Internet requests go through the proxy server, the proxy can cache the response and return cached content when a similar request is found, thus saving bandwidth and time.

A reverse proxy is the exact opposite of a forward proxy. It protects internal servers from the outside world. A reverse proxy accepts requests from external clients and routes them to servers behind the proxy. External clients can see a single entity serving requests, but internally, it can be multiple servers working behind the proxy and sharing the load. More details about reverse proxies are covered in article 3, Working with Web Servers.

In this recipe, we will discuss how to install a squid server. Squid is a well-known application in the forward proxy world and works well as a caching proxy. It supports HTTP, HTTPS, FTP, and other popular network protocols.

Getting ready

As always, you will need access to a root account or an account with sudo privileges.

How to do it…

Following are the steps to setup and configure Squid proxy:

Squid is quite an old, mature, and commonly used piece of software. It is generally shipped as a default package with various Linux distributions. The Ubuntu package repository contains the necessary pre-compiled binaries, so the installation is as easy as two commands.

First, update the apt cache and then install squid as follows:

$ sudo apt-get update

$ sudo apt-get install squid3

Edit the /etc/squid3/squid.conf file:

$ sudo nano /etc/squid3/squid.conf

Ensure that the cache_dir directive is not commented out:

cache_dir ufs /var/spool/squid3 100 16 256

Optionally, change the http_port directive to your desired TCP port:

http_port 8080

Optionally, change the squid hostname:

visible_hostname proxy1

Save changes with Ctrl + O and exit with Ctrl + X.

Restart the squid server:

$ sudo service squid3 restart

Make sure that you have allowed the selected http_port on firewall.

Next, configure your browser using the squid server as the http/https proxy.

How it works…

Squid is available as a package in the Ubuntu repository, so you can directly install it with the apt-get install squid command. After installing squid, we need to edit the squid.conf file for some basic settings. The squid.conf file is quite a big file and you can find a large number of directives listed with their explanation. It is recommended to create a copy of the original configuration file as a reference before you do any modifications.

In our example, we are changing the port squid listens on. The default port is 3128. This is just a security precaution and it's fine if you want to run squid on the default port. Secondly, we have changed the hostname for squid.

Other important directive to look at is cache_dir. Make sure that this directive is enabled, and also set the cache size. The following example sets cache_dir to /var/spool/suid3 with the size set to 100MB:

cache_dir ufs /var/spool/squid3 100 16 256

To check the cache utilization, use the following command:

$ sudo du /var/spool/squid3

There's more…

Squid provides lot more features than a simple proxy server. Following is a quick list of some important features:

Access control list

With squid ACLs, you can set the list of IP addresses allowed to use squid. Add the following line at the bottom of the acl section of /etc/squid3/squid.conf:

acl developers src 192.168.2.0/24

Then, add the following line at the top of the http_access section in the same file:

http_access allow developers

Set cache refresh rules

You can change squid's caching behavior depending on the file types. Add the following line to cache all image files to be cached—the minimum time is an hour and the maximum is a day:

refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 3600 90% 86400

This line uses a regular expression to find the file names that end with any of the listed file extensions (gif, png, and etc)

Sarg – tool to analyze squid logs

Squid Analysis Report Generator is an open source tool to monitor the squid server usages. It parses the logs generated by Squid and converts them to easy-to-digest HTML-based reports. You can track various metrics such as bandwidth used per user, top sites, downloads, and so on. Sarg can be quickly installed with the following command:

$ sudo apt-get install sarg

The configuration file for Sarg is located at /etc/squid/sarg.conf. Once installed, set the output_dir path and run sarg. You can also set cron jobs to execute sarg periodically. The generated reports are stored in output_dir and can be accessed with the help of a web server.

Squid guard

Squid guard is another useful plugin for squid server. It is generally used to block a list of websites so that these sites are inaccessible from the internal network. As always, it can also be installed with a single command, as follows:

$ sudo apt-get install squidguard

The configuration file is located at /etc/squid/squidGuard.conf.

See also

Check out the squid manual pages with the man squid command

Check out the Ubuntu community page for squid guard at https://help.ubuntu.com/community/SquidGuard

Transcoding means converting media from one format to another. Suppose your music files are in a format different to MP3 and your media player only understands MP3 format. In that case, you need to convert your music files to MP3. This conversion task is done by transcoder programs. There are various transcoding programs available, such as ffmpeg and avconv. These programs need codec before they can convert media from source format to destination format. We need to separately install and configure these components.

Ampache supports on-the-fly transcoding of media files. That is, your music that is not in an MP3 format can be converted into the MP3 format just before it is delivered to your music player, and your high definition video content can be optimized for mobile consumption to reduce bandwidth use.

In this recipe, we will learn how to install and configure transcoding programs with Ampache.

Getting ready

Make sure you have working a setup of the Ampache server.

You will need access to a root account or an account with root privileges.

How to do it…

Ampache depends on external libraries for transcoding to work. We will first install the dependencies and then configure Ampache to work with them:

First, add the ffmpeg PPA to the Ubuntu installation sources:

$ sudo apt-add-repository ppa:mc3man/trusty-media

$ sudo apt-get update

Now, install ffmpeg and other required codecs:

$ sudo apt-get install flac mp3splt lame faad ffmpeg vorbis- tools

Next, we need to configure Ampache and enable transcoding. Open the configuration file located at /var/www/ampache/config/ampache.cfg.php, find the following lines in the file, and uncomment them:

max_bit_rate = 576

min_bit_rate = 48

transcode_flac = required

transcode_mp3 = allowed

encode_target = mp3

transcode_cmd = "ffmpeg"

Here, we have set ffmpeg for the encoding/decoding of media files. You can choose any encoder of your choice. Change the value of transcode_cmd respectively.

Next, enable debug mode to get details of the transcoding. Find the debug section in the configuration file and set it as follows:

debug = true

Enable log file path which is, by default, set to null

log_path = "/var/log/ampache"

Save the changes to the configuration file and reload the Apache web server:

$ sudo service apache2 reload

Now your transcoding setup should be working. You should be able to upload media in a different format and play it as MP3 or other respective formats.

It often happens that we have content in a format that is not supported by the device we are using for playback. Maybe the device does not have the required codec or the hardware is not capable of playing a high bit rate. We may even need to convert content to a lower bit rate and reduce the bandwidth used to stream. The transcoding feature of Ampache helps us to cover these scenarios.

With transcoding, you can convert the content to the desired device-supported format before actually starting the streaming. This is called on-the-fly transcoding. The contents are encoded in a new format as and when needed. Once the conversion is completed, the new format can be cached for repeat use. In the above example, we set Ampache to convert FLAC files to MP3 using the ffmpeg tool. Now whenever we request a file that is originally available in the FLAC format, Ampache will convert it to MP3 before streaming it to our device.

Ampache uses external, well-established media conversion tools for transcoding. Any tool that works with the Ubuntu command line can be configured to work with Ampache. Refer to the Ampache configuration file to get more details and configuration examples.

In this article, you will see how to add new users to the Ubuntu server, update existing users, and set permissions for users. You will get to know the default setting for new users and how to change them. Also, you will take a look at secure shell (SSH) access and securing user profiles.

LXC is a system built on the modern Linux kernel and enables the creation and management of virtual Linux systems or containers. As discussed earlier, LXC is not a full virtualization system and shares the kernel with the host operating system, providing lightweight containerization. LXC uses Linux namespaces to separate and isolate the processes running inside containers. This provides much better security than simple chroot-based filesystem isolation. These containers are portable and can easily be moved to another system with a similar processor architecture.

Ubuntu 15.04 unveiled a new tool named LXD, which is a wrapper around LXC. The official page calls it a container hypervisor and a new user experience for LXC. Ubuntu 16.04 comes preinstalled with its latest stable release, LXD 2.0. With LXD, you no longer need to work directly with lower-level LXC tools.

LXD adds some important features to LXC containers. First, it runs unprivileged containers by default, resulting in improved security and better isolation for containers. Second, LXD can manage multiple LXC hosts and can be used as an orchestration tool. It also supports the live migration of containers across hosts.

LXD provides a central daemon named LXD and a command-line client named lxc. Containers can be managed with the command-line client or the REST APIs provided by the LXD daemon. It also provides an OpenStack plugin, nova-compute-lxd, to deploy containers on the OpenStack cloud.

In this recipe, we will learn to install and configure the LXD daemon. This will set up a base for the next few recipes in this article.

Getting ready

You will need access to the root account or an account with sudo privileges.

Make sure that you have enough free space available on disk.

How to do it…

Ubuntu 16.04 ships with the latest release of LXD preinstalled. We just need to initialize the LXD daemon to set the basic settings.

First, update the apt cache and try to install LXD. This should install updates to the LXD package, if any:

$ sudo apt-get update

$ sudo apt-get install lxd

Along with LXD, we will need one more package named ZFS—the most important addition to Ubuntu 16.04. We will be using ZFS as a storage backend for LXD:

$ sudo apt-get install zfsutils-linux

Once LXD has been installed, we need to configure the daemon before we start using it. Use lxd init to start the initialization process. This will ask some questions about the LXD configuration:

$ sudo lxd init

Name of the storage backend to use (dir or zfs): zfs

Create a new ZFS pool (yes/no)? yes

Name of the new ZFS pool: lxdpool

Would you like to use an existing block device (yes/no)? no

Size in GB of the new loop device (1GB minimum): 10

Would you like LXD to be available over the network (yes/no)? no

Do you want to configure the LXD bridge (yes/no)? yes

Warning: Stopping lxd.service, but it can still be activated by: lxd.socket

LXD has been successfully configured.

Now, we have our LXD setup configured and ready to use. In the next recipe, we will start our first container with LXD.

How it works…

Ubuntu 16.04 comes preinstalled with LXD and makes it even easier to start with system containers or operating system virtualization. In addition to LXD, Ubuntu now ships with inbuilt support for ZFS (OpenZFS), a filesystem with support for various features that improve the containerization experience. With ZFS, you get faster clones and snapshots with copy-on-write, data compression, disk quotas, and automated filesystem repairs.

LXD is a wrapper around lower-level LXC or Linux containers. It provides the REST API for communicating and managing LXC components. LXD runs as a central daemon and adds some important features, such as dynamic resource restrictions and live migrations between multiple hosts. Containers started with LXD are unprivileged containers by default, resulting in improved security and isolation.

This recipe covers the installation and initial configuration of the LXD daemon. As mentioned previously, LXD comes preinstalled with Ubuntu 16. The installation commands should fetch updates to LXD, if any. We have also installed zfsutils-linux, a user space package to interact with ZFS. After the installation, we initialized the LXD daemon to set basic configuration parameters, such as the default storage backend and network bridge for our containers.

We selected ZFS as the default storage backend and created a new ZFS pool called lxdpool, backed by a simple loopback device. In a production environment, you should opt for a physical device or separate partition. If you have already created a ZFS pool, you can directly name it by choosing no for Create new ZFS pool. To use a separate storage device or partition, choose yes when asked for block storage.

ZFS is the recommended storage backend, but LXD also works with various other options, such as Logical Volume Manager (LVM) and btrfs (pronounced "butter F S"), that offer nearly the same features as ZFS or a simple directory-based storage system.

Next, you can choose to make LXD available on the network. This is necessary if you are planning a multi-host setup and support for migration. The initialization also offers to configure the lxdbr0 bridge interface, which will be used by all containers. By default, this bridge is configured with IPv6 only. Containers created with the default configuration will have their veth0 virtual Ethernet adapter attached to lxdbr0 through a NAT network. This is the gateway for containers to communicate with the outside world. LXD also installs a local DHCP server and the dnsmasq package. DHCP is used to assign IP addresses to containers, and dnsmasq acts as a local name-resolution service.

If you misplace the network bridge configuration or need to update it, you can use the following command to get to the network configuration screen:

$ sudo dpkg-reconfigure -p medium lxd

There's more…

The LXD 2.0 version, which ships with Ubuntu 16, is an LTS version. If you want to get your hands on the latest release, then you can install stable versions from the following repository:

$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable

For development releases, change the PPA to ppa:ubuntu-lxc/lxd-git-master.

For more information, visit the LXC download page at https://linuxcontainers.org/lxc/downloads/ .

If you still want to install LXC, you can. Use the following command:

$ sudo apt-get install lxc

This will install the required user space package and all the commands necessary to work directly with LXC. Note that all LXC commands are prefixed with lxc-, for example, lxc-create and lxc-info. To get a list of all commands, type lxc- in your terminal and press Tab twice.

See also

For more information, check the LXD page of the Ubuntu Server guide: https://help.ubuntu.com/lts/serverguide/lxd.html

The LXC blog post series is at https://www.stgraber.org/2013/12/20/lxc-1-0-blog-post-series/

The LXD 2.0 blog post series is at https://www.stgraber.org/2016/03/11/lxd-2-0-blog-post-series-012/

Ubuntu 16.04 switched to Systemd, which provides its own container framework, systemd-nspawn; read more about systemd containers on its Ubuntu man page at http://manpages.ubuntu.com/manpages/xenial/man1/systemd-nspawn.1.html

See how to get started with systemd containers at https://community.flockport.com/topic/32/systemd-nspawn-containers

In this recipe, we will learn how to install the File Transfer Protocol (FTP) server and configure it to use SSL encryption.

Getting ready

You will need access to a root account or an account with sudo privileges.

How to do it…

Follow these steps to install the secure FTP server:

Install vsftpd with the following command:

$ sudo apt-get update

$ sudo apt-get install vsftpd

After installation, we can configure vsftpd by editing /etc/vsftpd.conf.

First create the SSL certificate for the FTP server:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 - keyout /etc/ssl/private/vsftpd.pem -out /etc/ssl/private/vsftpd.pem

Next, configure Vsftpd. Add or edit the following lines in vsftpd.conf:

anonymous_enable=no

local_enable=yes

write_enable=yes

chroot_local_user=yes

Add the SSL certificate created in the previous step:

rsa_cert_file=/etc/ssl/private/vsftpd.pem

rsa_private_key_file=/etc/ssl/private/vsftpd.pem

ssl_enable=yes

ssl_ciphers=high

force_local_data_ssl=yes

force_local_logins_ssl=yes

Save and exit the configuration file.

Restart the Vsftpd server:

$ sudo service vsftpd restart

Now you can use any FTP client that supports the SFTP protocol to connect to your FTP server. The following is the configuration screen for SFTP client FileZilla:

How it works…

FTP is an insecure protocol and you should avoid using it, especially in a production environment. Limit use of FTP to downloads only and use more secure methods, such as SCP, to upload and transfer files on servers. If you have to use FTP, make sure that you have disabled anonymous access and enable SFTP to secure your data and login credentials.

In this recipe, we have installed Vsftpd, which is a default FTP package in the Ubuntu repository. Vsftpd stands for very secure FTP daemon, and it is designed to protect against possible FTP vulnerabilities. It supports both FTP and SFTP protocols.

As Vsftpd is available in the Ubuntu package repository, installation is very simple, using only a single command. After Vsftpd installed, we created an SSL certificate to be used with an FTP server. With this configuration, we will be using the SFTP protocol, which is more secure than FTP. You can find more details about SSL certificates in article 3, Working with Web Servers.

Under the Vsftpd configuration, we have modified some settings to disable anonymous logins, allowed local users to use FTP, enabled write access, and used chroot for local users. Next, we have set a path for previously generated SSL certificates and enabled the use of SSL. Additionally, you can force the use of TLS over SSL by adding the following lines to the configuration file:

ssl_tlsv1=yes

ssl_sslv2=no

ssl_sslv3=no

There's more…

This recipe covers FTP as a simple and easy-to-use tool for network storage. FTP is inherently insecure and you must avoid its use in a production environment. Server deployments can easily be automated with simple Git hooks or the sophisticated integration of continuous deployment tools such Chef, Puppet, or Ansible.

See also

Ubuntu server FTP guide at https://help.ubuntu.com/lts/serverguide/ftp-server.html

In this recipe, we will learn how to install and configure the MySQL database on an Ubuntu server.

Getting ready

You will need access to a root account or an account with sudo privileges.

Make sure that the MySQL default port 3306 is available and not blocked by any firewall.

How to do it…

Follow these steps to install the relational database MySQL:

To install the MySQL server, use the following command:

$ sudo apt-get update

$ sudo apt-get install mysql-server-5.7

The installation process will download the necessary packages and then prompt you to enter a password for the MySQL root account. Choose a strong password:

Once the installation process is complete, you can check the server status with the following command. It should return an output similar to the following:

$ sudo service mysql status

mysql.service - MySQL Community Server

Loaded: loaded (/lib/systemd/system/mysql.service

Active: active (running) since Tue 2016-05-10 05:

Next, create a copy of the original configuration file:

$ cd /etc/mysql/mysql.conf.d

$ sudo cp mysqld.cnf mysqld.cnf.bkp

Set MySQL to listen for a connection from network hosts. Open the configuration file /etc/mysql/mysql.conf.d/mysqld.cnf and change bind-address under the [mysqld] section to your server’s IP address:

$ sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

bind-address = 10.0.2.6

Optionally, you can change the default port used by the MySQL server. Find the [mysqld] section in the configuration file and change the value of the port variable as follows:

port = 30356

Make sure that the selected port is available and open under firewall.

Save the changes to the configuration file and restart the MySQL server:

$ sudo service mysql restart

Now open a connection to the server using the MySQL client. Enter the password when prompted:

$ mysql -u root -p

To get a list of available commands, type \h:

mysql> \h

How it works…

MySQL is a default database server available in Ubuntu. If you are installing the Ubuntu server, you can choose MySQL to be installed by default as part of the LAMP stack. In this recipe, we have installed the latest production release of MySQL (5.7) from the Ubuntu package repository. Ubuntu 16.04 contains MySQL 5.7, whereas Ubuntu 14.04 defaults to MySQL version 5.5.

If you prefer to use an older version on Ubuntu 16, then use following command:

$ sudo add-apt-repository ‘deb http://archive.ubuntu.com/ubuntu trusty universe’

$ sudo apt-get update

$ sudo apt-get install mysql-server-5.6

After installation, configure the MySQL server to listen for connections from external hosts. Make sure that you open your database installation to trusted networks such as your private network. Making it available on the Internet will open your database to attackers.

There’s more…

Securing MySQL installation

MySQL provides a simple script to configure basic settings related to security. Execute this script before using your server in production:

$ mysql_secure_installation

This command will start a basic security check, starting with changing the root password. If you have not set a strong password for the root account, you can do it now. Other settings include disabling remote access to the root account and removing anonymous users and unused databases.

MySQL is popularly used with PHP. You can easily install PHP drivers for MySQL with the following command:

$ sudo apt-get install php7.0-mysql

See also

The Ubuntu server guide mysql page at https://help.ubuntu.com/14.04/serverguide/mysql.html