Ubuntu

Memory is another important component of system performance. All files and data that are currently being used are kept in the system main memory for faster access. The CPU performance also depends on the availability of enough memory. Swap, on the other hand, is an extension to main memory. Swap is part of persistent storage, such as hard drives or solid state drives. It is utilized only when the system is low on main memory.

In this article, we will learn how to monitor system memory and swap utilization.

Getting ready

You may need sudo privileges for some commands.

How to do it…

In the last recipe, we used commands top and vmstat to monitor CPU utilization. These commands also provided details of memory usage. Let's start with the top command again:

Run the top command and check for the Mem and Swap rows:

The memory line displays the size of total available memory, size of used memory, free memory, and the memory used for buffers and the file system cache. Similarly, swap row should display the allocated size of the swap if you have enabled the swapping. Along with these two lines, top shows per process memory utilization as well. The columns VIRT, RES, SHR, and %MEM all show different memory allocation for each process:

Similar to the top command, you can query memory statistics for a specific PID or program by using the pidstat command. By default, pidstat displays only CPU statistics for a given process. Use flag -r to query memory utilization and page faults:

$ pidstat -C mysql -r

Next, we will go through the vmstat command. This is an abbreviation of virtual memory statistics. Enter the command vmstat in your console and you should see output similar to the following screenshot:

Using vmstat without any option returns a single line report of memory, swap, io, and CPU utilization. Under the memory column, it shows the amount of swap, free memory, and the memory used for cache and buffers. It also display a separate swap column with Swap In (si) and Swap Out (so) details.

To get detailed statistics of memory and event counters, use flag -s. This should display a table, as follows:

$ vmstat -s

Another handy command is free, which displays the amount of used and available memory in the system. Use it as follows, with the -h flag to get human-friendly units:

$ free -h

Finally, command sar can give you periodic reports of memory utilization. Simply enable sar to collect all reports and then extract memory reports from it or set a specific command to log only memory and swap details.

Finally, use sar to monitor current memory and swap utilizations. The following command will query the current memory (-r) and swap (-S) utilization:

$ sar -rS 1 5

For more details on using sar, check Monitoring the CPU recipe or read the manual pages using the man sar command. The command sar is available in the package sysstat; you will need to install it separately if not already installed.

All these tools show process-level memory statistics. If you are interested in memory allocation inside a particular process, then the command pmap can help you. It reports the memory mapping of a process, including details of any shared libraries in use and any program extensions with their respective memory consumptions. Use pmap along with the PID you want to monitor as follows:

$ sudo pmap -x 1322

How it works…

System memory is the primary storage for processes in execution. It is the fastest available storage medium, but is volatile and limited in storage space. The limited storage is generally extended with the help of slower, disk-based Swap files. Processes that are not being actively executed are swapped to disk so that active processes get more space in the faster main memory. Similar to other operating systems, Ubuntu provides various tools to monitor system-wide memory utilization as well as memory uses by process. Commonly used tools include top, vmstat, and free.

We have used the top command to monitor CPU uses and know that top provides a summarized view of system resource utilization. Along with a CPU summary, top also provides the memory statistics. This includes overall memory utilization plus per process usage. The summary section in the top output displays the total available and used memory. It also contains a separate row for swap. By default, all Ubuntu systems enable the swap partition with nearly the same size as main memory. Some cloud service providers disable the cache for performance reasons.

The details section of top shows per process memory usage separated into multiple columns:

Column VIRT shows the virtual memory assigned to a task or process; this includes memory assigned for program code, data, and shared libraries, plus memory that is assigned but not used.

Column RES shows the non-swapped physical memory used by processes. Whereas column SHR shows the amount of shared memory, this is the memory that can be shared with other processes through shared libraries.

The column %MEM shows the percentage of main memory assigned to a specific process. This is a percentage of RES memory available to task out of total available memory.

By default, all memory values are shown in the lowest units, KB. This can be changed using the key combination, Shift + E for summary rows and E for process columns.

Similar to top, the command ps lists running processes but without refreshing the list. Without any options, ps shows the list of processes owned by the current user. Use it as follows to get a list of all running processes:

$ ps aux

The command vmstat gives you overall detail regarding memory and swap utilization. The memory column shows the amount of available memory. Next to the memory column, the swap column indicates the amount of memory read from disk (si) or written to disk (so) per second. Any activity in the si and so columns indicates active swap utilization. In that case, you should either increase the physical memory of the system or reduce the number of processes running. Large numbers under the swap column may also indicate higher CPU utilization, where the CPU waits for IO operations (wa) to complete. As seen before, you can specify the delay and interval options to repeatedly query vmstat reports.

One more command, named free, shows the current state of system memory. This shows overall memory utilization in the first row and swap utilization in the second row. You may get confused by looking at the lower values in the free column and assume higher memory uses. Part of free memory is being used by Linux to improve file system performance by caching frequently used files. The memory used for file caching is reflected in the buff/cache column and is available to other programs when required. Check the last column, named available, for the actual free memory.

The second row of free output displays the swap utilization. You may see swap being used under the used column. This is the amount of swap allocated but not effectively used. To check if your system is effectively swapping, use the command vmstat 1 and monitor si/so columns for any swap activity.

System swapping behavior also depends on the value of the kernel parameter named vm.swappiness. Its value can range between 0 to 100, where 0 configures the kernel to avoid swapping as much as possible and 100 sets it to swap aggressively. You can read the current swappiness value using the following command:

$ sudo sysctl vm.swappiness

vm.swappiness = 60

To modify the swappiness value for the current session, use the sysctl command with a new value, as follows. It is a good idea to use lower values and avoid swapping as much as possible:

$ sudo sysctl vm.swappiness=10

vm.swappiness = 10

To permanently set swappiness, you need to edit the /etc/sysctl.conf file and add or uncomment vm.swappiness=10 to it. Once the file is updated, use the following command to read and set a new value from the configuration file:

$ sudo sysctl -p

Check the swapon and swapoff commands if you need to enable swapping or disable it.

There's more…

Most of these statistics are read from the /proc partition. The two main files listing details of memory and swap are /proc/meminfo and /proc/swaps.

The command lshw (list hardware) can give you the details of actual hardware. This includes the physical memory configuration, the firmware version, CPU details, such as clock speed, the cache, and various other hardware information. Use lshw as follows:

$ sudo lshw

See also

Check the swapon and swapoff commands to enable or disable swap files:

$ man swapon

$ man swapoff

In this recipe, we will learn how to install and set up Nginx as a web server. We will also install PHP to be able to serve dynamic content. We need to install PHP_FPM (FastCGI Process Manager), as Nginx doesn't support the native execution of PHP scripts. We will install the latest stable version available from the Nginx package repository.

Getting ready

You will need access to a root account or an account with sudo privileges.

How to do it…

Follow these steps to install Nginx with PHP_FPM:

Update the apt package repository and install Nginx. As of writing this Ubuntu 16.04 repository contains latest stable release of Nginx with version 1.10.0:

$ sudo apt-get update

$ sudo apt-get install nginx

Check if Nginx is properly installed and running:

$ sudo service nginx status

Check the installed version of Nginx:

$ nginx -v

You may want to point your browser to the server IP or domain. You should see a default Nginx welcome page:

Next, proceed with installing PHP_FPM:

$ sudo apt-get install php7.0-fpm

Configure Nginx to use the PHP processor. Nginx sites are listed at /etc/nginx/sites-available. We will modify the default site:

$ sudo nano /etc/nginx/sites-available/default

Find a line stating the priority of the index file and add index.php as a first option:

index index.php index.html index.htm;

Next, add the following two location directives:

location / {

try_files $uri $uri/ /index.php;

}

location ~ \.php$ {

include fastcgi_params;

fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

fastcgi_param QUERY_STRING $query_string;

fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;

}

Save the changes and exit the file. It should look similar to this:

Change the PHP settings to disable PATH_TRANSLATED support. Find an option, cgi.fix_pathinfo, and uncomment it with the value set to 0:

$ sudo nano /etc/php/7.0/fpm/php.ini

cgi.fix_pathinfo=0

Now, restart PHP_FPM and Nginx for the changes to take effect:

$ sudo service php7.0-fpm restart

$ sudo service nginx restart

Create an index.php file with some PHP code in it at the path mentioned in the default site configuration:

$ sudo nano /var/www/html/index.php

Open your browser and point it to your server. You should see the result of your PHP script:

How it works…

Here, we have installed the latest stable version of the Nginx server with PHP_FPM to support dynamic content scripted with PHP. The Ubuntu repository for version 16.04 contains the latest stable release of Nginx, So installing Nginx is as easy as a single command. If you are interested in more recent versions Nginx maintains their own package repository for mainline packages. You just need to add repository, the rest of the installation process is similar to a single apt-get install nginx command.

After installing Nginx, we need to configure it to support dynamic content. Here, we have selected PHP as a dynamic content processor. PHP is a popular scripting language and very commonly used with web servers for dynamic content processing. You can also add support for other modules by installing their respective processors. After installing PHP_FPM, we have configured Nginx to use PHP_FPM and pass all PHP requests to the FPM module on a socket connection.

We have used two location blocks in configuration. The first block search is for static content, such as files and directories, and then if nothing matches, the request is forwarded to index.php, which is in turn forwarded to the FastCGI module for processing. This ensures that Nginx serves all static content without executing PHP, and only requests that are not static files and directories are passed to the FPM module.

The following is a brief description of the parameters used under FastCGI configuration:

The parameter try_files configures Nginx to return 404 pages, that is, the page not found error, for any requests that do not match website content. This is limited to static files.

With the parameter fastcgi_param, you can forward the script name and query string to the PHP FPM process.

One more optional parameter is cgi.fix_pathinfo=0, under the PHP configuration file php.ini. By default, PHP is set to search for the exact script filename and then search for the closest match if the exact name is not found. This may become a security risk by allowing an attacker to execute random scripts with simple guesswork for script names. We have disabled this by setting its value to 0.

Finally, after we restart PHP_FPM and Nginx, our server is ready to process static as well as dynamic content. All static content will be handled by Nginx itself, and requests for URLs that end with .php will be forwarded to PHP_FPM for processing. Nginx may cache the processed result for future use.

There's more…

If you are running Ubuntu 12.10, you may need to install the following dependencies before adding the Nginx repository to the installation sources:

Install python-software-properties and software-properties-common:

$ sudo apt-get install python-software-properties

$ sudo apt-get install software-properties-common

You may want to remove your Apache installation completely. Use the following commands to remove Apache:

$ sudo service apache2 stop

$ sudo apt-get remove --purge apache2 apache2-utils apache2.2- bin apache2-common

Nginx maintains their own package repositories for stable and mainline releases. These repositories can be used to get the latest updates of Nginx as and when available. Use the stable repository, - $ sudo add-apt-repository ppa:nginx/stable.

Use the mainline repository - $ sudo add-apt-repository ppa:nginx/development.

See also

Common Nginx pitfalls at http://wiki.nginx.org/Pitfalls

Nginx Quick start guide at http://wiki.nginx.org/QuickStart

Up to now, we have learned how to create a local Git repository and add or update files to it. In this recipe, we will learn how to set up a remote repo and synchronize local code with it. We will be using GitHub to host our remote repository; feel free to choose any other code hosting service.

Getting ready

You will need a GitHub account. Sign up for a free account if you do not already have one.

How to do it…

To create a new repository on GitHub, log in to your GitHub account and create a new public repository:

Click the Create repository button. Make sure that the checkbox Initialize this repository with a README is unchecked. The new repository form should look something like this:

On the next page, you will be given an option to initialize this repository. We already have a local repository, so we will use the ... or push an existing repository from the command line option:

Copy both commands and execute them on a local Git repository:

$ git remote add origin https://github.com/sawantuday/mynewproject.git

$ git push -u origin master

The first command, git remote, adds a reference to the remote repository on GitHub and sets it as its origin. The next command, git push, synchronizes all local content with the remote repository. The git push command will show the details, as follows:

You will be prompted to authenticate with your GitHub account from the command line. Enter your GitHub username and password. This ensures that you are allowed to push the changes to the repository. Alternatively, you can add your local SSH public key to your GitHub account to avoid manual authentication.

Now you can use your GitHub repository to share code with others or clone it to some other system. On the GitHub page, check the code tab to take a look at files in the repository.

How it works…

Local repositories are good for personal work. A single person can work with them easily. A centrally hosted repository is required when you need to share the code base with a group of people. Everyone can make a local copy of the central code base and send their changes back to the central copy. GitHub solves this problem by hosting repositories that are accessible over the Internet. You can simply create a free public repository and share its URL with colleagues. Through access control, you can select who can check in their code. You can also set up your own centrally hosted repository. All you need is a system accessible over your network or Internet.

Here, we have created a central shared repository on GitHub. GitHub provides various options to initialize a repository and add code to it. As we already have our local repository ready, we just need to add a reference to the remote repo and synchronize our changes with git push. The git remote command is used to add a reference to the remote repository. We have set the remote repository as origin, that is, the default remote repository. When using git push or git pull commands, if we do not specify any remote name it is assumed to be origin. Also, by default, Git marks the first remote as origin.

Next, we used Git push to push or synchronize our local contents to a remote copy. We have explicitly mentioned the remote name as origin and the remote branch as master. By default, Git always pushes to a remote named origin and branch master.

There's more…

You can create your own remote copy on a local shared server. All you need is a normal user account on that server.

Log in to the shared server and create a bare repository with the following command:

$ git init --bare shared_repo

This will create an empty bare repository under the shared_repo directory. If you check its contents, you will find all Git-specific files and directories.

Now you can clone this repo from your workstation or use the git remote add command to add a remote to your already initialized repository. Use the following command to clone the repo. Replace the username with the user account on a shared server:

$ git clone ssh://user@ server_ip_or_name/full/path/to/repo

This command will ask for the password of the user account you have used in the username. Additionally, you can remove the password prompt by setting key-based SSH authentication with a shared server.

GitHub pages

You can host your own simple static website with GitHub for free. All you need is a Git repository hosted on GitHub. Follow these steps to get your own GitHub page:

Create a new repository with the name username.github.io, where username should be your GitHub username.

Clone this repository to your local system. If you already have a project created on your local system, you can add this repository as a remote. Check this recipe for how to add a remote.

Create index.html if you do not have one. Add some content to index.html.

Stage all content, commit to the local repository, and then push to GitHub.

Next, point your browser to username.github.io. You should see the content of index.html.

GitHub pages works with websites generated using static website generators such as Jeykyll, Hugo, and Octopress. By default, you get a github.io sub-domain, but you can use your own domain name as well.

See also

Check the manual pages for git remote and git push with man git-remote and man git-push respectively:

Read more about generating SSH keys: https://help.github.com/articles/generating-ssh-keys/

Get free hosting for your static website at GitHub pages: https://pages.github.com/