Linux Tutorial

Upgrading cvs.gentoo.org

Introduction

In the first article on LVM, I explained the concepts behind LVM. Now is the time to put into action what we saw on LVM. In this article, we will set up LVM on the official Gentoo Linux cvs server, cvs.gentoo.org. Although cvs.gentoo.org has only one hard drive, the flexibility of LVM also an incredible improvement over classical techniques of static partitioning. Will show all the steps of the process of converting to LVM, so that those interested can make a similar transition on one of their machines.

Note: As the implementing LVM is a big change for the system (which involves the creation of new partitions and other potentially hazardous actions) may be a good idea to perform a full system backup before starting the journey. If you do not intend to make a backup, the author hopes that the car used in the experiments is intended and does not contain important data. It is important to note that the author has never encountered any problems in the transition to LVM, but it is always better to be prepared in case something goes wrong.

Without the necessary prerequisites, you can proceed. Before starting the conversion process, was made an update to cvs.gentoo.org were used so that the following packages. The moment has been made the transition to LVM, these were the latest versions available (see Resources later in this article):

• Linux kernel 2.4.1-ac19
• LVM 0.9.1_beta5
• reiserfs-utils 3.6.25

To start, the hard drive. cvs.gentoo.org has a good hard drive new IBM 45 GB in it, however, when it was installed Gentoo Linux on cvs, were partitioned only about 10 gigabytes of hard disk, leaving the remaining 35 GB for future partitions. These are some little tricks that the user needs when not using LVM: left parts of the disk is not partitioned is a rough but effective way to allow for future expansions. However, LVM, there is a better approach.

The space problem

In the past few weeks, it was noted that the root ReiserFS partition had become slowly filling up, as you can see from this printed df:

Code 1.1: Free space running out

Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda3              9765200   6989312   2775888  72% /
tmpfs                   269052         0    269052   0% /dev/shm

In fact, a root partition filled to 72% is not exactly a crisis, but not a great situation. ReiserFS, like many other file systems, performance degrades gradually fills it, and it was only a matter of time, before the root filesystem was completely full, and his performance would suffer a meltdown.

It was therefore decided to remedy the problem by using LVM to create a new logical volume out of the 35 GB of unpartitioned space that is currently at the end of the hard drive. Thus was created a filesystem on this volume and have been moved a good part of the contents of /dev/hda3 on it.

If the reader is going to make a similar transition on one of their machines, the first thing they need is to find a part of your root filesystem suitable to move a logical volume. For the author, the choice was easy: the tree under /home occupied about 5.7 GB. Moving /home/home, the root filesystem should probably be roughly around 20% of its capacity, a situation very promising.

The beginning of a solution

To begin the conversion, was added to the first partition the unused space at the end of the disk drive. Using cfdisk, you created a 35 GB partition (/dev/hda5) and set the type of that partition to 8E (the type for the official LVM). After this change, the partition table was as follows:

Code Listing 1.2: The new partition table

# sfdisk -l
Disk /dev/hda: 89355 cylinders, 16 heads, 63 sectors/track
Units = cylinders of 516096 bytes, blocks of 1024 bytes, counting from 0
   Device Boot Start     End   #cyls   #blocks   Id  System
/dev/hda1   *      0+    247     248-   124960+  83  Linux
/dev/hda2        248     743     496    249984   82  Linux swap
/dev/hda3        744   20119   19376   9765504   83  Linux
/dev/hda4      20120   89354   69235  34894440    5  Extended
/dev/hda5      20120+  89354   69235- 34894408+  8e  Linux LVM

From the time that there was an empty partition of 35 GB, it was possible to proceed with its initialization for use with LVM. The steps are: first, initialization of the 35 gigabytes as a physical volume, thus creating a volume group using the above-mentioned physical volume, and finally, allocation of some entity in the volume group, thus going to create a logical volume that should contain the new file system and host all the files located at /home.

To begin the process, we used the pvcreate command to initialize /dev/hda5 as a physical volume:

Code Listing 1.3: Creating the physical volume

# pvcreate /dev/hda5
pvcreate -- physical volume "/dev/hda5" successfully created

pvcreate in this case sets up a special area of ​​”management” on /dev/hda5, called VGDA (volume group descriptor area). LVM uses this to keep track of how physical entities are allocated, among other things.

The next step is to create the volume group and add /dev/hda5 to this group. The volume group should function as a pool of entities (parts of blocks of memory). Once you create the volume group, logical volumes can be created many as you want. In this case, it was decided that the volume group should be called “main”:

Code Listing 1.4: Creating the volume group

# vgcreate main /dev/hda5
vgcreate -- INFO: using default physical extent size 4 MB
vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte
vgcreate -- doing automatic backup of volume group "main"
vgcreate -- volume group "main" successfully created and activated

The vgcreate command does a couple of things. In addition to creating the volume group “main”, also imposes a /dev/hda5 entities to use 4 MB, the default size for the entities. This means that each logical volume that will be created by the specific volume group can be expanded and reduced by increments of 4 MB.

Because of the limitations of the kernel, the size of the entity determines the maximum size that a logical volume can take. As you can see from the above example, a size of 4 MB per entity imposes a restriction on the logical volume size of 256 gigabytes, which is a logical volume size to be easily reached if you add components to its high-capacity group volume. If there is a possibility that the volumes end up by becoming larger than 256 GB apiece, it is advisable to specify a larger size for use at the time of vgcreate. The size of the entity may vary freely between 8 MB and 512 MB, and must always be a multiple of two. Increasing the size of the entities above the 4 MB, the maximum size for physical volumes will be scaled accordingly, up to a maximum of 1 petabyte (even when the current limit in the real world is equal to 2 terabytes on x86 systems). For example, to create a volume group with a magnitude equal to 32 megabytes in size, must be typed:

Code Listing 1.5: Setting a larger scale for

# vgcreate -s 32M main /dev/hda5

32 MB is a good size for size, since a granularity of 32 MB is still manageable and brings the maximum size for logical volume to 2 terabytes startup. Once the volume group is created, you can view the information by typing vgdisplay:

Code Listing 1.6: Review the information volume

# vgdisplay
--- Volume group ---
VG Name               main
VG Access             read/write
VG Status             available/resizable
VG #                  0
MAX LV                256
Cur LV                0
Open LV               0
MAX LV Size           255.99 GB
Max PV                256
Cur PV                1
Act PV                1
VG Size               33.28 GB
PE Size               4 MB
Total PE              8519
Alloc PE / Size       0 / 0
Free  PE / Size       8519 / 33.28 GB
VG UUID               2qC2H2-iA8s-qW6F-cwXx-JVIh-I6VC-VVCGmn

Now that the volume group is ready, you can create a logical volume. In the specific case was initially decided to make it equal to 8 gigabytes in size and call it “lv_home”

Code Listing 1.7: Creating the logical volume

# lvcreate -L8G -nlv_home main
lvcreate -- doing automatic backup of "main"
lvcreate -- logical volume "/dev/main/lv_home" successfully created

So, you created a filesystem on the volume:

Code Listing 1.8: Creating the filesystem

# mkreiserfs /dev/main/lv_home

   Block size 4096 bytes
   Block count 2097152
   Used blocks 8275
           Journal - 8192 blocks (18-8209), journal header is in block 8210
           Bitmaps: 17, 32768, 65536, 98304, 131072, 163840,
           196608, 229376, 262144, 294912, 327680, 360448,
           393216, 425984, 458752, 491520, 524288, 557056,
           589824, 622592, 655360, 688128, 720896, 753664,
           786432, 819200, 851968, 884736, 917504, 950272,
           983040, 1015808, 1048576, 1081344, 1114112,
           1146880, 1179648, 1212416, 1245184, 1277952,
           1310720, 1343488, 1376256, 1409024, 1441792,
           1474560, 1507328, 1540096, 1572864, 1605632,
           1638400, 1671168, 1703936, 1736704, 1769472,
           1802240, 1835008, 1867776, 1900544, 1933312,
           1966080, 1998848, 2031616, 2064384
   Root block 8211
Hash function "r5"
ATTENTION: ALL DATA WILL BE LOST ON '/dev/main/lv_home'! (y/n)y
journal size 8192 (from 18)
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..done.

Once you create the filesystem can be mounted on /mnt/newhome:

Code Listing 1.9: Mount the new volume

# mkdir /mnt/newhome
# mount /dev/main/lv_home /mnt/newhome
# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda3              9765200   6989840   2775360  72% /
tmpfs                   291388         0    291388   0% /dev/shm
/dev/main/lv_home      8388348     32840   8355508   1% /mnt/newhome

How was it possible to see before, everything is almost ready to copy over all data to /home. Before you begin, it is better to go to runlevel 1 to ensure that no user or process wants to access or modify files in /home as they are copied:

Code Listing 1.10: Switch to runlevel 1

# init 1

At this point, it begins copying files:

Code Listing 1.11: Copying files to new folder

# cp -avx /home/* /mnt/newhome

In such case, the copy was completed in about ten minutes. So, was made a backup copy of the original folder /home/home.old, in case something went wrong while copying. It was then created a new mount point, and remounted the new home on /home:

Code Listing 1.12: The new mount point

# cd /
# mv home home.old
# mkdir home
# umount /mnt/newhome
# mount /dev/main/lv_home /home

Now, it’s time to fine-tune the server so that the new /home partition will be available whenever the machine is started. First of all, must be modified /etc/fstab to include the /home:

Code Listing 1.13: Editing fstab

#fs                 mountpoint       type         opts          dump/pass
/dev/hda3           /                reiserfs     defaults      1 1
/dev/main/lv_home   /home            reiserfs     defaults      2 2
/dev/hda2           none             swap         sw            0 0
/dev/hda1           /boot            reiserfs     noauto        0 0
/dev/cdrom          /mnt/cdrom       iso9660      noauto,ro     0 0
proc                /proc            proc         defaults      0 0
none                /dev/pts         devpts       mode=620      0 0
tmpfs               /dev/shm         tmpfs        defaults      0 0

Then, go make minor changes to the initialization scripts. This is achieved by modifying the boot script “checkroot” so the following command can be executed immediately after the root partition was remounted read /write:

Code Listing 1.14: Editing the Startup Script

/sbin/vgscan
/sbin/vgchange -a y

Still, the script must be modified to unmount filesystems at shutdown is called, so the following command can be executed immediately after all filesystems have been removed:

Code Listing 1.15: Edit the script shutdown

/sbin/vgchange -a n

Once you complete these steps, just reboot the machine, and note with pleasure that everything works perfectly. After a day or more without any problem, can be canceled /home.old to free up some ‘space on the root filesystem. The transition to LVM is successful.

The beauty of LVM

While the transition to LVM you can not define trivial, once the transition is complete, managing filesystems becomes tremendously easier. For example, the author decided to resize your logical volume on /home, adding about 2 gigabytes of additional space at the end of the file system. First, additional capacity has added to its logical volume “lv_home“, and then used the program resize_reiserfs to expand the filesystem so that he could use this additional capacity. Here are two commands that allow this:

Code 1.16: Use additional space

# lvextend -L+2G /dev/main/lv_home
# resize_reiserfs -f /dev/main/lv_home

In about a second, the filesystem on /home has been expanded to 2 GB, surprisingly, there was no need to reboot, go to runlevel 1, or even unmount /home to resize. Everything continued to work as he had done before. How cool is that? Following is the current state of the filesystem, after the operation:

Code 1.17: space on the filesystem

# df
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/hda3              9765200   1413340   8351860  15% /
/dev/main/lv_home     10485436   5609836   4875600  54% /home

You can find out how LVM can actually make the job of an administrator much easier. In the future, there is the hope of being able to move additional parts of the root filesystem on LVM, and eventually convert the root file system within a logical volume on LVM.

Magic memory management with Logical Volume Management

Introduction to LVM

In this series, I will show how to install and use the new integrated support for Logical Volume Management in Linux kernel 2.4. If the reader has had no previous experience with LVM, you will find an interesting treatise, this is a fantastic technology. Before actually working to recover and make LVM, you should explain exactly what it is and how it works. As a result, LVM can be thoroughly tested and analyzed.

For many, as the author, the experience with UNIX and Linux began on a PC platform, rather than on large commercial UNIX servers and workstations. On the PC base, there was always need to get busy with partitioning the hard disk. PC users are generally well informed about tools like fdisk, which is used to create and delete primary and extended partitions on hard disks. The partitioning of the hard drive is an unpleasant but accepted part of the process it takes to make a functioning operating system.

The partitioning of the hard disks can be annoying because to do a good job there is the need to accurately estimate how much space each partition needs. In case of under estimation, the Linux system might be unusable, and to solve the problem will probably need to make a full backup of the system, to clean the hard drive, then insert all data into a new (and presumably better ) partitioning scheme. Nice work. This is exactly the kind of situations that system administrators try to avoid as much as possible in the first place.

As the partitions were once static memory regions, fortunately, today we have a proliferation of computer tools for repartitioning (the product of PowerQuest Partition Magic is one of the most popular). These tools allow you to boot into a special disk, and dynamically resize partitions and filesystems. Once restarted, the partitions are available to change, hopefully escaping away from the problems of partitioning. These tools for resizing partitions are fine and resolve the problem of managing the storage space for many people. But I’m perfect? Not exactly.

Vehicles such as Partition Magic are great for workstations, but not really suitable for servers. First of all, will need to restart the system. This is something that many system administrators are trying desperately to avoid. What would happen if it was not possible to simply restart the machine each time the memory needs to change, if that memory needs drastic changes every week? What would happen if it were necessary to expand the filesystem so that it can cover more than one hard drive, or if you need to dynamically grow or shrink the capacity of a physical volume while allowing Apache to continue to serve Web pages? In a dynamic environment with high availability, a basic partition resizer is not enough. For these and other situations, Logical Volume Manager is an excellent (if not perfect) solution.

Within the LVM

Now you need to look at how LVM solves these problems. Creating an LVM logical volume is a three-step process. First, you must select the physical storage resources that will be used for LVM. Typically, these are standard partitions but can also be Linux software RAID volumes created previously. In LVM terminology, these storage resources are called “physical volumes.” The first step in creating LVM includes properly initializing these partitions so that they can be recognized by the LVM system. This involves setting the correct type of partition if that is added is a physical partition, then run the pvcreate command.

Once in possession of one or more physical volumes initialized for use by LVM, you can proceed to step two, the creation of a volume group. You can consider the group as a container volume of memory that consists of one or more physical volumes. While LVM is running, additional physical volumes can be added to the volume group or even removed. However, you can not mount filesystem directly on or create a volume group. Rather, you can ask for LVM to create one or more “logical volumes” using the container’s storage volume group:

Figure 1.1: A volume group is created out of the physical volumes

Create an LVM logical volume is really easy, once created, then you can proceed by adding a file system on it, mount it and start using the volume to store your files. To create a logical volume, you must use the lvcreate command, specifying the name of the new volume, the size you want for it, and the volume group that you want the particular logical volume is part. The LVM system then allocates memory from the specified volume group and create the new volume, which is then ready for use. Once created, it can be put on a ext2 or ReiserFS, it can be mounted, can be used as the best choice.

Figure 1.2: Create two logical volumes from the existing volume group

Entity

Behind the curtain, the LVM system allocates memory in “slices” of equal size, called entities (or extent). It is possible to specify the size to use for the specific entity at the time of creation. The default size is 4 Mb for an entity, already perfect for many users. One of the highlights of LVM is that the physical location of memory used for an entity of the logical volumes (in other words, are stored on which disk) can be changed dynamically while the logical volume is mounted and in use. The system ensures that the LVM logical volumes will continue to operate perfectly while allowing the administrator to physically change the place where everything is stored.

Certainly, until everything is created out of equally sized entities, it’s really easy to allocate some additional amount for a logical volume already exists, or in other words, “grow” the volume dynamically:

Figure 1.3: Add additional amount from its volume group, expanding the size of the particular logical volume

Once the logical volume is expanded, you can then expand your ext2 or ReiserFS to take advantage of this new space. If you plan to use a program like resize_reiserfs, this extension of the file system can also occur when the volume is mounted and in use. Really Surprising: LVM tools and expansion of the hot file systems (also known as online), you will not need to reboot your system, or switch to runlevel 1 to change your memory configuration.

The only time you will need to shutdown your system is when you will need to add new physical disks. Once the new disks have been added, you can then add these new physical volumes to your volume group (or groups to its volume) to create a new reserve amount.

Configure LVM

At this point, LVM is installed. LVM consists of two parts: a kernel component and a set of tools that operate in user-space. To begin, go to the main page of the LVM and download the latest version of the archive of the LVM (lvm_0.9.1_beta3.tar.gz at the time) present. The archive contains all the LVM tools that operate in user-space, as well as a set of patches for the kernel. Here is where things start to get interesting.

If you already have a 2.4 kernel installed, you may have also already provides support for LVM on your system, but if not, recompile your kernel to enable LVM support will be a trivial matter. However, you may not want to use LVM support 2.4 kernel included in its preparation (or provided by the distribution). If the user wants to use the latest version of LVM, you probably will want to apply the patch in the archive to your LVM source tree for the current 2.4 kernel. The following explains how you can do.

To begin, you must log in your kernel source directory (/usr/src/linux) and create a folder called extras. Next, enter this directory and extract your archive LVM:

Code Listing 1.1: Remove the patch

# cd /usr/src/linux
# Mkdir extras
# Cd extras
# Tar xzvf /path/to/location/of/lvm_0.9.1_beta3.tar.gz

Once you’ve done the above, you will notice a new folder called extras LVM that contains another folder whose name is equivalent to the version of LVM you just extracted. Go up into these two folders to get to the source of the LVM:

Code Listing 1.2: Getting to the source of LVM

# cd LVM/0.9.1_beta3
# ls
ABSTRACT      COPYING      INSTALL     Makefile     README    autoconf      config.status  kernel         make.tmpl.in
CHANGELOG     COPYING.LIB  KNOWN_BUGS  Makefile.in  TODO      config.cache  configure      lvm_input_msg  scripts
CONTRIBUTORS  FAQ          LVM-HOWTO   PATCHES      WHATSNEW  config.log    configure.in   make.tmpl      tools

There are several text files, scripts, and directories of sources. You can find instructions for installation in the INSTALL file, this text will guide you through this process. First of all, to be run the configure script, as follows:

Code Listing 1.3: Configure LVM sources

# ./configure --prefix=/ --mandir=/usr/man

Patching

After running the above command, will be created and configured the Makefile to install all the LVM tools in /sbin and man pages in /usr/man. If your man pages in /usr/share/man (as in FHS 2.1), we need to correct the above path in this direction. And, if the kernel sources are in --with-kernel_dir=/percorso/per/usr/src/linux on the command line. Once the configure script completes its task, you can proceed with the installation of the instruments and create patches for the current kernel. First we need to apply various patches to the kernel. Enter the PATCHES directory:

Code Listing 1.4: Changing directory

# cd PATCHES

Now, simply type the make command. The Makefile will generate a patch specifically for the specific sources of the kernel 2.4 series:

Code Listing 1.5: Creating the patch

# make

The patch will be called lvm-[versione_lvm]-[versione_kernel].patch. For example, using the version 0.9.1_beta3 LVM and kernel 2.4.0-ac11, the patch will be called lvm-0.9.1_beta3-2.4.0-ac11.patch. The following can be found in the current folder. Now is the time to apply the patch. To do this, you need to change directories in passing that on to the kernel sources and use the patch command as shown below:

Code Listing 1.6: The patch command

# cd /usr/src/linux
# patch -l -p1 < /usr/src/linux/extras/LVM/0.9.1_beta3/PATCHES/lvm-0.9.1_beta3-2.4.0-ac11.patch

Although the information found in the INSTALL file does not contain the following LVM, the author usually adds the -l option to the patch command. This option allows the patch program to compensate for any changes in the white spaces (such as minor changes in indentation) that may commonly cause the failure of some parts of the patch. If the previous command ends without a single line “FAILED”, will have reached the time to install the tools that operate in user-space. If not, we will instead need to explore its own directory /usr /src/linux to search for files .rej and insert the components within the excluded sources by hand using a text editor. However, in almost all cases the patch will be applied without any problems and you can then proceed.

Configure, build and install

You are now in possession of a kernel that is patched and therefore has available the latest LVM code. At this point, will want to configure your kernel so as to enable support for LVM. It is recommended to compile support directly into the LVM kernel rather than configuring it to be compiled as a module. We must then start the preferred method for configuring the Linux kernel:

Code Listing 1.7: Configuring the kernel

# cd /usr/src/linux
# make menuconfig

The options for LVM can be found under the section “Multi-device support (RAID and LVM)”. Once enabled the first option:

Code Listing 1.8: Support for RAID and LVM

[*] Multiple devices driver support (RAID and LVM)

The following options will appear, which can in turn be empowered to:

Code Listing 1.9: Enable support for LVM

 Logical volume manager (LVM) support

Depending on the version LVM, there may be other options for the LVM, which you might want to enable. Once this is done, just save your kernel configuration and continue the procedure followed to compile the default kernel, then reboot. Congratulations, now support for LVM is enabled in the kernel. Now, we need to build and install tools that operate in user-space. This step is easy:

Code Listing 1.10: Creating tools that operate in user-space

# cd / usr/src/linux/extras/LVM/0.9.1_beta3
# make
# make install

There remains only one thing to be done, and is optional. If you want to carry out multiple operations on LVM, you must add the following lines to your startup rc scripts:

Code Listing 1.11: Edit the startup script rc

/sbin/vgscan
/sbin/vgchange -a y

These lines will seek all available volume groups and activates them. Then, add the following line to the script rc-off, and make sure it is done after all filesystems have been removed:

Code Listing 1.12: Edit the script rc-off

/sbin/vgchange -a n

If you want to test only LVM, these steps can be skipped. Just remember to type after every reboot and vgscan vgchange-ay as the root user first to make available for their use logical volumes.

It’s all for this article. In the next, will be shown how to create logical volumes and how their personal unleash the power of LVM. Do not miss it.