Deividson on Databases: Stored Procedures

By Deividson Luiz Okopnik

Stored Procedures

Stored Procedures are pre-compiled subroutines that are stored inside the database. They allow you to select and manipulate data, and, with the use of control structures and loops, perform complex computations and return the calculated result to the client. This saves considerable amounts of client/server communication.

PostgreSQL allows Stored Procedures to be written in several different Procedural Languages, including Perl, Python, TCL, and pgSQL - the PostgreSQL internal procedure language. User-defined Procedural Languages can also be used, and several of these languages are easily downloadable, e.g. PL/Java.

In this article, we will be using PL/pgSQL. PL/pgSQL is very similar to normal SQL, but adds many more features to it, like control structures and user-defined data types and functions.

Example 1: The Basic Stored Procedure

Let's get started with a very basic stored procedure that returns "Hello World!" - not very useful, I know, but it will get us started with the basic syntax of PL/pgSQL. Here's the code:

create or replace function hello() RETURNS text AS $$
DECLARE
  hello text;
begin
  hello := 'Hello World!';
  return hello;	
END;
$$ LANGUAGE plpgsql;

Here's what it does:

create or replace function hello() RETURNS text AS $$

Creates the function called hello which receives no parameters and returns text. You must always define what the function returns; use VOID if you don't need to return anything.

DECLARE

Opens the variable declarations block.

hello text;

Declares a variable called "hello" of type "text". To define multiple variables, use ";" as the separator. You can use any of standard types used in tables, like integer and float, and even user-defined types or domains.

	BEGIN

Starts the actual function code.

	hello := 'Hello World!';

Pupulates the variable "hello" with 'Hello World!'. Note that you have to use single quotes for string/text values.

return hello; 

Returns our value.

END;

Ends the function.

$$ LANGUAGE plpgsql;	

Defines what language we used - 'plpgsql' in this case. To call that function, you use the following SQL code.

select * from hello();

The output will be a text field called "hello", with the value of "Hello World!".

Example 2: Populating a Table with Test Data

This is another use of a Stored Procedure ('SP' from now on) - generating test data for your tables. Let's use last month's article as an example - we used a SP to generate 500K rows of data for one of our tables. Here's the code:

create or replace function test_data_computer()
RETURNS integer AS $$
DECLARE
  count integer;
  sql text;
begin
  count = 1;
  LOOP 
    sql = 'insert into computer(computer_id, computer_ram, cpu_id, video_id) values';
    sql = sql || '('|| count ||', ' || random()*1024 || ', ' || (random()*49999)+1 || ', ' || (random()*49999)+1 || ')';
    EXECUTE sql;
    count = count + 1;
    EXIT WHEN count > 500000;
  END LOOP;  
  return count;	
END;
$$ LANGUAGE plpgsql;	

It starts much like our previous example, but this time we declare 2 variables instead of one. Things become different at line 8, where we introduce the LOOP statement. The loop is a basic repeating structure: it repeats the code inside indefinitely, until it finds a EXIT or EXIT WHEN clause.

Lines 9 and 10 are used to generate the SQL code to include a simple record in our tables. The double pipes ("||") is the concatenation operator. Random() generates a random float number between 0 and 1 (so "random()*49999)+1" will generate a random number between 1 and 50000).

Line 11 executes the SQL code stored inside the sql variable, adding the registry to the table.

Lines 12 and 13 are used to control the flow of the LOOP, and if omitted will make the loop an infinite one. "EXIT WHEN count > 500000;" makes the loop stop when the condition is met (when "count" goes over 500000 in this case.)

Line 14 closes the LOOP block, making the function go back to line 8, executing everything that is inside the loop again (and again, and again).

Line 15 returns the number of added registries (plus one in this case).

Example 3: Calculations and Date/Time Handling

Let's make up a scenario for this one. Imagine that you are building a system for a doctor, and one of the bits of data he wants is exactly how much time he spends with his patients (NOT just idling in the office.) Even more, he wants to be able to select the data for a given date or date interval, and he wants the option of selecting the records of either a single patient or all of them. Complex scenario, right? Well, we can solve it all with a single SP. These are the tables our database will have:

create table patient (
patient_id serial primary key, 
patient_name text );

create table visits (
v_id serial  primary key, 
patient_id integer references patient,
v_date date,
v_time_start time,
v_time_end time );

One for the patients, another one to store the visits, with the date, start, and end time. Let's now populate the tables with some data:

insert into patient (patient_name) values ('Deividson');
insert into patient (patient_name) values ('John');
insert into patient (patient_name) values ('Benjamin');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (1, '10/04/2008', '08:00', '09:00');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (1, '14/04/2008', '13:00', '13:45');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (1, '18/04/2008', '10:00', '10:15');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (2, '11/04/2008', '14:00', '15:00');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (2, '12/04/2008', '14:00', '15:45');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (2, '17/04/2008', '14:00', '15:15');
insert into visits (patient_id, v_date, v_time_start, v_time_end) values (3, '15/04/2008', '08:00', '12:00');

Three patients, seven records - enough to test our SP. Here is the code:

CREATE OR REPLACE FUNCTION total( date1 date, date2 date, patient integer ) 
RETURNS interval AS $$
DECLARE
  total interval;
  rec record;
  sql text;
BEGIN
  total = '00:00:00'::time;
  sql = 'select * from visits';
	
  if date1 is not null OR patient is not null then
		sql = sql || ' where ';
  end if;

  if patient is not null then
		sql = sql || '(patient_id = ' || patient || ')';
  end if;

  if date2 is not null AND date1 is not null then
		if patient is not null then
			sql = sql || ' AND ';
		end if;
		sql = sql || '(v_date between ''' || date1 || ''' and ''' || date2 || ''')';	
  else
		if date1 is not null then
			if patient is not null then
				sql = sql || ' AND ';
			end if;
			sql = sql || '(v_date = ''' || date1 || ''')';	
		end if;  
  end if;

  for rec in EXECUTE sql loop
    total = total + (rec.v_time_end - rec.v_time_start);
  end loop;
	
  return total;
END;
$$ LANGUAGE plpgsql;

Wow! Big one this time, eh? Let's take a look at it. The start of the code is pretty similar to the other examples, but we have 3 variables this time. 'total' will store the total time to return to the client, and 'rec' (of type record) is a variable that will hold the result of the query we will run.

On line 8, we start the variable total with the value of 00:00:00 - the "::" is a typecast - ":: time" means the string we're passing ("00:00:00") needs to be turned into a time.

From line 9 all the way down to line 31, all we are doing is creating the SQL statement that will select the data we want. Here, we use another type of structure - the IF. IFs are basic flow-control structures, and its syntax is (as in most programming languages):

IF (condition) THEN (commands) [ELSE (commands)] END IF;

The condition can be any logical comparison ( <, > , =, IS NULL, or IS NOT NULL), and you can combine multiple conditions using the logical operators (AND, OR, etc). If the condition is true, then the execution will continue with the commands inside the THEN clause, or if it's false, execution will move to the commands in the ELSE (if it exists), or to after the END IF.

It's in those IFs that we create the conditions (single date, date interval, single patient, etc.)

On line 33, we execute the SQL code we generated, looping over each of the records of the result. We then add the duration of each visit in the 'total' variable, and return the result when there are no more records available.

We can call this SP in one of several different ways, each way selecting a different data set and giving us a different result:

# All the records, from all patients
select * from total(NULL, NULL, NULL);

# All the records, from patient #3 only
select * from total(NULL, NULL, 3);

# Records from '14/04/2008', all patients
select * from total('14/04/2008', NULL, NULL);

# Records from '14/04/2008', patient #1 only
select * from total('14/04/2008', NULL, 1);

# Records from '14/04/2008' through '17/04/2008', all patients
select * from total('14/04/2008', '17/04/2008', NULL);

# Records from '14/04/2008' through '17/04/2008', patient #2 only.
select * from total('14/04/2008', '17/04/2008', 2);

Conclusion

Stored Procedures are powerful and flexible, and can be a very good way to help you pre-select and pre-process data, as well as allowing you to manipulate data and run code directly on the server.

PostgreSQL offers a comprehensive manual on their site, including a chapter about PL/pgSQL. You can find it here: http://www.postgresql.org/docs/8.3/static/plpgsql.html

That's it for Stored Procedures - see you next month, when we'll discuss Triggers!

Talkback: Discuss this article with The Answer Gang

Knoppix 5.3.1

By Edgar Howell

At the recent CeBIT computer fair, Klaus Knopper finally released a new version of his popular Knoppix live-CD/DVD variety of GNU/Linux. The initial version was fine for the CeBIT crowd but had some problems, the most obvious of which was a mixture of languages in menus if you booted with the parameter "lang=" (by the way, it is now "lang=us" rather than "lang=en"; this will take getting used to after many years of using "lang=en".)

The vital statistics on the post-CeBIT version, 5.3.1, are:

• 2.6.24.4 kernel,
• KDE 3.5.8 as default,
• KDE 4 Beta on board as are Gnome, IceWM, and various other GUIs.

The DVD is close to full and the software is distributed between 2 compressed directories. This DVD should include just about everything but the proverbial kitchen sink.

New to me (absent appropriate hardware) but not new to the 5.x.x Knoppix series is recognition of dual-core CPUs: on boot, Knoppix shows you Tux twice. However, new to 5.3 (at least since 5.0) is the use of a key that for me has always been a dead-key: the "flag" to the left of "alt" to the left of the space-bar. This key now activates the start menu as if you had clicked on the icon at the far left of the panel. Those who have learned to use the keyboard ought to appreciate that progress away from mouse-dependency.

To my knowledge virtualization is also a new topic with this release. You will find KVM as well as a XEN-kernel and VirtualBox. I played briefly with VirtualBox, and it was necessary to load the 'vboxdrv' module manually before starting the graphics interface.

For what it's worth, during the last nine months or so everything I've done in the office here other than USB and burning CD/DVD has been under VMs. It is a topic well worth investigating, particularly in this environment.

By now, most readers certainly have heard about Knoppix and ought to have some idea of what it is all about - and this was the first time that I encountered any significant problems with a new release of Knoppix. So, I think that topic deserves some comment this time around.

Linux has a certain reputation - it's supposed to behave well on older hardware. While this is indeed true in large measure, if all you have is Knoppix on a DVD, you won't be able to do much with it on a machine that can't boot from DVD or doesn't even have a DVD drive! But this was the first time that I ever had trouble with hardware that has worked fine with various distributions for many years.

And it was a strange experience indeed. The desktop failed in two ways, both error messages from KDesktop: "Unable to create io-slave" and "Signal 6 SIGABRT". Unfortunately the former error didn't prevent Konqueror from starting and hiding the small window with the error message which I initially missed because I am typically busy with other things as Knoppix boots from DVD. These errors occurred to one degree or another on a 5-year-old PC, a 3-year-old notebook and a notebook I've only had for a couple of weeks. In desperation I even tried the beta of KDE 4 to no avail.

As it turns out, KDE development seems to be moving in a direction that I have no use for anyhow: the best I can say about 'compiz' is that it seems to be a significant waste of developer's time and CPU cycles. Trying to re-position a window is like trying to push a wet noodle. The floppy window and flames that delay closing don't make much sense to me either. Fortunately, Knoppix lets us set the "no3d" parameter.

The beta of KDE 4 was a turn-off from the start menu. It looks just like SuSE 10.3 for crying out loud! Why!? Why would any sane person think it reasonable to use 5 times as much space as necessary for a line of text in a menu? This just forces some entries into another level in the hierarchy thus hiding things deeper and more mouse-clicks away. That makes about as much sense as adding another level of bureaucracy to the government...

So, the time had come to abandon KDE and re-master the standard release of Knoppix with a default configuration that I like. Isn't that why we all use Linux? Because we are in charge?

Re-mastering a distribution may sound like pretty heavy-duty guru-type stuff, but it needn't be difficult at all. If you want to add software to one of the standard locations (rather than just putting it somewhere on the DVD) it is pretty much a 2-pass algorithm since you have to add something to a directory that then is converted into a compressed file which then has to fit along with everything else into an iso-image - not too big for the DVD or CD. But in this case, all that is necessary is to make a tiny alteration to a boot parameter.

Here are the commands needed, to be executed as root as usual:

mount -o loop /media/hda3/KNOPPIX_V5.3.1DVD-2008-03-26-EN.iso /mnt
mkdir TEMP_KNOPPIX
cd TEMP_KNOPPIX
mkdir KNOPPIX_master
cp -Rp /mnt/* KNOPPIX_master/
vi KNOPPIX_master/boot/isolinux/isolinux.cfg
/usr/bin/mkisofs -pad -l -r -J -v -V "Knoppix 5.3.1" -P "Linux Gazette"   \
                 -no-emul-boot -boot-load-size 4 -boot-info-table         \
                 -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat  \
                 -hide-rr-moved -o knoppix.iso KNOPPIX_master/ 

This is what's going on here:

• The 'mount' command references an ISO-image downloaded from one of the mirrors on the Internet instead of a CD or DVD in a drive.
• The first 'mkdir' command gives us a convenient and readily-identifiable (for later deletion) directory in which to work.
• After changing into it we need one to hold Knoppix where it can be modified.
• The 'cp' command merely copies everything there while preserving permissions.
• What you do with the 'vi' command (or some other editor) is up to you, below.
• And, finally, the 'mkisofs' command creates the ISO-image, "knoppix.iso", that you can then burn to a DVD.

For my purposes it was sufficient to add the following part (bold text) at the end of the second line of "isolinux.cfg":

APPEND ramdisk_size=100000 init=/etc/init lang=us apm=power-off \
  vga=791 initrd=minirt.gz nomce highres=off loglevel=0         \
  libata.atapi_enabled=1 quiet SELINUX_INIT=NO nmi_watchdog=0   \
  BOOT_IMAGE=knoppix keyboard=de desktop=icewm

After having made this change, and burning the ISO-image to a DVD of course, booting from the DVD without entering any parameters at all brings up Knoppix with the keyboard needed and IceWM as the desktop.

Given the parameters one can use when booting Knoppix, re-mastering really isn't absolutely necessary. But as easy as it is to do and as convenient as it makes later use of Knoppix, why not?

An alternative to re-mastering would be to create a USB-device with persistent settings; I've used this technique in the past and it works nicely. But on one occasion a release change introduced an incompatibility and I decided it is just as easy to have a USB-device with a script or two that can be executed as needed: if the notebook is attached to the office LAN and there is a need for the network printer then it is easy to run the script that sets up CUPS.

So when all is said and done in spite of a couple of "minor" problems I am still an enthusiastic user of Knoppix. It definitely belongs in every toolbox.

And the fact that I don't like what is happening with KDE is absolutely irrelevant: the world is a big place, there is room for lots of different opinions. It's just a matter of taste. You might like it; you might not want to re-master - that's great too. Someone else might want Gnome instead of IceWM. The point is that with Linux, you have options. Go for it!

Talkback: Discuss this article with The Answer Gang

Edgar is a consultant in the Cologne/Bonn area in Germany. His day job involves helping a customer with payroll, maintaining ancient IBM Assembler programs, some occasional COBOL, and otherwise using QMF, PL/1 and DB/2 under MVS.

(Note: mail that does not contain "linuxgazette" in the subject will be rejected.)

Virtualizing without Virtualizing

By Kapil Hari Paranjape

Introduction

Nowadays one reads a lot about virtualization. The Gazette even ran an article on it. Most of these reports talk about Xen, Vserver, OpenVZ, Qemu, User Mode Linux or co-Linux. Each of those modern technologies has its place no doubt, but this article will concentrate on the humble chroot way of running programs in a virtual environment.

Here is the EULA¹ that you must agree to if you want to read further:

• The software you want to run “virtually” can run with your currently running kernel.
• Your super-user avatar trusts that the software you want to run is no less trustworthy than the software that is currently running on the system.
• You have enough disk-space to replicate a complete installation.
• You use Debian or a derived distribution. (These days that last line probably reads: “You run Ubuntu!”)

Here some possible scenarios where one might want to run programs in a virtual context in spite of these (admittedly rather stringent) conditions:

• You have installed the amd64 flavour of Debian but you need to compile and/or use 32-bit programs. (Read as: You want to see those flash thingies that everyone is talking about!).
• You have installed Debian stable but you want to “try out” some packages from testing or unstable; and you don’t really want to mess up your stable installation. (apt-get is great for resolving dependencies for upgrades; those who tried a downgrade have not lived to tell the tale).
• You really like Debian but the BDFL² has waltzed into town looking for new employees for Canonical and you want to impress him by running the Hardy Heron.
• “Girls³ just want to have fun!”. This is often the best reason of all — after all LG stands for Fun with Linux!

Ask the super

Your super-user avatar (SU) must carry out the following steps for you to enter the virtual context.

Allocate disk space

Either there is already enough disk space on some mounted partition or you have an unused partition. You will need this empty space to appear in some fixed location like /srv/schroot. SU can use mount (with the --bind option in the former case) to set this up.

Create a new Debian installation

Do not reach for that Debian boot CD! Here is a way that does not destroy your uptime record. SU should install debootstrap:

apt-get install debootstrap

and run it:

debootstrap --include=iceweasel,mozplugger lenny /srv/schroot

SU can of course replace iceweasel and mozplugger by any comma-separated list of programs which need to be installed. It is nice to add a local mirror at the tail end of the debootstrap invocation so as to get a faster download.

The main program

Since you run Debian, installing schroot is as easy as SU running:

apt-get install schroot

You also need to configure schroot; there are a number of rather interesting options. The following configuration stanza seemed to be just right:

[lenny]
description=Debian lenny (testing)
location=/srv/schroot
type=directory
users=luser
root-groups=root
root-users=root
aliases=testing,default
run-setup-scripts=true
run-exec-scripts=true

If you are planning to use a 32-bit chroot under a 64-bit system then you need to have personality=linux32 as part of this configuration. The above stanza is placed in the file schroot.conf in the configuration directory /etc/schroot/. It says that SU allows the user luser to use schroot to run under the directory /src/schroot after various standard setup and startup scripts have been executed.

You should also look through the setup script 10mount in the setup.d sub-directory of the above directory as you may need to create some additional mounts. For example, adding the line

do_mount "-o rw,bind" "/dev/snd"    "${CHROOT_PATH}/dev/snd"

to 10mount together with the creation of the dev/snd subdirectory of /srv/schroot ensures that the (ALSA) sound devices are accessible in the chroot. This is extremely important for flash!

That completes the setup that the super-user needs to do. After this, luser can go ahead and play in this newly created “sand-box”.

All play and no work

The authorized user luser can execute a shell by running

schroot -p -c lenny

The shell will normally run in an environment where the users home directories, /tmp and /dev will be mounted from the base system. Hence it should be possible to execute commands with need the X window environment as well. (The -p option given above is required to preserve the environment that includes the DISPLAY variable).

Another way to run a command like iceweasel directly is

schroot -p -c lenny iceweasel

Note that each such command creates a new schroot “session”. To re-use an already created session, you must save the session identifier and use it. For example, you can start a new session, without any command:

SCHROOT_SESSION=$(schroot -b -c lenny)

If you then issue the command

schroot -p --chroot $(SCHROOT_SESSION) iceweasel

the weasel will start up and run in that session. If you run this command another time, you will not create a new session.

Philosophy

The chroot command has existed for “eons”, but it was often felt that it is “for the super-user only”. By using schroot it becomes quite safely accessible to the regular user of the system. Using this kind of minimal virtualisation is certainly not in the same league security-wise as the “real” virtualisation techniques but has no overhead (except disk-usage); I hope the article demonstrates that schroot is at least as easy to set up.

Using schroot is a good solution to the frequently asked question:

How do I run the late-esht version of <name your favourite rapidly developing application> on Debian?

In my opinion, the above solution is to be preferred over running a mixed stable/testing version of Debian. Even backports are slightly worse, as a mixed stable/backports environment is not what the packages are being tested in by most developers.

The motivation to write this article came from a discussion I saw in the letters to Linux Weekly News where people said that Debian was hindering those who want to run 32-bit programs on a 64-bit system. If indeed mixed library setups are well packaged and maintained, then that is easier than the solution herein. All the same, Debian does have this solution!

The title of this article is inspired by John Archibald Wheeler, one of the most fascinating physicists of the 20th century, who passed away recently.

1

This is a dyslexic acronym for “Lookout — Advanced Experimental Usage”

2

Mark Shuttleworth: Benevolent Dictator For Life for Ubuntu

3

or boys as the case may be

Talkback: Discuss this article with The Answer Gang

Kapil Hari Paranjape has been a ``hack''-er since his punch-card days. Specifically, this means that he has never written a ``real'' program. He has merely tinkered with programs written by others. After playing with Minix in 1990-91 he thought of writing his first program---a ``genuine'' *nix kernel for the x86 class of machines. Luckily for him a certain L. Torvalds got there first---thereby saving him the trouble (once again) of actually writing code. In eternal gratitude he has spent a lot of time tinkering with and promoting Linux and GNU since those days---much to the dismay of many around him who think he should concentrate on mathematical research---which is his paying job. The interplay between actual running programs, what can be computed in principle and what can be shown to exist continues to fascinate him.

Lockpicking

By Aurelian Melinte

"When two trains approach each other at a crossing, both shall come to a
full stop and neither shall start up again until the other has gone."
 -- old Kansas statute

Overview

The dynamic linker allows us to override functions exported by the shared objects that are used by programs. In this article, we will use this interposition functionality and build a library that wraps around the 'pthreads' library to diagnose mutex-related problems, including the well-known deadlock.

Deadlocks and how to avoid them

Simply put, a deadlock is a run-time condition where threads are waiting for resources in a circular chain. The well-known solution to this is to assign a process-wide order to these resources and have each thread acquire the resources it needs in that particular order; threads should release the taken resources in reverse order.

To keep things simple for our debugging library, let's assume the resources that threads are competing for are all mutexes. But the techniques presented here can be extended to other type of resources threads would be competing for (e.g. semaphores, condition variables).

With pthreads, a thread can also self-deadlock if it attempts to lock a mutex it already owns. The ptreads library can help with self-lock: when creating the mutex, you can create it:

• As type PTHREAD_MUTEX_ERRORCHECK. A thread attempting to relock this type of mutex without first unlocking it will get an error. A thread attempting to unlock a mutex which another thread has locked will get an error. A thread attempting to unlock an unlocked mutex will also get an error.
• As type PTHREAD_MUTEX_RECURSIVE. A thread attempting to relock this mutex without first unlocking it will succeed in locking the mutex. A thread attempting to unlock a mutex which another thread has locked will get an error. A thread attempting to unlock an unlocked mutex will still get an error.

One certainly should understand exactly why and whether a particular thread needs to multiple lock the mutex before making usage of such techniques.

Library interposition

The dynamic linker allows users to intercept any function call an application makes to any shared library that the application uses. The dynamic linker will first load libraries specified in the LD_PRELOAD environment variable (the interposition libraries) and the linker will use these libraries before any other when it resolves symbols. This functionality offered by the linker is used traditionally for debugging purposes, just like we will do shortly.

Functions exported by the interposition library will get called instead of the functions in the shared objects (here pthreads) that the application normally uses. Thus, in the interposition library we can wrap around the "real" functions; or replace them outright.

Hooking pthreads

We need to know which thread acquires and releases what and when. At a minimum, we have to hook pthread_mutex_lock(), pthread_mutex_trylock() and pthread_mutex_unlock(). Other candidates might be pthread_mutex_init(), pthread_mutex_destroy(), pthread_cond_timedwait(), pthread_cond_wait(). But since we decided to tackle only mutexes, the first three might well be enough, depending on your strategy to communicate the resources' acquiring order to the debug library. Also, hooking pthread_mutex_destroy() is worthwhile because attempting to destroy a locked mutex is undefined behavior according to the standard (aside from being a programming error).

Below is the code used to hook our mutex functions, stripped of error checking code. We use the dlsym() function to dig out the real function, so that we can call it from within our wrapper function. The hooking function in the interposition library looks like this:

    #define _GNU_SOURCE
    #include <dlfcn.h>

    typedef int (*lp_pthread_mutex_func)(pthread_mutex_t *mutex); 

    int lp_hookfunc(lp_pthread_mutex_func *fptr, const char *fname) 
    {
        char *msg = NULL; 

        *fptr = dlsym(RTLD_NEXT, fname); 
        if ((*fptr == NULL) || ((msg = dlerror()) != NULL)) {
            lp_report("dlsym %s failed : %s\n", fname, msg); 
            return -1; 
        } else {
            return 0; 
        }
    }

We'll use it to hook the pthreads-exported functions we need. This will be done as soon as the linker loads the debugging library, well before the main() function of the application gets called:

    static lp_pthread_mutex_func next_pthread_mutex_lock = NULL; 
    static lp_pthread_mutex_func next_pthread_mutex_trylock = NULL; 
    static lp_pthread_mutex_func next_pthread_mutex_unlock = NULL; 

    void _init()
    {
        lp_hookfunc(&next_pthread_mutex_lock,    "pthread_mutex_lock"); 
        lp_hookfunc(&next_pthread_mutex_trylock, "pthread_mutex_trylock"); 
        lp_hookfunc(&next_pthread_mutex_unlock,  "pthread_mutex_unlock"); 
        /*And check for errors.*/
    }

Next, we'll compile the code in a shared object:

    $ gcc -g -Wall -fPIC -I. -c lp_hooks.c -o lp_hooks.o
    $ ld -o liblockpick.so -shared lp_hooks.c 

Finally, we load this diagnostic shared object so that it takes over pthreads. At a Bourne shell prompt:

    $ LD_PRELOAD=./liblockpick.so ./deadlock

Now, each time the application will call pthreads_mutex_lock(), the overriding/wrapping pthreads_mutex_lock() function from the debugging library will be called instead of the real one. Having taken control over the locking and unlocking functions, we can start keeping track of what is going on. The overriding function looks like this:

    int pthread_mutex_lock(pthread_mutex_t *mutex)
    {
        int rc = EINVAL;

        if (mutex != NULL) {
            lp_lock_precheck(mutex);

            /* Call the real pthread_mutex_lock() */
            rc = next_pthread_mutex_lock(mutex);

            lp_lock_postcheck(mutex, rc);
        } else {
            lp_report("%s(): mutex* is NULL.\n", __FUNCTION__ );
        }

        return rc;
    }

Resource ordering & bookkeeping

One important issue to consider is how to pass the resource-ordering information down to the debug library. There are various solutions that would require making the application aware of the debugging library and thus would require modifications of the application itself. These solutions would eventually make the library less generic too.

This resource-ordering information is not needed to diagnose deadlock situations, but we need it if we want to check for out-of-order locking issues.

The code accompanying this article is strictly application-agnostic: the order given to mutexes is the order in which the library finds out about the existence of these mutexes. The mutex to be acquired first is the first one it finds out about. Practical for exemplifying this article -- we need to hook only three functions -- but maybe not so practical for a real application.

For each of the functions hooked, there are two moments when we do bookkeeping and proceed to check the sanity of the application: before the real call is made and after the real call returns. Making the most of the time before the call proceeds is important as the call might block indefinitely and thus we can detect deadlocks before actually placing the call.

Before the call, we can check:

• whether the mutex will be acquired or released out of order;
• whether the thread tries to lock a mutex it already acquired;
• or whether it tries to unlock a mutex it does not owns;

Before the call, we also store the information about which mutex the thread is trying to acquire. After the call, we keep evidence of which mutexes have been acquired or released by which thread.

    void lp_lock_precheck_diagnose(const pthread_mutex_t *mutex)
    {
        int rc = -1; 
        /*Highest ranking mutex currently acquired by 'me'*/
        int maxmidx = LP_INVALID_IDX; 
        int midx = LP_INVALID_IDX; 
        pthread_t me = pthread_self(); 

        pthread_t             thr = LP_INVALID_THREAD; 
        const pthread_mutex_t *mtx = NULL; 

        /* Thread tries to lock a mutex it has already locked? */
        if ((rc=lp_is_mutex_locked(mutex, me)) != 0) {
            lp_report("Mutex M%lx is already locked by thread.\n", mutex); 
            lp_dump();
        } 

        /* Is mutex order respected? */
        maxmidx = lp_max_idx_mutex_locked(me); 
        midx = lp_mutex_idx(mutex); 
        if (midx < maxmidx) {
            lp_report("Mutex M%lx will be locked out of order (idx=%d, crt max=%d).\n", 
                    mutex, midx, maxmidx); 
            lp_dump();
        }

        /* Will deadlock? Check for a circular chain. */
        lp_report("Checking if it will deadlock...\n");
        thr = me; 
        mtx = mutex; 
        while ((thr=lp_thread_owning(mtx)) != LP_INVALID_THREAD) {
            lp_report("  Mutex M%lx is owned by T%lx.\n", mtx, thr); 
            mtx = lp_mutex_wanted(thr); 
            lp_report("  Thread T%lx waits for M%lx.\n", thr, mtx); 

            if (mtx == LP_INVALID_MUTEX) 
                break; /*no circular dead; */

            if (0 != pthread_equal(thr, me)) {
                lp_report("  Deadlock condition detected.\n");
                lp_dump();
                break; 
            }
        }
    }

    void lp_unlock_precheck_diagnose(const pthread_mutex_t *mutex) 
    {
        int rc = -1; 
        int maxmidx = LP_INVALID_IDX, midx = LP_INVALID_IDX; 

        /* Thread tries to unlock a mutex it does not have? */
        if ((rc=lp_is_mutex_locked(mutex, pthread_self())) == 0) {
            lp_report("Attempt to release M%lx NOT locked by thread.\n", mutex); 
            lp_dump();
        } 

        /* Are mutexes released in reverse order? */
        maxmidx = lp_max_idx_mutex_locked(pthread_self());
        midx = lp_mutex_idx(mutex); 
        if (midx < maxmidx) {
            lp_report("Mutex M%lx will be released out of order (idx=%d, crt max=%d).\n",
                    mutex, midx, maxmidx); 
            lp_dump();
        }
    }

A sample of the results obtained at run-time can be seen below:

$ make test

...
Tx40571bb0: Mutex M8049e48 will be locked out of order (idx=1, crt max=2). 
...
Tx40571bb0: Checking if it will deadlock... 
Tx40571bb0:   Mutex M8049e48 is owned by T40370bb0. 
Tx40571bb0:   Thread T40370bb0 waits for M8049e30. 
Tx40571bb0:   Mutex M8049e30 is owned by T40571bb0. 
Tx40571bb0:   Thread T40571bb0 waits for M8049e48. 
Tx40571bb0:   Deadlock condition detected. 
Tx40571bb0: Mutexes: 
Tx40571bb0:   [00] M4001d180 [owned by:T40571bb0 02] 
Tx40571bb0:   [01] M08049e48 [owned by:T40370bb0 01] 
Tx40571bb0:   [02] M08049e30 [owned by:T40571bb0 02] 
Tx40571bb0:   [03] M08049e60 [owned by:T40772bb0 03] 
Tx40571bb0: Threads: 
Tx40571bb0:   [00] T4016f6c0 [owns:M0000000000000000000000000000000][wants:M00000000] 
Tx40571bb0:   [01] T40370bb0 [owns:M0100000000000000000000000000000][wants:M08049e30] 
Tx40571bb0:   [02] T40571bb0 [owns:M1010000000000000000000000000000][wants:M08049e48] 
Tx40571bb0:   [03] T40772bb0 [owns:M0001000000000000000000000000000][wants:M00000000] 
...
Tx40772bb0: Mutex M8049e60 is already locked by thread. 

On Heisenberg's uncertainty principle

A word of caution: the Heisenberg's uncertainty principle, in its general form, applies to software! In plain words, Heisenberg stated that observation of a system does modify the behavior of the observed system. In our case, the debug library affects the behavior of the application in at least two ways:

• First, there is now an extra mutex protecting the data collected by the debug library. This mutex has to be acquired now by the observed threads and thus comes to play into the deadlock game.
• Secondly, bookkeeping takes time, no matter how fast and smart are the data structures at the core of the library. And these extra times will affect the way threads interact.

Thus, it might happen that the debug library makes it easier to reproduce the deadlock. Or, on the contrary, it may make it harder to reproduce.

Resources

• The dlsym() man page, The Open Group Base Specifications Issue 6
• Pthreads, The Open Group Base Specifications Issue 6
• Lock hierarchies (DDJ).

Talkback: Discuss this article with The Answer Gang

Searching for Text (Part II)

By René Pfeiffer

In the first part of this series, we took a look at various methods that help us to index text and to facilitate text searches. The approaches were focused on SQL databases. Provided you don't get nervous about abandoning SQL queries, I can show you another way to build an index and to query it for your favourite phrase. Prepare to meet Lucene!

Collecting and Converting Files

I've already said that organising your data is of premium importance. This starts with a smart directory structure (or database design) and ends with proper document summaries and tagging data or having sensible categories. No matter how you store your files, if you start to build an index you need to collect them somehow. A lot of documents are stored in directory trees (on file servers for example); some servers hold thousands or millions of files. Indexing this much documentation requires a good strategy - e.g., you don't want to keep rebuilding indexes for data that hasn't changed.

When it comes down to indexing you need plain text; this means you have to convert your file content. Since there is no such thing as plain text anymore, we also have to think about the character encoding. Will it be sufficient to use 7-bit US ASCII? Do we use ISO-8859-1 or even ISO-8859-15 because we'd like to have the Euro sign? It might be a good idea to use UTF-8. It is a variable-length character encoding for Unicode that 'feels' like 7-bit US ASCII as long as no special character is encountered and has some extra room for a lot of strange letters. This is the reason why a lot of people use it. Be aware that if you use an Unicode encoding in one place, you should use it everywhere. It makes life a lot easier.

When looking at a pile of a hundred thousand files on a server you probably don't want all of them; the ones you want either contain text or can be converted into text. A picture might contain text fields (such as JPEG EXIF tags or PNG iTXt/tEXt chunks), but often it is useful to stick to text file formats. How do we choose? Well, the easiest way is to look for the file extension. This isn't very accurate since the extensions ".jpg" and ".jpeg" both most certainly mark JPEG files, but determining a file type by analysing its content is not easy (the GNU tool file does a good job if you choose to go that way.) For now we will stick to extension for the sake of simplicity.

So what do we do now? Well, we follow the UNIX tradition and split the problem into smaller, manageable parts. First we do a little walking and collecting - a filesystem walk to be exact. We need to create a list of interesting files for later indexing. Let's start with the syntax of the config file that lists all interesting extensions.

extension = [ ".doc", ".htm", ".html", ".odp", ".ods", ".odt", ".pdf", ".txt", ".xls" ];
nice      = 13;
output    = "filelist.txt";

The config option extension lists all interesting extensions. nice is the nice level for your program. A filesystem walk creates more than enough load, so we'll let other processes have the CPU first. output tells our program where to write the list of interesing files. It will be written to the file filelist.txt. The syntax of the config file originates from the C/C++ library libconfig. It parses the file for use and extracts the options. The parser isn't that difficult to write, but we have some more complex tasks ahead. Later in the article, I'll provide a link to the source code for filelist, the helper code files helper.h and helper.cc, the command line options configuration filelist.ggo and a Makefile to build everything. You will need the Boost filesystem and iostreams libraries as well as the gengetopt skeleton generator from the GNU project. gengetopt and especially the Boost libraries are extremely useful to make your life in C/C++ a lot easier.

Lucene and its Ports

The Apache Lucene project is a collection of tools for building software that features search functions. The core component is Lucene itself, which is a Java-based indexing and search library. The project offers additional code to build web search applications and dealing with metadata. Lucene has been ported to Perl (PLucene), Python (PyLucene), C (Lucy), and C++ (CLucene), so you can access a Lucene index with your favourite tool chain. The Java code also contains classes that facilitate the import of documents by using stream classes. Creating and maintaining an index is very easy, and the index format can be used across different platforms and accessed by various ports of Lucene without the need for conversion.

We'll focus on the C++ port of Lucene in order to index our documents.

Indexing with CLucene

CLucene introduces some concepts that should be understood before using the API. Every object that is to be indexed is called a document; in an ideal word this document consists of pure text to make the work of the indexer easier. CLucene analyses the content and uses different algorithms to extract useful words or tokens from the indexed document. The analysers are classes of their own. They contain the algorithms such as dividing by white spaces, using stop words, replacing special characters, and other methods.

Every document can be described by a list of fields with content. The fields can have arbitrary names. These names are used to access the content of the field, very similar to associative arrays or hashes.

Field      | Content
-----------+-------------------------------
title      | My notes from the conference
author     | R. Pfeiffer
content    | Lots and lots of text, ...
timestamp  | 1207958005
type       | UTF-8 Unicode English text
...        |

This means that the document is a container for all kinds of information added by the fields. It is not necessary to put the whole content into the index, but it helps if your aim is to have a full text search. It is also useful to pack additional metadata into the document. The search covers all fields or a selection of them, so that you can tune your search later in order to reduce the number of hits.

The index as a whole resides in a directory and is completely managed by CLucene; there are no user-serviceable parts, all access is done through CLucene (or one of the other ports). Moving the index around is as easy as copying a file. You can also copy the index directory between different platforms and it should work without conversion.

The CLucene library works with Unicode. The strings are typed with wchar_t *, which means they consist of wide character literals (marked with a L such as L'x'). Therefore I suggest that you use Unicode when interfacing with CLucene.

Simple Indexing Strategy

How do we get from our list of interesting files to a properly-filled index? Simple, we need a little strategy.

1. Walk through the list of files one by one
2. Check if the file has been modified since the last indexing run
3. Determine file type (by extension or more elaborate means)
4. Convert file to plain text with a given character encoding
5. Add file name, content, timestamps, file type, etc. to index

Now we know what our next piece of code should do. I will describe some important aspects of the task.

Converting Documents to Plain Text

I've already said that we need text; this means we have to convert PDF, PostScript, and anything that is not text into text. We should also ensure that the end result is suitably Unicode encoded (UTF-8 for example). In order to avoid doing this in our C++ code, we rely on external programs. This isn't very elegant, but it helps to maintain a flexible approach to future data formats that we wish to index. The external helper tools are defined by file extension in a configuration file.

// List of known file extensions and their converters to
// plain text
pdf=(pdftotext -q -eol unix -enc UTF-8 $IN - > $OUT)
ps=(pstotext $IN | iconv -f ISO-8859-1 -t UTF-8 -o $OUT -)
doc=(antiword $IN > $OUT)
html=(html2text -nobs -o $OUT $IN)
htm=(html2text -nobs -o $OUT $IN)
odp=(ooo_as_text $IN > $OUT)
ods=(ooo_as_text $IN > $OUT)
odt=(ooo_as_text $IN > $OUT)
php=(html2text -nobs -o $OUT $IN)
rtf=(unrtf --nopict --text $IN > $OUT)
txt=(cat $IN > $OUT)
xls=(py_xls2txt $IN > $OUT)
xml=(cat $IN > $OUT)

$IN is the file to be converted. $OUT is the output file, which will be a temporary file. The extension is to the left of the equal sign. The commands inside the brackets will be executed by the indexer prior to feeding the content to the CLucene index. An alternative would be to use classes that understand file types, read them, and convert them; the Strigi project has code for this. They call it JStreams. JStreams provide a standardised interface for accessing the contents of different file types. The approach with external tools is a bit more generic. Note that all OpenOffice document formats can be converted with a single tool (ooo_as_text is still not released but part of the OOoPy project. Kindly ask the author of OOoPy for a copy if it's not contained in the downloads).

The configuration file is parsed by a parser generated with Boost's Spirit library. The whole parser is defined by using templates. struct filter_grammar in helper.cc contains the full rules for the parser. Once you understand how the templates work the Spirit library is a convenient way to build your own parsers.

Maintaining a Database of Timestamps

When you consider the documents stored on a file server, you will realise that most of them don't change very often. Maybe some change more often when someone works on them. Once you have a kind of "library", most of the documents will stay the same. So it's a good idea to keep track of the modification time of the files. By doing this we can decide to update the document depending on its last modification timestamp. This saves a lot of I/O when indexing a large collection of documents.

The indexer keeps track of the timestamps by a SQLite database. The table could also be replaced by a hash, but I wanted to do more in SQL and then didn't do it. indexer.cc has the creation statement.

CREATE TABLE IF NOT EXISTS fileaccess ( filename TEXT PRIMARY KEY, mtime INT(8) )

SQL is overkill for that, but that's why we use SQLite. We use a transaction during the whole indexing process. The changes are not committed unless all documents are indexed without error.

Creating and Writing to a CLucene Index

Before you can do anything with an index you have to open it. This is very similar to other resources such as files or sockets. However, before you can open or create it, you have to select an analyser object. This analyser defines how you wish to process the data fed to the index. Since we don't know exactly what we are indexing, we'll use a whitespace analyser. Then we open the index.

// This is the analyser we use.
WhitespaceAnalyzer analyser;

// Initialise CLucene index writer
IndexWriter::IndexWriter index_repository( args_info.index_arg, &analyser, new_index, true );

args_info.index_arg is a string containing the directory where the index will live. &analyser is our analyser and new_index is a boolean flag that indicates whether we open an existing index or create a new one. The last argument is described with closeDir in CLucene's Doxygen documentation.

The index can now be filled with document objects. A CLucene document is simply a container for fields describing something.

// Fields we want to put into the index
lucene::document::Field *field_filename;
lucene::document::Field *field_file_content;
lucene::document::Field *field_mtime;
lucene::document::Field *field_type;
...

 // Create Lucene document for adding to index directory.
 file_document = new lucene::document::Document;
 // Add fields to document object.
 if ( file_has_content ) {
     file_document->add( *field_file_content );
 }
 file_document->add( *field_filename );
 file_document->add( *field_mtime );
 file_document->add( *field_type );
 index_repository.addDocument( file_document, &analyser );
...

You can add as many fields as you like. Adding metadata is a good idea since you probably don't want to search for content all of the time. The addDocument() method adds the document to the index repository.

Since CLucene maintains the index repository, it's also a good idea to call the optimise method if you change a lot of data.

// Optimising the index should be done after there were changes to the index.
index_repository.optimize();

// Close index.
index_repository.close();

That's the short introduction to CLucene, and these are the basics you need to get started. The library can do many more things.

Reading from a CLucene index

We won't read from the index (yet), but reading is as easy as writing. You open the index repository, send search queries and retrieve documents. A fragment of code performing a search and retrieving all hits looks like this.

using namespace lucene::index;
using namespace lucene::analysis;
using namespace lucene::util;
using namespace lucene::store;
using namespace lucene::document;
using namespace lucene::search;
using namespace lucene::queryParser;

wstring search_string = L"Where is it?";

lucene::index::IndexReader    *index_reader;
lucene::search::IndexSearcher *index_searcher;
Query                         *index_query;
Hits                          *index_hits;
WhitespaceAnalyzer            analyser;

index_reader   = IndexReader::open( args_info.index_arg );
index_searcher = new IndexSearcher(index_reader);
index_query    = QueryParser::parse( search_string.c_str(), L"content", &analyser );
index_hits     = index_searcher->search(index_query);
if ( index_hits->length() > 0 ) {
    for( long i=0; i < index_hits->length(); i++ ) {
        Document &doc = index_hits->doc(i);
        wcout << "FOUND: " << doc.get(L"filename") << endl;
    }
}

delete index_hits;
delete index_query;

index_reader->close();
delete index_searcher;

It is important to use the same analyser as the indexing process did. In our case this is the WhitespaceAnalyzer again. IndexReader::open() opens the index, QueryParser::parse() performs the search, and CLucene returns Query objects whose content, the Document objects, can be retrieved. As you can see, all strings are wide strings, so using Unicode really is important.

If you do some debugging with your created CLucene indices, you might want to try Luke, the Lucene Index Toolbox. It provides a Java tool that can display the contents of an index. You can browse the documents, look at the fields, and perform search queries.

The Code

Since this article is already much longer than anticipated, I'll just provide a link to the complete tar archive of all the code I've shown above. It also contains a Makefile to facilitate compiling. I used the GCC/G++ 4.1.2 for development (and I'd like to see what Intel's compiler says about my code). If you use a Debian system you will need the following packages:

• gengetopt
• libboost-filesystem-dev
• libboost-iostreams-dev
• libboost-regex-dev

I compiled SQLite, libconfig++, and CLucene from source, because I wasn't happy with the packages in Debian Etch. The newer SQLite version is especially interesting, since its provides a new API to some functions (marked with ..._v2()). If you want to compile the Boost library as well, you have to add the path to its include files (which is /usr/local/include/boost-1_35 for the current version if installed from source). Since the Boost libraries consist mainly of templates, the compile process is fairly short despite the size of Boost's distribution.

Test Run with a little "Benchmark"

And now for the final question: Why all the fuss? How fast is it? Do we need to care which port we use? I can't answer these questions. All I can do is run the indexer over a list of files (this is not a benchmark, and has no statistical significance.) The directory looks like this:

rpfeiffer@miranda:/nfs/Bibliothek$ du -h --max-depth=1
1.3M    ./Lyrics
703M    ./Security
0       ./Biometrie
16M     ./Sysadmin
172K    ./Misc
403M    ./Teaching
12M     ./Programming
57M     ./Hardware
3.9M    ./Reports
7.3M    ./Networks
92K     ./Chaos
32K     ./Gfx
12K     ./UTF-8
23M     ./VoIP
1.8M    ./Science
1.2M    ./Manuals
1.2G    .
rpfeiffer@miranda:/nfs/Bibliothek$ 

The documents reside on our file server in the office and are accessed via NFSv3 and Gigabit Ethernet. The machine running the indexer is a Core2 Duo with 2.13 GHz and 2 GB RAM. filelist counts 539 interesting documents (by using the extensions I listed earlier). Let's try to run the indexer and create a new index. Note that the directory was cached in part due to my du command and that the 539 documents may be less than the 1.2 GB because we only look for specific file extensions.

rpfeiffer@miranda:~/code$ time ./indexer -c ./indexer.cfg -i /var/tmp/i -n 1 -l ./filelist.txt 

real    1m48.767s
user    1m12.337s
sys     0m17.849s
rpfeiffer@miranda:~/code$ 

The index looks like this:

rpfeiffer@miranda:/var/tmp/i$ ls -lh
total 5.2M
-rwxr-xr-x 1 rpfeiffer rpfeiffer    4 2008-04-18 23:56 deletable
-rwxr-xr-x 1 rpfeiffer rpfeiffer 5.2M 2008-04-18 23:56 _gk.cfs
-rwxr-xr-x 1 rpfeiffer rpfeiffer   28 2008-04-18 23:56 segments
rpfeiffer@miranda:/var/tmp/i$ 

So, the indexer did something and stored something to disk. An inspection with Luke shows familiar documents and content.

A Word about Alpha Code and Bugs

Please be aware that the code shown in this article is of alpha quality. The source contains some dead code and still needs some improvement (especially the execution of the external helper binaries). So far, it works, and it doesn't segfault that often anymore - but that's about it. It is built on solid and stable libraries, but it isn't meant to be in production status (yet). It's also a bit messy and should be cleaned up, because it took me a while to understand the libraries that I used. If you have suggestions, just send patches - that's what the GPL is for. If you have no code but a lot of good ideas, let's hear them! Preferably in an article for one of our next issues.

Useful resources

• Boost C++ libraries
• CLucene - a C++ port of Lucene.
• gengetopt
• libconfig - a C/C++ configuration file library.
• Lucene - provides Java-based indexing and search technology.
• Luke - Lucene Index Toolbox.
• SQLite - a lightweight database library.
• Strigi - a desktop and indexer independent desktop search engine.

Talkback: Discuss this article with The Answer Gang

Joey's Notes: Guide to adding a new partition or drive to an existing system

By Joey Prestia

The basic steps involved in this process are:

1. Determine what partitions need to be created and where.
2. Create the partitions (I use fdisk here but any Linux disk partitioning tool should work)
3. Re-read the partition table either with partprobe or by a reboot
4. Make a filesystem on the partition, label it, and create the necessary mount points
5. Add the appropriate entries to /etc/fstab so the partitions are mounted upon reboot

Scenario

Imagine that we have a server running RHEL 4 and our supervisor comes over and wants a 10 GB partition created for the data processing department. This is in addition to what the server currently has allocated, so we can either create a partition out of unpartitioned space on the existing disk (experienced Linux system administrators will leave unpartitioned disk space for future expansion) if available, or we can add another drive. This scenario actually happens quite frequently in the production world, so this is a valuable skill to have even if you administer nothing more than your home machines.

We'll assume that your supervisor has given you the latitude of deciding which of the above options you'll use, so your first task is to check to see if space is available on your existing media. We'll run "fdisk -l" to see the size of the disk; the data we need is on the first line of output.

[root@station17 ~]# fdisk -l

Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          38      305203+  83  Linux
/dev/sda2              39        7687    61440592+  83  Linux
/dev/sda3            7688        7942     2048287+  82  Linux swap

From this we can see the size of our drive is 80.0 GB. Now, we'll use "df -h" to calculate the size of the partitions that are on our system. We only need to be concerned with the rows that have a device label, the others (labeled with "none") don't concern us. The column labeled "Size" has the numbers we'll need to add up to get a overall size.

[root@station17 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              58G  6.5G   49G  12%   /
/dev/sda1             289M   17M  258M   6%   /boot
none                 1013M     0 1013M   0%   /dev/shm

[root@station17 ~]# 

From this, we can see that /dev/sda2 is 58G and /dev/sda1 is 289M - a total of 58.3GB. Now we need to add in our swap size; "cat /proc/swaps" will tell us what size our swap partition is.

[ If you feel like using an actual system utility for this, "swapon -s" will do the same thing. -- Ben ]

[root@station17 ~]# cat /proc/swaps 
Filename                              Type            Size    Used    Priority
/dev/sda3                             partition       2048276    0       -1
[root@station17 ~]# 

Adding in the 2GB from this means that we have 19.7 GB to work with - well over what we need. Now, let's move on to creating our partition: "fdisk /dev/sda" will open our drive's partition table for modification. Since we're already using 3 partitions on the drive, we'll have to make our 4th one an extended one - a container to house any additional partitions, including the one we are creating now. We'll want to accept the defaults on this extended partition, which will make the whole rest of the drive available for our new partitions. We'll be using an ext3 filesystem, so we also need to keep this in mind: the "mkfs" command reserves 5% of the blocks for root. Given all that, we'll make our new partition 11.5GB to compensate for the blocks reserved for root plus a little extra.

[root@station17 ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 9726.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
e
Selected partition 4
First cylinder (7943-9726, default 7943): 
Using default value 7943
Last cylinder or +size or +sizeM or +sizeK (7943-9726, default 9726): 
Using default value 9726

Here you can see where I selected "n" for a new partition and "e" to make a extended partition. I then accepted the defaults for both the starting cylinder and again for the ending cylinder.

Command (m for help): n
First cylinder (7943-9726, default 7943): 
Using default value 7943
Last cylinder or +size or +sizeM or +sizeK (7943-9726, default 9726):.+11500M

Next, I hit "n" to create a new partition; then, when prompted to use a starting cylinder, I hit 'enter' to accept the default. For the ending cylinder I entered "+11500M" to specify the size. The plus is important - without it, you will get an error. Its a good idea to hit "p" to at this point to get "fdisk" to print the partition table. This will show what we have done before saving our changes.

Command (m for help):.p

Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          38      305203+  83  Linux
/dev/sda2              39        7687    61440592+  83  Linux
/dev/sda3            7688        7942     2048287+  82  Linux swap
/dev/sda4            7943        9726    14329980    5  Extended
/dev/sda5            7943        9341    11237436   83  Linux

If there are any mistakes just quit "fdisk" with a "q" and no changes will be saved. This looks right - so lets write our changes with a "w".

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@station17 ~]# 

This warning can be remedied by using the 'partprobe' command to force the kernel to reread the partition table. Remember - if this were a production machine, we wouldn't want to have to reboot it.

[root@station17 ~]# partprobe

At this point our 11.5G partition is /dev/sda5 and raw - it has neither a file system nor a label descriptor - so let's format it and give it a label. Giving the partition a label can be done at the same time that the file system is being created with the -L option, but I prefer to do it in a separate step.

[root@station17 ~]# mkfs.ext3 /dev/sda5
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1406272 inodes, 2809359 blocks
140467 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2877292544
86 block groups
32768 blocks per group, 32768 fragments per group
16352 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@station17 ~]# 

Now we'll give it label ("/data"):

[root@station17 ~]# e2label /dev/sda5 /data
[root@station17 ~]# 

Next, we need to create a mount point in our filesystem and make sure that it's mounted at boot time. Let's create a directory on our system called /data.

[ The usual method of allocating new space is often much more complex than that - at least in the planning stages. In fact, creating a non-standard directory name in the root of the filesystem as suggested here is incorrect and violates the Filesystem Hierarchy Standard (FHS). As an example of a more typical situation, if an administrator finds that a shared machine's drive is running out of room, he may first examine the machine to see where the most activity/space consumption is occuring. Assuming that it's in the space assigned to users (i.e., "/home"), he would most likely back up the data in that subdirectory, restore it to the newly-created partition, delete "/home", and mount the new partition as "/home". This would recover all the space used by the original "/home" and leave it available for the rest of the system to use - and most users would not even realize that any change had been made. This approach doesn't require rebooting the machine either. -- Ben ]

[ I do understand that the partitioning is inconsistent with the FHS, but our RedHat course materials do instruct us to create directories in / for simplicity and ease in the aid backups. We are also led by instruction to do things such as specialized partitioning schemes for different things this way here at the RedHat academy. -- Joey ]

[root@station17 ~]# mkdir /data
[root@station17 ~]# 

Now we put it in the file system table, '/etc/fstab', so it gets mounted on every boot.

[root@station17 ~]# vi /etc/fstab

# This file is edited by fstab-sync - see 'man fstab-sync' for details
LABEL=/                /                       ext3    defaults        1 1
LABEL=/data            /data                   ext3    defaults        1 1
LABEL=/boot            /boot                   ext3    defaults        1 2
none                   /dev/pts                devpts  gid=5,mode=620  0 0
none                   /dev/shm                tmpfs   defaults        0 0
none                   /proc                   proc    defaults        0 0
none                   /sys                    sysfs   defaults        0 0
LABEL=SWAP-sda3         swap                   swap    defaults        0 0
/dev/scd0   /media/cdrecorder   auto    pamconsole,exec,noauto,managed 0 0

I used the root partition as a guide in this sample. The label is in the first column, the mount point is in the second, then we have the file system type and the mount options. The last two numbers are the dump indicator and the fsck indicator; they determine when the system gets backed up if you're using 'dump', and when the system gets checked for errors. Basically, you can copy these numbers and options just as I have. Write your changes and exit the editor. Then, to make sure that there were no errors, run "mount -a" to mount all the partitions listed in /etc/fstab. Any errors would be reported at this point.

[root@station17 ~]# mount -a

Since we didn't get any errors, let's do a "df -h" and see how everything looks.

[root@station17 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              58G  6.6G   49G  12% /
/dev/sda1             289M   17M  258M   6% /boot
none                 1013M     0 1013M   0% /dev/shm
/dev/sda5              11G   59M   10G   1% /data

[root@station17 ~]# 

That's it - we are now ready to start using this new partition, keeping in mind we may have to modify permissions as needed for our users and groups. This is a very common task, one that all Linux users should become familiar with because you will almost certainly be faced with needing more room. This process is very similar to adding another disk - you would simply substitute your device labels as required.

Talkback: Discuss this article with The Answer Gang

Hands-on Linux Software RAID

By Amit Kumar Saha

What is Software RAID?

Software RAID is RAID implemented with software - no additional hardware such as a RAID controller is needed. Thus, software RAID is a good starting point to start getting some hands-on RAID experience. Also, software RAID is independent of proprietary management software - maintaining a software RAID works the same way on all machines that run Linux. However, there is something to think about too: when considering software RAID, think about performance. All RAID algorithms are done by the system CPU and every block has to be copied over the system's data bus (i.e. sda1 <-> IO controller <-> RAM, possibly CPU <-> IO controller <-> sdb1). (Thanks to René Pfeiffer of the Answer Gang for pointing that out.)

Enabling your Kernel to support RAID

I am using Ubuntu 7.10 with the "stock" kernel for my experiments. The test machine has an 80GB SATA HDD.

First, check whether the RAID support is enabled in your kernel:

cat /proc/mdtstat

If you get a message saying:

cat: /proc/mdstat: No such file or directory

then you need to enable RAID support. There are two possiblities:

1. RAID support was disabled while compiling the kernel and you will have to recompile it
2. You will have to insert the multiple disk (md) support module manually. Check whether the "md*" modules exist under /lib/modules/$(uname -r)/kernel/drivers/md/ and insert the module as follows:

   $ sudo modprobe md-mod 
   Password:
   

   (Thanks to Kapil of the Answer Gang for this one)

Now, you can verify whether RAID support is active:

amit@amit-desktop:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
unused devices: 

This means that now we have RAID support in the kernel.

Tools for manipulating RAID arrays

Now that you have got a RAID enabled Kernel, you will need to use some user-space tools to help you out to play with RAID.

Slightly outdated 'raidtools' and the newer, better 'mdadm' are the tools available to you. My focus in this article will be on 'mdadm'. For more information on using 'raidtools' and a comparison of the two, please refer to the How-To mentioned in the References.

Installing 'mdadm'

amit@amit-desktop:~$ sudo apt-get install mdadm
Reading package lists... Done
Building dependency tree  
Reading state information... Done
Recommended packages:
mail-transport-agent
The following NEW packages will be installed:
mdadm
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 220kB of archives.
After unpacking 627kB of additional disk space will be used.
Get:1 http://in.archive.ubuntu.com gutsy/main mdadm 2.6.2-1ubuntu2 [220kB]
Fetched 220kB in 48s (4515B/s)                                            
Preconfiguring packages ...
Selecting previously deselected package mdadm.
(Reading database ... 88932 files and directories currently installed.)
Unpacking mdadm (from .../mdadm_2.6.2-1ubuntu2_i386.deb) ...
Setting up mdadm (2.6.2-1ubuntu2) ...
Generating array device nodes... done.
Generating mdadm.conf... done.
Removing any system startup links for /etc/init.d/mdadm-raid ...
update-initramfs: deferring update (trigger activated)
* Starting MD monitoring service mdadm --monitor                        [ OK ]

Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.22-14-generic

Creating a RAID device

My disk setup now is as follows:

 Name      Flags       Part Type    FS Type                 [Label]     Size (MB)
---------------------------------------------------------------------------------
sda1       Boot        Primary      NTFS                    []           20612.56      
sda5                   Logical      W95 FAT32                            20579.66
sda6                   Logical      W95 FAT32                            20587.88
sda7                   Logical      Linux ext3                           12000.69
sda8                   Logical      Linux swap / Solaris                  1019.94
sda9                   Logical      Linux                                 2048.10
sda10                  Logical      Linux                                 2048.10
sda11                  Logical      Linux                                 3446.40

I will now combine sda9 and sda10 to form one large logical device to form a RAID array. For the purpose of demonstration, and also since 0 is always a good point to start, creating a level-0 RAID is next.

[ Note the type of the partition. The Linux RAID kernel driver can automatically start a RAID device if the type of the partition is marked as 0xFD meaning "Linux RAID partition with autodetect using persistent superblock". -- René ]

Combining 2 consecutive partitions to form a RAID is not a smart thing to do, I was told by the Answer Gang. But till I find it why, I shall persist.

[ The purpose of having a RAID is to distribute the I/O load of any read/write operations over multiple disks. Hard disks are slow, and take a while to complete commands given to them. Depending on the I/O operation, a RAID will allow the system to let the disks in a RAID work in parallel. This is especially true when reading from a RAID0 or RAID1.
If you create a RAID device on the same physical device the RAID driver doesn't notice. The problem you have then is that you put the poor drive under a lot of load, since the driver now thinks it can issue a command in parallel while in reality the is no parallelism. This means that the heads of the drive will probably move a lot - and this is a bad idea as a friend of mine who does professional data recovery once explained to me.
So, it's OK to do this for educational purposes, but please, don't ever ever put live data on a production server into a RAID consisting of partitions on the same physical drive. -- René]

Creating a Level-0 RAID

amit@amit-desktop:~$ sudo mdadm --create --verbose /dev/md0 --level=0 --raid-devices=2 /dev/sda9 /dev/sda10
[sudo] password for amit:
mdadm: chunk size defaults to 64K
mdadm: array /dev/md0 started.

Let us now check the RAID array we just created:

amit@amit-desktop:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid0 sda10[1] sda9[0]
     3999872 blocks 64k chunks
    
unused devices: 

Now, we'll create a filesystem on the new RAID device:

amit@amit-desktop:~$ sudo mkfs -t ext3 /dev/md0
[sudo] password for amit:
mke2fs 1.40.2 (12-Jul-2007)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
500960 inodes, 999968 blocks
49998 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1027604480
31 block groups
32768 blocks per group, 32768 fragments per group
16160 inodes per group
Superblock backups stored on blocks:
       32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done                           
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

amit@amit-desktop:~$ sudo mkdir /media/RAID0
amit@amit-desktop:~$ mount /dev/md0 /media/RAID0/
mount: only root can do that
amit@amit-desktop:~$ sudo mount /dev/md0 /media/RAID0/

amit@amit-desktop:~$ df

Filesystem           1K-blocks      Used Available Use% Mounted on
.
.

/dev/md0               3936940     73440   3663508   2% /media/RAID0

let us now use 'mdadm' to get some details on the RAID array:

amit@amit-desktop:~$ sudo mdadm --query /dev/md0 --detail
/dev/md0: 3.81GiB raid0 2 devices, 0 spares. Use mdadm --detail for more detail.

amit@amit-desktop:~$ sudo mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.03
 Creation Time : Tue Mar 11 13:05:22 2008
    Raid Level : raid0
    Array Size : 3999872 (3.81 GiB 4.10 GB)
  Raid Devices : 2
 Total Devices : 2
Preferred Minor : 0
   Persistence : Superblock is persistent

   Update Time : Tue Mar 11 13:05:22 2008
         State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
 Spare Devices : 0

    Chunk Size : 64K

          UUID : f77bd177:706b589c:2a7af8c6:cbd32339 (local to host amit-desktop)
        Events : 0.1

   Number   Major   Minor   RaidDevice State
      0       8        9        0      active sync   /dev/sda9
      1       8       10        1      active sync   /dev/sda10

Looking ahead

A RAID experimental bed is now ready for us. In some future articles, I shall try to share my experiments on the RAID setup. You may also consider visiting my blog posts on RAID here.

References

1. What is RAID?
2. Software RAID How-To
3. 'mdadm' manual page

Acknowledgements

Thanks to the Answer Gang (TAG) for discussions on RAID a while back. Though none of the cool suggestions has been tried by me, the next article shall have them tried, tested and appreciated. I also had the privilege to get my article "live-edited" by the Answer Gang which I believe was a limited period offer . Thanks, guys!

Talkback: Discuss this article with The Answer Gang

The author is a freelance technical writer. He mainly writes on the Linux kernel, Network Security and XML.

XKCD

By Randall Munroe

Click here to see the full-sized image

Talkback: Discuss this article with The Answer Gang

HelpDex

By Shane Collinge

These images are scaled down to minimize horizontal scrolling.

All HelpDex cartoons are at Shane's web site, www.shanecollinge.com.

Talkback: Discuss this article with The Answer Gang

Part computer programmer, part cartoonist, part Mars Bar. At night, he runs around in his brightly-coloured underwear fighting criminals. During the day... well, he just runs around in his brightly-coloured underwear. He eats when he's hungry and sleeps when he's sleepy.