
This is the Nth public release of MD, which stands for Multiple
Devices. Its main goal is to group several disks or partitions
together, making them look like a single block device.  Furthermore,
it is interface independent, so it is possible to mix IDE
(MFM/RLL/ESDI/AT-BUS), SCSI, and even old XT-like disks.

Note that Roger E. Wolff <R.E.Wolff@et.tudelft.nl> wrote a similar
thing named red. This main difference between the two projects is
that md does things much more efficiently (at the expense of some
memory, since it makes a massive use of hash tables).

**WARNING** : This is some **VERY ALPHA** software. Don't use it
unless you exactly know what you're doing. This patch works for me,
and I trust my data to it, but I do very frequent backups. So, if this
damned thing goes wild, I won't loose a lot. Please backup your own
system BEFORE using this software.

WARNING : versions 0.14 and 0.15 have a nasty bug in the way chunks
are distributed on devices (striped mode only, linear is safe).  If
you are upgrading from those versions, you won't be able to retrieve
your data. So, you MUST backup your files living on your md devices
FIRST, compile the new kernel and reboot. THEN mkfs your md devices
and reinstall your files. If your using md devices as swap devices, do
a mkswap on them.

PLEASE don't use 0.14 and 0.15 versions anymore. The bug won't crash
your system, but will give you I/O errors in some circumstances. The
best thing to do is to upgrade to 1.2.5 (at least...) and then to
apply the new md patch.

I repeat : SAVE YOUR FILES BEFORE USING POST-0.15 VERSIONS !!!

--

Theory of operations :
	The kernel patch included in the package create a new block
device driver called "md" (device major=9, which is now the official
and registered major number). The main difference between this driver
and the other ones like "hd" and "sd" (among others...) is that "md"
never access the disks itself. The trick is to requeue the requests
that the upper layer sends to the md driver back to the physical
driver. With a little help from a hash table (a friend of mine...),
you shouldn't notice too much difference about speed.

	There's currently three modes (personalities) to manage such
devices : linear, raid0, and raid1.

	Linear means that real devices are appended to each
other. This kind of device should be easily expandable (see 'Future
extensions' section), but gives no or very small speed improvement.

	Raid0 does a classic (and rather efficient) striping on
disks (i.e. contiguous blocks on the md device are spread across real
devices). It gives rather good performances on SCSI disks, specially
with concurrent disk access.

	Raid1 add mirroring to raid0 striping. Note that it is not
complete yet (no rebuild tools, error trapping is incomplete). It's
also kwown to be rather slow when writing.

	Please note that, since all of this is done by software, it
can't be as fast as a hardware implementation of these RAID levels.
Anyway, it is better than nothing at all...

--

Using it :
	Once the kernel is patched and running, you can use a small
command set to manage your md-devices :
- mdadd md-dev block-dev1 block-dev2 ... : add blocks devices to
md-dev.
- mdrun -px md-dev : make the md device usable as a block device.
- mdstop md-dev : stop the device (if started) and cancel the group,
flushing all buffers.

For example, that's what I used to have in my /etc/rc :
	/sbin/mdadd /dev/md0 /dev/sdb1 /dev/sdc2

It groups /dev/sdb1 and /dev/sdc2 in a single device called /dev/md0.
Note that it could also be written :
        /sbin/mdadd /dev/md0 /dev/sdb1
        /sbin/mdadd /dev/md0 /dev/sdc2

BE CAREFUL ! :
        /sbin/mdadd /dev/md0 /dev/sdb1 /dev/sdc2
is NOT equivalent to
        /sbin/mdadd /dev/md0 /dev/sdc2 /dev/sdb1

It produces a device that have the same size, but that will be very
different, specially if you have already put a filesystem on it. So
once you've configured a md-device with an arbitrary order, ALWAYS USE
THAT ORDER, or you won't retrieve your data. You been warned !!

Note that the device is NOT yet usable. It has to be started with :
	/sbin/mdrun -px /dev/md0

where x is :
	l : starts the device in linear mode,
	0 : starts the device in raid-0 mode,
	1 : starts the device in raid-1 mode.

So running this device in linear mode is just a matter of :
        /sbin/mdrun -pl /dev/md0

To create a 'raid0' device, you should have used
        /sbin/mdrun -p0 /dev/md0

Two other options can also be used :
	-cn (raid-0 and raid-1 only) : set the chunk factor. The chunk
factor indicate the size of a chunk on a real device, according to the
following formula :
	chunk_size = PAGE_SIZE << chunk factor
So, on a 386, a 0 chunk factor indicate a chunk size of 4096 bytes, a
1 chunk factor indicate a chunk size of 8192 bytes, a 2 chunk factor
indicate a chunk size of 16384 bytes... Default is -c0.

	-fn (raid-1 only) : set the maximum number of faults that a
physical device can generate before being permanently disabled.
Default is -f0.

It is also a good idea to create a /etc/mtab file that contains
entries to run mdadd on. Each line describe a md device with the
following syntax :

md_dev		mode,c,f	dev1 dev2 ... devn

where	mode is one of linear, raid0, raid1
	c is the chunk factor (optional)
	f is the fault number (optional)

For example, here is my own /etc/mdtab :

# mdtab for wild-wind #1 & #2

/dev/md0	raid0,0	/dev/sdb1	/dev/sdc2	# /usr/local
/dev/md1        linear	/dev/hda6	/dev/hda7	# /mnt for swapfile
/dev/md2	raid1,0,1 /dev/sdd1 	/dev/sde1

So I can simply do
	/sbin/mdadd -a
	/sbin/mdrun -a

You can even use the '-r' flag to automagically start the device once
it has been successfully completed. In this case, the above becomes :
	/sbin/mdrun -ar

To use a particular device, use the syntax
	/sbin/mdadd [-r] md_device

mdstop is not very useful, but it's a good way of being sure that the
md-device has been sync-ed.

You can see the exact state of your md-devices by looking at
/proc/mdstat which looks like this on a test machine running in linear
mode :
	$ cat /proc/mdstat
	Personalities : [1 linear] [2 raid0]
	read_ahead 8 sectors
	md0 : active linear hda6 hda7 10360 blocks
	md1 : inactive
	md2 : inactive
	md3 : inactive

First, you have the suported personalities (linear and striped, also
called raid0), followed by the current read_ahead value (currently for
debbuging purpose, should go away...). Following is the state of each
device the kernel is configured for. Here, /dev/md0 is active and
/dev/md[1-3] are stopped. Then you see the devices used for the
current md-device and the total size in 1024 bytes blocks. You can
obtain much more informations by defining MD_DEBUG in linear_status
(in linear.c) (respectively in raid0_status (in raid0.c)).

Now, the same machine running in striped mode  :

	$ cat /proc/mdstat
	Personalities : [1 linear] [2 raid0]
	read_ahead 8 sectors
	md0 : active raid0 hda6 hda7 10360 blocks
	md1 : inactive
	md2 : inactive
	md3 : inactive

You can now do whatever you want with an active md-device (create a
filesystem, a swapfile, or even a swap partition (the last one seems
to be really useful, since someone reported a big speed improvement
when swapping to a md device running in striped mode, but this mostly
is an untested feature...).

--

Install :
	In the package, you should have found some files :

- README   : this file,
- ChangeLog: history of the project,
- mdpatch  : patch to kernel against version 1.2.4 (may require some
	     minor changes to work with previous versions)
- mdadd.c  : source for mdadd, mdrun and mdstop,
- mdparse.c: parser for mdadd & co
- mdtab	   : Example for /etc/mdtab
- mdadd.8  : man page for mdadd, mdstop and mdrun,
- mdtab.5  : man page for /etc/mdtab,
- md.lsm   : lsm entry for md,
- Makefile : guess what...

As you're used to, do a
		$ patch -p0 -s < wherever_it_lives/mdpatch
from the directory that contains your linux tree (supposed to be clean...).
You can now compile your brand new (bugged ? ;-) kernel (don't forget
to do a 'make config' first and to answer 'yes' to the questions about
md-device).

Kernel modules are now supported. You must have at least support for
the md driver compiled into the kernel, and then load personalities at
run time, thanks to the modules package. If you want to use that
feature, just issue a
		$ make modules
after the kernel compilation, and install modules as describe in the
modules package.  Note that each personality eats about 4k of kernel
space, so this can be a win if your kernel is getting too big for lilo
to load it succesfully.

In the meantime (Yes sir, that's what I call multitasking !), get back
to the package, edit Makefile to suit your taste, and do a
		$ make
		$ make install
(the latest may require from you to be logged as root).

You shouldn't get any warning during both of the compilations. It also
creates entries in /dev called md[0-3]. If you need some more, add
them and change the MAX_MD_DEV constant in md.h. Also, if you want
more than 8 real devices per md-device, adjust the MAX_REAL constant
to suit your needs.

--

Bugs :
	There is at least one known bug with mkdosfs, which asks the
device about physical geometry. This is, of course, not relevant with
a md-dev. So the max size of a DOSFS on such a device is sometimes
limited to a fraction of the avaible size. If there's a DOSFS guru out
there that can help me, please have a look at the HDIO_GETGEO ioctl in
md.c. But who will use DOSFS on such a device anyway ?

	The code heavily depends on 1024 bytes blocks and 512 bytes
sectors. I hope to change it soon.

	You CANNOT use floppies as real devices for md, since they are
not managed the same way hard-disks are. Maybe some day, I'll dig into
that, but don't be too sure, since it has a rather low priority on my
'Things-to-do-one-day-when-I-have-time-to-give-a-look' list ;-).

	In striped mode, chunk size depends on hardware page size.
So a striped device with a 0 factor on an i386 (PAGE_SIZE=4k) cannot
be used on an Alpha (PAGE_SIZE=8k).

	RAID-1 doesn't support mixed SCSI/non-SCSI devices. You'll
have to choose one of those worlds (believe me, SCSI is better...).

	If you find a bug, report as fast a possible. Please include :
- the kernel version you're running,
- the kind of real devices you're using,
- a copy of your partition tables (using fdisk),
- a copy of your /proc/mdtab (with MD_DEBUG enabled),
- if the bug results in a crash, kernel info as described in
/usr/src/linux/README, since this will help a lot,
- any application message showing that there's a problem.

	This file itself is a known bug ;-).

--

Future extensions :
	Well, what I'd like to have now is a way to extend an existing
file system by just adding a new device to a md-dev running in linear
mode (of course, no data loss !!). If Remy Card read this, I'd be glad
to have his own opinion about it (Yes, I can read french ;-).

	Some people asked me about full RAID support. I'm thinking
hard about it, but don't have much time yet. Anyway, raid0 personality
is a first step in that direction. Raid1 is almost working, but I'd
like to have some help, since I only have 24 hours a day, not enough
disk space, and a lot of other things to do (like any other Linux
hacker, I think... ;-).

Also feel free to send me any idea about what you'd like to see in a
future version. Any positive feedback would be very nice too !

--

Thanks to :
	Linus Torvalds and others for writing such a beast,
	All the courageous people that have been testing early versions,
	pc, jpl, arthur & jmm for everything, including missed train ;-)
	bb for love and understanding (sometimes).

--

Please send any idea, opinion or bug report to
<zyngier@amertume.ufr-info-p7.ibp.fr> (preferred) or to
<maz@gloups.fdn.fr>

Please don't expect any immediate answer, as I can only access my mail
account once a week :-(, but mailing on Friday gives you a chance to
receive an answer during week-end !

			Marc ZYNGIER

