baikgdµÄ²©¿Í

The goal of this post is to set up an architecture composed of two nodes that can both simultaneously have read/write access to a synchronized partition. Each node will access the synchronized partition as any other classical partition, the only difference is that every modification will also be forwarded to the other node.

DRBD8 configuration file

• drbd.conf
  

System requirements

• 2.6.24 kernel at least

[tux]# apt-get update
[tux]# apt-get install linux-image-2.6.24-etchnhalf.1-686
[tux]# apt-get install linux-headers-2.6.24-etchnhalf.1-686
[tux]# reboot

Configuration requirements

• Adjust hostname, that is to say specify the same hostname as in /etc/drbd.conf in /etc/hostname.
• Every resource has to be accessible from its hostname, that¡¯s why you have to define every hostname in hosts section:

[tux]# vi /etc/hosts
192.168.9.xx hostname1.domain.org hostname1
192.168.9.xx hostname2.domain.org hostname2

Update your source list to be able to get packages from debian backports

• Add the following line to your /etc/apt/sources.list:

deb http://www.backports.org/debian etch-backports main contrib non-free

• If you are using etch and you want apt to verify the downloaded backports you can import backports.org archive¡¯s key into apt then run apt-get update:

[tux]# apt-get install debian-backports-keyring
[tux]# apt-get update

Install DRBD8

• Install DRBD8 source and DRBD8 utils:

[tux]# apt-get install drbd8-source drbd8-utils

• Build DRBD8 module from previously installed sources using the module-assistant:

[tux]# module-assistant auto-install drbd8

Prepare the partition that will be synchronized thanks to DRBD

• Generally it¡¯s better to think about the DRBD partition earlier and prepare a dedicated partition during operating system installation. Since my machines are hosted on VMWare, I simply add a new virtual disk. Then we will process to its configuration. Do the same on both nodes as usual!

[tux]# fdisk -l    Disk /dev/sda: 6442 MB, 6442450944 bytes  255 heads, 63 sectors/track, 783 cylinders  Units = cylinders of 16065 * 512 = 8225280 bytes    Device       Boot   Start    End     Blocks    Id                 System  /dev/sda1       *       1    743    5968116    83                  Linux  /dev/sda2             744    783     321300     5               Extended  /dev/sda5             744    783    321268+    82   Linux swap / Solaris    Disk /dev/sdb: 1073 MB, 1073741824 bytes  255 heads, 63 sectors/track, 130 cylinders  Units = cylinders of 16065 * 512 = 8225280 bytes    Disk /dev/sdb doesn't contain a valid partition table

• Let¡¯s configure the new disk added entitled /dev/sdb. First, create a logical partition on an extended one that will completely fill the new partition:

[tux]# fdisk /dev/sdb    Command (m for help): n  Command action  e   extended  p   primary partition (1-4)  e  Partition number (1-4): 1  First cylinder (1-130, default 1):  Using default value 1  Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):  Using default value 130    Command (m for help): n  Command action  l   logical (5 or over)  p   primary partition (1-4)    l  First cylinder (1-130, default 1):  Using default value 1    Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):    Using default value 130    Command (m for help): w  The partition table has been altered!    Calling ioctl() to re-read partition table.  Syncing disks.

• In our case, the logical partition that will be synchronized by DRBD is entitled /dev/sdb5:

[tux]# fdisk -l /dev/sdb    Disk /dev/sdb: 1073 MB, 1073741824 bytes  255 heads, 63 sectors/track, 130 cylinders  Units = cylinders of 16065 * 512 = 8225280 bytes    Device       Boot   Start    End     Blocks    Id      System  /dev/sdb1               1    130   1044193+     5    Extended  /dev/sdb5               1    130    1044162    83       Linux

• /dev/sdb5 is ready to be used by DRBD. Thanks to the comment of Mark, we know that if there is no existing data on the device, we should not create a filesystem on the underlying device! It will be done later on the DRBD device itself.

Configure DRBD8

• Adjust drbd.conf according to your system, that is to say specify your local partition as well as the correct hostnames and IP addresses, then copy it in /etc directory.

on hostname1 {    device /dev/drbd0;    disk /dev/sdb5;    address 192.168.9.xx:7789;    meta-disk internal;  }  on hostname2 {    device /dev/drbd0;    disk /dev/sdb5;    address 192.168.9.xx:7789;    meta-disk internal;  }

Create DRBD resource

• First of all create your resource. Resource name corresponds to what you specified in your drbd.conf (here the resource name is r0). The DRBD partition must have the same size on both nodes to work correctly.
• If there is no filesystem on /dev/sdb5, following commands should do the trick (on both nodes):

[tux]# drbdadm create-md r0  You want me to create a v08 style flexible-size internal meta data block.  There apears to be a v08 flexible-size internal meta data block  already in place on /dev/sdb5 at byte offset 106917888  Do you really want to overwrite the existing v08 meta-data?  [need to type 'yes' to confirm] yes    Writing meta data...  initialising activity log  NOT initialized bitmap  New drbd meta data block sucessfully created.

• If any error like the following one occurs:

[tux]# drbdadm create-md r0  md_offset 1069215744  al_offset 1069182976  bm_offset 1069150208    Found ext3 filesystem which uses 1044160 kB  current configuration leaves usable 1044092 kB    Device size would be truncated, which  would corrupt data and result in  'access beyond end of device' errors.  You need to either  * use external meta data (recommended)  * shrink that filesystem first  * zero out the device (destroy the filesystem)  Operation refused.    Command 'drbdmeta /dev/drbd0 v08 /dev/sdb5 internal create-md'  terminated with exit code 40  drbdadm aborting

• It means there was some data on /dev/sdb5 or at least a filesystem. In this kind of scenario there are two ways to solve the problem: adjust partition size or backup and erase existing data.

1. Adjust partition size is the most dangerous option! Use it at your own risks. Many people encountered ¡®accessing beyond end of device¡¯ errors when the device was almost full! (thank you for your comment Mark ). If you still want to do it,  use the current configuration leaves usable information returned by previous command (1044092) to resize the device. Please perform the same commands on each node.

[tux]# e2fsck -f /dev/sdb5 && resize2fs /dev/sdb5 1044092K

2. The second option is the safer one: backup up existing data on another device, then erase its current content:

[tux]# dd if=/dev/zero of=/dev/sdb5

• And then you can successfully create your resource on both machines:

[tux]# drbdadm create-md r0

Start DRBD8 and ensure it works correctly

• You can start DRBD this way (will produce an error since at first startup you have to force only one node into primary state):

[hostname1]# /etc/init.d/drbd start
[hostname2]# /etc/init.d/drbd start

• Or make it manually (attach and connect the resource r0):

[hostname1]# drbdadm attach r0
[hostname1]# drbdadm connect r0
[hostname2]# drbdadm attach r0
[hostname2]# drbdadm connect r0

• In order to get consistent data, force DRBD partition into primary state for only one of the two nodes, this will trigger the synchronization process. (¡°drbdadm ¡ª ¨Coverwrite-data-of-peer primary r0¡å is equivalent to the following command)

[hostame1]# drbdsetup /dev/drbd0 primary -o

• Wait the end of the synchronization.

[hostname1]# /etc/init.d/drbd status  drbd driver loaded OK; device status:  version: 8.0.13 (api:86/proto:86)  GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by phil@fat-tyre,  2008-08-04 15:28:07  m:res  cs         st                ds                    p mounted fstype  ...    sync'ed:   20.8%             (830012/1044092)K  0:r0   SyncSource Primary/Secondary UpToDate/Inconsistent C

• When 100% is reached, you can successfully set the secondary node into primary state in order to get two primaries:

[hostame2]# drbdadm primary r0

• Verify that everything works fine:

[tux]# /etc/init.d/drbd status  drbd driver loaded OK; device status:  version: 8.0.13 (api:86/proto:86)  GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by phil@fat-tyre,  2008-08-04 15:28:07  m:res cs        st              ds                p mounted fstype  0:r0  Connected Primary/Primary UpToDate/UpToDate C

• Now it¡¯s time to create your EXT3 filesystem on the DRBD device! (only in the case there was no existing data on /dev/sdb5)

[hostname1]# mke2fs -j /dev/drbd0

• You can finally check if your system works correctly. First create a new directory:

[hostname1]# mkdir /synchronized
[hostname2]# mkdir /synchronized

• Then mount the DRBD partition at this location. You should be able to access it read/write on both machines simultaneously.

[hostname1]# mount /dev/drbd0 /synchronized
[hostname2]# mount /dev/drbd0 /synchronized

• If you were unable to mount your partition and get the following error message, ensure you have correctly created a valid filesystem on the DRBD device as explained before.

[tux] # mount /dev/drbd0 /synchronized/
mount: you must specify the filesystem type

• If you followed every step carefully, your system must be set up correctly and the DRBD status command should return the following state on both nodes. You can see that DRBD is mounted on /synchronized.

[hostname1]# /etc/init.d/drbd status  drbd driver loaded OK; device status:  version: 8.0.13 (api:86/proto:86)  GIT-hash: ee3ad77563d2e87171a3da17cc002ddfd1677dbe build by phil@fat-tyre,  2008-08-04 15:28:07  m:res cs        st              ds                p mounted       fstype  0:r0  Connected Primary/Primary UpToDate/UpToDate C /synchronized ext3

• According to DRBD specifications, our system and configuration is valid. We can try to create files from both locations to see if it works how it should.

[hostname1]# touch /synchronized/from_host1
[hostname2]# touch /synchronized/from_host2
[hostname1]# ls /synchronized
lost+found from_host1
[hostname2]# ls /synchronized
lost+found from_host2

• As you can notice, our system does not work as expected, since we are not able to see the file added from the other node. I did not find a way to make it work properly using a standard Ext3 filesystem. Generally, you have to unmount the partition, put it into secondary state, put it back into primary state and mount it again to be sure to see the modifications of the other node. DRBD documentation specifies that in order to use the software safely with two primaries we have to couple DRBD with a specific filesystem that ensures integrity. I will go further and asserts that DRBD cannot work properly without a specific filesystem even if there is no corruption risk.

Other considerations

You have to keep in mind that there is no integrity guaranty between the two machines. Therefore we are in any case force to implement a lock mechanism to avoid data corruption. DRBD8 deals only with synchronization between nodes. In order to circumvent this drawback, the DRBD synchronized partition must use a specific file system like GFS2 for example if you want to see modifications from both nodes almost in real time. A next post will go into further details and explain how to build a shared nothing architecture between two nodes using DRBD8 + GFS2 on debian etch. I can already claim this time it will work as expected, so do not become discouraged.