gmane.linux.kernel.device-mapper.devel

path_checker within multipath {}

Matthew Kent — 2008-11-27T19:47:54

Am I missing something obvious here? /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.annotated says this should work, and I can't find a RHEL 5 issue for it: My config says: multipaths { multipath { wwid "22263000155ada9e9" path_grouping_policy failover path_checker tur } multipath { wwid "36090a018108277e8ed92e4e9cee1733e" path_grouping_policy multibus path_checker readsector0 } } but multipathd -k"show conf" says: multipaths { multipath { wwid 22263000155ada9e9 path_grouping_policy failover } multipath { wwid 36090a018108277e8ed92e4e9cee1733e path_grouping_policy multibus } and multipath -v3 shows them all sdb: path checker = readsector0 (config file default) while the grouping policy seems to work, pgpolicy = multibus (LUN setting) I'm sure I can use 'tur' on both multipaths via the defaults {} section and be fine, but from the docs (even checked the RHEL 5.3 beta SRPM) this should work.

Announce: unlimited number of shared snapshots

Mikulas Patocka — 2008-11-27T05:41:36

Hi I made first release of my snapshot storage that can hold unlimited number of snapshots and share data between them: http://people.redhat.com/mpatocka/patches/kernel/new-snapshots/ How-to-use is at the beginning of dm-multisnapshot.patch If you see some errors, report them. This storage uses btree with the key (chunk number, snapshot range). It uses log-structured storage, so that it is safe w.r.t. crash. TODO: - reclaim allocated space, allow deleting snapshots - writable snapshots Mikulas

[PATCH 0/9] dm snapshot: shared exception store (v4)

FUJITA Tomonori — 2008-11-26T16:17:30

This is the fourth version of the shared exception store (based on Zumastor code). You can find the previous submissions: https://www.redhat.com/archives/dm-devel/2008-August/msg00003.html https://www.redhat.com/archives/dm-devel/2008-August/msg00087.html https://www.redhat.com/archives/dm-devel/2008-October/msg00159.html Except for bug fixes, the major change since the previous version is fixing the disk endian (Zumastor simply uses the CPU endian for the disk format). The current code uses little endian for the disk format (I just slightly tested it with ppc64. It needs more tests, of course). I also modified the snapshot disk initialization code to reserve chunks for the journaling mechanism. I've not implemented the journaling mechanism but there is no need to change the disk format for that later. There are 9 patches and I modified only 7th patch (Zumastor shared snapshot code) since the last submission. IOW, I have not change the way to refactor the existing code. I'll think about how to integrate this with the existing code cleaner again but I think that the user-interface is far more important than how to integrate this with the existing in-kernel code. We don't want to change the user-interface after mainline mering while we can evolve (or clean up) the in-kernel structures any time (and it's not hard at all). For example, a way to handle the snapshots looks oK? - create one snapshot (the id is 0): vine:/home/fujita# dmsetup message work 0 snapshot create 0 - get the list of snapshots: vine:/home/fujita# dmsetup status work: 0 268430022 s-snap-origin 2 0 5 (this means we have two snapshots, their IDs are 0 and 5). With Zumastor code, the maximum number of snapshots is 64 so the above interface to get the list would work, but it might not work well for an shared-exception-snapshot implementation that support tons of snapshots (which someone might implement in the future). Any opinions? The patches are available (against the latest Linus git tree): http://www.kernel.org/pub/linux/kernel/people/tomo/dm-snap/2008-11-26/ If you like a git tree: git://git.kernel.org/pub/scm/linux/kernel/git/tomo/linux-2.6-misc.git dm-snap The following is the same instruction as the previous submission. = In this example, /dev/sda1 is used as an origin device and /dev/sdb1 is used as a cow device. - first, set up an origin and a cow: vine:/home/fujita# echo |dmesg create --notable work vine:/home/fujita# echo 0 `blockdev --getsize /dev/sda1` s-snap-origin /dev/sda1 /dev/sdb1 S 16|dmsetup load work vine:/home/fujita# dmsetup resume work - confirm the current status: vine:/home/fujita# dmsetup status work: 0 268430022 s-snap-origin : no snapshot - create one snapshot (the id is 0): vine:/home/fujita# dmsetup message work 0 snapshot create 0 - see if the snapshot is created: vine:/home/fujita# dmsetup status work: 0 268430022 s-snap-origin 1 0 - create one more snapshot (the id is 5) and see the result: vine:/home/fujita# dmsetup message work 0 snapshot create 5 vine:/home/fujita# dmsetup status work: 0 268430022 s-snap-origin 2 0 5 Note that creating a snapshot just means that you store some information on the cow device. You don't create a device for it yet. - create a device to access to the snapshot #5 vine:/home/fujita# echo |dmsetup create --notable snap5 vine:/home/fujita# echo 0 `blockdev --getsize /dev/sda1` s-snap /dev/sda1 5|dmsetup load snap5 vine:/home/fujita# dmsetup resume snap5 - confirm the current status: vine:/home/fujita# dmsetup status snap5: 0 268430022 s-snap Unknown work: 0 268430022 s-snap-origin 2 0 5 - remove the devices after you play with them: vine:/home/fujita# dmsetup remove snap5 vine:/home/fujita# dmsetup remove work vine:/home/fujita# dmsetup info No devices found - Again, setup the origin and the cow: vine:/home/fujita# echo |dmsetup create --notable work vine:/home/fujita# echo 0 `blockdev --getsize /dev/sda1` s-snap-origin /dev/sda1 /dev/sdb1 S 16|dmsetup load work vine:/home/fujita# dmsetup resume work vine:/home/fujita# dmsetup status work: 0 268430022 s-snap-origin 2 0 5 Note that you have two snapshots that you created. You can play with them again after creating devices for them. - delete the 5th snapshot vine:/home/fujita# dmsetup message work 0 snapshot delete 5 vine:/home/fujita# dmsetup status work: 0 268430022 s-snap-origin 1 0 *) By default, dmsetup perfomrs multiple tasks. I uses dmsetup commands that issue a single ioctl to make things clear.

[RFC][PATCH 1/4] dm-log: fix io_client_destroy leak

Takahiro Yasui — 2008-11-26T00:01:37

In create_log_context function, dm_io_client_destroy function needs to be called, when memory allocation of disk_header, sync_bits and recovering_bits failed, but dm_io_client_destroy is not called. This patch fixes it. Signed-off-by: Takahiro Yasui redhat.com> --- drivers/md/dm-log.c | 5 +++++ 1 file changed, 5 insertions(+) Index: linux-2.6.28-rc4/drivers/md/dm-log.c =================================================================== --- linux-2.6.28-rc4.orig/drivers/md/dm-log.c +++ linux-2.6.28-rc4/drivers/md/dm-log.c < at >< at > -467,6 +467,7 < at >< at > static int create_log_context(struct dm_ lc->disk_header = vmalloc(buf_size); if (!lc->disk_header) { DMWARN("couldn't allocate disk log buffer"); +dm_io_client_destroy(lc->io_req.client); kfree(lc); return -ENOMEM; } < at >< at > -483,6 +484,8 < at >< at > static int create_log_context(struct dm_ if (!dev) vfree(lc->clean_bits); vfree(lc->disk_header); +if (dev) +dm_io_client_destroy(lc->io_req.client); kfree(lc); return -ENOMEM; } < at >< at > -496,6 +499,8 < at >< at > static int create_log_context(struct dm_ if (!dev) vfree(lc->clean_bits); vfree(lc->disk_header); +if (dev) +dm_io_client_destroy(lc->io_req.client); kfree(lc); return -ENOMEM; }

[RFC][PATCH 3/4] dm-log: support multiple log devices

Takahiro Yasui — 2008-11-26T00:01:41

This patch introduces multiple log devices feature. * "log" member is added into log context to keep a device status and device info. * read_headers function reads log data from each log devices and check if they contain the same header values. * write_headers issues write I/O to all active log devices at the same time and check each result when all I/Os complete. * add parse_params to search "region size" in the parameter list. Signed-off-by: Takahiro Yasui redhat.com> --- drivers/md/dm-log.c | 579 +++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 416 insertions(+), 163 deletions(-) Index: linux-2.6.28-rc4/drivers/md/dm-log.c =================================================================== --- linux-2.6.28-rc4.orig/drivers/md/dm-log.c +++ linux-2.6.28-rc4/drivers/md/dm-log.c < at >< at > -249,6 +249,15 < at >< at > struct log_header { sector_t nr_regions; }; +struct log { +struct log_c *lc; +int failed; + +struct dm_dev *dev; +struct log_header header; +struct dm_io_region header_location; +}; + struct log_c { struct dm_target *ti; int touched; < at >< at > -270,17 +279,19 < at >< at > struct log_c { FORCESYNC,/* Force a sync to happen */ } sync; -struct dm_io_request io_req; - /* * Disk log fields */ -int log_dev_failed; -struct dm_dev *log_dev; -struct log_header header; +unsigned int nr_logs; -struct dm_io_region header_location; struct log_header *disk_header; +struct dm_io_request io_req; + +unsigned int nr_active_logs; +struct dm_io_region *io_regions; +struct log **io_logs;/* index log array of io_regions */ + +struct log log[0]; }; /* < at >< at > -323,72 +334,236 < at >< at > static void header_from_disk(struct log_ core->nr_regions = le64_to_cpu(disk->nr_regions); } -static int read_header(struct log_c *log) +static void update_io_regions(struct log_c *lc) +{ +struct log *l; +int count = 0; + +for (l = lc->log; l < lc->log + lc->nr_logs; l++) { +if (l->failed) +continue; + +lc->io_regions[count] = l->header_location; +lc->io_logs[count] = l; +count++; +} + +lc->nr_active_logs = count; +} + +static void fail_log_device(struct log *l) +{ +if (l->failed) +return; + +l->failed = 1; +dm_table_event(l->lc->ti->table); +} + +static void fail_all_devices(struct log_c *lc) { +struct log *l; + +for (l = lc->log; l < lc->log + lc->nr_logs; l++) +l->failed = 1; + +lc->nr_active_logs = 0; +dm_table_event(lc->ti->table); +} + +static int read_header(struct log *l) +{ +struct log_c *lc = l->lc; int r; -log->io_req.bi_rw = READ; +lc->io_req.bi_rw = READ; -r = dm_io(&log->io_req, 1, &log->header_location, NULL); -if (r) +r = dm_io(&lc->io_req, 1, &l->header_location, NULL); +if (r) { +DMWARN("Failed to read header on ditry " + "region log device, %s", l->dev->name); +fail_log_device(l); return r; +} -header_from_disk(&log->header, log->disk_header); +header_from_disk(&l->header, lc->disk_header); /* New log required? */ -if (log->sync != DEFAULTSYNC || log->header.magic != MIRROR_MAGIC) { -log->header.magic = MIRROR_MAGIC; -log->header.version = MIRROR_DISK_VERSION; -log->header.nr_regions = 0; +if (lc->sync != DEFAULTSYNC || l->header.magic != MIRROR_MAGIC) { +l->header.magic = MIRROR_MAGIC; +l->header.version = MIRROR_DISK_VERSION; +l->header.nr_regions = 0; } #ifdef __LITTLE_ENDIAN -if (log->header.version == 1) -log->header.version = 2; +if (l->header.version == 1) +l->header.version = 2; #endif -if (log->header.version != MIRROR_DISK_VERSION) { +if (l->header.version != MIRROR_DISK_VERSION) { DMWARN("incompatible disk log version"); +fail_log_device(l); return -EINVAL; } return 0; } -static inline int write_header(struct log_c *log) +/* + * read_headers + * + * Issue read I/Os sequentially and check their contents. + * + * return value: + * nr_regions ... the number of region stored on log disks + * -EIO ... all read I/Os failed and no active log exists + * -EINVAL ... header data are not consistent among logs + */ +static int read_headers(struct log_c *lc) { -log->io_req.bi_rw = WRITE; -return dm_io(&log->io_req, 1, &log->header_location, NULL); +struct log *l; +sector_t nr_regions = 0; +int active_logs = 0; + +/* + * read all log headers + * + * Read shoud be done sequentially, since one buffer is + * shared by all logs. + */ +for (l = lc->log; l < lc->log + lc->nr_logs; l++) +if (!l->failed && !read_header(l)) +active_logs++; + +if (!active_logs) { +DMWARN("All read I/Os to log disks failed.\n"); +fail_all_devices(lc); +return -EIO; +} + +/* + * check consistency of log headers + */ +for (l = lc->log; l < lc->log + lc->nr_logs; l++) { +if (l->failed || !l->header.nr_regions) +continue; + +if (!nr_regions) { +nr_regions = l->header.nr_regions; +continue; +} + +if (l->header.nr_regions != nr_regions) { +DMWARN("log %s has inconsistent region counts %ld" + " (expected %ld)", l->dev->name, + l->header.nr_regions, nr_regions); +fail_all_devices(lc); +return -EINVAL; +} +} + +/* + * Refresh log contents, since current data might contain + * data on a new log disk which does not have valid log data. + */ +if (active_logs > 1) { +for (l = lc->log; l < lc->log + lc->nr_logs; l++) { +if (l->failed || !l->header.nr_regions) +continue; +if (!read_header(l)) +break; +} + +if (unlikely(l == lc->log + lc->nr_logs)) +nr_regions = 0; + +/* initialize new log headers */ +for (l = lc->log; l < lc->log + lc->nr_logs; l++) +if (!l->failed) +l->header.nr_regions = nr_regions; +} + +update_io_regions(lc); + +return nr_regions; } -/*---------------------------------------------------------------- - * core log constructor/destructor +/* + * write_headers * - * argv contains region_size followed optionally by [no]sync - *--------------------------------------------------------------*/ + * Issue write I/Os to all active logs and return 0 if at lease + * one log has scceeded its I/O, othersize (no active logs) + * returns a return value of dm_io function. + */ +static int write_headers(struct log_c *lc) +{ +unsigned long error; +int i, r; + +lc->io_req.bi_rw = WRITE; + +r = dm_io(&lc->io_req, lc->nr_active_logs, lc->io_regions, + &error); +if (r) { +/* check error devices and disable them */ +for (i = 0; i < lc->nr_active_logs; i++) +if (test_bit(i, &error)) +fail_log_device(lc->io_logs[i]); + +update_io_regions(lc); + +if (!lc->nr_active_logs) +return r; +} + +return 0; +} + #define BYTE_SHIFT 3 -static int create_log_context(struct dm_dirty_log *log, struct dm_target *ti, - unsigned int argc, char **argv, - struct dm_dev *dev) +static inline size_t log_bitset_size(struct log_c *lc) { -enum sync sync = DEFAULTSYNC; +return dm_round_up(lc->region_count, + sizeof(*lc->clean_bits) << BYTE_SHIFT) +>> BYTE_SHIFT; +} -struct log_c *lc; -uint32_t region_size; -unsigned int region_count; -size_t bitset_size, buf_size; -int r; +static size_t log_buffer_size(struct log_c *lc) +{ +/* Buffer holds both header and bitset. */ +return dm_round_up((LOG_OFFSET << SECTOR_SHIFT) + + log_bitset_size(lc), + lc->ti->limits.hardsect_size); +} +static int parse_params(unsigned int argc, char **argv, +uint32_t *region_size, enum sync *sync) +{ +/* + * check number of parameters + */ if (argc < 1 || argc > 2) { DMWARN("wrong number of arguments to dirty region log"); return -EINVAL; } +/* + * get region size + */ +if (sscanf(argv[0], "%u", region_size) != 1) { +DMWARN("invalid region size string to dirty region log"); +return -EINVAL; +} + +/* + * get sync option + */ +*sync = DEFAULTSYNC; + if (argc > 1) { if (!strcmp(argv[1], "sync")) -sync = FORCESYNC; +*sync = FORCESYNC; else if (!strcmp(argv[1], "nosync")) -sync = NOSYNC; +*sync = NOSYNC; else { DMWARN("unrecognised sync argument to " "dirty region log: %s", argv[1]); < at >< at > -396,113 +571,180 < at >< at > static int create_log_context(struct dm_ } } -if (sscanf(argv[0], "%u", ®ion_size) != 1) { -DMWARN("invalid region size string"); -return -EINVAL; -} +return 0; +} -region_count = dm_sector_div_up(ti->len, region_size); +static struct log_c *create_log_context(struct dm_target *ti, +unsigned int nr_logs, +uint32_t region_size, +enum sync sync) +{ +struct log_c *lc; +size_t len; + +len = sizeof(*lc) + sizeof(lc->log[0]) * nr_logs; -lc = kmalloc(sizeof(*lc), GFP_KERNEL); +lc = kzalloc(len, GFP_KERNEL); if (!lc) { -DMWARN("couldn't allocate core log"); -return -ENOMEM; +DMWARN("couldn't allocate log context"); +return NULL; } lc->ti = ti; lc->touched = 0; lc->region_size = region_size; -lc->region_count = region_count; +lc->region_count = dm_sector_div_up(ti->len, region_size); + lc->sync = sync; +lc->nr_logs = nr_logs; + +return lc; +} + +static void destroy_log_context(struct log_c *lc) +{ +vfree(lc->recovering_bits); +vfree(lc->sync_bits); +vfree(lc->clean_bits); +kfree(lc); +} + +static void destroy_log_devices(struct log_c *lc) +{ +struct log *l; + +kfree(lc->io_logs); +kfree(lc->io_regions); + +if (lc->io_req.client) +dm_io_client_destroy(lc->io_req.client); + +vfree(lc->disk_header); +lc->clean_bits = NULL; + +for (l = lc->log; l < lc->log + lc->nr_logs; l++) +dm_put_device(l->lc->ti, l->dev); +} + +static int create_log_devices(struct log_c *lc, char **dev) +{ +struct log *l; +size_t buf_size = 0; +int r; /* - * Work out how many "unsigned long"s we need to hold the bitset. + * setup each log device + */ +for (l = lc->log; l < lc->log + lc->nr_logs; l++, dev++) { +r = dm_get_device(lc->ti, dev[0], 0, 0 /* FIXME */, + FMODE_READ | FMODE_WRITE, &l->dev); +if (r) { +lc->ti->error = "Device lookup failure"; + +while (--l >= lc->log) +dm_put_device(l->lc->ti, l->dev); + +return r; +} + +if (!buf_size) +buf_size = log_buffer_size(lc); + +l->lc = lc; +l->failed = 0; + +l->header.magic = 0; +l->header.version = 0; +l->header.nr_regions = 0; + +l->header_location.bdev = l->dev->bdev; +l->header_location.sector = 0; +l->header_location.count = buf_size >> SECTOR_SHIFT; +} + +/* + * setup common info */ -bitset_size = dm_round_up(region_count, - sizeof(*lc->clean_bits) << BYTE_SHIFT); -bitset_size >>= BYTE_SHIFT; +lc->nr_active_logs = lc->nr_logs; + +lc->disk_header = vmalloc(buf_size); +if (!lc->disk_header) { +DMWARN("couldn't allocate disk log buffer"); +destroy_log_devices(lc); +return -ENOMEM; +} + +lc->io_req.mem.type = DM_IO_VMA; +lc->io_req.mem.ptr.vma = lc->disk_header; +lc->io_req.notify.fn = NULL; +lc->io_req.client = dm_io_client_create(dm_div_up(buf_size, + PAGE_SIZE)); +if (IS_ERR(lc->io_req.client)) { +r = PTR_ERR(lc->io_req.client); +DMWARN("couldn't allocate disk io client"); +destroy_log_devices(lc); +return -ENOMEM; +} + +lc->io_regions = kmalloc(sizeof(*lc->io_regions) * lc->nr_logs, + GFP_KERNEL); +if (!lc->io_regions) { +DMWARN("couldn't allocate I/O regions"); +destroy_log_devices(lc); +return -ENOMEM; +} +lc->io_logs = kmalloc(sizeof(*lc->io_logs) * lc->nr_logs, + GFP_KERNEL); +if (!lc->io_logs) { +DMWARN("couldn't allocate I/O region index log array"); +destroy_log_devices(lc); +return -ENOMEM; +} + +return 0; +} + +static int setup_log_bitmaps(struct log_c *lc) +{ +size_t bitset_size; + +/* + * Work out how many "unsigned long"s we need to hold the bitset. + */ +bitset_size = log_bitset_size(lc); lc->bitset_uint32_count = bitset_size / sizeof(*lc->clean_bits); /* * Disk log? */ -if (!dev) { +if (!lc->nr_logs) { lc->clean_bits = vmalloc(bitset_size); if (!lc->clean_bits) { DMWARN("couldn't allocate clean bitset"); -kfree(lc); -return -ENOMEM; -} -lc->disk_header = NULL; -} else { -lc->log_dev = dev; -lc->log_dev_failed = 0; -lc->header_location.bdev = lc->log_dev->bdev; -lc->header_location.sector = 0; - -/* - * Buffer holds both header and bitset. - */ -buf_size = dm_round_up((LOG_OFFSET << SECTOR_SHIFT) + - bitset_size, ti->limits.hardsect_size); -lc->header_location.count = buf_size >> SECTOR_SHIFT; - -lc->io_req.mem.type = DM_IO_VMA; -lc->io_req.mem.ptr.vma = lc->disk_header; -lc->io_req.notify.fn = NULL; -lc->io_req.client = dm_io_client_create(dm_div_up(buf_size, - PAGE_SIZE)); -if (IS_ERR(lc->io_req.client)) { -r = PTR_ERR(lc->io_req.client); -DMWARN("couldn't allocate disk io client"); -kfree(lc); return -ENOMEM; } - -lc->disk_header = vmalloc(buf_size); -if (!lc->disk_header) { -DMWARN("couldn't allocate disk log buffer"); -dm_io_client_destroy(lc->io_req.client); -kfree(lc); -return -ENOMEM; -} - +} else lc->clean_bits = (void *)lc->disk_header + (LOG_OFFSET << SECTOR_SHIFT); -} memset(lc->clean_bits, -1, bitset_size); lc->sync_bits = vmalloc(bitset_size); if (!lc->sync_bits) { DMWARN("couldn't allocate sync bitset"); -if (!dev) -vfree(lc->clean_bits); -vfree(lc->disk_header); -if (dev) -dm_io_client_destroy(lc->io_req.client); -kfree(lc); return -ENOMEM; } -memset(lc->sync_bits, (sync == NOSYNC) ? -1 : 0, bitset_size); -lc->sync_count = (sync == NOSYNC) ? region_count : 0; +memset(lc->sync_bits, (lc->sync == NOSYNC) ? -1 : 0, bitset_size); +lc->sync_count = (lc->sync == NOSYNC) ? lc->region_count : 0; lc->recovering_bits = vmalloc(bitset_size); if (!lc->recovering_bits) { DMWARN("couldn't allocate sync bitset"); -vfree(lc->sync_bits); -if (!dev) -vfree(lc->clean_bits); -vfree(lc->disk_header); -if (dev) -dm_io_client_destroy(lc->io_req.client); -kfree(lc); return -ENOMEM; } memset(lc->recovering_bits, 0, bitset_size); lc->sync_search = 0; -log->context = lc; return 0; } < at >< at > -510,51 +752,78 < at >< at > static int create_log_context(struct dm_ static int core_ctr(struct dm_dirty_log *log, struct dm_target *ti, unsigned int argc, char **argv) { -return create_log_context(log, ti, argc, argv, NULL); -} +struct log_c *lc; +uint32_t region_size; +enum sync sync; +int r; -static void destroy_log_context(struct log_c *lc) -{ -vfree(lc->sync_bits); -vfree(lc->recovering_bits); -kfree(lc); +r = parse_params(argc, argv, ®ion_size, &sync); +if (r) +return r; + +lc = create_log_context(ti, 0, region_size, sync); +if (!lc) +return -ENOMEM; + +r = setup_log_bitmaps(lc); +if (r) { +destroy_log_context(lc); +return r; +} + +log->context = lc; + +return 0; } static void core_dtr(struct dm_dirty_log *log) { struct log_c *lc = (struct log_c *) log->context; -vfree(lc->clean_bits); destroy_log_context(lc); } /*---------------------------------------------------------------- - * disk log constructor/destructor + * disks log constructor/destructor * * argv contains log_device region_size followed optionally by [no]sync *--------------------------------------------------------------*/ static int disk_ctr(struct dm_dirty_log *log, struct dm_target *ti, unsigned int argc, char **argv) { +struct log_c *lc; +uint32_t region_size; +enum sync sync; int r; -struct dm_dev *dev; -if (argc < 2 || argc > 3) { -DMWARN("wrong number of arguments to disk dirty region log"); +if (!argc) { +DMWARN("wrong number of arguments to dirty region log"); return -EINVAL; } -r = dm_get_device(ti, argv[0], 0, 0 /* FIXME */, - FMODE_READ | FMODE_WRITE, &dev); +r = parse_params(argc-1, argv+1, ®ion_size, &sync); if (r) return r; -r = create_log_context(log, ti, argc - 1, argv + 1, dev); +lc = create_log_context(ti, 1, region_size, sync); +if (!lc) +return -ENOMEM; + +r = create_log_devices(lc, argv); +if (r) { +destroy_log_context(lc); +return r; +} + +r = setup_log_bitmaps(lc); if (r) { -dm_put_device(ti, dev); +destroy_log_devices(lc); +destroy_log_context(lc); return r; } +log->context = lc; + return 0; } < at >< at > -562,9 +831,7 < at >< at > static void disk_dtr(struct dm_dirty_log { struct log_c *lc = (struct log_c *) log->context; -dm_put_device(lc->ti, lc->log_dev); -vfree(lc->disk_header); -dm_io_client_destroy(lc->io_req.client); +destroy_log_devices(lc); destroy_log_context(lc); } < at >< at > -578,45 +845,31 < at >< at > static int count_bits32(uint32_t *addr, return count; } -static void fail_log_device(struct log_c *lc) -{ -if (lc->log_dev_failed) -return; - -lc->log_dev_failed = 1; -dm_table_event(lc->ti->table); -} - static int disk_resume(struct dm_dirty_log *log) { -int r; +int r = 0; unsigned i; struct log_c *lc = (struct log_c *) log->context; +struct log *l; size_t size = lc->bitset_uint32_count * sizeof(uint32_t); +unsigned int nr_regions = 0; -/* read the disk header */ -r = read_header(lc); -if (r) { -DMWARN("%s: Failed to read header on dirty region log device", - lc->log_dev->name); -fail_log_device(lc); -/* - * If the log device cannot be read, we must assume - * all regions are out-of-sync. If we simply return - * here, the state will be uninitialized and could - * lead us to return 'in-sync' status for regions - * that are actually 'out-of-sync'. - */ -lc->header.nr_regions = 0; +if (lc->nr_active_logs) { +r = read_headers(lc); +if (r < 0) { +DMWARN("Failed to read dirty region log"); +nr_regions = 0; +} else +nr_regions = r; } /* set or clear any new bits -- device has grown */ if (lc->sync == NOSYNC) -for (i = lc->header.nr_regions; i < lc->region_count; i++) +for (i = nr_regions; i < lc->region_count; i++) /* FIXME: amazingly inefficient */ log_set_bit(lc, lc->clean_bits, i); else -for (i = lc->header.nr_regions; i < lc->region_count; i++) +for (i = nr_regions; i < lc->region_count; i++) /* FIXME: amazingly inefficient */ log_clear_bit(lc, lc->clean_bits, i); < at >< at > -630,17 +883,17 < at >< at > static int disk_resume(struct dm_dirty_l lc->sync_search = 0; /* set the correct number of regions in the header */ -lc->header.nr_regions = lc->region_count; - -/* update disk headers */ -header_to_disk(&lc->header, lc->disk_header); +for (l = lc->log; l < lc->log + lc->nr_logs; l++) +l->header.nr_regions = lc->region_count; -/* write the new header */ -r = write_header(lc); -if (r) { -DMWARN("%s: Failed to write header on dirty region log device", - lc->log_dev->name); -fail_log_device(lc); +if (lc->nr_active_logs) { +/* update disk headers */ +header_to_disk(&lc->io_logs[0]->header, lc->disk_header); + +/* write the new header */ +r = write_headers(lc); +if (r) +DMWARN("Failed to write dirty region log"); } return r; < at >< at > -683,12 +936,12 < at >< at > static int disk_flush(struct dm_dirty_lo struct log_c *lc = (struct log_c *) log->context; /* only write if the log has changed */ -if (!lc->touched) +if (!lc->touched || !lc->nr_active_logs) return 0; -r = write_header(lc); +r = write_headers(lc); if (r) -fail_log_device(lc); +DMWARN("Failed to write dirty region log"); else lc->touched = 0; < at >< at > -784,13 +1037,13 < at >< at > static int disk_status(struct dm_dirty_l switch(status) { case STATUSTYPE_INFO: -DMEMIT("3 %s %s %c", log->type->name, lc->log_dev->name, - lc->log_dev_failed ? 'D' : 'A'); +DMEMIT("3 %s %s %c", log->type->name, lc->log[0].dev->name, + lc->log[0].failed ? 'D' : 'A'); break; case STATUSTYPE_TABLE: DMEMIT("%s %u %s %u ", log->type->name, - lc->sync == DEFAULTSYNC ? 2 : 3, lc->log_dev->name, + lc->sync == DEFAULTSYNC ? 2 : 3, lc->log[0].dev->name, lc->region_size); DMEMIT_SYNC; }

[RFC][PATCH 4/4] dm-log: interface update for multiplelog devices

Takahiro Yasui — 2008-11-26T00:01:43

This patch introduces a new interface for multi-log devices. disk log has two interfaces, constructor and status, and disk_ctr function and disk_status functions are extended to handle multiple log device paths. - Current interface disk_ctr(): log_path region_size [[no]sync] disk_status() - STATUSTYPE_INFO: nr_params disk log_path log_status:"A" or "D" disk_status() - STATUSTYPE_TABLE: disk nr_params log_path region_size [[no]sync] - Proposed interface disk_ctr(): [log_path]{1,} region_size [[no]sync] disk_status() - STATUSTYPE_INFO: nr_params disk [log_path]{1,} [log_status:"A" or "D"]{1,} disk_status() - STATUSTYPE_TABLE: disk nr_params [log_path]{1,} region_size [[no]sync] Signed-off-by: Takahiro Yasui redhat.com> --- drivers/md/dm-log.c | 75 ++++++++++++++++++++++++++++++++++++++------------ drivers/md/dm-raid1.c | 2 - 2 files changed, 59 insertions(+), 18 deletions(-) Index: linux-2.6.28-rc4/drivers/md/dm-log.c =================================================================== --- linux-2.6.28-rc4.orig/drivers/md/dm-log.c +++ linux-2.6.28-rc4/drivers/md/dm-log.c < at >< at > -11,7 +11,7 < at >< at > #include #include #include - +#include #include #define DM_MSG_PREFIX "dirty region log" < at >< at > -536,8 +536,40 < at >< at > static size_t log_buffer_size(struct log } static int parse_params(unsigned int argc, char **argv, -uint32_t *region_size, enum sync *sync) +unsigned int *nr_logs, uint32_t *region_size, +enum sync *sync) { +unsigned long tmp_region; +int i; + +if (argc && nr_logs) { +/* + * case of disk log + * + * find region_size param from the end as a number string + * and get the number of log devices + */ +for (i = argc - 1; i >= 0; i--) +if (!strict_strtoul(argv[i], 10, &tmp_region)) +break; + +if (i < 0) { +DMWARN("invalid region size to dirty region log"); +return -EINVAL; +} + +/* limit the same number as mirror disk */ +if (i < 1 || i > DM_KCOPYD_MAX_REGIONS + 1) { +DMWARN("Invalid number of dirty region log devices"); +return -EINVAL; +} + +*nr_logs = i; + +argc -= i; +argv += i; +} + /* * check number of parameters */ < at >< at > -757,7 +789,7 < at >< at > static int core_ctr(struct dm_dirty_log enum sync sync; int r; -r = parse_params(argc, argv, ®ion_size, &sync); +r = parse_params(argc, argv, NULL, ®ion_size, &sync); if (r) return r; < at >< at > -786,26 +818,22 < at >< at > static void core_dtr(struct dm_dirty_log /*---------------------------------------------------------------- * disks log constructor/destructor * - * argv contains log_device region_size followed optionally by [no]sync + * argv contains [log_path]{1,} region_size [[no]sync] *--------------------------------------------------------------*/ static int disk_ctr(struct dm_dirty_log *log, struct dm_target *ti, unsigned int argc, char **argv) { struct log_c *lc; +unsigned int nr_logs; uint32_t region_size; enum sync sync; int r; -if (!argc) { -DMWARN("wrong number of arguments to dirty region log"); -return -EINVAL; -} - -r = parse_params(argc-1, argv+1, ®ion_size, &sync); +r = parse_params(argc, argv, &nr_logs, ®ion_size, &sync); if (r) return r; -lc = create_log_context(ti, 1, region_size, sync); +lc = create_log_context(ti, nr_logs, region_size, sync); if (!lc) return -ENOMEM; < at >< at > -1032,19 +1060,32 < at >< at > static int core_status(struct dm_dirty_l static int disk_status(struct dm_dirty_log *log, status_type_t status, char *result, unsigned int maxlen) { -int sz = 0; struct log_c *lc = log->context; +char buffer[lc->nr_logs + 1]; +unsigned int i; +int sz = 0; +int params; switch(status) { case STATUSTYPE_INFO: -DMEMIT("3 %s %s %c", log->type->name, lc->log[0].dev->name, - lc->log[0].failed ? 'D' : 'A'); +DMEMIT("%d %s ", lc->nr_logs + 2, log->type->name); +for (i = 0; i < lc->nr_logs; i++) { +DMEMIT("%s ", lc->log[i].dev->name); +buffer[i] = lc->log[i].failed ? 'D' : 'A'; +} +buffer[i] = '\0'; +DMEMIT("%s", buffer); break; case STATUSTYPE_TABLE: -DMEMIT("%s %u %s %u ", log->type->name, - lc->sync == DEFAULTSYNC ? 2 : 3, lc->log[0].dev->name, - lc->region_size); +params = lc->nr_logs + 1; +params += (lc->sync == DEFAULTSYNC) ? 0 : 1; +DMEMIT("%s %d ", log->type->name, params); + +for (i = 0; i < lc->nr_logs; i++) +DMEMIT("%s ", lc->log[i].dev->name); + +DMEMIT("%u ", lc->region_size); DMEMIT_SYNC; } Index: linux-2.6.28-rc4/drivers/md/dm-raid1.c =================================================================== --- linux-2.6.28-rc4.orig/drivers/md/dm-raid1.c +++ linux-2.6.28-rc4/drivers/md/dm-raid1.c < at >< at > -926,7 +926,7 < at >< at > static int parse_features(struct mirror_ * [#features ] * * log_type is "core" or "disk" - * #log_params is between 1 and 3 + * #log_params is more than 0 * * If present, features must be "handle_errors". */

[RFC][PATCH 2/4] dm-log: unify rw_header toread/write_header

Takahiro Yasui — 2008-11-26T00:01:39

rw_header function updates three members of io_req data every time when I/O is processed. bi_rw and notify.fn are never modified once they get initialized, and also they can be set in advance. This patch removes unnecessary update operations. Signed-off-by: Takahiro Yasui redhat.com> --- drivers/md/dm-log.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) Index: linux-2.6.28-rc4/drivers/md/dm-log.c =================================================================== --- linux-2.6.28-rc4.orig/drivers/md/dm-log.c +++ linux-2.6.28-rc4/drivers/md/dm-log.c < at >< at > -323,20 +323,13 < at >< at > static void header_from_disk(struct log_ core->nr_regions = le64_to_cpu(disk->nr_regions); } -static int rw_header(struct log_c *lc, int rw) -{ -lc->io_req.bi_rw = rw; -lc->io_req.mem.ptr.vma = lc->disk_header; -lc->io_req.notify.fn = NULL; - -return dm_io(&lc->io_req, 1, &lc->header_location, NULL); -} - static int read_header(struct log_c *log) { int r; -r = rw_header(log, READ); +log->io_req.bi_rw = READ; + +r = dm_io(&log->io_req, 1, &log->header_location, NULL); if (r) return r; < at >< at > -364,8 +357,8 < at >< at > static int read_header(struct log_c *log static inline int write_header(struct log_c *log) { -header_to_disk(&log->header, log->disk_header); -return rw_header(log, WRITE); +log->io_req.bi_rw = WRITE; +return dm_io(&log->io_req, 1, &log->header_location, NULL); } /*---------------------------------------------------------------- < at >< at > -454,7 +447,10 < at >< at > static int create_log_context(struct dm_ buf_size = dm_round_up((LOG_OFFSET << SECTOR_SHIFT) + bitset_size, ti->limits.hardsect_size); lc->header_location.count = buf_size >> SECTOR_SHIFT; + lc->io_req.mem.type = DM_IO_VMA; +lc->io_req.mem.ptr.vma = lc->disk_header; +lc->io_req.notify.fn = NULL; lc->io_req.client = dm_io_client_create(dm_div_up(buf_size, PAGE_SIZE)); if (IS_ERR(lc->io_req.client)) { < at >< at > -636,6 +632,9 < at >< at > static int disk_resume(struct dm_dirty_l /* set the correct number of regions in the header */ lc->header.nr_regions = lc->region_count; +/* update disk headers */ +header_to_disk(&lc->header, lc->disk_header); + /* write the new header */ r = write_header(lc); if (r) {

[RFC][PATCH 0/4] dm-log: support multi-log devices

Takahiro Yasui — 2008-11-26T00:01:35

Hi All, This is my first post to this mailing list, but let me introduce a patch set to implement multi-log devices for dm-mirror. I appreciate your kind comments and suggestions on this patch set. PATCH SET ========= 1/4: dm-log: fix no io_client_destroy 2/4: dm-log: remove unnecessary updates of io_req members 3/4: dm-log: introduce multi-log devices 4/4: dm-log: update interface to control multi-log devices BACKGROUND ========== device mapper mirroring (dm-mirror) uses "log" to keep status of consistency among mirror devices. There are two types of log, "core" and "disk". The former keep its data only on memory, and the latter can keep data on a device. When system starts booting and setup disks managed by device mapper, device mapper checks log data if each region is in sync state among mirror devices. And if not, it executes data replication from default-mirror device to others. However, once log disk breaks down, data replications are required for a whole data disks, and if a size of data disk is huge, it takes long time for disk replication, and it utilizes much system resources, such as I/O bandwidth, CPU, memory. That might cause system performance degradation until disk replication completes. This patch introduces multi-log devices for mirror target, which stores log data on multiple log devices, and decreases probability of disk replication even if one log disk has broken down. DESIGN OVERVIEW =============== * maximum number of log devices Nine devices, the same maximum number as mirror devices, can be used for log devices. * error handling during setting up a mirror device All log devices should be detected Linux kernel. If any disks are not found, a mirror construction will fail. Also, log headers are checked if all log devices have the same region numbers. New log devices are excluded from this check. (They are usually initialized by "0") * error handling during I/O operation An error has been detected on a log device after a mirror device is constructed, the device gets marked "fail". I/O operations are done only to valid log devices, and no I/O is issued on those failed devices. If all log devices fails, disk log works much like "core" log. * ioctl interface This patch does not affect "core" log interface, but change "disk" log interface. To keep backward compatibility, path names of log device are just listed without a number of log devices. disk_ctr(): log_path region_size [[no]sync] disk_status() - STATUSTYPE_INFO: nr_params disk log_path log_status:"A" or "D" disk_status() - STATUSTYPE_TABLE: disk nr_params log_path region_size [[no]sync] disk_ctr(): [log_path]{1,} region_size [[no]sync] disk_status() - STATUSTYPE_INFO: nr_params disk [log_path]{1,} [log_status:"A" or "D"]{1,} disk_status() - STATUSTYPE_TABLE: disk nr_params [log_path]{1,} region_size [[no]sync] EXAMPLES ======== * create mirror with one log device (the same operation as usual) # dmsetup create mirror --table \ "0 2097152 mirror disk 2 253:0 1024 2 253:1 0 253:2 0" # dmsetup status mirror 0 2097152 mirror 2 253:1 253:2 960/2048 1 AA 3 disk 253:0 A * create mirror with two log devices [NEW] # dmsetup create mirror --table \ "0 2097152 mirror disk 3 253:0 253:1 1024 2 253:2 0 253:3 0" # dmsetup status mirror 0 2097152 mirror 2 253:2 253:3 2048/2048 1 AA 4 disk 253:0 253:1 AA * create mirror with four log devices [NEW] # dmsetup create mirror --table \ "0 2097152 mirror disk 5 253:0 253:1 253:2 253:3 1024 \ 2 253:4 0 253:5 0" # dmsetup status mirror 0 2097152 mirror 2 253:4 253:5 316/2048 1 AA 6 disk \ 253:0 253:1 253:2 253:3 AAAA FUTURE WORKS ============ * Independent header and bitmap I/O Currently disk_header holds header and bitmap and issue I/O together. But header can be modified during construction and resume procedures, and we can remove header I/O. * Partial bitmap update bitmap size could be larger if the amount of mirror devices are huge or region size is small. In the current implementation, disk log handles bitmap as one buffer and updates a whole bitmap every time. Therefore, the amount of I/O issued to a log device could be larger than to mirror devices. Partial bitmap update can issue I/O of updated sectors. --- Takahiro Yasui Hitachi Computer Products (America) Inc.

[PATCH] Read the verbosity level from multipath.confconfiguration file

Ritesh Raj Sarraf — 2008-11-21T10:54:30

With this patch, multipathd can read the verbosity level from the multipath.conf configuration file. For example: verbosity 5 Signed-off-by: Ritesh Raj Sarraf netapp.com> --- libmultipath/config.c | 2 +- libmultipath/defaults.h | 1 + libmultipath/dict.c | 20 ++++++++++++++++++++ multipath/multipath.conf.5 | 5 +++++ 4 files changed, 27 insertions(+), 1 deletions(-) diff --git a/libmultipath/config.c b/libmultipath/config.c index 6d731d2..b49cc90 100644 --- a/libmultipath/config.c +++ b/libmultipath/config.c < at >< at > -420,7 +420,7 < at >< at > load_config (char * file) * internal defaults */ if (!conf->verbosity) -conf->verbosity = 2; +conf->verbosity = DEFAULT_VERBOSITY; conf->dev_type = DEV_NONE; conf->minio = 1000; diff --git a/libmultipath/defaults.h b/libmultipath/defaults.h index 87b155e..a943673 100644 --- a/libmultipath/defaults.h +++ b/libmultipath/defaults.h < at >< at > -11,6 +11,7 < at >< at > #define DEFAULT_NO_PATH_RETRY NO_PATH_RETRY_UNDEF #define DEFAULT_PGTIMEOUT -PGTIMEOUT_NONE #define DEFAULT_USER_FRIENDLY_NAMES 0 +#define DEFAULT_VERBOSITY2 #define DEFAULT_CHECKINT5 #define MAX_CHECKINT(a)(a << 2) diff --git a/libmultipath/dict.c b/libmultipath/dict.c index 2429a93..a6ec494 100644 --- a/libmultipath/dict.c +++ b/libmultipath/dict.c < at >< at > -33,6 +33,17 < at >< at > polling_interval_handler(vector strvec) } static int +verbosity_handler(vector strvec) +{ +char * buff; + +buff = VECTOR_SLOT(strvec, 1); +conf->verbosity = atoi(buff); + +return 0; +} + +static int udev_dir_handler(vector strvec) { conf->udev_dir = set_value(strvec); < at >< at > -1316,6 +1327,14 < at >< at > snprint_def_polling_interval (char * buff, int len, void * data) } static int +snprint_def_verbosity (char * buff, int len, void * data) +{ +if (conf->verbosity == DEFAULT_VERBOSITY) +return 0; +return snprintf(buff, len, "%i", conf->verbosity); +} + +static int snprint_def_udev_dir (char * buff, int len, void * data) { if (!conf->udev_dir) < at >< at > -1549,6 +1568,7 < at >< at > void init_keywords(void) { install_keyword_root("defaults", NULL); +install_keyword("verbosity", &verbosity_handler, &snprint_def_verbosity); install_keyword("polling_interval", &polling_interval_handler, &snprint_def_polling_interval); install_keyword("udev_dir", &udev_dir_handler, &snprint_def_udev_dir); install_keyword("multipath_dir", &multipath_dir_handler, &snprint_def_multipath_dir); diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5 index 39639dd..6c53d9d 100644 --- a/multipath/multipath.conf.5 +++ b/multipath/multipath.conf.5 < at >< at > -74,6 +74,11 < at >< at > interval between two path checks in seconds; default is directory where udev creates its device nodes; default is .I /dev .TP +.B verbosity +default verbosity. Higher values increase the verbosity level. Valid +levels are between 0 and 6; default is 2. +.I /dev +.TP .B selector The default path selector algorithm to use; they are offered by the kernel multipath target. The only currently implemented is

[PATCH] Read the verbosity level from multipath.confconfiguration file

2008-11-20T12:41:15

From: Ritesh Raj Sarraf researchut.com> With this patch, multipathd can read the verbosity level from the multipath.conf configuration file. For example: verbosity 5 Signed-off-by: Ritesh Raj Sarraf researchut.com> --- libmultipath/config.c | 2 +- libmultipath/defaults.h | 1 + libmultipath/dict.c | 20 ++++++++++++++++++++ multipath/multipath.conf.5 | 5 +++++ 4 files changed, 27 insertions(+), 1 deletions(-) diff --git a/libmultipath/config.c b/libmultipath/config.c index 6d731d2..b49cc90 100644 --- a/libmultipath/config.c +++ b/libmultipath/config.c < at >< at > -420,7 +420,7 < at >< at > load_config (char * file) * internal defaults */ if (!conf->verbosity) -conf->verbosity = 2; +conf->verbosity = DEFAULT_VERBOSITY; conf->dev_type = DEV_NONE; conf->minio = 1000; diff --git a/libmultipath/defaults.h b/libmultipath/defaults.h index 87b155e..a943673 100644 --- a/libmultipath/defaults.h +++ b/libmultipath/defaults.h < at >< at > -11,6 +11,7 < at >< at > #define DEFAULT_NO_PATH_RETRY NO_PATH_RETRY_UNDEF #define DEFAULT_PGTIMEOUT -PGTIMEOUT_NONE #define DEFAULT_USER_FRIENDLY_NAMES 0 +#define DEFAULT_VERBOSITY2 #define DEFAULT_CHECKINT5 #define MAX_CHECKINT(a)(a << 2) diff --git a/libmultipath/dict.c b/libmultipath/dict.c index 2429a93..9d26e9e 100644 --- a/libmultipath/dict.c +++ b/libmultipath/dict.c < at >< at > -33,6 +33,17 < at >< at > polling_interval_handler(vector strvec) } static int +verbosity_handler(vector strvec) +{ +char * buff; + +buff = VECTOR_SLOT(strvec, 1); +conf->verbosity = atoi(buff); + +return 0; +} + +static int udev_dir_handler(vector strvec) { conf->udev_dir = set_value(strvec); < at >< at > -1316,6 +1327,14 < at >< at > snprint_def_polling_interval (char * buff, int len, void * data) } static int +snprint_def_verbosity (char * buff, int len, void * data) +{ +if (conf->checkint == DEFAULT_VERBOSITY) +return 0; +return snprintf(buff, len, "%i", conf->verbosity); +} + +static int snprint_def_udev_dir (char * buff, int len, void * data) { if (!conf->udev_dir) < at >< at > -1549,6 +1568,7 < at >< at > void init_keywords(void) { install_keyword_root("defaults", NULL); +install_keyword("verbosity", &verbosity_handler, &snprint_def_verbosity); install_keyword("polling_interval", &polling_interval_handler, &snprint_def_polling_interval); install_keyword("udev_dir", &udev_dir_handler, &snprint_def_udev_dir); install_keyword("multipath_dir", &multipath_dir_handler, &snprint_def_multipath_dir); diff --git a/multipath/multipath.conf.5 b/multipath/multipath.conf.5 index 39639dd..6c53d9d 100644 --- a/multipath/multipath.conf.5 +++ b/multipath/multipath.conf.5 < at >< at > -74,6 +74,11 < at >< at > interval between two path checks in seconds; default is directory where udev creates its device nodes; default is .I /dev .TP +.B verbosity +default verbosity. Higher values increase the verbosity level. Valid +levels are between 0 and 6; default is 2. +.I /dev +.TP .B selector The default path selector algorithm to use; they are offered by the kernel multipath target. The only currently implemented is

[PATCH] Change occurences of netapp to ontap.

Ritesh Raj Sarraf — 2008-11-20T09:59:16

From: Ritesh Raj Sarraf researchut.com> Change all occurences of netapp to ontap to be consistent. This was proposed earlier, before the libprio change, but was deferred for compatibility reasons. Also update maintainers. --- libmultipath/prioritizers/netapp.c | 243 ------------------------------------ libmultipath/prioritizers/netapp.h | 7 - libmultipath/prioritizers/ontap.c | 243 ++++++++++++++++++++++++++++++++++++ libmultipath/prioritizers/ontap.h | 7 + 4 files changed, 250 insertions(+), 250 deletions(-) delete mode 100644 libmultipath/prioritizers/netapp.c delete mode 100644 libmultipath/prioritizers/netapp.h create mode 100644 libmultipath/prioritizers/ontap.c create mode 100644 libmultipath/prioritizers/ontap.h diff --git a/libmultipath/prioritizers/netapp.c b/libmultipath/prioritizers/netapp.c deleted file mode 100644 index 812ecf6..0000000 --- a/libmultipath/prioritizers/netapp.c +++ /dev/null < at >< at > -1,243 +0,0 < at >< at > -/* - * Copyright 2005 Network Appliance, Inc., All Rights Reserved - * Author: David Wysochanski available at davidw< at >netapp.com - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * General Public License v2 for more details. - */ - -#include -#include -#include -#include -#include -#include - -#include -#include -#include - -#define INQUIRY_CMD0x12 -#define INQUIRY_CMDLEN6 -#define DEFAULT_PRIOVAL10 -#define RESULTS_MAX256 -#define SG_TIMEOUT30000 - -#define pp_netapp_log(prio, fmt, args...) \ - condlog(prio, "%s: netapp prio: " fmt, dev, ##args) - -static void dump_cdb(unsigned char *cdb, int size) -{ -int i; -char buf[10*5+1]; -char * p = &buf[0]; - -condlog(0, "- SCSI CDB: "); -for (i=0; imasked_status, -io_hdr->host_status, io_hdr->driver_status); -if (io_hdr->sb_len_wr > 0) { -condlog(0, "- SCSI sense data: "); -for (i=0; isb_len_wr; i++) { -p += snprintf(p, 128*(io_hdr->sb_len_wr-i), "0x%02x ", - io_hdr->sbp[i]); -} -condlog(0, "%s", buf); -} -} - -/* - * Returns: - * -1: error, errno set - * 0: success - */ -static int send_gva(const char *dev, int fd, unsigned char pg, - unsigned char *results, int *results_size) -{ -unsigned char sb[128]; -unsigned char cdb[10] = {0xc0, 0, 0x1, 0xa, 0x98, 0xa, - pg, sizeof(sb), 0, 0}; -struct sg_io_hdr io_hdr; -int ret = -1; - -memset(&io_hdr, 0, sizeof (struct sg_io_hdr)); -io_hdr.interface_id = 'S'; -io_hdr.cmd_len = sizeof (cdb); -io_hdr.mx_sb_len = sizeof (sb); -io_hdr.dxfer_direction = SG_DXFER_FROM_DEV; -io_hdr.dxfer_len = *results_size; -io_hdr.dxferp = results; -io_hdr.cmdp = cdb; -io_hdr.sbp = sb; -io_hdr.timeout = SG_TIMEOUT; -io_hdr.pack_id = 0; -if (ioctl(fd, SG_IO, &io_hdr) < 0) { -pp_netapp_log(0, "SG_IO ioctl failed, errno=%d", errno); -dump_cdb(cdb, sizeof(cdb)); -goto out; -} -if (io_hdr.info & SG_INFO_OK_MASK) { -pp_netapp_log(0, "SCSI error"); -dump_cdb(cdb, sizeof(cdb)); -process_sg_error(&io_hdr); -goto out; -} - -if (results[4] != 0x0a || results[5] != 0x98 || - results[6] != 0x0a ||results[7] != 0x01) { -dump_cdb(cdb, sizeof(cdb)); -pp_netapp_log(0, "GVA return wrong format "); -pp_netapp_log(0, "results[4-7] = 0x%02x 0x%02x 0x%02x 0x%02x", -results[4], results[5], results[6], results[7]); -goto out; -} -ret = 0; - out: -return(ret); -} - -/* - * Retuns: - * -1: Unable to obtain proxy info - * 0: Device _not_ proxy path - * 1: Device _is_ proxy path - */ -static int get_proxy(const char *dev, int fd) -{ -unsigned char results[256]; -unsigned char sb[128]; -unsigned char cdb[INQUIRY_CMDLEN] = {INQUIRY_CMD, 1, 0xc1, 0, - sizeof(sb), 0}; -struct sg_io_hdr io_hdr; -int ret = -1; - -memset(&results, 0, sizeof (results)); -memset(&io_hdr, 0, sizeof (struct sg_io_hdr)); -io_hdr.interface_id = 'S'; -io_hdr.cmd_len = sizeof (cdb); -io_hdr.mx_sb_len = sizeof (sb); -io_hdr.dxfer_direction = SG_DXFER_FROM_DEV; -io_hdr.dxfer_len = sizeof (results); -io_hdr.dxferp = results; -io_hdr.cmdp = cdb; -io_hdr.sbp = sb; -io_hdr.timeout = SG_TIMEOUT; -io_hdr.pack_id = 0; -if (ioctl(fd, SG_IO, &io_hdr) < 0) { -pp_netapp_log(0, "ioctl sending inquiry command failed, " -"errno=%d", errno); -dump_cdb(cdb, sizeof(cdb)); -goto out; -} -if (io_hdr.info & SG_INFO_OK_MASK) { -pp_netapp_log(0, "SCSI error"); -dump_cdb(cdb, sizeof(cdb)); -process_sg_error(&io_hdr); -goto out; -} - -if (results[1] != 0xc1 || results[8] != 0x0a || - results[9] != 0x98 || results[10] != 0x0a || - results[11] != 0x0 || results[12] != 0xc1 || - results[13] != 0x0) { -pp_netapp_log(0,"proxy info page in unknown format - "); -pp_netapp_log(0,"results[8-13]=0x%02x 0x%02x 0x%02x 0x%02x " -"0x%02x 0x%02x", -results[8], results[9], results[10], -results[11], results[12], results[13]); -dump_cdb(cdb, sizeof(cdb)); -goto out; -} -ret = (results[19] & 0x02) >> 1; - - out: -return(ret); -} - -/* - * Returns priority of device based on device info. - * - * 4: FCP non-proxy, FCP proxy unknown, or unable to determine protocol - * 3: iSCSI HBA - * 2: iSCSI software - * 1: FCP proxy - */ -static int netapp_prio(const char *dev, int fd) -{ -unsigned char results[RESULTS_MAX]; -int results_size=RESULTS_MAX; -int rc; -int is_proxy; -int is_iscsi_software; -int is_iscsi_hardware; -int tot_len; - -is_iscsi_software = is_iscsi_hardware = is_proxy = 0; - -memset(&results, 0, sizeof (results)); -rc = send_gva(dev, fd, 0x41, results, &results_size); -if (rc == 0) { -tot_len = results[0] << 24 | results[1] << 16 | - results[2] << 8 | results[3]; -if (tot_len <= 8) { -goto try_fcp_proxy; -} -if (results[8] != 0x41) { -pp_netapp_log(0, "GVA page 0x41 error - " -"results[8] = 0x%x", results[8]); -goto try_fcp_proxy; -} -if ((strncmp((char *)&results[12], "ism_sw", 6) == 0) || - (strncmp((char *)&results[12], "iswt", 4) == 0)) { -is_iscsi_software = 1; -goto prio_select; -} -else if (strncmp((char *)&results[12], "ism_sn", 6) == 0) { -is_iscsi_hardware = 1; -goto prio_select; -} -} - - try_fcp_proxy: -rc = get_proxy(dev, fd); -if (rc >= 0) { -is_proxy = rc; -} - - prio_select: -if (is_iscsi_hardware) { -return 3; -} else if (is_iscsi_software) { -return 2; -} else { -if (is_proxy) { -return 1; -} else { -/* Either non-proxy, or couldn't get proxy info */ -return 4; -} -} -} - -int getprio (struct path * pp) -{ -return netapp_prio(pp->dev, pp->fd); -} diff --git a/libmultipath/prioritizers/netapp.h b/libmultipath/prioritizers/netapp.h deleted file mode 100644 index ec38821..0000000 --- a/libmultipath/prioritizers/netapp.h +++ /dev/null < at >< at > -1,7 +0,0 < at >< at > -#ifndef _NETAPP_H -#define _NETAPP_H - -#define PRIO_NETAPP "netapp" -int prio_netapp(struct path * pp); - -#endif diff --git a/libmultipath/prioritizers/ontap.c b/libmultipath/prioritizers/ontap.c new file mode 100644 index 0000000..812ecf6 --- /dev/null +++ b/libmultipath/prioritizers/ontap.c < at >< at > -0,0 +1,243 < at >< at > +/* + * Copyright 2005 Network Appliance, Inc., All Rights Reserved + * Author: David Wysochanski available at davidw< at >netapp.com + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License v2 for more details. + */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#define INQUIRY_CMD0x12 +#define INQUIRY_CMDLEN6 +#define DEFAULT_PRIOVAL10 +#define RESULTS_MAX256 +#define SG_TIMEOUT30000 + +#define pp_netapp_log(prio, fmt, args...) \ + condlog(prio, "%s: netapp prio: " fmt, dev, ##args) + +static void dump_cdb(unsigned char *cdb, int size) +{ +int i; +char buf[10*5+1]; +char * p = &buf[0]; + +condlog(0, "- SCSI CDB: "); +for (i=0; imasked_status, +io_hdr->host_status, io_hdr->driver_status); +if (io_hdr->sb_len_wr > 0) { +condlog(0, "- SCSI sense data: "); +for (i=0; isb_len_wr; i++) { +p += snprintf(p, 128*(io_hdr->sb_len_wr-i), "0x%02x ", + io_hdr->sbp[i]); +} +condlog(0, "%s", buf); +} +} + +/* + * Returns: + * -1: error, errno set + * 0: success + */ +static int send_gva(const char *dev, int fd, unsigned char pg, + unsigned char *results, int *results_size) +{ +unsigned char sb[128]; +unsigned char cdb[10] = {0xc0, 0, 0x1, 0xa, 0x98, 0xa, + pg, sizeof(sb), 0, 0}; +struct sg_io_hdr io_hdr; +int ret = -1; + +memset(&io_hdr, 0, sizeof (struct sg_io_hdr)); +io_hdr.interface_id = 'S'; +io_hdr.cmd_len = sizeof (cdb); +io_hdr.mx_sb_len = sizeof (sb); +io_hdr.dxfer_direction = SG_DXFER_FROM_DEV; +io_hdr.dxfer_len = *results_size; +io_hdr.dxferp = results; +io_hdr.cmdp = cdb; +io_hdr.sbp = sb; +io_hdr.timeout = SG_TIMEOUT; +io_hdr.pack_id = 0; +if (ioctl(fd, SG_IO, &io_hdr) < 0) { +pp_netapp_log(0, "SG_IO ioctl failed, errno=%d", errno); +dump_cdb(cdb, sizeof(cdb)); +goto out; +} +if (io_hdr.info & SG_INFO_OK_MASK) { +pp_netapp_log(0, "SCSI error"); +dump_cdb(cdb, sizeof(cdb)); +process_sg_error(&io_hdr); +goto out; +} + +if (results[4] != 0x0a || results[5] != 0x98 || + results[6] != 0x0a ||results[7] != 0x01) { +dump_cdb(cdb, sizeof(cdb)); +pp_netapp_log(0, "GVA return wrong format "); +pp_netapp_log(0, "results[4-7] = 0x%02x 0x%02x 0x%02x 0x%02x", +results[4], results[5], results[6], results[7]); +goto out; +} +ret = 0; + out: +return(ret); +} + +/* + * Retuns: + * -1: Unable to obtain proxy info + * 0: Device _not_ proxy path + * 1: Device _is_ proxy path + */ +static int get_proxy(const char *dev, int fd) +{ +unsigned char results[256]; +unsigned char sb[128]; +unsigned char cdb[INQUIRY_CMDLEN] = {INQUIRY_CMD, 1, 0xc1, 0, + sizeof(sb), 0}; +struct sg_io_hdr io_hdr; +int ret = -1; + +memset(&results, 0, sizeof (results)); +memset(&io_hdr, 0, sizeof (struct sg_io_hdr)); +io_hdr.interface_id = 'S'; +io_hdr.cmd_len = sizeof (cdb); +io_hdr.mx_sb_len = sizeof (sb); +io_hdr.dxfer_direction = SG_DXFER_FROM_DEV; +io_hdr.dxfer_len = sizeof (results); +io_hdr.dxferp = results; +io_hdr.cmdp = cdb; +io_hdr.sbp = sb; +io_hdr.timeout = SG_TIMEOUT; +io_hdr.pack_id = 0; +if (ioctl(fd, SG_IO, &io_hdr) < 0) { +pp_netapp_log(0, "ioctl sending inquiry command failed, " +"errno=%d", errno); +dump_cdb(cdb, sizeof(cdb)); +goto out; +} +if (io_hdr.info & SG_INFO_OK_MASK) { +pp_netapp_log(0, "SCSI error"); +dump_cdb(cdb, sizeof(cdb)); +process_sg_error(&io_hdr); +goto out; +} + +if (results[1] != 0xc1 || results[8] != 0x0a || + results[9] != 0x98 || results[10] != 0x0a || + results[11] != 0x0 || results[12] != 0xc1 || + results[13] != 0x0) { +pp_netapp_log(0,"proxy info page in unknown format - "); +pp_netapp_log(0,"results[8-13]=0x%02x 0x%02x 0x%02x 0x%02x " +"0x%02x 0x%02x", +results[8], results[9], results[10], +results[11], results[12], results[13]); +dump_cdb(cdb, sizeof(cdb)); +goto out; +} +ret = (results[19] & 0x02) >> 1; + + out: +return(ret); +} + +/* + * Returns priority of device based on device info. + * + * 4: FCP non-proxy, FCP proxy unknown, or unable to determine protocol + * 3: iSCSI HBA + * 2: iSCSI software + * 1: FCP proxy + */ +static int netapp_prio(const char *dev, int fd) +{ +unsigned char results[RESULTS_MAX]; +int results_size=RESULTS_MAX; +int rc; +int is_proxy; +int is_iscsi_software; +int is_iscsi_hardware; +int tot_len; + +is_iscsi_software = is_iscsi_hardware = is_proxy = 0; + +memset(&results, 0, sizeof (results)); +rc = send_gva(dev, fd, 0x41, results, &results_size); +if (rc == 0) { +tot_len = results[0] << 24 | results[1] << 16 | + results[2] << 8 | results[3]; +if (tot_len <= 8) { +goto try_fcp_proxy; +} +if (results[8] != 0x41) { +pp_netapp_log(0, "GVA page 0x41 error - " +"results[8] = 0x%x", results[8]); +goto try_fcp_proxy; +} +if ((strncmp((char *)&results[12], "ism_sw", 6) == 0) || + (strncmp((char *)&results[12], "iswt", 4) == 0)) { +is_iscsi_software = 1; +goto prio_select; +} +else if (strncmp((char *)&results[12], "ism_sn", 6) == 0) { +is_iscsi_hardware = 1; +goto prio_select; +} +} + + try_fcp_proxy: +rc = get_proxy(dev, fd); +if (rc >= 0) { +is_proxy = rc; +} + + prio_select: +if (is_iscsi_hardware) { +return 3; +} else if (is_iscsi_software) { +return 2; +} else { +if (is_proxy) { +return 1; +} else { +/* Either non-proxy, or couldn't get proxy info */ +return 4; +} +} +} + +int getprio (struct path * pp) +{ +return netapp_prio(pp->dev, pp->fd); +} diff --git a/libmultipath/prioritizers/ontap.h b/libmultipath/prioritizers/ontap.h new file mode 100644 index 0000000..ec38821 --- /dev/null +++ b/libmultipath/prioritizers/ontap.h < at >< at > -0,0 +1,7 < at >< at > +#ifndef _NETAPP_H +#define _NETAPP_H + +#define PRIO_NETAPP "netapp" +int prio_netapp(struct path * pp); + +#endif

[Patch 2 of 2]: Cluster-aware dirty log - log module

Jonathan Brassow — 2008-11-14T17:44:27

Please note that I have not yet tested this module (outside of compiling) with the latest kernel. The only concern I have is with the newly refactored region hash code. I'll probably get to testing this next week. brassow This patch contains a cluster-aware log module. When used by dm-raid1, device-mapper mirroring can be cluster-aware. There is a kernel component (provided in this patch) and a user space component. The kernel component implements the logging interface and passes all requests to userspace via 'connector' (a netlink wrapper). The userspace daemon is built upon OpenAIS for cluster communication and is fault tolerant. The logging interface code (found in dm-clog.c) is fairly simple and should be close to completion. I would really enjoy feedback on: 1) Connector code (found in dm-clog-tfr.c) 2) Suggestions on possible break-ups of dm-clog-tfr.h, since some of that code is meant to be used by both userspace and kernel space, and some is meant only for kernel space... (I think the code meant for both should go in include/linux, while the code just meant for kernel space should go in drivers/md... yes?) Still-needs-testing-by: Jonathan Brassow redhat.com> Index: linux-2.6/drivers/md/dm-clog-tfr.c =================================================================== --- /dev/null +++ linux-2.6/drivers/md/dm-clog-tfr.c < at >< at > -0,0 +1,272 < at >< at > +/* + * Copyright (C) 2006-2008 Red Hat, Inc. + * + * This file is released under the LGPL. + */ + +#include +#include +#include +#include +#include +#include + +#include +#include "dm-clog-tfr.h" + +#include /* Unnecessary */ + +#define SHORT_UUID(x) (strlen(x) > 8) ? ((x) + (strlen(x) - 8)) : (x) + +static uint32_t seq = 0; + +/* + * Pre-allocated space for speed + */ +#define DM_CLOG_PREALLOCED_SIZE 512 +static struct cn_msg *prealloced_cn_msg = NULL; +static struct clog_tfr *prealloced_clog_tfr = NULL; + +static struct cb_id cn_clog_id = { 0x4, 0x1 }; +static DEFINE_MUTEX(_lock); + +struct receiving_pkg { +struct list_head list; +struct completion complete; + +uint32_t seq; + +int error; +int *data_size; +char *data; +}; + +static spinlock_t receiving_list_lock = SPIN_LOCK_UNLOCKED; +static struct list_head receiving_list; + +static int dm_clog_sendto_server(struct clog_tfr *tfr) +{ +int r; +int size; +struct cn_msg *msg = prealloced_cn_msg; + +if (tfr != prealloced_clog_tfr) { +size = sizeof(struct cn_msg) + +sizeof(struct clog_tfr) + tfr->data_size; +msg = kmalloc(size, GFP_NOIO); +if (!msg) +return -ENOMEM; +memcpy((msg + 1), tfr, + sizeof(struct clog_tfr) + tfr->data_size); +} + +memset(msg, 0, sizeof(struct cn_msg)); + +msg->id.idx = cn_clog_id.idx; +msg->id.val = cn_clog_id.val; +msg->ack = 0; +msg->seq = tfr->seq; +msg->len = sizeof(struct clog_tfr) + tfr->data_size; + +r = cn_netlink_send(msg, 0, gfp_any()); + +if (msg != prealloced_cn_msg) +kfree(msg); + +return r; +} + +/* + * fill_pkg + * < at >msg + * < at >tfr + * + * Parameters can be either msg or tfr, but not both. This + * function fills in the reply for a waiting request. If just + * msg is given, then the reply is simply an ACK from userspace + * that the request was received. + * + * Returns: 0 on success, -ENOENT on failure + */ +static int fill_pkg(struct cn_msg *msg, struct clog_tfr *tfr) +{ +uint32_t rtn_seq = (msg) ? msg->seq : (tfr) ? tfr->seq : 0; +struct receiving_pkg *pkg; + +list_for_each_entry(pkg, &receiving_list, list) { +if (rtn_seq != pkg->seq) +continue; + +if (msg) { +pkg->error = -msg->ack; +/* + * If we are trying again, we will need to know our + * storage capacity. Otherwise, along with the + * error code, we make explicit that we have no data. + */ +if (pkg->error != -EAGAIN) +*(pkg->data_size) = 0; +} else if (tfr->data_size > *(pkg->data_size)) { +DMERR("Insufficient space to receive package [%s]::", + RQ_TYPE(tfr->request_type)); +DMERR(" tfr->data_size = %u", tfr->data_size); +DMERR(" *(pkg->data_size) = %u", *(pkg->data_size)); + +*(pkg->data_size) = 0; +pkg->error = -ENOSPC; +} else { +pkg->error = tfr->error; +memcpy(pkg->data, tfr->data, tfr->data_size); +*(pkg->data_size) = tfr->data_size; +} +complete(&pkg->complete); +return 0; +} + +return -ENOENT; +} + +/* + * cn_clog_callback + * < at >data + * + * This is the connector callback that delivers data + * that was sent from userspace. + */ +static void cn_clog_callback(void *data) +{ +struct cn_msg *msg = (struct cn_msg *)data; +struct clog_tfr *tfr = (struct clog_tfr *)(msg + 1); + +spin_lock(&receiving_list_lock); +if (msg->len == 0) +fill_pkg(msg, NULL); +else if (msg->len < sizeof(*tfr)) +DMERR("Incomplete message received: [%u]", msg->seq); +else +fill_pkg(NULL, tfr); +spin_unlock(&receiving_list_lock); +} + +/* + * dm_clog_consult_server + * < at >uuid: log's uuid (must be DM_UUID_LEN in size) + * < at >request_type: + * < at >data: data to tx to the server + * < at >data_size: size of data in bytes + * < at >rdata: place to put return data from server + * < at >rdata_size: value-result (amount of space given/amount of space used) + * + * Only one process at a time can communicate with the server. + * rdata_size is undefined on failure. + * + * Returns: 0 on success, -EXXX on failure + */ +int dm_clog_consult_server(const char *uuid, int request_type, + char *data, int data_size, + char *rdata, int *rdata_size) +{ +int r = 0; +int dummy = 0; +int overhead_size = sizeof(struct clog_tfr *) + sizeof(struct cn_msg); +struct clog_tfr *tfr = prealloced_clog_tfr; +struct receiving_pkg pkg; + +if (data_size > (DM_CLOG_PREALLOCED_SIZE - overhead_size)) { +DMINFO("Size of tfr exceeds preallocated size"); +/* FIXME: is kmalloc sufficient if we need this much space? */ +tfr = kzalloc(data_size + sizeof(*tfr), GFP_NOIO); +} + +if (!tfr) +return -ENOMEM; + +if (!rdata_size) +rdata_size = &dummy; +resend: +/* + * We serialize the sending of requests so we can + * use the preallocated space. + */ +mutex_lock(&_lock); + +memset(tfr, 0, DM_CLOG_PREALLOCED_SIZE - overhead_size); +memcpy(tfr->uuid, uuid, DM_UUID_LEN); +tfr->seq = seq++; +tfr->request_type = request_type; +tfr->data_size = data_size; +if (data && data_size) +memcpy(tfr->data, data, data_size); + +memset(&pkg, 0, sizeof(pkg)); +init_completion(&pkg.complete); +pkg.seq = tfr->seq; +pkg.data_size = rdata_size; +pkg.data = rdata; +spin_lock(&receiving_list_lock); +list_add(&(pkg.list), &receiving_list); +spin_unlock(&receiving_list_lock); + +r = dm_clog_sendto_server(tfr); + +mutex_unlock(&_lock); + +if (r) { +DMERR("Unable to send cluster log request [%s] to server: %d", + RQ_TYPE(request_type), r); +spin_lock(&receiving_list_lock); +list_del_init(&(pkg.list)); +spin_unlock(&receiving_list_lock); + +goto out; +} + +r = wait_for_completion_timeout(&(pkg.complete), 15 * HZ); +spin_lock(&receiving_list_lock); +list_del_init(&(pkg.list)); +spin_unlock(&receiving_list_lock); +if (!r) { +DMWARN("[%s] Request timed out: [%s/%u] - retrying", + SHORT_UUID(uuid), RQ_TYPE(request_type), pkg.seq); +goto resend; +} + +r = pkg.error; +if (r == -EAGAIN) +goto resend; + +out: +if (tfr != (struct clog_tfr *)prealloced_clog_tfr) +kfree(tfr); + +return r; +} + +int dm_clog_tfr_init(void) +{ +int r; +void *prealloced; + +INIT_LIST_HEAD(&receiving_list); + +prealloced = kmalloc(DM_CLOG_PREALLOCED_SIZE, GFP_KERNEL); +if (!prealloced) +return -ENOMEM; + +prealloced_cn_msg = prealloced; +prealloced_clog_tfr = prealloced + sizeof(struct cn_msg); + +r = cn_add_callback(&cn_clog_id, "clulog", cn_clog_callback); +if (r) { +cn_del_callback(&cn_clog_id); +return r; +} + +return 0; +} + +void dm_clog_tfr_exit(void) +{ +cn_del_callback(&cn_clog_id); +kfree(prealloced_cn_msg); +} Index: linux-2.6/drivers/md/dm-clog-tfr.h =================================================================== --- /dev/null +++ linux-2.6/drivers/md/dm-clog-tfr.h < at >< at > -0,0 +1,18 < at >< at > +/* + * Copyright (C) 2006-2008 Red Hat, Inc. + * + * This file is released under the LGPL. + */ + +#ifndef __DM_CLOG_TFR_H__ +#define __DM_CLOG_TFR_H__ + +#define DM_MSG_PREFIX "dm-log-clustered" + +int dm_clog_tfr_init(void); +void dm_clog_tfr_exit(void); +int dm_clog_consult_server(const char *uuid, int request_type, + char *data, int data_size, + char *rdata, int *rdata_size); + +#endif /* __DM_CLOG_TFR_H__ */ Index: linux-2.6/drivers/md/dm-clog.c =================================================================== --- /dev/null +++ linux-2.6/drivers/md/dm-clog.c < at >< at > -0,0 +1,786 < at >< at > +/* + * Copyright (C) 2006-2008 Red Hat, Inc. + * + * This file is released under the LGPL. + */ + +#include /* for sector_div, which is used in dm-dirty-log.h */ +#include +#include +#include + +#include +#include "dm-clog-tfr.h" + +struct flush_entry { +int type; +region_t region; +struct list_head list; +}; + +struct log_c { +struct dm_target *ti; +uint32_t region_size; +region_t region_count; +char uuid[DM_UUID_LEN]; + +char *ctr_str; /* Gives ability to restart if userspace dies */ +uint32_t ctr_size; + +/* + * in_sync_hint gets set when doing is_remote_recovering. It + * represents the first region that needs recovery. IOW, the + * first zero bit of sync_bits. This can be useful for to limit + * traffic for calls like is_remote_recovering and get_resync_work, + * but be take care in its use for anything else. + */ +uint64_t in_sync_hint; + +spinlock_t flush_lock; +struct list_head flush_list; /* only for clear and mark requests */ + +struct dm_dev *disk_log; +}; + +static mempool_t *flush_entry_pool = NULL; + +static void *flush_entry_alloc(gfp_t gfp_mask, void *pool_data) +{ +return kmalloc(sizeof(struct flush_entry), gfp_mask); +} + +static void flush_entry_free(void *element, void *pool_data) +{ +kfree(element); +} + +int cluster_do_request(struct log_c *lc, const char *uuid, int request_type, + char *data, int data_size, char *rdata, int *rdata_size) +{ +int r; + +/* + * If the server isn't there, -ESRCH is returned, + * and we must keep trying until the server is + * restored. + */ +retry: +r = dm_clog_consult_server(uuid, request_type, data, + data_size, rdata, rdata_size); + +if (r != -ESRCH) +return r; + +DMERR(" Userspace cluster log server not found."); +while (1) { +set_current_state(TASK_INTERRUPTIBLE); +schedule_timeout(2*HZ); +DMWARN("Attempting to contact cluster log server..."); +r = dm_clog_consult_server(uuid, DM_CLOG_CTR, lc->ctr_str, + lc->ctr_size, NULL, NULL); +if (!r) +break; +} +DMINFO("Reconnected to cluster log server... CTR complete"); +r = dm_clog_consult_server(uuid, DM_CLOG_RESUME, NULL, + 0, NULL, NULL); +if (!r) +goto retry; + +DMERR("Error trying to resume cluster log: %d", r); + +return -ESRCH; +} + +static int cluster_ctr(struct dm_dirty_log *log, struct dm_target *ti, + unsigned int argc, char **argv, + struct dm_dev *disk_log) +{ +int i; +int r = 0; +int str_size; +int offset = (disk_log) ? 1 : 0; +char *ctr_str = NULL; +struct log_c *lc = NULL; +uint32_t region_size; +region_t region_count; + +/* Already checked argument count */ + +if (sscanf(argv[offset], "%u", ®ion_size) != 1) { +DMWARN("Invalid region size string"); +return -EINVAL; +} + +region_count = dm_sector_div_up(ti->len, region_size); + +lc = kmalloc(sizeof(*lc), GFP_KERNEL); +if (!lc) { +DMWARN("Unable to allocate cluster log context."); +return -ENOMEM; +} + +lc->ti = ti; +lc->region_size = region_size; +lc->region_count = region_count; +lc->disk_log = disk_log; + +/* FIXME: Need to check size of uuid arg */ +memcpy(lc->uuid, argv[1 + offset], DM_UUID_LEN); +spin_lock_init(&lc->flush_lock); +INIT_LIST_HEAD(&lc->flush_list); + +for (i = 0, str_size = 0; i < argc; i++) +str_size += strlen(argv[i]) + 1; /* +1 for space between args */ + +str_size += 20; /* Max number of chars in a printed u64 number */ + +ctr_str = kzalloc(str_size, GFP_KERNEL); +if (!ctr_str) { +DMWARN("Unable to allocate memory for constructor string"); +kfree(lc); +return -ENOMEM; +} + +for (i = 0, str_size = 0; i < argc; i++) +str_size += sprintf(ctr_str + str_size, "%s ", argv[i]); +str_size += sprintf(ctr_str + str_size, "%llu", + (unsigned long long)ti->len); + +/* Send table string */ +r = dm_clog_consult_server(lc->uuid, DM_CLOG_CTR, + ctr_str, str_size, NULL, NULL); + +if (r == -ESRCH) +DMERR(" Userspace cluster log server not found"); + +if (r) { +kfree(lc); +kfree(ctr_str); +} else { +lc->ctr_str = ctr_str; +lc->ctr_size = str_size; +log->context = lc; +} + +return r; +} + +/* + * cluster_core_ctr + * < at >log + * < at >ti + * < at >argc + * < at >argv + * + * argv contains: + * [[no]sync] + * + * Returns: 0 on success, -XXX on failure + */ +static int cluster_core_ctr(struct dm_dirty_log *log, struct dm_target *ti, + unsigned int argc, char **argv) +{ +int i, r; +if ((argc < 2) || (argc > 3)) { +DMERR("Too %s arguments to clustered-core mirror log type.", + (argc < 2) ? "few" : "many"); +DMERR(" %d arguments supplied:", argc); +for (i = 0; i < argc; i++) +DMERR(" %s", argv[i]); +return -EINVAL; +} + +r = cluster_ctr(log, ti, argc, argv, NULL); + +return r; +} + + +/* + * cluster_core_ctr + * < at >log + * < at >ti + * < at >argc + * < at >argv + * + * argv contains: + * [[no]sync] + * + * Returns: 0 on success, -XXX on failure + */ +static int cluster_disk_ctr(struct dm_dirty_log *log, struct dm_target *ti, + unsigned int argc, char **argv) +{ +int r, i; +struct dm_dev *dev; + +if ((argc < 3) || (argc > 4)) { +DMERR("Too %s arguments to clustered-disk mirror log type.", + (argc < 3) ? "few" : "many"); +DMERR(" %d arguments supplied:", argc); +for (i = 0; i < argc; i++) +DMERR(" %s", argv[i]); +return -EINVAL; +} + +r = dm_get_device(ti, argv[0], 0, 0, FMODE_READ | FMODE_WRITE, &dev); +if (r) +return r; + +r = cluster_ctr(log, ti, argc, argv, dev); +if (r) +dm_put_device(ti, dev); + +return r; +} + +/* + * cluster_dtr + * < at >log + */ +static void cluster_dtr(struct dm_dirty_log *log) +{ +int r; +struct log_c *lc = (struct log_c *)log->context; + +r = dm_clog_consult_server(lc->uuid, DM_CLOG_DTR, + NULL, 0, + NULL, NULL); + +if (lc->disk_log) +dm_put_device(lc->ti, lc->disk_log); +kfree(lc->ctr_str); +kfree(lc); + +return; +} + +/* + * cluster_presuspend + * < at >log + */ +static int cluster_presuspend(struct dm_dirty_log *log) +{ +int r; +struct log_c *lc = (struct log_c *)log->context; + +r = dm_clog_consult_server(lc->uuid, DM_CLOG_PRESUSPEND, + NULL, 0, + NULL, NULL); + +return r; +} + +/* + * cluster_postsuspend + * < at >log + */ +static int cluster_postsuspend(struct dm_dirty_log *log) +{ +int r; +struct log_c *lc = (struct log_c *)log->context; + +r = dm_clog_consult_server(lc->uuid, DM_CLOG_POSTSUSPEND, + NULL, 0, + NULL, NULL); + +return r; +} + +/* + * cluster_resume + * < at >log + */ +static int cluster_resume(struct dm_dirty_log *log) +{ +int r; +struct log_c *lc = (struct log_c *)log->context; + +lc->in_sync_hint = 0; +r = dm_clog_consult_server(lc->uuid, DM_CLOG_RESUME, + NULL, 0, + NULL, NULL); + +return r; +} + +/* + * cluster_get_region_size + * < at >log + * + * Only called during mirror construction, ok to block. + * + * Returns: region size (doesn't fail) + */ +static uint32_t cluster_get_region_size(struct dm_dirty_log *log) +{ +struct log_c *lc = (struct log_c *)log->context; + +return lc->region_size; +} + +/* + * cluster_is_clean + * < at >log + * < at >region + * + * Check whether a region is clean. If there is any sort of + * failure when consulting the server, we return not clean. + * + * Returns: 1 if clean, 0 otherwise + */ +static int cluster_is_clean(struct dm_dirty_log *log, region_t region) +{ +int r; +int is_clean; +int rdata_size; +struct log_c *lc = (struct log_c *)log->context; + +rdata_size = sizeof(is_clean); +r = cluster_do_request(lc, lc->uuid, DM_CLOG_IS_CLEAN, + (char *)®ion, sizeof(region), + (char *)&is_clean, &rdata_size); + +return (r) ? 0 : is_clean; +} + +/* + * cluster_in_sync + * < at >log + * < at >region + * < at >can_block: if set, return immediately + * + * Check if the region is in-sync. If there is any sort + * of failure when consulting the server, we assume that + * the region is not in sync. + * + * Returns: 1 if in-sync, 0 if not-in-sync, -EWOULDBLOCK + */ +static int cluster_in_sync(struct dm_dirty_log *log, region_t region, int can_block) +{ +int r; +int in_sync; +int rdata_size; +struct log_c *lc = (struct log_c *)log->context; + +/* + * We can never respond directly - even if in_sync_hint is + * set. This is because another machine could see a device + * failure and mark the region out-of-sync. If we don't go + * to userspace to ask, we might think the region is in-sync + * and allow a read to pick up data that is stale. (This is + * very unlikely if a device actually fails; but it is very + * likely if a connection to one device from one machine fails.) + * + * There still might be a problem if the mirror caches the region + * state as in-sync... but then this call would not be made. So, + * that is a mirror problem. + */ +if (!can_block) +return -EWOULDBLOCK; + +rdata_size = sizeof(in_sync); +r = cluster_do_request(lc, lc->uuid, DM_CLOG_IN_SYNC, + (char *)®ion, sizeof(region), + (char *)&in_sync, &rdata_size); +return (r) ? 0 : in_sync; +} + +/* + * cluster_flush + * < at >log + * + * This function is ok to block. + * The flush happens in two stages. First, it sends all + * clear/mark requests that are on the list. Then it + * tells the server to commit them. This gives the + * server a chance to optimise the commit to the cluster + * and/or disk, instead of doing it for every request. + * + * Additionally, we could implement another thread that + * sends the requests up to the server - reducing the + * load on flush. Then the flush would have less in + * the list and be responsible for the finishing commit. + * + * Returns: 0 on success, < 0 on failure + */ +static int cluster_flush(struct dm_dirty_log *log) +{ +int r = 0; +unsigned long flags; +struct log_c *lc = (struct log_c *)log->context; +LIST_HEAD(flush_list); +struct flush_entry *fe, *tmp_fe; + +spin_lock_irqsave(&lc->flush_lock, flags); +list_splice_init(&lc->flush_list, &flush_list); +spin_unlock_irqrestore(&lc->flush_lock, flags); + +if (list_empty(&flush_list)) +return 0; + +/* + * FIXME: Count up requests, group request types, + * allocate memory to stick all requests in and + * send to server in one go. Failing the allocation, + * do it one by one. + */ + +list_for_each_entry(fe, &flush_list, list) { +r = cluster_do_request(lc, lc->uuid, fe->type, + (char *)&fe->region, + sizeof(fe->region), + NULL, NULL); +if (r) +goto fail; +} + +r = cluster_do_request(lc, lc->uuid, DM_CLOG_FLUSH, + NULL, 0, NULL, NULL); + +fail: +/* + * We can safely remove these entries, even if failure. + * Calling code will receive an error and will know that + * the log facility has failed. + */ +list_for_each_entry_safe(fe, tmp_fe, &flush_list, list) { +list_del(&fe->list); +mempool_free(fe, flush_entry_pool); +} + +if (r) +dm_table_event(lc->ti->table); + +return r; +} + +/* + * cluster_mark_region + * < at >log + * < at >region + * + * This function should avoid blocking unless absolutely required. + * (Memory allocation is valid for blocking.) + */ +static void cluster_mark_region(struct dm_dirty_log *log, region_t region) +{ +unsigned long flags; +struct log_c *lc = (struct log_c *)log->context; +struct flush_entry *fe; + +/* Wait for an allocation, but _never_ fail */ +fe = mempool_alloc(flush_entry_pool, GFP_NOIO); +BUG_ON(!fe); + +spin_lock_irqsave(&lc->flush_lock, flags); +fe->type = DM_CLOG_MARK_REGION; +fe->region = region; +list_add(&fe->list, &lc->flush_list); +spin_unlock_irqrestore(&lc->flush_lock, flags); + +return; +} + +/* + * cluster_clear_region + * < at >log + * < at >region + * + * This function must not block. + * So, the alloc can't block. In the worst case, it is ok to + * fail. It would simply mean we can't clear the region. + * Does nothing to current sync context, but does mean + * the region will be re-sync'ed on a reload of the mirror + * even though it is in-sync. + */ +static void cluster_clear_region(struct dm_dirty_log *log, region_t region) +{ +unsigned long flags; +struct log_c *lc = (struct log_c *)log->context; +struct flush_entry *fe; + +/* + * If we fail to allocate, we skip the clearing of + * the region. This doesn't hurt us in any way, except + * to cause the region to be resync'ed when the + * device is activated next time. + */ +fe = mempool_alloc(flush_entry_pool, GFP_ATOMIC); +if (!fe) { +DMERR("Failed to allocate memory to clear region."); +return; +} + +spin_lock_irqsave(&lc->flush_lock, flags); +fe->type = DM_CLOG_CLEAR_REGION; +fe->region = region; +list_add(&fe->list, &lc->flush_list); +spin_unlock_irqrestore(&lc->flush_lock, flags); + +return; +} + +/* + * cluster_get_resync_work + * < at >log + * < at >region + * + * Get a region that needs recovery. It is valid to return + * an error for this function. + * + * Returns: 1 if region filled, 0 if no work, <0 on error + */ +static int cluster_get_resync_work(struct dm_dirty_log *log, region_t *region) +{ +int r; +int rdata_size; +struct log_c *lc = (struct log_c *)log->context; +struct { int i; region_t r; } pkg; + +if (lc->in_sync_hint >= lc->region_count) +return 0; + +rdata_size = sizeof(pkg); +r = cluster_do_request(lc, lc->uuid, DM_CLOG_GET_RESYNC_WORK, + NULL, 0, + (char *)&pkg, &rdata_size); + +*region = pkg.r; +return (r) ? r : pkg.i; +} + +/* + * cluster_set_region_sync + * < at >log + * < at >region + * < at >in_sync + * + * Set the sync status of a given region. This function + * must not fail. + */ +static void cluster_set_region_sync(struct dm_dirty_log *log, + region_t region, int in_sync) +{ +int r; +struct log_c *lc = (struct log_c *)log->context; +struct { region_t r; int i; } pkg; + +pkg.r = region; +pkg.i = in_sync; + +r = cluster_do_request(lc, lc->uuid, DM_CLOG_SET_REGION_SYNC, + (char *)&pkg, sizeof(pkg), + NULL, NULL); + +/* + * It would be nice to be able to report failures. + * However, it is easy emough to detect and resolve. + */ +return; +} + +/* + * cluster_get_sync_count + * < at >log + * + * If there is any sort of failure when consulting the server, + * we assume that the sync count is zero. + * + * Returns: sync count on success, 0 on failure + */ +static region_t cluster_get_sync_count(struct dm_dirty_log *log) +{ +int r; +int rdata_size; +region_t sync_count; +struct log_c *lc = (struct log_c *)log->context; + +rdata_size = sizeof(sync_count); +r = cluster_do_request(lc, lc->uuid, DM_CLOG_GET_SYNC_COUNT, + NULL, 0, + (char *)&sync_count, &rdata_size); + +if (r) +return 0; + +if (sync_count >= lc->region_count) +lc->in_sync_hint = lc->region_count; + +return sync_count; +} + +/* + * cluster_status + * < at >log + * < at >status_type + * < at >result + * < at >maxlen + * + * Returns: amount of space consumed + */ +static int cluster_status(struct dm_dirty_log *log, status_type_t status_type, + char *result, unsigned int maxlen) +{ +int r = 0; +unsigned int sz = maxlen; +struct log_c *lc = (struct log_c *)log->context; + +switch(status_type) { +case STATUSTYPE_INFO: +r = cluster_do_request(lc, lc->uuid, DM_CLOG_STATUS_INFO, + NULL, 0, + result, &sz); +/* + * FIXME: If we fail to contact server, we should still + * populate this with parsible results + */ +break; +case STATUSTYPE_TABLE: +r = cluster_do_request(lc, lc->uuid, DM_CLOG_STATUS_TABLE, + NULL, 0, + result, &sz); +break; +} +return (r) ? 0: sz; +} + +/* + * cluster_is_remote_recovering + * < at >log + * < at >region + * + * Returns: 1 if region recovering, 0 otherwise + */ +static int cluster_is_remote_recovering(struct dm_dirty_log *log, +region_t region) +{ +int r; +struct log_c *lc = (struct log_c *)log->context; +static unsigned long long limit = 0; +struct { int is_recovering; uint64_t in_sync_hint; } pkg; +int rdata_size = sizeof(pkg); + +/* + * Once the mirror has been reported to be in-sync, + * it will never again ask for recovery work. So, + * we can safely say there is not a remote machine + * recovering if the device is in-sync. (in_sync_hint + * must be reset at resume time.) + */ +if (region < lc->in_sync_hint) +return 0; +else if (jiffies < limit) +return 1; + +limit = jiffies + (HZ / 4); +r = cluster_do_request(lc, lc->uuid, DM_CLOG_IS_REMOTE_RECOVERING, + (char *)®ion, sizeof(region), + (char *)&pkg, &rdata_size); +if (r) +return 1; + +lc->in_sync_hint = pkg.in_sync_hint; + +return pkg.is_recovering; +} + +static struct dm_dirty_log_type _clustered_core_type = { +.name = "clustered-core", +.module = THIS_MODULE, +.ctr = cluster_core_ctr, +.dtr = cluster_dtr, +.presuspend = cluster_presuspend, +.postsuspend = cluster_postsuspend, +.resume = cluster_resume, +.get_region_size = cluster_get_region_size, +.is_clean = cluster_is_clean, +.in_sync = cluster_in_sync, +.flush = cluster_flush, +.mark_region = cluster_mark_region, +.clear_region = cluster_clear_region, +.get_resync_work = cluster_get_resync_work, +.set_region_sync = cluster_set_region_sync, +.get_sync_count = cluster_get_sync_count, +.status = cluster_status, +.is_remote_recovering = cluster_is_remote_recovering, +}; + +static struct dm_dirty_log_type _clustered_disk_type = { +.name = "clustered-disk", +.module = THIS_MODULE, +.ctr = cluster_disk_ctr, +.dtr = cluster_dtr, +.presuspend = cluster_presuspend, +.postsuspend = cluster_postsuspend, +.resume = cluster_resume, +.get_region_size = cluster_get_region_size, +.is_clean = cluster_is_clean, +.in_sync = cluster_in_sync, +.flush = cluster_flush, +.mark_region = cluster_mark_region, +.clear_region = cluster_clear_region, +.get_resync_work = cluster_get_resync_work, +.set_region_sync = cluster_set_region_sync, +.get_sync_count = cluster_get_sync_count, +.status = cluster_status, +.is_remote_recovering = cluster_is_remote_recovering, +}; + +static int __init cluster_dirty_log_init(void) +{ +int r = 0; + +flush_entry_pool = mempool_create(100, flush_entry_alloc, + flush_entry_free, NULL); + +if (!flush_entry_pool) { +DMWARN("Unable to create flush_entry_pool: No memory."); +return -ENOMEM; +} + +r = dm_clog_tfr_init(); +if (r) { +DMWARN("Unable to initialize cluster log communications"); +mempool_destroy(flush_entry_pool); +return r; +} + +r = dm_dirty_log_type_register(&_clustered_core_type); +if (r) { +DMWARN("Couldn't register clustered-core dirty log type"); +dm_clog_tfr_exit(); +mempool_destroy(flush_entry_pool); +return r; +} + +r = dm_dirty_log_type_register(&_clustered_disk_type); +if (r) { +DMWARN("Couldn't register clustered-disk dirty log type"); +dm_dirty_log_type_unregister(&_clustered_core_type); +dm_clog_tfr_exit(); +mempool_destroy(flush_entry_pool); +return r; +} + +DMINFO("(built %s %s) installed", __DATE__, __TIME__); +return 0; +} + +static void __exit cluster_dirty_log_exit(void) +{ +dm_dirty_log_type_unregister(&_clustered_disk_type); +dm_dirty_log_type_unregister(&_clustered_core_type); +dm_clog_tfr_exit(); +mempool_destroy(flush_entry_pool); +DMINFO("(built %s %s) removed", __DATE__, __TIME__); +return; +} + +module_init(cluster_dirty_log_init); +module_exit(cluster_dirty_log_exit); + +MODULE_DESCRIPTION(DM_NAME " cluster-aware dirty log"); +MODULE_AUTHOR("Jonathan Brassow redhat.com>"); +MODULE_LICENSE("GPL"); Index: linux-2.6/drivers/md/Kconfig =================================================================== --- linux-2.6.orig/drivers/md/Kconfig +++ linux-2.6/drivers/md/Kconfig < at >< at > -264,6 +264,15 < at >< at > config DM_MIRROR Allow volume managers to mirror logical volumes, also needed for live data migration tools such as 'pvmove'. +config DM_CLOG + tristate "Mirror cluster logging (EXPERIMENTAL)" + depends on DM_MIRROR && EXPERIMENTAL + ---help--- + Cluster logging allows device-mapper mirroring to be + cluster-aware. Mirror devices can be used by multiple + machines at the same time. Note: this will not make + your applications cluster-aware. + config DM_ZERO tristate "Zero target" depends on BLK_DEV_DM Index: linux-2.6/drivers/md/Makefile =================================================================== --- linux-2.6.orig/drivers/md/Makefile +++ linux-2.6/drivers/md/Makefile < at >< at > -8,6 +8,7 < at >< at > dm-multipath-objs := dm-path-selector.o dm-snapshot-objs := dm-snap.o dm-exception-store.o dm-exstore-shared-objs := dm-ex-store-shared.o dm-mirror-objs:= dm-raid1.o +dm-log-clustered-objs := dm-clog.o dm-clog-tfr.o md-mod-objs := md.o bitmap.o raid456-objs:= raid5.o raid6algos.o raid6recov.o raid6tables.o \ raid6int1.o raid6int2.o raid6int4.o \ < at >< at > -37,6 +38,7 < at >< at > obj-$(CONFIG_DM_MULTIPATH)+= dm-multipa obj-$(CONFIG_DM_SNAPSHOT)+= dm-snapshot.o obj-$(CONFIG_DM_EXSTORE_SHARED)+= dm-exstore-shared.o obj-$(CONFIG_DM_MIRROR)+= dm-mirror.o dm-log.o dm-region-hash.o +obj-$(CONFIG_DM_CLOG)+= dm-log-clustered.o obj-$(CONFIG_DM_ZERO)+= dm-zero.o quiet_cmd_unroll = UNROLL $< at > Index: linux-2.6/include/linux/dm-cluster-log.h =================================================================== --- /dev/null +++ linux-2.6/include/linux/dm-cluster-log.h < at >< at > -0,0 +1,66 < at >< at > +/* + * Copyright (C) 2006-2008 Red Hat, Inc. + * + * This file is released under the LGPL. + */ + +#ifndef __DM_CLUSTER_LOG_H__ +#define __DM_CLUSTER_LOG_H__ + +#include /* For DM_UUID_LEN */ + +#define DM_CLOG_TFR_SIZE 1024 + +#define DM_CLOG_CTR 1 +#define DM_CLOG_DTR 2 +#define DM_CLOG_PRESUSPEND 3 +#define DM_CLOG_POSTSUSPEND 4 +#define DM_CLOG_RESUME 5 +#define DM_CLOG_GET_REGION_SIZE 6 +#define DM_CLOG_IS_CLEAN 7 +#define DM_CLOG_IN_SYNC 8 +#define DM_CLOG_FLUSH 9 +#define DM_CLOG_MARK_REGION 10 +#define DM_CLOG_CLEAR_REGION 11 +#define DM_CLOG_GET_RESYNC_WORK 12 +#define DM_CLOG_SET_REGION_SYNC 13 +#define DM_CLOG_GET_SYNC_COUNT 14 +#define DM_CLOG_STATUS_INFO 15 +#define DM_CLOG_STATUS_TABLE 16 +#define DM_CLOG_IS_REMOTE_RECOVERING 17 + +#define RQ_TYPE(x) \ +((x) == DM_CLOG_CTR) ? "DM_CLOG_CTR" : \ +((x) == DM_CLOG_DTR) ? "DM_CLOG_DTR" : \ +((x) == DM_CLOG_PRESUSPEND) ? "DM_CLOG_PRESUSPEND" : \ +((x) == DM_CLOG_POSTSUSPEND) ? "DM_CLOG_POSTSUSPEND" : \ +((x) == DM_CLOG_RESUME) ? "DM_CLOG_RESUME" : \ +((x) == DM_CLOG_GET_REGION_SIZE) ? "DM_CLOG_GET_REGION_SIZE" : \ +((x) == DM_CLOG_IS_CLEAN) ? "DM_CLOG_IS_CLEAN" : \ +((x) == DM_CLOG_IN_SYNC) ? "DM_CLOG_IN_SYNC" : \ +((x) == DM_CLOG_FLUSH) ? "DM_CLOG_FLUSH" : \ +((x) == DM_CLOG_MARK_REGION) ? "DM_CLOG_MARK_REGION" : \ +((x) == DM_CLOG_CLEAR_REGION) ? "DM_CLOG_CLEAR_REGION" : \ +((x) == DM_CLOG_GET_RESYNC_WORK) ? "DM_CLOG_GET_RESYNC_WORK" : \ +((x) == DM_CLOG_SET_REGION_SYNC) ? "DM_CLOG_SET_REGION_SYNC" : \ +((x) == DM_CLOG_GET_SYNC_COUNT) ? "DM_CLOG_GET_SYNC_COUNT" : \ +((x) == DM_CLOG_STATUS_INFO) ? "DM_CLOG_STATUS_INFO" : \ +((x) == DM_CLOG_STATUS_TABLE) ? "DM_CLOG_STATUS_TABLE" : \ +((x) == DM_CLOG_IS_REMOTE_RECOVERING) ? \ +"DM_CLOG_IS_REMOTE_RECOVERING" : NULL + +struct clog_tfr { +uint64_t private[2]; +char uuid[DM_UUID_LEN]; /* Ties a request to a specific mirror log */ + +int error; /* Used by server to inform of errors */ +uint32_t originator; /* Cluster ID of this machine */ + +uint32_t seq; /* Sequence number for request */ +uint32_t request_type; /* DM_CLOG_* */ +uint32_t data_size; /* How much data (not including this struct) */ + +char data[0]; +}; + +#endif /* __DM_CLUSTER_LOG_H__ */

[Patch 1 of 2]: Cluster-aware dirty log - Log APIaddition

Jonathan Brassow — 2008-11-14T17:44:21

This patch can be considered by itself. The second patch I am sending will be the actual cluster log module, but these 2 patches don't necessarily need to go together. brassow The logging API needs an extra function to make cluster mirroring possible. This new function allows us to check whether a mirror region is being recovered on another machine in the cluster. This helps us prevent simultaneous recovery I/O and process I/O to the same locations on disk. Cluster-aware log modules will implement this function. Single machine log modules will not. So, there is no performance penalty for single machine mirrors. Signed-off-by: Jonathan Brassow redhat.com> Index: linux-2.6/drivers/md/dm-raid1.c =================================================================== --- linux-2.6.orig/drivers/md/dm-raid1.c +++ linux-2.6/drivers/md/dm-raid1.c < at >< at > -588,6 +588,9 < at >< at > static void do_writes(struct mirror_set int state; struct bio *bio; struct bio_list sync, nosync, recover, *this_list = NULL; +struct bio_list requeue; +struct dm_dirty_log *log = dm_rh_dirty_log(ms->rh); +region_t region; if (!writes->head) return; < at >< at > -598,10 +601,18 < at >< at > static void do_writes(struct mirror_set bio_list_init(&sync); bio_list_init(&nosync); bio_list_init(&recover); +bio_list_init(&requeue); while ((bio = bio_list_pop(writes))) { -state = dm_rh_get_state(ms->rh, -dm_rh_bio_to_region(ms->rh, bio), 1); +region = dm_rh_bio_to_region(ms->rh, bio); + +if (log->type->is_remote_recovering && + log->type->is_remote_recovering(log, region)) { +bio_list_add(&requeue, bio); +continue; +} + +state = dm_rh_get_state(ms->rh, region, 1); switch (state) { case DM_RH_CLEAN: case DM_RH_DIRTY: < at >< at > -621,6 +632,16 < at >< at > static void do_writes(struct mirror_set } /* + * Add bios that are delayed due to remote recovery + * back on to the write queue + */ +if (unlikely(requeue.head)) { +spin_lock_irq(&ms->lock); +bio_list_merge(&ms->writes, &requeue); +spin_unlock_irq(&ms->lock); +} + +/* * Increment the pending counts for any regions that will * be written to (writes to recover regions are going to * be delayed). Index: linux-2.6/include/linux/dm-dirty-log.h =================================================================== --- linux-2.6.orig/include/linux/dm-dirty-log.h +++ linux-2.6/include/linux/dm-dirty-log.h < at >< at > -113,6 +113,16 < at >< at > struct dm_dirty_log_type { */ int (*status)(struct dm_dirty_log *log, status_type_t status_type, char *result, unsigned maxlen); + +/* + * is_remote_recovering is necessary for cluster mirroring. It provides + * a way to detect recovery on another node, so we aren't writing + * concurrently. This function is likely to block (when a cluster log + * is used). + * + * Returns: 0, 1 + */ +int (*is_remote_recovering)(struct dm_dirty_log *log, region_t region); }; int dm_dirty_log_type_register(struct dm_dirty_log_type *type);

[PATCHES] new solution for dm_any_congested crash

Mikulas Patocka — 2008-11-14T01:55:27

Hi The Chandra's patch was correct, but the problem is more serious (the same crash could happen in dm_merge_bvec, dm_unplug_all or at some other dm places), so I had to rework reference counting. These are three patches. 1. reverts Chadra's changes 2. just a little swap of two calls, to prepare for the third 3. the reference counting rework Chandra, please test the patches at your system (without your original patch) and verify that they avoid the crashes as well as your patch does. MikulasRevert dm-avoid-destroying-table-in-dm_any_congested.patch The patch is correct, however the problem is more serious. The same lazy destructor calls can happen in dm_merge_bvec, dm_unplug_all and in many other device mapper places. So, a more generalized solution needs to be developed. Signed-off-by: Mikulas Patocka redhat.com> --- drivers/md/dm.c | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-) Index: linux-2.6.28-rc4-devel/drivers/md/dm.c =================================================================== --- linux-2.6.28-rc4-devel.orig/drivers/md/dm.c2008-11-14 02:39:36.000000000 +0100 +++ linux-2.6.28-rc4-devel/drivers/md/dm.c2008-11-14 02:39:09.000000000 +0100 < at >< at > -941,24 +941,16 < at >< at > static void dm_unplug_all(struct request static int dm_any_congested(void *congested_data, int bdi_bits) { -int r = bdi_bits; -struct mapped_device *md = congested_data; -struct dm_table *map; - -atomic_inc(&md->pending); - -if (!test_bit(DMF_BLOCK_IO, &md->flags)) { -map = dm_get_table(md); -if (map) { -r = dm_table_any_congested(map, bdi_bits); -dm_table_put(map); -} -} +int r; +struct mapped_device *md = (struct mapped_device *) congested_data; +struct dm_table *map = dm_get_table(md); -if (!atomic_dec_return(&md->pending)) -/* nudge anyone waiting on suspend queue */ -wake_up(&md->wait); +if (!map || test_bit(DMF_BLOCK_IO, &md->flags)) +r = bdi_bits; +else +r = dm_table_any_congested(map, bdi_bits); +dm_table_put(map); return r; } Swap these two reference decrements ... so that the last reference to the table is dropped in __unbind. This is required for the following patch, dm-table-rework-reference-counting.patch, which will change the logic in such a way that table destructor is called only at specific points in the code. Signed-off-by: Mikulas Patocka redhat.com> --- drivers/md/dm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Index: linux-2.6.28-rc4-devel/drivers/md/dm.c =================================================================== --- linux-2.6.28-rc4-devel.orig/drivers/md/dm.c2008-11-14 02:39:09.000000000 +0100 +++ linux-2.6.28-rc4-devel/drivers/md/dm.c2008-11-14 02:43:10.000000000 +0100 < at >< at > -1322,8 +1322,8 < at >< at > void dm_put(struct mapped_device *md) dm_table_presuspend_targets(map); dm_table_postsuspend_targets(map); } -__unbind(md); dm_table_put(map); +__unbind(md); free_dev(md); } } Rework table reference counting. The code before the patch uses a reference counters on a table. When the last reference is dropped and the counter reaches zero, the table destructor is called. Table reference counters are acquired/released from upcalls from other kernel code (dm_any_congested, dm_merge_bvec, dm_unplug_all). If the reference counter reaches zero in one of the upcalls, the table destructor is called from almost random kernel code. This leads to various problems: * dm_any_congested being called under a spinlock, which calls the destructor, which calls some sleeping function. * the destructor attempting to take a lock that is already taken by the same process. * stale reference from some other kernel code keeps the table constructed, which keeps some devices open, even after successful return from "dmsetup remove". This can confuse lvm and prevent closing of underlying devices or reusing device minor numbers. The patch changes reference counting so that there is the table destructor can be called only at predetermined places. The table has always exactly one reference from either mapped_device->map or hash_cell->new_map. This reference is not counted in table->holders. A pair of dm_create_table/dm_destroy_table functions is used for table creation/destruction. Temporary references from the other code increase table->holders. A pair of dm_table_get/dm_table_put functions is used to manipulate it. When the table is about to be destroyed, we wait for table->holders to reach 0. Then, we call the table destructor. This way, the destructor is called only at specific point (dm_table_destroy function) and the above problems associated with lazy destructing can't happen. Signed-off-by: Mikulas Patocka redhat.com> --- drivers/md/dm-ioctl.c | 10 ++++------ drivers/md/dm-table.c | 27 +++++++++++++++++++++++---- drivers/md/dm.c | 7 ++++--- drivers/md/dm.h | 1 + 4 files changed, 32 insertions(+), 13 deletions(-) Index: linux-2.6.28-rc4-devel/drivers/md/dm-ioctl.c =================================================================== --- linux-2.6.28-rc4-devel.orig/drivers/md/dm-ioctl.c2008-11-14 02:36:45.000000000 +0100 +++ linux-2.6.28-rc4-devel/drivers/md/dm-ioctl.c2008-11-14 02:43:14.000000000 +0100 < at >< at > -233,7 +233,7 < at >< at > static void __hash_remove(struct hash_ce } if (hc->new_map) -dm_table_put(hc->new_map); +dm_table_destroy(hc->new_map); dm_put(hc->md); free_cell(hc); } < at >< at > -827,8 +827,8 < at >< at > static int do_resume(struct dm_ioctl *pa r = dm_swap_table(md, new_map); if (r) { +dm_table_destroy(new_map); dm_put(md); -dm_table_put(new_map); return r; } < at >< at > -836,8 +836,6 < at >< at > static int do_resume(struct dm_ioctl *pa set_disk_ro(dm_disk(md), 0); else set_disk_ro(dm_disk(md), 1); - -dm_table_put(new_map); } if (dm_suspended(md)) < at >< at > -1080,7 +1078,7 < at >< at > static int table_load(struct dm_ioctl *p } if (hc->new_map) -dm_table_put(hc->new_map); +dm_table_destroy(hc->new_map); hc->new_map = t; up_write(&_hash_lock); < at >< at > -1109,7 +1107,7 < at >< at > static int table_clear(struct dm_ioctl * } if (hc->new_map) { -dm_table_put(hc->new_map); +dm_table_destroy(hc->new_map); hc->new_map = NULL; } Index: linux-2.6.28-rc4-devel/drivers/md/dm-table.c =================================================================== --- linux-2.6.28-rc4-devel.orig/drivers/md/dm-table.c2008-11-14 02:36:45.000000000 +0100 +++ linux-2.6.28-rc4-devel/drivers/md/dm-table.c2008-11-14 02:43:14.000000000 +0100 < at >< at > -60,6 +60,21 < at >< at > struct dm_table { }; /* + * New table reference rules: + * + * The table has always exactly one reference from either mapped_device->map + * or hash_cell->new_map. This reference is not counted in table->holders. + * A pair of dm_create_table/dm_destroy_table functions is used for table + * creation/destruction. + * + * Temporary references from the other code increase table->holders. A pair + * of dm_table_get/dm_table_put functions is used to manipulate it. + * + * When the table is about to be destroyed, we wait for table->holders to + + */ + +/* * Similar to ceiling(log_size(n)) */ static unsigned int int_log(unsigned int n, unsigned int base) < at >< at > -223,7 +238,7 < at >< at > int dm_table_create(struct dm_table **re return -ENOMEM; INIT_LIST_HEAD(&t->devices); -atomic_set(&t->holders, 1); +atomic_set(&t->holders, 0); if (!num_targets) num_targets = KEYS_PER_NODE; < at >< at > -253,10 +268,14 < at >< at > static void free_devices(struct list_hea } } -static void table_destroy(struct dm_table *t) +void dm_table_destroy(struct dm_table *t) { unsigned int i; +while (atomic_read(&t->holders)) +yield(); +smp_mb(); + /* free the indexes (see dm_table_complete) */ if (t->depth >= 2) vfree(t->index[t->depth - 2]); < at >< at > -294,8 +313,8 < at >< at > void dm_table_put(struct dm_table *t) if (!t) return; -if (atomic_dec_and_test(&t->holders)) -table_destroy(t); +smp_mb__before_atomic_dec(); +atomic_dec(&t->holders); } /* Index: linux-2.6.28-rc4-devel/drivers/md/dm.c =================================================================== --- linux-2.6.28-rc4-devel.orig/drivers/md/dm.c2008-11-14 02:43:10.000000000 +0100 +++ linux-2.6.28-rc4-devel/drivers/md/dm.c2008-11-14 02:43:14.000000000 +0100 < at >< at > -1208,10 +1208,11 < at >< at > static int __bind(struct mapped_device * __set_size(md, size); -if (size == 0) +if (size == 0) { +dm_table_destroy(t); return 0; +} -dm_table_get(t); dm_table_event_callback(t, event_callback, md); write_lock(&md->map_lock); < at >< at > -1233,7 +1234,7 < at >< at > static void __unbind(struct mapped_devic write_lock(&md->map_lock); md->map = NULL; write_unlock(&md->map_lock); -dm_table_put(map); +dm_table_destroy(map); } /* Index: linux-2.6.28-rc4-devel/drivers/md/dm.h =================================================================== --- linux-2.6.28-rc4-devel.orig/drivers/md/dm.h2008-11-14 02:36:45.000000000 +0100 +++ linux-2.6.28-rc4-devel/drivers/md/dm.h2008-11-14 02:43:14.000000000 +0100 < at >< at > -36,6 +36,7 < at >< at > struct dm_table; /*----------------------------------------------------------------- * Internal table functions. *---------------------------------------------------------------*/ +void dm_table_destroy(struct dm_table *t); void dm_table_event_callback(struct dm_table *t, void (*fn)(void *), void *context); struct dm_target *dm_table_get_target(struct dm_table *t, unsigned int index);

[git pull] 2.6.28-rc5 device-mapper fixes

Alasdair G Kergon — 2008-11-13T23:49:33

Please pull the following device-mapper fixes from: master.kernel.org:/pub/scm/linux/kernel/git/agk/linux-2.6-dm.git Chandra Seetharaman (3): dm mpath: avoid attempting to activate null path dm mpath: warn if args ignored dm: avoid destroying table in dm_any_congested Heinz Mauelshagen (1): dm stripe: fix init failure Mikulas Patocka (2): dm raid1: flush workqueue before destruction dm: move pending queue wake_up end_io_acct drivers/md/dm-mpath.c | 8 ++++++-- drivers/md/dm-raid1.c | 1 + drivers/md/dm-stripe.c | 4 +++- drivers/md/dm.c | 34 +++++++++++++++++++++------------- 4 files changed, 31 insertions(+), 16 deletions(-) Thanks, Alasdair