gmane.linux.iscsi.iscsi-target.devel

Re: iscsitarget boot differencing virtual harddisk

Ross Walker — 2013-05-09T12:12:15


No, IET only supports regular/sparse files and block devices.

If you can get a differencing VHD to appear as a block device then you can export as an iSCSI target, but IET doesn't directly support VHD/VMDK images and their variants.

-Ross

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

iscsitarget boot differencing virtual hard disk

2013-05-09T06:35:06

Hi
I want to share a differencing virtual hard disk by iscsitarget?
Is iscsitarget support?

Neal
thanks
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

Re: Performance

Ross Walker — 2013-05-08T14:53:50


Also, why MaxConnections = 1? For VMware the more connections the better. I
have 2-4 connections to each target from each ESX host in the cluster.

For an example here is my ietd.conf from one of my storage servers that has
been running IET+ESXi for years:

Target iqn.1999-04.prv.mfg:storage.disk.exch2.data1
        Lun 0 Path=/dev/r10_sas.1/exch2_data1,Type=blockio
        MaxConnections          8
        InitialR2T              No
        ImmediateData           Yes
        MaxRecvDataSegmentLength 65536
        MaxXmitDataSegmentLength 65536
        MaxBurstLength          1048576
        FirstBurstLength        262144
        MaxOutstandingR2T       1
        HeaderDigest            None
        DataDigest              None
        NOPInterval             60
        NOPTimeout              180
        Wthreads                8
        QueuedCommands          64

Target iqn.1999-04.prv.mfg:storage.disk.exch1.data1
        Lun 0 Path=/dev/r10_sas.1/exch1_data1,Type=blockio
        MaxConnections

Re: Performance

Ross Walker — 2013-05-08T13:03:20


Most definitely.

Having all 6 disks as LUNs under the same target means not only will the IOs fully randomize, but you will only see 1/6 the throughput under full load.

I would never use multiple LUNs in a target, the only exception being a spanned volume where the disks will act as a single drive anyways.

-Ross

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

Re: Performance

John Jore — 2013-05-08T12:35:37

Hi!

RAID Controller Cache: (3Ware 9690SA) 3x disks, 256KB Stripe size. Available Memory = 448MB (I believe this is the controllers cache)
tw_cli /c2/u5 show wrcache
/c2/u5 Write Cache = on

tw_cli /c2/u5 show rdcache
/c2/u5 Read Cache = Intelligent

ietd.conf (6x LUNs defined):
        HeaderDigest None
        DataDigest None
        MaxConnections 1
        InitialR2T Yes
        ImmediateData No
        MaxRecvDataSegmentLength 131072
        MaxXmitDataSegmentLength 131072
        MaxBurstLength 262144
        FirstBurstLength 262144
        DefaultTime2Wait 2
        DefaultTime2Retain 20
        MaxOutstandingR2T 0
        DataPDUInOrder Yes
        DataSequenceInOrder Yes
        ErrorRecoveryLevel 0

Should I split the LUNs into 6x targets?

JJ

From: Ross Walker [mailto:rswwalker< at >gmail.com]
Sent: 8. mai 2013 00:25
To: John Jore
Cc: iscsitarget-devel< at >lists.sourceforge.net
Subject: Re: [Iscsitarget-devel] Performance

On Tue, May 7, 2013 at 3:05 AM, John Jore <john< at >jore.no<mailto:john< at >jore.no>> wrote

Re: Question regarding iet_seq_start

Ross Walker — 2013-05-07T17:20:21


Sure.

-Ross
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

Re: Question regarding iet_seq_start

Arne Redlich — 2013-05-07T17:15:20

2013/5/7 Ross Walker <rswwalker< at >gmail.com>:

[...]


Yes, I realized that too after hitting send. This is what the
seq_file.txt documentation has to say:

"It's worth noting that the iterator value returned by start() and
manipulated by the other functions is considered to be completely opaque by
the seq_file code. It can thus be anything that is useful in stepping
through the data to be output."

So I think the patch is ok. Will you apply it?

Thanks,
Arne

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may

Re: Question regarding iet_seq_start

Ross Walker — 2013-05-07T16:51:35

On Tue, May 7, 2013 at 12:46 PM, Arne Redlich
<arne.redlich< at >googlemail.com>wrote:


Fell off list in replies...

Kernel code shows seq_list_next returns either the next on the list or
NULL, no ERR_PTR handling at all.

-Ross
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

Re: Performance

Ross S. W. Walker — 2013-05-07T14:59:49

Chris Siebenmann [mailto:cks< at >cs.toronto.edu] wrote:

Well I am going off the fact of irrespective of how many IOs can be
queued at a time and how long they spend on queue, the data pump in
kernel will wait an average of 17ms for each IO to complete before
returning.


You can't trace the iSCSI events, but I have used blktrace in the
past to diagnose problems with iet kernel threads and IO ordering.

Blktrace is only good at finding where in the block layer the problem
lies once you know there is a problem in the block layer.


True, but it almost always ends up being writes starving reads due
to slow underlying disks.

A good write cache and a large read cache will help. I suggested the
OP verify whether his onboard write cache is actually enabled and I
might add that using Type=fileio on the VMFS would help provide the
iSCSI with a read cache via the page cache.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto,

Re: Performance

Chris Siebenmann — 2013-05-07T14:41:28

| > If disks arenÂt overloaded; and the queue to service I/O is 75% of the
| > time, it sounds like CPU is the issue?
| 
| I think the problem is that iostat's svctm field is completely
| guess-a-mated, so we can't go by that.

 Yes, this. iostat svctm is completely made up using bogus assumptions.

| The only thing we can go by then is atime, which shows 17ms, which means
| the volume can only handle an average IOPS of 58 under load.

 Isn't this true only if the disk handles only one request at a time?
Also, 'atime' is the total time in the queue from queue entry to the IO
finishing, including the time the IO waited before being dispatched to
the drive. The kernel sadly does not record the actual 'in flight in
the drive' time as a general stat, although you can generate this number
yourself (generally post-facto) with blktrace.

(If you're really digging into an IO problem at a low level, you very
much want to look into blktrace. Unfortunately as far as I can see there
is no way to hook IETD into blktra

Re: Performance

Ross Walker — 2013-05-07T14:25:28


Well that blows that idea.



I think the problem is that iostat's svctm field is completely
guess-a-mated, so we can't go by that.

The only thing we can go by then is atime, which shows 17ms, which means
the volume can only handle an average IOPS of 58 under load.

Does your RAID controller have write-back cache? Is it enabled?



Can you post your ietd.conf?

-Ross
------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may_______________________________________________
Iscsitarget-devel mailing list
Iscsitarget-devel< at >lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel

Re: Performance

Chris Siebenmann — 2013-05-07T14:20:27

| If disks aren't overloaded; and the queue to service I/O is 75% of the
| time, it sounds like CPU is the issue?

 Your performance stats say that your CPU is snoring; it is idle 80% of
the time or so, assuming that these stats were gathered during a time of
VM activity and high load (if not, you need to gather all of these stats
at that point).

 Note that for things like network utilization the important thing is
network utilization and traffic during activity, not from boot. You
will need to work this out by, eg, taking a snapshot of eth1's stats
before you start up a VM and then afterwards.

 You may be interested either or both of the following:
    http://utcc.utoronto.ca/~cks/space/blog/linux/SeeingNetworkBandwidth
    https://github.com/siebenmann/mxiostat/

- cks

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page bo

Re: Performance

John Jore — 2013-05-07T07:05:25

Ross,
Thanks for looking it over, but I think I'm already using the deadline scheduler:

cat /sys/block/sdh/queue/scheduler
noop anticipatory [deadline] cfq

And the kernel line in grub has this: elevator=deadline

If disks aren't overloaded; and the queue to service I/O is 75% of the time, it sounds like CPU is the issue?

The server is based on OpenFiler and doesn't "do" much. Primary function is to serve up iSCSI disks to 2 ESX hosts.
Minor CIFS traffic, some NFS. Nothing what I would consider major. (Only 1 user)

I'll grab a picture next time.

JJ

From: Ross Walker [mailto:rswwalker< at >gmail.com]
Sent: 7. mai 2013 01:54
To: John Jore
Cc: iscsitarget-devel< at >lists.sourceforge.net
Subject: Re: [Iscsitarget-devel] Performance

On Sun, May 5, 2013 at 3:27 AM, John Jore <john< at >jore.no<mailto:john< at >jore.no>> wrote:
Ross/Chris:
Thanks for pointers and help:

RAID group is healthy. Configuration is blockio and IOMode=wt. And yes, there is a UPS attached and the raid controller has a battery too.
It's my understanding

Re: Question regarding iet_seq_start

Shivaram Upadhyayula — 2013-05-06T19:14:52

Hi Arne,


On Mon, May 6, 2013 at 11:51 PM, Arne Redlich
<arne.redlich< at >googlemail.com>wrote:


The easiest fix :-)
The only issue is if the target_list_mutex can never be acquired, possibly
due to the mutex holder never completing its task. Can this happen ?

Regards,
Shivaram

Re: Question regarding iet_seq_start

Arne Redlich — 2013-05-06T18:21:21

2013/5/5 Shivaram Upadhyayula <shivaram.u< at >quadstor.com>

Shivaram,

I think your analysis is spot on - nice catch indeed. It's been quite
a while since I went near that code, but avoiding the -_interruptible
version of mutex_lock seems to be the straightforward fix (at the
expense of not being interruptible from userspace, but I don't think
that this is a problem here). Or would you suggest sth. else?

Thanks,
Arne

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and 
their applications. This 200-page book is written by three acclaimed 
leaders in the field. The early access version is available now. 
Download your free book today! http://p.sf.net/sfu/neotech_d2d_may

Re: Performance

Ross Walker — 2013-05-06T15:54:26


Yes, it is safer in case of a kernel crash. It all depends on how stable
your system is and how comfortable you are with it.

Do you have a suitably sized write-back cache on your RAID controller? If
so, then write-through on ietd is the best way to go.




I don't think there are published RealTek best practices, just collective
experiences on the different forums on the Net.



There is/was some debate on the reliability of iostat's numbers, but if the
above is to be believed, the average wait time for each IO is 17.72ms but
the average service time is 3.41ms, so it looks like they are spending over
14ms on queue! The disk utilization is only calculated at 33%, so it
doesn't look like the disks are overloaded.

What IO scheduler are you using? For storage servers you should definitely
use 'deadline' as CFQ has performance issues on storage servers.


Jumbo frames: Not enabled. No transmissions errors on any nic’s or the

The above vmstat numbers basically show low interrupts and context switches
(in/cs

Question regarding iet_seq_start

Shivaram Upadhyayula — 2013-05-05T14:41:10

Hi,

I believe that there is a possibility of an invalid unlock on
target_list_sem


        err = mutex_lock_interruptible(&target_list_mutex);
        if (err < 0)
                return ERR_PTR(err);

If acquiring the mutex fails we return an error, however it looks like that
iet_seq_stop() will still be called which unlocks the mutex. I looked at
linux/fs/seq_file.c  and it seems that irrespective of whether start()
succeeds or not, stop() will be called. I guess we just need a plain
mutex_lock()

Would my analysis be correct ? My apologies if i have missed something
obvious.

Regards,
Shivaram

Re: Performance

John Jore — 2013-05-05T07:27:35

Ross/Chris:
Thanks for pointers and help:

RAID group is healthy. Configuration is blockio and IOMode=wt. And yes, there is a UPS attached and the raid controller has a battery too.
It's my understanding that wt = 'write to disk directly, not via cache' = slower than wb, but safer if there's a crash.

RealTek best practice guides:  My google skills aren't very good as I didn't find any :(. Working my way through Brendan Gregg's posts to extract the performance numbers. Linux performance tools aren't my strength as I'm more on the Windows side, but it's never too late to learn a new skill :)


Uptime (fluxuated while grabbing stats):
15:54:35 up 22:43,  2 users,  load average: 5.75, 6.07, 7.16
17:12:22 up 1 day, 1 min,  2 users,  load average: 10.12, 10.54, 11.39

iostat -p /dev/sdh 1 1
Linux 2.6.32-131.17.1.el6-0.11.smp.gcc4.4.x86_64 ()         05/05/2013

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.58    0.00   10.00    5.00    0.00   84.42

Device:            tps   Blk_read/s   Blk

Re: Performance

Chris Siebenmann — 2013-05-03T15:55:16

| What's real issue I'm facing, is the T5600 CPU no longer able to keep
| up, or is the high load a symptom of slow disks? (or is it IET?)

 If you suspect the CPU is the limiting factor, the first step is to use
top or vmstat to check for what the CPU is doing. I suspect that you'll
find that it's mostly waiting for IO, although I could be wrong.

| On the iSCSI server, when the VMs do nothing, which is most of the
| time, 'uptime' shows a load of 0.xx something. As soon as I logon
| to a VM, the load on the iSCSI server goes up pretty fast: "load
| average: 7.18, 5.10, 3.79". And "everything" within the VMs are pretty
| sluggish and slow. It usually peak around 8 or 9 and slowly goes down
| again. This is just from logging on. (or powering up a VM)

 Note that load average is a misleading measure of CPU consumption
because it also counts processes that are waiting for the disk. Your
high 'load average' may be because a lot of IETD's kernel threads are
busy waiting for IO (all of the istdN and istiodN proce

Re: Performance

Ross S. W. Walker — 2013-05-03T13:45:15

Hi!
Looking for some guidance on whether the disk, CPU (or IET) is my issue.

First some history:
Started with two ESX hosts with max 4GB of memory each, single socket, single core. All has worked ok for a long time.
Swapped out the two ESX hosts with two new ones, 16GB memory each, single socket, quad core. Many VMs have been updated to the latest windows and latest application layer and most  of the 32GB memory is now used up. Most of the time the VMs don’t _do_ much, except tick over.

Gigabit Ethernet, and unfortunately a RealTek network card of some description. Its embedded and no slots available for a quad port Intel card I have on a desk.

The iSCSI “server” has remained mostly unchanged through all of this. 3.3GB RAM, CPU: T5600 (x64 CPU and OS).
Disks for the VMs is a small dedicated RAID5; ST3750330NS, ST3750330NS and a  ST3750528AS on a 3Ware 9690SA (Yes, it’s not an optimal setup, but has worked ok for a long time).

On the iSCSI server, when the VMs do nothing, which is most of the tim

Performance

John Jore — 2013-05-03T10:54:09

Hi!
Looking for some guidance on whether the disk, CPU (or IET) is my issue.

First some history:
Started with two ESX hosts with max 4GB of memory each, single socket, single core. All has worked ok for a long time.
Swapped out the two ESX hosts with two new ones, 16GB memory each, single socket, quad core. Many VMs have been updated to the latest windows and latest application layer and most  of the 32GB memory is now used up. Most of the time the VMs don't _do_ much, except tick over.

Gigabit Ethernet, and unfortunately a RealTek network card of some description. Its embedded and no slots available for a quad port Intel card I have on a desk.

The iSCSI "server" has remained mostly unchanged through all of this. 3.3GB RAM, CPU: T5600 (x64 CPU and OS).
Disks for the VMs is a small dedicated RAID5; ST3750330NS, ST3750330NS and a  ST3750528AS on a 3Ware 9690SA (Yes, it's not an optimal setup, but has worked ok for a long time).

On the iSCSI server, when the VMs do nothing, which is most of the time, 'uptim