http://blog.gmane.org/gmane.linux.drivers.openib hourly 1 1901-01-01T00:00+00:00 http://gmane.org/img/gmane-25t.png http://gmane.org http://comments.gmane.org/gmane.linux.drivers.openib/58875 Hi all, As far as I can tell, the only way to create an ipoib interface w/ a non-default pkey is to: 1) goto /sys/class/net/ibX/ 2) echo $MY_NEW_PKEY > create_child 3) then bring up ibX.MY_NEW_PKEY interface 3a) assuming I don't want the original ib0, bring it down (although leaving it up, I guess may not harm anything) It seems somewhat cumbersome for an administrator to script this all up if they only want 1 ipoib interface w/ a non-default pkey. The attached patch creates a module option called "default_pkey" to allow ipoib to default to a different pkey. If nothing is input, it still uses the pkey at index 0. Al Al Chu 2008-12-02T18:07:19 http://comments.gmane.org/gmane.linux.drivers.openib/58874 uverbs: return ENOSYS for unimplemented commands (not EINVAL) In the original commit (883a99c7024c5763d6d4f22d9239c133893e8d74) (Add a mask of device methods allowed for userspace), the driver returned EINVAL for unimplemented commands. This creates a problem that there is no way to differentiate between an unimplemented command and an implemented one which is incorrectly invoked (which also returns EINVAL). The fix is to have unimplemented commands return ENOSYS. Signed-off-by: Jack Morgenstein <jackm< at >dev.mellanox.co.il> --- Roland, We've got a bit of a salad here (d--ned if we do, d--ned if we don't). In userspace, we have low-level libraries put NULL in the virtual function table for unimplemented verbs. libibverbs then returns NULL for those unimplemented verbs which expect a pointer return (e.g., ibv_create_srq) (also a problem since this does not differentiate between a missing verb and an incorrectly-invoked one), and ENOSYS for verbs which expect an int returned (e.g., resize_cq). This is not consistent with what was done in the kernel for unimplemented verbs (where EINVAL is returned). Additionally, what was done in the kernel (returning EINVAL) is problematic in that it makes it impossible to differentiate an unimplemented verb from one which simply had errors in the calling parameters. IMHO, the correct fix is to have unimplemented kernel verbs return ENOSYS. **(MPI already checks for ENOSYS when deciding whether or not to use resize-cq)**. I'm not sure, though, how to handle returns from older kernels -- should all user apps check for either ENOSYS or EINVAL? What about cases where the verb actually is implemented, but incorrectly called -- in older kernels this is already a problem. What about apps which checked for ENOSYS (consistent with userspace usage), and then run with a new low-level library over an older kernel, and suddenly start getting EINVAL returns (for resize_cq, there is no "activation" bit anywhere which gets passed from kernel to the user-level)? Ouch! Any ideas? Index: infiniband/drivers/infiniband/core/uverbs_main.c =================================================================== --- infiniband.orig/drivers/infiniband/core/uverbs_main.c +++ infiniband/drivers/infiniband/core/uverbs_main.c < at >< at > -584,10 +584,12 < at >< at > static ssize_t ib_uverbs_write(struct fi if (hdr.command < 0|| hdr.command >= ARRAY_SIZE(uverbs_cmd_table) || - !uverbs_cmd_table[hdr.command]|| - !(file->device->ib_dev->uverbs_cmd_mask & (1ull << hdr.command))) + !uverbs_cmd_table[hdr.command]) return -EINVAL; +if (!(file->device->ib_dev->uverbs_cmd_mask & (1ull << hdr.command))) +return -ENOSYS; + if (!file->ucontext && hdr.command != IB_USER_VERBS_CMD_GET_CONTEXT) return -EINVAL; Jack Morgenstein 2008-12-02T17:43:44 http://comments.gmane.org/gmane.linux.drivers.openib/58872 OFED Dec 1, 2008 meeting minutes on OFED 1.4 release status =============================================== Meeting minutes on the web: http://www.openfabrics.org/txt/documentation/linux/EWG_meeting_minutes/ Meeting Summary: ============== - OFED 1.4 release: GA on Dec 9 (delayed in one day to be after the OFED release meeting) - UNH Logo testing: Rupert will ask them to start with the SW testing, so we can get a report by middle of this week Details: ======= 1. Bug list review: 1421 blo raisch< at >de.ibm.com ipoib/ehca crash when bouncing network - fixed 1413 cri jackm< at >mellanox.co.il rmmod mlx4_ib deadlock - very rare case - reduced to normal 1384 maj eli< at >mellanox.co.il netperf latency small messages increase 5% - UD mode issue - under investigation 1386 maj eli< at >mellanox.co.il ofed 1.4 - iperf tcp connected mode BW large messages dec... - under investigation 1395 maj vu< at >mellanox.com kernel panic during SRP HA test - on work 1419 maj vlad< at >mellanox.co.il Iperf-2.0.4 fails: page allocation failure. order:5 1423 maj jackm< at >mellanox.co.il RDMA_Write RC small message bw degradation jitter of 6% - doing more measurements 2. UNH Logo testing update - Have not run a lot of tests on RC5 - Should do it this week with RC6 - Need to send the results of the SW part in the middle of this week so we will know we are OK for the GA release next week 3. Issue with new Intel MPI Benchmark (IMB, known as Pallas). In OFED 1.3 we used IMB 3.0 and in OFED 1.4 we upgraded the IMB to 3.1 version. During our performance verification we found 5% latency degradation with IMB 3.1 multi PingPong benchmark (we compare the result to IMB 3.0 version). Do we wish to go back to IMB 3.0? Will wait for decision according to reply of Intel IMB owner Tziporet Tziporet Koren 2008-12-02T15:29:49 http://comments.gmane.org/gmane.linux.drivers.openib/58869 This is not really bug fix, but more cosmetic change. There is ipv6_addr_any function in the kernel, so I propose to use it instead of the same code duplicated in cma_zero_addr. From: Aleksey Senin <alekseys< at >voltaire.com> Date: Tue, 2 Dec 2008 14:52:27 +0200 Subject: [PATCH] cma_zero_addr optimized Using builtin kernel function to check zero address Signed-off-by: Aleksey Senin <alekseys< at >voltaire.com> :100644 100644 f69dda4... 17864d6... Mdrivers/infiniband/core/cma.c diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index f69dda4..17864d6 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c < at >< at > -623,16 +623,12 < at >< at > EXPORT_SYMBOL(rdma_init_qp_attr); static inline int cma_zero_addr(struct sockaddr *addr) { -struct in6_addr *ip6; - if (addr->sa_family == AF_INET) return ipv4_is_zeronet( ((struct sockaddr_in *)addr)->sin_addr.s_addr); -else { -ip6 = &((struct sockaddr_in6 *) addr)->sin6_addr; -return (ip6->s6_addr32[0] | ip6->s6_addr32[1] | -ip6->s6_addr32[2] | ip6->s6_addr32[3]) == 0; -} +else +return ipv6_addr_any( +&((struct sockaddr_in6 *)addr)->sin6_addr); } static inline int cma_loopback_addr(struct sockaddr *addr) Aleksey Senin 2008-12-02T12:58:00 http://comments.gmane.org/gmane.linux.drivers.openib/58867 From: Aleksey Senin <alekseys< at >voltaire.com> Date: Mon, 1 Dec 2008 13:55:01 +0200 Subject: [PATCH] IB addr IPv6 support Support for network discovery in addr_send_arp function. Local IPv6 address resolution. Added remote IPv6 address resolusion for RDMA CM. Function addr_resolve_remote used as wrapper for two other functions: addr4_resolve_remote ( original addr_resolve_remote ) addr6_resolve_remote ( new function ) Signed-off-by: Aleksey Senin <alekseys< at >voltaire.com> :100644 100644 f95d21f... d9170c6... Mdrivers/infiniband/core/addr.c :100644 100644 f95d21f... bc1a97b... Mdrivers/infiniband/core/addr.c diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c index f95d21f..bc1a97b 100644 --- a/drivers/infiniband/core/addr.c +++ b/drivers/infiniband/core/addr.c < at >< at > -43,6 +43,7 < at >< at > #include <net/netevent.h> #include <net/addrconf.h> #include <rdma/ib_addr.h> +#include <net/ip6_route.h> MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("IB Address Translation"); < at >< at > -172,27 +173,42 < at >< at > static void queue_req(struct addr_req *req) mutex_unlock(&lock); } -static void addr_send_arp(struct sockaddr_in *dst_in) +static void addr_send_arp(struct sockaddr *dst_in) { struct rtable *rt; struct flowi fl; -__be32 dst_ip = dst_in->sin_addr.s_addr; +struct dst_entry *dst; memset(&fl, 0, sizeof fl); -fl.nl_u.ip4_u.daddr = dst_ip; -if (ip_route_output_key(&init_net, &rt, &fl)) -return; +if (dst_in->sa_family == AF_INET) { +fl.nl_u.ip4_u.daddr = +((struct sockaddr_in *)dst_in)->sin_addr.s_addr; -neigh_event_send(rt->u.dst.neighbour, NULL); -ip_rt_put(rt); +if (ip_route_output_key(&init_net, &rt, &fl)) +return; + +neigh_event_send(rt->u.dst.neighbour, NULL); +ip_rt_put(rt); + +} else { +fl.nl_u.ip6_u.daddr = +((struct sockaddr_in6 *)dst_in)->sin6_addr; + +dst = ip6_route_output(&init_net, NULL, &fl); +if (!dst) +return; + +neigh_event_send(dst->neighbour, NULL); +dst_release(dst); +} } -static int addr_resolve_remote(struct sockaddr *src_in, - struct sockaddr *dst_in, +static int addr4_resolve_remote(struct sockaddr_in *src_in, + struct sockaddr_in *dst_in, struct rdma_dev_addr *addr) { -__be32 src_ip = ((struct sockaddr_in *)src_in)->sin_addr.s_addr; -__be32 dst_ip = ((struct sockaddr_in *)dst_in)->sin_addr.s_addr; +__be32 src_ip = src_in->sin_addr.s_addr; +__be32 dst_ip = dst_in->sin_addr.s_addr; struct flowi fl; struct rtable *rt; struct neighbour *neigh; < at >< at > -223,8 +239,8 < at >< at > static int addr_resolve_remote(struct sockaddr *src_in, } if (!src_ip) { -src_in->sa_family = dst_in->sa_family; -((struct sockaddr_in *)src_in)->sin_addr.s_addr = rt->rt_src; +src_in->sin_family = dst_in->sin_family; +src_in->sin_addr.s_addr = rt->rt_src; } ret = rdma_copy_addr(addr, neigh->dev, neigh->ha); < at >< at > -236,6 +252,47 < at >< at > out: return ret; } +static int addr6_resolve_remote(struct sockaddr_in6 *src_in, + struct sockaddr_in6 *dst_in, + struct rdma_dev_addr *addr) +{ +struct flowi fl; +struct neighbour *neigh; +struct dst_entry *dst; +int ret = -ENODATA; + +memset(&fl, 0, sizeof fl); +fl.nl_u.ip6_u.daddr = dst_in->sin6_addr; +fl.nl_u.ip6_u.saddr = src_in->sin6_addr; + +dst = ip6_route_output(&init_net, NULL, &fl); +if (!dst) +return ret; + +if (dst->dev->flags & IFF_NOARP) { +ret = rdma_copy_addr(addr, dst->dev, NULL); +} else { +neigh = dst->neighbour; +if (neigh && (neigh->nud_state & NUD_VALID)) +ret = rdma_copy_addr(addr, neigh->dev, neigh->ha); +} + +dst_release(dst); +return ret; +} + +static int addr_resolve_remote(struct sockaddr *src_in, +struct sockaddr *dst_in, +struct rdma_dev_addr *addr) +{ +if (src_in->sa_family == AF_INET) { +return addr4_resolve_remote((struct sockaddr_in *)src_in, +(struct sockaddr_in *)dst_in, addr); +} else +return addr6_resolve_remote((struct sockaddr_in6 *)src_in, +(struct sockaddr_in6 *)dst_in, addr); +} + static void process_req(struct work_struct *work) { struct addr_req *req, *temp_req; < at >< at > -279,29 +336,58 < at >< at > static int addr_resolve_local(struct sockaddr *src_in, struct rdma_dev_addr *addr) { struct net_device *dev; -__be32 src_ip = ((struct sockaddr_in *)src_in)->sin_addr.s_addr; -__be32 dst_ip = ((struct sockaddr_in *)dst_in)->sin_addr.s_addr; int ret; -dev = ip_dev_find(&init_net, dst_ip); -if (!dev) -return -EADDRNOTAVAIL; +if (dst_in->sa_family == AF_INET) { +__be32 src_ip = ((struct sockaddr_in *)src_in)->sin_addr.s_addr; +__be32 dst_ip = ((struct sockaddr_in *)dst_in)->sin_addr.s_addr; -if (ipv4_is_zeronet(src_ip)) { -src_in->sa_family = dst_in->sa_family; -((struct sockaddr_in *)src_in)->sin_addr.s_addr = dst_ip; -ret = rdma_copy_addr(addr, dev, dev->dev_addr); -} else if (ipv4_is_loopback(src_ip)) { -ret = rdma_translate_ip(dst_in, addr); -if (!ret) -memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +dev = ip_dev_find(&init_net, dst_ip); +if (!dev) +return -EADDRNOTAVAIL; + +if (ipv4_is_zeronet(src_ip)) { +src_in->sa_family = dst_in->sa_family; +((struct sockaddr_in *)src_in)->sin_addr.s_addr = dst_ip; +ret = rdma_copy_addr(addr, dev, dev->dev_addr); +} else if (ipv4_is_loopback(src_ip)) { +ret = rdma_translate_ip(dst_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} else { +ret = rdma_translate_ip(src_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} +dev_put(dev); } else { -ret = rdma_translate_ip(src_in, addr); -if (!ret) -memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +struct in6_addr *a = &((struct sockaddr_in6 *)dst_in)->sin6_addr; + +for_each_netdev(&init_net, dev) +if (ipv6_chk_addr(&init_net, &((struct sockaddr_in6 *) addr)->sin6_addr, dev, 1)) +break; + +if (!dev) +return -EADDRNOTAVAIL; + +a = &((struct sockaddr_in6 *)src_in)->sin6_addr; + +if (ipv6_addr_any(a)) { +src_in->sa_family = dst_in->sa_family; +((struct sockaddr_in6 *)src_in)->sin6_addr = +((struct sockaddr_in6 *)dst_in)->sin6_addr; +ret = rdma_copy_addr(addr, dev, dev->dev_addr); +} else if (ipv6_addr_loopback(a)) { +ret = rdma_translate_ip(dst_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} else { +ret = rdma_translate_ip(src_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} } -dev_put(dev); return ret; } < at >< at > -344,7 +430,7 < at >< at > int rdma_resolve_ip(struct rdma_addr_client *client, case -ENODATA: req->timeout = msecs_to_jiffies(timeout_ms) + jiffies; queue_req(req); -addr_send_arp((struct sockaddr_in *)dst_in); +addr_send_arp(dst_in); break; default: ret = req->status; Aleksey Senin 2008-12-02T12:44:51 http://comments.gmane.org/gmane.linux.drivers.openib/58866 Fix for bug 1311 https://bugs.openfabrics.org/show_bug.cgi?id=1311 Signed-off-by: Nicolas Morey-Chaisemartin <nicolas.morey-chaisemartin< at >ext.bull.net> --- drivers/infiniband/ulp/sdp/sdp_bcopy.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) Nicolas Morey Chaisemartin 2008-12-02T11:37:04 http://comments.gmane.org/gmane.linux.drivers.openib/58865 This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.19 Passed on ppc64 with linux-2.6.18-8.el5 Failed: Vladimir Sokolovsky (Mellanox 2008-12-02T11:25:37 http://comments.gmane.org/gmane.linux.drivers.openib/58863 Todd Bowman 2008-12-01T23:28:43 http://comments.gmane.org/gmane.linux.drivers.openib/58859 Sasha, Minor patch to declare prefix and guid in network rather than host endian order. Hal Rosenstock 2008-12-01T18:43:47 http://comments.gmane.org/gmane.linux.drivers.openib/58857 Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This will get two fixes to the ehca driver for problems in patches added in the 2.6.28 cycle, and two fixes for mlx4, one for a regression introduced in the 2.6.28 cycle, and one for a resource leak that is serious enough and has a simple enough fix to target -stable. Jack Morgenstein (2): mlx4_core: Save/restore default port IB capability mask IB/mlx4: Fix MTT leakage in resize CQ Joachim Fenkes (1): IB/ehca: Change misleading error message on memory hotplug Roland Dreier (1): Merge branches 'ehca' and 'mlx4' into for-linus Stefan Roscher (1): IB/ehca: Fix problem with generated flush work completions drivers/infiniband/hw/ehca/ehca_classes.h | 4 ++- drivers/infiniband/hw/ehca/ehca_main.c | 3 +- drivers/infiniband/hw/ehca/ehca_qp.c | 26 +++++++++++--- drivers/infiniband/hw/ehca/ehca_reqs.c | 51 +++++++++++++++++------------ drivers/infiniband/hw/mlx4/cq.c | 5 +++ drivers/net/mlx4/main.c | 8 ++++ drivers/net/mlx4/mlx4.h | 1 + drivers/net/mlx4/port.c | 39 +++++++++++++++++++++- include/linux/mlx4/device.h | 1 + 9 files changed, 107 insertions(+), 31 deletions(-) diff --git a/drivers/infiniband/hw/ehca/ehca_classes.h b/drivers/infiniband/hw/ehca/ehca_classes.h index 4df887a..7fc35cf 100644 --- a/drivers/infiniband/hw/ehca/ehca_classes.h +++ b/drivers/infiniband/hw/ehca/ehca_classes.h < at >< at > -163,7 +163,8 < at >< at > struct ehca_mod_qp_parm { /* struct for tracking if cqes have been reported to the application */ struct ehca_qmap_entry { u16 app_wr_id; -u16 reported; +u8 reported; +u8 cqe_req; }; struct ehca_queue_map { < at >< at > -171,6 +172,7 < at >< at > struct ehca_queue_map { unsigned int entries; unsigned int tail; unsigned int left_to_poll; +unsigned int next_wqe_idx; /* Idx to first wqe to be flushed */ }; struct ehca_qp { diff --git a/drivers/infiniband/hw/ehca/ehca_main.c b/drivers/infiniband/hw/ehca/ehca_main.c index bb02a86..bec7e02 100644 --- a/drivers/infiniband/hw/ehca/ehca_main.c +++ b/drivers/infiniband/hw/ehca/ehca_main.c < at >< at > -994,8 +994,7 < at >< at > static int ehca_mem_notifier(struct notifier_block *nb, if (printk_timed_ratelimit(&ehca_dmem_warn_time, 30 * 1000)) ehca_gen_err("DMEM operations are not allowed" - "as long as an ehca adapter is" - "attached to the LPAR"); + "in conjunction with eHCA"); return NOTIFY_BAD; } } diff --git a/drivers/infiniband/hw/ehca/ehca_qp.c b/drivers/infiniband/hw/ehca/ehca_qp.c index 9e05ee2..cadbf0c 100644 --- a/drivers/infiniband/hw/ehca/ehca_qp.c +++ b/drivers/infiniband/hw/ehca/ehca_qp.c < at >< at > -435,9 +435,13 < at >< at > static void reset_queue_map(struct ehca_queue_map *qmap) { int i; -qmap->tail = 0; -for (i = 0; i < qmap->entries; i++) +qmap->tail = qmap->entries - 1; +qmap->left_to_poll = 0; +qmap->next_wqe_idx = 0; +for (i = 0; i < qmap->entries; i++) { qmap->map[i].reported = 1; +qmap->map[i].cqe_req = 0; +} } /* < at >< at > -1121,6 +1125,7 < at >< at > static int calc_left_cqes(u64 wqe_p, struct ipz_queue *ipz_queue, void *wqe_v; u64 q_ofs; u32 wqe_idx; +unsigned int tail_idx; /* convert real to abs address */ wqe_p = wqe_p & (~(1UL << 63)); < at >< at > -1133,12 +1138,17 < at >< at > static int calc_left_cqes(u64 wqe_p, struct ipz_queue *ipz_queue, return -EFAULT; } +tail_idx = (qmap->tail + 1) % qmap->entries; wqe_idx = q_ofs / ipz_queue->qe_size; -if (wqe_idx < qmap->tail) -qmap->left_to_poll = (qmap->entries - qmap->tail) + wqe_idx; -else -qmap->left_to_poll = wqe_idx - qmap->tail; +/* check all processed wqes, whether a cqe is requested or not */ +while (tail_idx != wqe_idx) { +if (qmap->map[tail_idx].cqe_req) +qmap->left_to_poll++; +tail_idx = (tail_idx + 1) % qmap->entries; +} +/* save index in queue, where we have to start flushing */ +qmap->next_wqe_idx = wqe_idx; return 0; } < at >< at > -1185,10 +1195,14 < at >< at > static int check_for_left_cqes(struct ehca_qp *my_qp, struct ehca_shca *shca) } else { spin_lock_irqsave(&my_qp->send_cq->spinlock, flags); my_qp->sq_map.left_to_poll = 0; +my_qp->sq_map.next_wqe_idx = (my_qp->sq_map.tail + 1) % +my_qp->sq_map.entries; spin_unlock_irqrestore(&my_qp->send_cq->spinlock, flags); spin_lock_irqsave(&my_qp->recv_cq->spinlock, flags); my_qp->rq_map.left_to_poll = 0; +my_qp->rq_map.next_wqe_idx = (my_qp->rq_map.tail + 1) % +my_qp->rq_map.entries; spin_unlock_irqrestore(&my_qp->recv_cq->spinlock, flags); } diff --git a/drivers/infiniband/hw/ehca/ehca_reqs.c b/drivers/infiniband/hw/ehca/ehca_reqs.c index 6492807..00a648f 100644 --- a/drivers/infiniband/hw/ehca/ehca_reqs.c +++ b/drivers/infiniband/hw/ehca/ehca_reqs.c < at >< at > -179,6 +179,7 < at >< at > static inline int ehca_write_swqe(struct ehca_qp *qp, qmap_entry->app_wr_id = get_app_wr_id(send_wr->wr_id); qmap_entry->reported = 0; +qmap_entry->cqe_req = 0; switch (send_wr->opcode) { case IB_WR_SEND: < at >< at > -203,8 +204,10 < at >< at > static inline int ehca_write_swqe(struct ehca_qp *qp, if ((send_wr->send_flags & IB_SEND_SIGNALED || qp->init_attr.sq_sig_type == IB_SIGNAL_ALL_WR) - && !hidden) + && !hidden) { wqe_p->wr_flag |= WQE_WRFLAG_REQ_SIGNAL_COM; +qmap_entry->cqe_req = 1; +} if (send_wr->opcode == IB_WR_SEND_WITH_IMM || send_wr->opcode == IB_WR_RDMA_WRITE_WITH_IMM) { < at >< at > -569,6 +572,7 < at >< at > static int internal_post_recv(struct ehca_qp *my_qp, qmap_entry = &my_qp->rq_map.map[rq_map_idx]; qmap_entry->app_wr_id = get_app_wr_id(cur_recv_wr->wr_id); qmap_entry->reported = 0; +qmap_entry->cqe_req = 1; wqe_cnt++; } /* eof for cur_recv_wr */ < at >< at > -706,27 +710,34 < at >< at > repoll: goto repoll; wc->qp = &my_qp->ib_qp; +qmap_tail_idx = get_app_wr_id(cqe->work_request_id); +if (!(cqe->w_completion_flags & WC_SEND_RECEIVE_BIT)) +/* We got a send completion. */ +qmap = &my_qp->sq_map; +else +/* We got a receive completion. */ +qmap = &my_qp->rq_map; + +/* advance the tail pointer */ +qmap->tail = qmap_tail_idx; + if (is_error) { /* * set left_to_poll to 0 because in error state, we will not * get any additional CQEs */ -ehca_add_to_err_list(my_qp, 1); +my_qp->sq_map.next_wqe_idx = (my_qp->sq_map.tail + 1) % +my_qp->sq_map.entries; my_qp->sq_map.left_to_poll = 0; +ehca_add_to_err_list(my_qp, 1); +my_qp->rq_map.next_wqe_idx = (my_qp->rq_map.tail + 1) % +my_qp->rq_map.entries; +my_qp->rq_map.left_to_poll = 0; if (HAS_RQ(my_qp)) ehca_add_to_err_list(my_qp, 0); -my_qp->rq_map.left_to_poll = 0; } -qmap_tail_idx = get_app_wr_id(cqe->work_request_id); -if (!(cqe->w_completion_flags & WC_SEND_RECEIVE_BIT)) -/* We got a send completion. */ -qmap = &my_qp->sq_map; -else -/* We got a receive completion. */ -qmap = &my_qp->rq_map; - qmap_entry = &qmap->map[qmap_tail_idx]; if (qmap_entry->reported) { ehca_warn(cq->device, "Double cqe on qp_num=%#x", < at >< at > -738,10 +749,6 < at >< at > repoll: wc->wr_id = replace_wr_id(cqe->work_request_id, qmap_entry->app_wr_id); qmap_entry->reported = 1; -/* this is a proper completion, we need to advance the tail pointer */ -if (++qmap->tail == qmap->entries) -qmap->tail = 0; - /* if left_to_poll is decremented to 0, add the QP to the error list */ if (qmap->left_to_poll > 0) { qmap->left_to_poll--; < at >< at > -805,13 +812,14 < at >< at > static int generate_flush_cqes(struct ehca_qp *my_qp, struct ib_cq *cq, else qmap = &my_qp->rq_map; -qmap_entry = &qmap->map[qmap->tail]; +qmap_entry = &qmap->map[qmap->next_wqe_idx]; while ((nr < num_entries) && (qmap_entry->reported == 0)) { /* generate flush CQE */ + memset(wc, 0, sizeof(*wc)); -offset = qmap->tail * ipz_queue->qe_size; +offset = qmap->next_wqe_idx * ipz_queue->qe_size; wqe = (struct ehca_wqe *)ipz_qeit_calc(ipz_queue, offset); if (!wqe) { ehca_err(cq->device, "Invalid wqe offset=%#lx on " < at >< at > -850,11 +858,12 < at >< at > static int generate_flush_cqes(struct ehca_qp *my_qp, struct ib_cq *cq, wc->qp = &my_qp->ib_qp; -/* mark as reported and advance tail pointer */ +/* mark as reported and advance next_wqe pointer */ qmap_entry->reported = 1; -if (++qmap->tail == qmap->entries) -qmap->tail = 0; -qmap_entry = &qmap->map[qmap->tail]; +qmap->next_wqe_idx++; +if (qmap->next_wqe_idx == qmap->entries) +qmap->next_wqe_idx = 0; +qmap_entry = &qmap->map[qmap->next_wqe_idx]; wc++; nr++; } diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c index d0866a3..1830849 100644 --- a/drivers/infiniband/hw/mlx4/cq.c +++ b/drivers/infiniband/hw/mlx4/cq.c < at >< at > -343,6 +343,7 < at >< at > int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata) { struct mlx4_ib_dev *dev = to_mdev(ibcq->device); struct mlx4_ib_cq *cq = to_mcq(ibcq); +struct mlx4_mtt mtt; int outst_cqe; int err; < at >< at > -376,10 +377,13 < at >< at > int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata) goto out; } +mtt = cq->buf.mtt; + err = mlx4_cq_resize(dev->dev, &cq->mcq, entries, &cq->resize_buf->buf.mtt); if (err) goto err_buf; +mlx4_mtt_cleanup(dev->dev, &mtt); if (ibcq->uobject) { cq->buf = cq->resize_buf->buf; cq->ibcq.cqe = cq->resize_buf->cqe; < at >< at > -406,6 +410,7 < at >< at > int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata) goto out; err_buf: +mlx4_mtt_cleanup(dev->dev, &cq->resize_buf->buf.mtt); if (!ibcq->uobject) mlx4_ib_free_cq_buf(dev, &cq->resize_buf->buf, cq->resize_buf->cqe); diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 468921b..90a0281 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c < at >< at > -753,6 +753,7 < at >< at > static int mlx4_setup_hca(struct mlx4_dev *dev) struct mlx4_priv *priv = mlx4_priv(dev); int err; int port; +__be32 ib_port_default_caps; err = mlx4_init_uar_table(dev); if (err) { < at >< at > -852,6 +853,13 < at >< at > static int mlx4_setup_hca(struct mlx4_dev *dev) } for (port = 1; port <= dev->caps.num_ports; port++) { +ib_port_default_caps = 0; +err = mlx4_get_port_ib_caps(dev, port, &ib_port_default_caps); +if (err) +mlx4_warn(dev, "failed to get port %d default " + "ib capabilities (%d). Continuing with " + "caps = 0\n", port, err); +dev->caps.ib_port_def_cap[port] = ib_port_default_caps; err = mlx4_SET_PORT(dev, port); if (err) { mlx4_err(dev, "Failed to set port %d, aborting\n", diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index 56a2e21..34c909d 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h < at >< at > -385,5 +385,6 < at >< at > void mlx4_init_mac_table(struct mlx4_dev *dev, struct mlx4_mac_table *table); void mlx4_init_vlan_table(struct mlx4_dev *dev, struct mlx4_vlan_table *table); int mlx4_SET_PORT(struct mlx4_dev *dev, u8 port); +int mlx4_get_port_ib_caps(struct mlx4_dev *dev, u8 port, __be32 *caps); #endif /* MLX4_H */ diff --git a/drivers/net/mlx4/port.c b/drivers/net/mlx4/port.c index e2fdab4..0a057e5 100644 --- a/drivers/net/mlx4/port.c +++ b/drivers/net/mlx4/port.c < at >< at > -258,6 +258,42 < at >< at > out: } EXPORT_SYMBOL_GPL(mlx4_unregister_vlan); +int mlx4_get_port_ib_caps(struct mlx4_dev *dev, u8 port, __be32 *caps) +{ +struct mlx4_cmd_mailbox *inmailbox, *outmailbox; +u8 *inbuf, *outbuf; +int err; + +inmailbox = mlx4_alloc_cmd_mailbox(dev); +if (IS_ERR(inmailbox)) +return PTR_ERR(inmailbox); + +outmailbox = mlx4_alloc_cmd_mailbox(dev); +if (IS_ERR(outmailbox)) { +mlx4_free_cmd_mailbox(dev, inmailbox); +return PTR_ERR(outmailbox); +} + +inbuf = inmailbox->buf; +outbuf = outmailbox->buf; +memset(inbuf, 0, 256); +memset(outbuf, 0, 256); +inbuf[0] = 1; +inbuf[1] = 1; +inbuf[2] = 1; +inbuf[3] = 1; +*(__be16 *) (&inbuf[16]) = cpu_to_be16(0x0015); +*(__be32 *) (&inbuf[20]) = cpu_to_be32(port); + +err = mlx4_cmd_box(dev, inmailbox->dma, outmailbox->dma, port, 3, + MLX4_CMD_MAD_IFC, MLX4_CMD_TIME_CLASS_C); +if (!err) +*caps = *(__be32 *) (outbuf + 84); +mlx4_free_cmd_mailbox(dev, inmailbox); +mlx4_free_cmd_mailbox(dev, outmailbox); +return err; +} + int mlx4_SET_PORT(struct mlx4_dev *dev, u8 port) { struct mlx4_cmd_mailbox *mailbox; < at >< at > -273,7 +309,8 < at >< at > int mlx4_SET_PORT(struct mlx4_dev *dev, u8 port) ((u8 *) mailbox->buf)[3] = 6; ((__be16 *) mailbox->buf)[4] = cpu_to_be16(1 << 15); ((__be16 *) mailbox->buf)[6] = cpu_to_be16(1 << 15); -} +} else +((__be32 *) mailbox->buf)[1] = dev->caps.ib_port_def_cap[port]; err = mlx4_cmd(dev, mailbox->dma, port, is_eth, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B); diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index bd9977b..371086f 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h < at >< at > -179,6 +179,7 < at >< at > struct mlx4_caps { intnum_ports; intvl_cap[MLX4_MAX_PORTS + 1]; intib_mtu_cap[MLX4_MAX_PORTS + 1]; +__be32ib_port_def_cap[MLX4_MAX_PORTS + 1]; u64def_mac[MLX4_MAX_PORTS + 1]; inteth_mtu_cap[MLX4_MAX_PORTS + 1]; intgid_table_len[MLX4_MAX_PORTS + 1]; Roland Dreier 2008-12-01T18:16:10 http://comments.gmane.org/gmane.linux.drivers.openib/58856 This is the Agenda for the OFED meeting today: 1. Bug list review: 1421 blo raisch< at >de.ibm.com ipoib/ehca crash when bouncing network 1413 cri jackm< at >mellanox.co.il rmmod mlx4_ib deadlock - on work 1384 maj eli< at >mellanox.co.il netperf latency small messages increase 5% - UD mode issue, seems caused by LRO 1386 maj eli< at >mellanox.co.il ofed 1.4 - iperf tcp connected mode BW large messages dec... - RC mode - not reproduced for now 1395 maj vu< at >mellanox.com kernel panic during SRP HA test - on work 1419 maj vlad< at >mellanox.co.il Iperf-2.0.4 fails: page allocation failure. order:5 1423 maj jackm< at >mellanox.co.il RDMA_Write RC small message bw degradation jitter of 6% - doing more measurements 2. UNH Logo testing update 3. Issue with new Intel MPI Benchmark (IMB, known as Pallas). In OFED 1.3 we used IMB 3.0 and in OFED 1.4 we upgraded the IMB to 3.1 version. During our performance verification we found 5% latency degradation with IMB 3.1 multi PingPong benchmark (we compare the result to IMB 3.0 version). Do we wish to go back to IMB 3.0? 4. Open discussion Tziporet Tziporet Koren 2008-12-01T16:21:17 http://comments.gmane.org/gmane.linux.drivers.openib/58855 Does anyone know why this value is being altered by OFED? "net.ipv4.tcp_timestamps" is being set to 0 during OFED installation.  Default value of this parameter is set to 1 on standard RHEL/SLES distros which OFED installation script modifies to 0.  Also, when OFED is uninstalled, it does not reset these sysctl parameters to their original values. This parameter is specifically recommended to be turned ON for High performance network.  This is a TCP option that can be used to calculate the Round Trip Measurement in a better and more accurate way than the retransmission timeout method can.  Accurate value of retransmission timeout should be determined to avoid unnecessary retransmissions and hence to improve TCP performance.  RFC 1323 talks about this TCP extension for High Performance.  When the parameter net.ipv4.tcp_timestamps=1, then it adds extra 12 bytes into TCP header increasing its size.  This has an obvious effect of decrease in bandwidth as we have some extra data flowing.  Is this the reason why OFED turns it OFF to net.ipv4.tcp_timestamps=0. John Russo 2008-12-01T15:33:27 http://comments.gmane.org/gmane.linux.drivers.openib/58854 John Russo 2008-12-01T15:29:03 http://comments.gmane.org/gmane.linux.drivers.openib/58852 Set from two patches for support IPv6 protocol in RMDA CM Changes: Use two patches instead of six. This patch adds IPv6 support for IB address resolution. From: Aleksey Senin <alekseys< at >voltaire.com> Date: Mon, 1 Dec 2008 13:55:01 +0200 Subject: [PATCH] IB addr IPv6 support Support for network discovery in addr_send_arp function. Local IPv6 address resolution. Added remote IPv6 address resolusion for RDMA CM. Function addr_resolve_remote used as wrapper for two other functions: addr4_resolve_remote ( original addr_resolve_remote ) addr6_resolve_remote ( new function ) Signed-off-by: Aleksey Senin <alekseys< at >voltaire.com> :100644 100644 f95d21f... d9170c6... Mdrivers/infiniband/core/addr.c diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c index f95d21f..d9170c6 100644 --- a/drivers/infiniband/core/addr.c +++ b/drivers/infiniband/core/addr.c < at >< at > -43,6 +43,7 < at >< at > #include <net/netevent.h> #include <net/addrconf.h> #include <rdma/ib_addr.h> +#include <net/ip6_route.h> MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("IB Address Translation"); < at >< at > -172,27 +173,42 < at >< at > static void queue_req(struct addr_req *req) mutex_unlock(&lock); } -static void addr_send_arp(struct sockaddr_in *dst_in) +static void addr_send_arp(struct sockaddr *dst_in) { struct rtable *rt; struct flowi fl; -__be32 dst_ip = dst_in->sin_addr.s_addr; +struct dst_entry *dst; memset(&fl, 0, sizeof fl); -fl.nl_u.ip4_u.daddr = dst_ip; -if (ip_route_output_key(&init_net, &rt, &fl)) -return; +if (dst_in->sa_family == AF_INET) { +fl.nl_u.ip4_u.daddr = +((struct sockaddr_in *)dst_in)->sin_addr.s_addr; -neigh_event_send(rt->u.dst.neighbour, NULL); -ip_rt_put(rt); +if (ip_route_output_key(&init_net, &rt, &fl)) +return; + +neigh_event_send(rt->u.dst.neighbour, NULL); +ip_rt_put(rt); + +} else { +fl.nl_u.ip6_u.daddr = +((struct sockaddr_in6 *)dst_in)->sin6_addr; + +dst = ip6_route_output(&init_net, NULL, &fl); +if (!dst) +return; + +neigh_event_send(dst->neighbour, NULL); +dst_release(dst); +} } -static int addr_resolve_remote(struct sockaddr *src_in, - struct sockaddr *dst_in, +static int addr4_resolve_remote(struct sockaddr_in *src_in, + struct sockaddr_in *dst_in, struct rdma_dev_addr *addr) { -__be32 src_ip = ((struct sockaddr_in *)src_in)->sin_addr.s_addr; -__be32 dst_ip = ((struct sockaddr_in *)dst_in)->sin_addr.s_addr; +__be32 src_ip = src_in->sin_addr.s_addr; +__be32 dst_ip = dst_in->sin_addr.s_addr; struct flowi fl; struct rtable *rt; struct neighbour *neigh; < at >< at > -223,8 +239,8 < at >< at > static int addr_resolve_remote(struct sockaddr *src_in, } if (!src_ip) { -src_in->sa_family = dst_in->sa_family; -((struct sockaddr_in *)src_in)->sin_addr.s_addr = rt->rt_src; +src_in->sin_family = dst_in->sin_family; +src_in->sin_addr.s_addr = rt->rt_src; } ret = rdma_copy_addr(addr, neigh->dev, neigh->ha); < at >< at > -236,6 +252,47 < at >< at > out: return ret; } +static int addr6_resolve_remote(struct sockaddr_in6 *src_in, + struct sockaddr_in6 *dst_in, + struct rdma_dev_addr *addr) +{ +struct flowi fl; +struct neighbour *neigh; +struct dst_entry *dst; +int ret = -ENODATA; + +memset(&fl, 0, sizeof fl); +fl.nl_u.ip6_u.daddr = dst_in->sin6_addr; +fl.nl_u.ip6_u.saddr = src_in->sin6_addr; + +dst = ip6_route_output(&init_net, NULL, &fl); +if (!dst) +return ret; + +if (dst->dev->flags & IFF_NOARP) { +ret = rdma_copy_addr(addr, dst->dev, NULL); +} else { +neigh = dst->neighbour; +if (neigh && (neigh->nud_state & NUD_VALID)) +ret = rdma_copy_addr(addr, neigh->dev, neigh->ha); +} + +dst_release(dst); +return ret; +} + +static int addr_resolve_remote(struct sockaddr *src_in, +struct sockaddr *dst_in, +struct rdma_dev_addr *addr) +{ +if (src_in->sa_family == AF_INET) { +return addr4_resolve_remote((struct sockaddr_in *)src_in, +(struct sockaddr_in *)dst_in, addr); +} else +return addr6_resolve_remote((struct sockaddr_in6 *)src_in, +(struct sockaddr_in6 *)dst_in, addr); +} + static void process_req(struct work_struct *work) { struct addr_req *req, *temp_req; < at >< at > -279,29 +336,58 < at >< at > static int addr_resolve_local(struct sockaddr *src_in, struct rdma_dev_addr *addr) { struct net_device *dev; -__be32 src_ip = ((struct sockaddr_in *)src_in)->sin_addr.s_addr; -__be32 dst_ip = ((struct sockaddr_in *)dst_in)->sin_addr.s_addr; -int ret; +int ret = -EADDRNOTAVAIL; -dev = ip_dev_find(&init_net, dst_ip); -if (!dev) -return -EADDRNOTAVAIL; +if (dst_in->sa_family == AF_INET) { +__be32 src_ip = ((struct sockaddr_in *)src_in)->sin_addr.s_addr; +__be32 dst_ip = ((struct sockaddr_in *)dst_in)->sin_addr.s_addr; + +dev = ip_dev_find(&init_net, dst_ip); +if (!dev) +return -EADDRNOTAVAIL; -if (ipv4_is_zeronet(src_ip)) { -src_in->sa_family = dst_in->sa_family; -((struct sockaddr_in *)src_in)->sin_addr.s_addr = dst_ip; -ret = rdma_copy_addr(addr, dev, dev->dev_addr); -} else if (ipv4_is_loopback(src_ip)) { -ret = rdma_translate_ip(dst_in, addr); -if (!ret) -memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +if (ipv4_is_zeronet(src_ip)) { +src_in->sa_family = dst_in->sa_family; +((struct sockaddr_in *)src_in)->sin_addr.s_addr = dst_ip; +ret = rdma_copy_addr(addr, dev, dev->dev_addr); +} else if (ipv4_is_loopback(src_ip)) { +ret = rdma_translate_ip(dst_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} else { +ret = rdma_translate_ip(src_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} +dev_put(dev); } else { -ret = rdma_translate_ip(src_in, addr); -if (!ret) -memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +struct in6_addr *a = &((struct sockaddr_in6 *)dst_in)->sin6_addr; + +for_each_netdev(&init_net, dev) +if (ipv6_chk_addr(&init_net, &((struct sockaddr_in6 *) addr)->sin6_addr, dev, 1)) +break; + +if (!dev) +return -EADDRNOTAVAIL; + +a = &((struct sockaddr_in6 *)src_in)->sin6_addr; + +if (ipv6_addr_any(a)) { +src_in->sa_family = dst_in->sa_family; +((struct sockaddr_in6 *)src_in)->sin6_addr = +((struct sockaddr_in6 *)dst_in)->sin6_addr; +ret = rdma_copy_addr(addr, dev, dev->dev_addr); +} else if (ipv6_addr_loopback(a)) { +ret = rdma_translate_ip(dst_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} else { +ret = rdma_translate_ip(src_in, addr); +if (!ret) +memcpy(addr->dst_dev_addr, dev->dev_addr, MAX_ADDR_LEN); +} } -dev_put(dev); return ret; } < at >< at > -344,7 +430,7 < at >< at > int rdma_resolve_ip(struct rdma_addr_client *client, case -ENODATA: req->timeout = msecs_to_jiffies(timeout_ms) + jiffies; queue_req(req); -addr_send_arp((struct sockaddr_in *)dst_in); +addr_send_arp(dst_in); break; default: ret = req->status; Aleksey Senin 2008-12-01T13:16:44 http://comments.gmane.org/gmane.linux.drivers.openib/58851 mlx4_ib: Fix MTT leakage in resize_cq. MTTs associated with the old CQE buffer were not deallocated (i.e., returned to the free pool). As a result, the MTT free pool was eventually completely emptied (for users of resize_cq). Once the resize_cq command returns successfully from FW, FW no longer accesses the old CQE buffer, so it is safe to deallocate the MTT entries used by the old CQE buffer. Finally, if the resize_cq command fails, the MTTs allocated for the new CQEs buffer also need to be de-allocated. Signed-off-by: Jack Morgenstein <jackm< at >dev.mellanox.co.il> --- Roland, Resize_cq is already in kernel 2.6.27, so I think this fix should go into a "latest stable" kernel version (say, 2.6.27.8). For heavy resize_cq users, such as MPI, this fix is significant. There is no need to call mlx4_mtt_cleanup separately for userspace and kernel-space paths -- this way the call is made in only one place. Actually, we cannot call mlx4_mtt_cleanup inside a spinlock anyway, because it may sleep (it indirectly invokes mlx4_UNMAP_ICM). There is also no need to protect the mlx4_mtt_cleanup call against parallel access (spinlock), since the cleanup only is relevant to FW access to the buffer (MTT entries); the buffer itself is still available (or not!) to the driver. Finally, I save the mtt for cleanup because the switchover to the new buffer may have already occurred (in mlx4_ib_poll_one) when the resize_cq call returns (in which case cq->buf.mtt has already been overwritten with the new CQE buffer info). diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c index d0866a3..20d6833 100644 --- a/drivers/infiniband/hw/mlx4/cq.c +++ b/drivers/infiniband/hw/mlx4/cq.c < at >< at > -343,6 +343,7 < at >< at > int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata) { struct mlx4_ib_dev *dev = to_mdev(ibcq->device); struct mlx4_ib_cq *cq = to_mcq(ibcq); +struct mlx4_mtt mtt; int outst_cqe; int err; < at >< at > -376,10 +377,12 < at >< at > int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata) goto out; } +mtt = cq->buf.mtt; err = mlx4_cq_resize(dev->dev, &cq->mcq, entries, &cq->resize_buf->buf.mtt); if (err) goto err_buf; +mlx4_mtt_cleanup(dev->dev, &mtt); if (ibcq->uobject) { cq->buf = cq->resize_buf->buf; cq->ibcq.cqe = cq->resize_buf->cqe; < at >< at > -406,6 +409,7 < at >< at > int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata) goto out; err_buf: +mlx4_mtt_cleanup(dev->dev, &cq->resize_buf->buf.mtt); if (!ibcq->uobject) mlx4_ib_free_cq_buf(dev, &cq->resize_buf->buf, cq->resize_buf->cqe); diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c diff --git a/drivers/infiniband/hw/mthca/mthca_mad.c b/drivers/infiniband/hw/mthca/mthca_mad.c Jack Morgenstein 2008-12-01T12:32:38 http://comments.gmane.org/gmane.linux.drivers.openib/58850 This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.19 Passed on ppc64 with linux-2.6.18-8.el5 Failed: Vladimir Sokolovsky (Mellanox 2008-12-01T11:20:34 http://comments.gmane.org/gmane.linux.drivers.openib/58847 Hi, This is not necessary the best place to post it but I was wondering why is ofed's gitweb so slow on the main page? It takes only a few seconds to display all the repository on kernel.org (and there's a lot more) but it takes nearly a minute to display the OFED git main page... I know it's probably not the most critical issue you have to work on but I connect quite often on this page and it starts to be really bugging me. And I'm probably not the only one ;) Thanks in advance Nicolas Morey-Chaisemartin Nicolas Morey Chaisemartin 2008-12-01T07:11:31 http://comments.gmane.org/gmane.linux.drivers.openib/58835 Sasha, This patch adds language to show_usage for the --do_mesh_analysis flag. Regards, Bob Pearson Signed-off-by: Bob Pearson <rpearson< at >systemfabricworks.com> Robert Pearson 2008-11-30T18:51:53 http://comments.gmane.org/gmane.linux.drivers.openib/58834 Sasha, This patch adds some descriptive language to current_routing.txt and opensm.8.in. Regards, Bob Pearson Signed-off-by: Bob Pearson <rpearson< at >systemfabricworks.com> Robert Pearson 2008-11-30T18:39:53 http://comments.gmane.org/gmane.linux.drivers.openib/58822 This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.19 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Failed: Vladimir Sokolovsky (Mellanox 2008-11-30T11:20:52 http://comments.gmane.org/gmane.linux.drivers.openib/58821 ib_mthca: Fix dispatch of IB_EVENT_LID_CHANGE When snooping a portinfo MAD, its client_reregister bit is checked. If the bit is ON then a CLIENT_REREGISTER event is dispatched, otherwise a LID_CHANGE event is dispatched. This ignores the cases where the MAD changes the LID along with an instruction to reregister (so a necessary LID_CHANGE event won't be dispatched), or the MAD is neither of these (and an unnecessary LID_CHANGE event is dispatched). This patch dispatches an event if the client_reregister bit is set. In addition, the patch compares the LID in the MAD to the current LID. If and only if they are not identical, a LID_CHANGE event is dispatched. From: Moni Shoua <monis< at >voltaire.com> Signed-off-by: Moni Shoua <monis< at >voltaire.com> Signed-off-by: Jack Morgenstein <jackm< at >dev.mellanox.co.il> Signed-off-by: Yossi Etigin <yosefe< at >voltaire.com> --- Roland, Here is Moni's patch separated into two patches, one for mlx4_ib and one for ib_mthca. Jack Index: infiniband/drivers/infiniband/hw/mthca/mthca_mad.c =================================================================== --- infiniband.orig/drivers/infiniband/hw/mthca/mthca_mad.c2008-11-04 10:21:02.000000000 +0200 +++ infiniband/drivers/infiniband/hw/mthca/mthca_mad.c2008-11-30 09:48:35.000000000 +0200 < at >< at > -104,7 +104,8 < at >< at > static void update_sm_ah(struct mthca_de */ static void smp_snoop(struct ib_device *ibdev, u8 port_num, - struct ib_mad *mad) + struct ib_mad *mad, + u16 prev_lid) { struct ib_event event; < at >< at > -114,6 +115,7 < at >< at > static void smp_snoop(struct ib_device * if (mad->mad_hdr.attr_id == IB_SMP_ATTR_PORT_INFO) { struct ib_port_info *pinfo = (struct ib_port_info *) ((struct ib_smp *) mad)->data; +u16 lid = be16_to_cpu(pinfo->lid); mthca_update_rate(to_mdev(ibdev), port_num); update_sm_ah(to_mdev(ibdev), port_num, < at >< at > -123,12 +125,15 < at >< at > static void smp_snoop(struct ib_device * event.device = ibdev; event.element.port_num = port_num; -if (pinfo->clientrereg_resv_subnetto & 0x80) +if (pinfo->clientrereg_resv_subnetto & 0x80) { event.event = IB_EVENT_CLIENT_REREGISTER; -else +ib_dispatch_event(&event); +} +if (prev_lid != lid) { event.event = IB_EVENT_LID_CHANGE; +ib_dispatch_event(&event); +} -ib_dispatch_event(&event); } if (mad->mad_hdr.attr_id == IB_SMP_ATTR_PKEY_TABLE) { < at >< at > -196,6 +201,8 < at >< at > int mthca_process_mad(struct ib_device * int err; u8 status; u16 slid = in_wc ? in_wc->slid : be16_to_cpu(IB_LID_PERMISSIVE); +u16 prev_lid = 0; +struct ib_port_attr pattr; /* Forward locally generated traps to the SM */ if (in_mad->mad_hdr.method == IB_MGMT_METHOD_TRAP && < at >< at > -233,6 +240,12 < at >< at > int mthca_process_mad(struct ib_device * return IB_MAD_RESULT_SUCCESS; } else return IB_MAD_RESULT_SUCCESS; +if ((in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_LID_ROUTED || + in_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE) && + in_mad->mad_hdr.method == IB_MGMT_METHOD_SET && + in_mad->mad_hdr.attr_id == IB_SMP_ATTR_PORT_INFO && + !ib_query_port(ibdev, port_num, &pattr)) +prev_lid = pattr.lid; err = mthca_MAD_IFC(to_mdev(ibdev), mad_flags & IB_MAD_IGNORE_MKEY, < at >< at > -252,7 +265,7 < at >< at > int mthca_process_mad(struct ib_device * } if (!out_mad->mad_hdr.status) { -smp_snoop(ibdev, port_num, in_mad); +smp_snoop(ibdev, port_num, in_mad, prev_lid); node_desc_override(ibdev, out_mad); } Jack Morgenstein 2008-11-30T08:29:01 Search the mailing list at Gmane query http://search.gmane.org/?group=$group=gmane.linux.drivers.openib