Skip to content

Commit 522cd6a

Browse files
torvaldstorvalds
committed
Merge tag '6.16-rc-part2-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6
Pull more smb client updates from Steve French: - multichannel/reconnect fixes - move smbdirect (smb over RDMA) defines to fs/smb/common so they will be able to be used in the future more broadly, and a documentation update explaining setting up smbdirect mounts - update email address for Paulo * tag '6.16-rc-part2-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6: cifs: update internal version number MAINTAINERS, mailmap: Update Paulo Alcantara's email address cifs: add documentation for smbdirect setup cifs: do not disable interface polling on failure cifs: serialize other channels when query server interfaces is pending cifs: deal with the channel loading lag while picking channels smb: client: make use of common smbdirect_socket_parameters smb: smbdirect: introduce smbdirect_socket_parameters smb: client: make use of common smbdirect_socket smb: smbdirect: add smbdirect_socket.h smb: client: make use of common smbdirect.h smb: smbdirect: add smbdirect.h with public structures smb: client: make use of common smbdirect_pdu.h smb: smbdirect: add smbdirect_pdu.h with protocol definitions
2 parents 538c429 + 8e9d6ef commit 522cd6a

File tree

16 files changed

+533
-288
lines changed

16 files changed

+533
-288
lines changed

‎.mailmap

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -602,6 +602,12 @@ Paul Mackerras <paulus@ozlabs.org> <paulus@samba.org>
602602
Paul Mackerras <paulus@ozlabs.org> <paulus@au1.ibm.com>
603603
Paul Moore <paul@paul-moore.com> <paul.moore@hp.com>
604604
Paul Moore <paul@paul-moore.com> <pmoore@redhat.com>
605+
Paulo Alcantara <pc@manguebit.org> <pcacjr@zytor.com>
606+
Paulo Alcantara <pc@manguebit.org> <paulo@paulo.ac>
607+
Paulo Alcantara <pc@manguebit.org> <pc@cjr.nz>
608+
Paulo Alcantara <pc@manguebit.org> <palcantara@suse.de>
609+
Paulo Alcantara <pc@manguebit.org> <palcantara@suse.com>
610+
Paulo Alcantara <pc@manguebit.org> <pc@manguebit.com>
605611
Pavankumar Kondeti <quic_pkondeti@quicinc.com> <pkondeti@codeaurora.org>
606612
Peter A Jonsson <pj@ludd.ltu.se>
607613
Peter Oruba <peter.oruba@amd.com>

‎Documentation/filesystems/smb/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@ CIFS
88

99
ksmbd
1010
cifsroot
11+
smbdirect

‎Documentation/filesystems/smb/smbdirect.rst

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
.. SPDX-License-Identifier: GPL-2.0
2+
3+
===========================
4+
SMB Direct - SMB3 over RDMA
5+
===========================
6+
7+
This document describes how to set up the Linux SMB client and server to
8+
use RDMA.
9+
10+
Overview
11+
========
12+
The Linux SMB kernel client supports SMB Direct, which is a transport
13+
scheme for SMB3 that uses RDMA (Remote Direct Memory Access) to provide
14+
high throughput and low latencies by bypassing the traditional TCP/IP
15+
stack.
16+
SMB Direct on the Linux SMB client can be tested against KSMBD - a
17+
kernel-space SMB server.
18+
19+
Installation
20+
=============
21+
- Install an RDMA device. As long as the RDMA device driver is supported
22+
by the kernel, it should work. This includes both software emulators (soft
23+
RoCE, soft iWARP) and hardware devices (InfiniBand, RoCE, iWARP).
24+
25+
- Install a kernel with SMB Direct support. The first kernel release to
26+
support SMB Direct on both the client and server side is 5.15. Therefore,
27+
a distribution compatible with kernel 5.15 or later is required.
28+
29+
- Install cifs-utils, which provides the `mount.cifs` command to mount SMB
30+
shares.
31+
32+
- Configure the RDMA stack
33+
34+
Make sure that your kernel configuration has RDMA support enabled. Under
35+
Device Drivers -> Infiniband support, update the kernel configuration to
36+
enable Infiniband support.
37+
38+
Enable the appropriate IB HCA support or iWARP adapter support,
39+
depending on your hardware.
40+
41+
If you are using InfiniBand, enable IP-over-InfiniBand support.
42+
43+
For soft RDMA, enable either the soft iWARP (`RDMA _SIW`) or soft RoCE
44+
(`RDMA_RXE`) module. Install the `iproute2` package and use the
45+
`rdma link add` command to load the module and create an
46+
RDMA interface.
47+
48+
e.g. if your local ethernet interface is `eth0`, you can use:
49+
50+
.. code-block:: bash
51+
52+
sudo rdma link add siw0 type siw netdev eth0
53+
54+
- Enable SMB Direct support for both the server and the client in the kernel
55+
configuration.
56+
57+
Server Setup
58+
59+
.. code-block:: text
60+
61+
Network File Systems --->
62+
<M> SMB3 server support
63+
[*] Support for SMB Direct protocol
64+
65+
Client Setup
66+
67+
.. code-block:: text
68+
69+
Network File Systems --->
70+
<M> SMB3 and CIFS support (advanced network filesystem)
71+
[*] SMB Direct support
72+
73+
- Build and install the kernel. SMB Direct support will be enabled in the
74+
cifs.ko and ksmbd.ko modules.
75+
76+
Setup and Usage
77+
================
78+
79+
- Set up and start a KSMBD server as described in the `KSMBD documentation
80+
<https://www.kernel.org/doc/Documentation/filesystems/smb/ksmbd.rst>`_.
81+
Also add the "server multi channel support = yes" parameter to ksmbd.conf.
82+
83+
- On the client, mount the share with `rdma` mount option to use SMB Direct
84+
(specify a SMB version 3.0 or higher using `vers`).
85+
86+
For example:
87+
88+
.. code-block:: bash
89+
90+
mount -t cifs //server/share /mnt/point -o vers=3.1.1,rdma
91+
92+
- To verify that the mount is using SMB Direct, you can check dmesg for the
93+
following log line after mounting:
94+
95+
.. code-block:: text
96+
97+
CIFS: VFS: RDMA transport established
98+
99+
Or, verify `rdma` mount option for the share in `/proc/mounts`:
100+
101+
.. code-block:: bash
102+
103+
cat /proc/mounts | grep cifs

‎MAINTAINERS

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5986,7 +5986,7 @@ X: drivers/clk/clkdev.c
59865986
COMMON INTERNET FILE SYSTEM CLIENT (CIFS and SMB3)
59875987
M: Steve French <sfrench@samba.org>
59885988
M: Steve French <smfrench@gmail.com>
5989-
R: Paulo Alcantara <pc@manguebit.com> (DFS, global name space)
5989+
R: Paulo Alcantara <pc@manguebit.org> (DFS, global name space)
59905990
R: Ronnie Sahlberg <ronniesahlberg@gmail.com> (directory leases, sparse files)
59915991
R: Shyam Prasad N <sprasad@microsoft.com> (multichannel)
59925992
R: Tom Talpey <tom@talpey.com> (RDMA, smbdirect)
@@ -9280,7 +9280,7 @@ F: include/linux/iomap.h
92809280

92819281
FILESYSTEMS [NETFS LIBRARY]
92829282
M: David Howells <dhowells@redhat.com>
9283-
M: Paulo Alcantara <pc@manguebit.com>
9283+
M: Paulo Alcantara <pc@manguebit.org>
92849284
L: netfs@lists.linux.dev
92859285
L: linux-fsdevel@vger.kernel.org
92869286
S: Supported

‎fs/smb/client/cifs_debug.c

Lines changed: 14 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -362,6 +362,10 @@ static int cifs_debug_data_proc_show(struct seq_file *m, void *v)
362362
c = 0;
363363
spin_lock(&cifs_tcp_ses_lock);
364364
list_for_each_entry(server, &cifs_tcp_ses_list, tcp_ses_list) {
365+
#ifdef CONFIG_CIFS_SMB_DIRECT
366+
struct smbdirect_socket_parameters *sp;
367+
#endif
368+
365369
/* channel info will be printed as a part of sessions below */
366370
if (SERVER_IS_CHAN(server))
367371
continue;
@@ -383,25 +387,26 @@ static int cifs_debug_data_proc_show(struct seq_file *m, void *v)
383387
seq_printf(m, "\nSMBDirect transport not available");
384388
goto skip_rdma;
385389
}
390+
sp = &server->smbd_conn->socket.parameters;
386391

387392
seq_printf(m, "\nSMBDirect (in hex) protocol version: %x "
388393
"transport status: %x",
389394
server->smbd_conn->protocol,
390-
server->smbd_conn->transport_status);
395+
server->smbd_conn->socket.status);
391396
seq_printf(m, "\nConn receive_credit_max: %x "
392397
"send_credit_target: %x max_send_size: %x",
393-
server->smbd_conn->receive_credit_max,
394-
server->smbd_conn->send_credit_target,
395-
server->smbd_conn->max_send_size);
398+
sp->recv_credit_max,
399+
sp->send_credit_target,
400+
sp->max_send_size);
396401
seq_printf(m, "\nConn max_fragmented_recv_size: %x "
397402
"max_fragmented_send_size: %x max_receive_size:%x",
398-
server->smbd_conn->max_fragmented_recv_size,
399-
server->smbd_conn->max_fragmented_send_size,
400-
server->smbd_conn->max_receive_size);
403+
sp->max_fragmented_recv_size,
404+
sp->max_fragmented_send_size,
405+
sp->max_recv_size);
401406
seq_printf(m, "\nConn keep_alive_interval: %x "
402407
"max_readwrite_size: %x rdma_readwrite_threshold: %x",
403-
server->smbd_conn->keep_alive_interval,
404-
server->smbd_conn->max_readwrite_size,
408+
sp->keepalive_interval_msec * 1000,
409+
sp->max_read_write_size,
405410
server->smbd_conn->rdma_readwrite_threshold);
406411
seq_printf(m, "\nDebug count_get_receive_buffer: %x "
407412
"count_put_receive_buffer: %x count_send_empty: %x",

‎fs/smb/client/cifsfs.h

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,6 @@ extern const struct export_operations cifs_export_ops;
145145
#endif /* CONFIG_CIFS_NFSD_EXPORT */
146146

147147
/* when changing internal version - update following two lines at same time */
148-
#define SMB3_PRODUCT_BUILD 54
149-
#define CIFS_VERSION "2.54"
148+
#define SMB3_PRODUCT_BUILD 55
149+
#define CIFS_VERSION "2.55"
150150
#endif /* _CIFSFS_H */

‎fs/smb/client/cifsglob.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1085,6 +1085,7 @@ struct cifs_chan {
10851085
};
10861086

10871087
#define CIFS_SES_FLAG_SCALE_CHANNELS (0x1)
1088+
#define CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES (0x2)
10881089

10891090
/*
10901091
* Session structure. One of these for each uid session with a particular host

‎fs/smb/client/connect.c

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -116,13 +116,9 @@ static void smb2_query_server_interfaces(struct work_struct *work)
116116
rc = server->ops->query_server_interfaces(xid, tcon, false);
117117
free_xid(xid);
118118

119-
if (rc) {
120-
if (rc == -EOPNOTSUPP)
121-
return;
122-
119+
if (rc)
123120
cifs_dbg(FYI, "%s: failed to query server interfaces: %d\n",
124121
__func__, rc);
125-
}
126122

127123
queue_delayed_work(cifsiod_wq, &tcon->query_interfaces,
128124
(SMB_INTERFACE_POLL_INTERVAL * HZ));

‎fs/smb/client/smb2ops.c

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -504,19 +504,22 @@ smb3_negotiate_wsize(struct cifs_tcon *tcon, struct smb3_fs_context *ctx)
504504
wsize = min_t(unsigned int, wsize, server->max_write);
505505
#ifdef CONFIG_CIFS_SMB_DIRECT
506506
if (server->rdma) {
507+
struct smbdirect_socket_parameters *sp =
508+
&server->smbd_conn->socket.parameters;
509+
507510
if (server->sign)
508511
/*
509512
* Account for SMB2 data transfer packet header and
510513
* possible encryption header
511514
*/
512515
wsize = min_t(unsigned int,
513516
wsize,
514-
server->smbd_conn->max_fragmented_send_size -
517+
sp->max_fragmented_send_size -
515518
SMB2_READWRITE_PDU_HEADER_SIZE -
516519
sizeof(struct smb2_transform_hdr));
517520
else
518521
wsize = min_t(unsigned int,
519-
wsize, server->smbd_conn->max_readwrite_size);
522+
wsize, sp->max_read_write_size);
520523
}
521524
#endif
522525
if (!(server->capabilities & SMB2_GLOBAL_CAP_LARGE_MTU))
@@ -552,19 +555,22 @@ smb3_negotiate_rsize(struct cifs_tcon *tcon, struct smb3_fs_context *ctx)
552555
rsize = min_t(unsigned int, rsize, server->max_read);
553556
#ifdef CONFIG_CIFS_SMB_DIRECT
554557
if (server->rdma) {
558+
struct smbdirect_socket_parameters *sp =
559+
&server->smbd_conn->socket.parameters;
560+
555561
if (server->sign)
556562
/*
557563
* Account for SMB2 data transfer packet header and
558564
* possible encryption header
559565
*/
560566
rsize = min_t(unsigned int,
561567
rsize,
562-
server->smbd_conn->max_fragmented_recv_size -
568+
sp->max_fragmented_recv_size -
563569
SMB2_READWRITE_PDU_HEADER_SIZE -
564570
sizeof(struct smb2_transform_hdr));
565571
else
566572
rsize = min_t(unsigned int,
567-
rsize, server->smbd_conn->max_readwrite_size);
573+
rsize, sp->max_read_write_size);
568574
}
569575
#endif
570576

‎fs/smb/client/smb2pdu.c

Lines changed: 32 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@
3636
#include "smb2glob.h"
3737
#include "cifspdu.h"
3838
#include "cifs_spnego.h"
39+
#include "../common/smbdirect/smbdirect.h"
3940
#include "smbdirect.h"
4041
#include "trace.h"
4142
#ifdef CONFIG_CIFS_DFS_UPCALL
@@ -411,14 +412,23 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon,
411412
if (!rc &&
412413
(server->capabilities & SMB2_GLOBAL_CAP_MULTI_CHANNEL) &&
413414
server->ops->query_server_interfaces) {
414-
mutex_unlock(&ses->session_mutex);
415-
416415
/*
417-
* query server network interfaces, in case they change
416+
* query server network interfaces, in case they change.
417+
* Also mark the session as pending this update while the query
418+
* is in progress. This will be used to avoid calling
419+
* smb2_reconnect recursively.
418420
*/
421+
ses->flags |= CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES;
419422
xid = get_xid();
420423
rc = server->ops->query_server_interfaces(xid, tcon, false);
421424
free_xid(xid);
425+
ses->flags &= ~CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES;
426+
427+
/* regardless of rc value, setup polling */
428+
queue_delayed_work(cifsiod_wq, &tcon->query_interfaces,
429+
(SMB_INTERFACE_POLL_INTERVAL * HZ));
430+
431+
mutex_unlock(&ses->session_mutex);
422432

423433
if (rc == -EOPNOTSUPP && ses->chan_count > 1) {
424434
/*
@@ -438,11 +448,8 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon,
438448
if (ses->chan_max > ses->chan_count &&
439449
ses->iface_count &&
440450
!SERVER_IS_CHAN(server)) {
441-
if (ses->chan_count == 1) {
451+
if (ses->chan_count == 1)
442452
cifs_server_dbg(VFS, "supports multichannel now\n");
443-
queue_delayed_work(cifsiod_wq, &tcon->query_interfaces,
444-
(SMB_INTERFACE_POLL_INTERVAL * HZ));
445-
}
446453

447454
cifs_try_adding_channels(ses);
448455
}
@@ -560,11 +567,18 @@ static int smb2_ioctl_req_init(u32 opcode, struct cifs_tcon *tcon,
560567
struct TCP_Server_Info *server,
561568
void **request_buf, unsigned int *total_len)
562569
{
563-
/* Skip reconnect only for FSCTL_VALIDATE_NEGOTIATE_INFO IOCTLs */
564-
if (opcode == FSCTL_VALIDATE_NEGOTIATE_INFO) {
570+
/*
571+
* Skip reconnect in one of the following cases:
572+
* 1. For FSCTL_VALIDATE_NEGOTIATE_INFO IOCTLs
573+
* 2. For FSCTL_QUERY_NETWORK_INTERFACE_INFO IOCTL when called from
574+
* smb2_reconnect (indicated by CIFS_SES_FLAG_SCALE_CHANNELS ses flag)
575+
*/
576+
if (opcode == FSCTL_VALIDATE_NEGOTIATE_INFO ||
577+
(opcode == FSCTL_QUERY_NETWORK_INTERFACE_INFO &&
578+
(tcon->ses->flags & CIFS_SES_FLAGS_PENDING_QUERY_INTERFACES)))
565579
return __smb2_plain_req_init(SMB2_IOCTL, tcon, server,
566580
request_buf, total_len);
567-
}
581+
568582
return smb2_plain_req_init(SMB2_IOCTL, tcon, server,
569583
request_buf, total_len);
570584
}
@@ -4449,10 +4463,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
44494463
#ifdef CONFIG_CIFS_SMB_DIRECT
44504464
/*
44514465
* If we want to do a RDMA write, fill in and append
4452-
* smbd_buffer_descriptor_v1 to the end of read request
4466+
* smbdirect_buffer_descriptor_v1 to the end of read request
44534467
*/
44544468
if (rdata && smb3_use_rdma_offload(io_parms)) {
4455-
struct smbd_buffer_descriptor_v1 *v1;
4469+
struct smbdirect_buffer_descriptor_v1 *v1;
44564470
bool need_invalidate = server->dialect == SMB30_PROT_ID;
44574471

44584472
rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter,
@@ -4466,8 +4480,8 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
44664480
req->ReadChannelInfoOffset =
44674481
cpu_to_le16(offsetof(struct smb2_read_req, Buffer));
44684482
req->ReadChannelInfoLength =
4469-
cpu_to_le16(sizeof(struct smbd_buffer_descriptor_v1));
4470-
v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0];
4483+
cpu_to_le16(sizeof(struct smbdirect_buffer_descriptor_v1));
4484+
v1 = (struct smbdirect_buffer_descriptor_v1 *) &req->Buffer[0];
44714485
v1->offset = cpu_to_le64(rdata->mr->mr->iova);
44724486
v1->token = cpu_to_le32(rdata->mr->mr->rkey);
44734487
v1->length = cpu_to_le32(rdata->mr->mr->length);
@@ -4975,10 +4989,10 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
49754989
#ifdef CONFIG_CIFS_SMB_DIRECT
49764990
/*
49774991
* If we want to do a server RDMA read, fill in and append
4978-
* smbd_buffer_descriptor_v1 to the end of write request
4992+
* smbdirect_buffer_descriptor_v1 to the end of write request
49794993
*/
49804994
if (smb3_use_rdma_offload(io_parms)) {
4981-
struct smbd_buffer_descriptor_v1 *v1;
4995+
struct smbdirect_buffer_descriptor_v1 *v1;
49824996
bool need_invalidate = server->dialect == SMB30_PROT_ID;
49834997

49844998
wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter,
@@ -4997,8 +5011,8 @@ smb2_async_writev(struct cifs_io_subrequest *wdata)
49975011
req->WriteChannelInfoOffset =
49985012
cpu_to_le16(offsetof(struct smb2_write_req, Buffer));
49995013
req->WriteChannelInfoLength =
5000-
cpu_to_le16(sizeof(struct smbd_buffer_descriptor_v1));
5001-
v1 = (struct smbd_buffer_descriptor_v1 *) &req->Buffer[0];
5014+
cpu_to_le16(sizeof(struct smbdirect_buffer_descriptor_v1));
5015+
v1 = (struct smbdirect_buffer_descriptor_v1 *) &req->Buffer[0];
50025016
v1->offset = cpu_to_le64(wdata->mr->mr->iova);
50035017
v1->token = cpu_to_le32(wdata->mr->mr->rkey);
50045018
v1->length = cpu_to_le32(wdata->mr->mr->length);

0 commit comments

Comments
 (0)