Skip to content

Commit ef3a43a

Browse files
Gregory Pricedavejiang
Gregory Price
authored and
davejiang
committed
cxl: docs/linux/cxl-driver - add example configurations
Add 4 example configurations: - single device - cross-host-bridge interleave - intra-host-bridge-interleave - multi-level interleave Signed-off-by: Gregory Price <gourry@gourry.net> Link: https://patch.msgid.link/20250512162134.3596150-11-gourry@gourry.net Signed-off-by: Dave Jiang <dave.jiang@intel.com>
1 parent 2e2865a commit ef3a43a

File tree

5 files changed

+1262
-0
lines changed

5 files changed

+1262
-0
lines changed

‎Documentation/driver-api/cxl/linux/cxl-driver.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -520,3 +520,13 @@ from HPA to DPA. This is why they must be aware of the entire interleave set.
520520

521521
Linux does not support unbalanced interleave configurations. As a result, all
522522
endpoints in an interleave set must have the same ways and granularity.
523+
524+
Example Configurations
525+
======================
526+
.. toctree::
527+
:maxdepth: 1
528+
529+
example-configurations/single-device.rst
530+
example-configurations/hb-interleave.rst
531+
example-configurations/intra-hb-interleave.rst
532+
example-configurations/multi-interleave.rst

‎Documentation/driver-api/cxl/linux/example-configurations/hb-interleave.rst

Lines changed: 314 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
.. SPDX-License-Identifier: GPL-2.0
2+
3+
============================
4+
Inter-Host-Bridge Interleave
5+
============================
6+
This cxl-cli configuration dump shows the following host configuration:
7+
8+
* A single socket system with one CXL root
9+
* CXL Root has Four (4) CXL Host Bridges
10+
* Two CXL Host Bridges have a single CXL Memory Expander Attached
11+
* The CXL root is configured to interleave across the two host bridges.
12+
13+
This output is generated by :code:`cxl list -v` and describes the relationships
14+
between objects exposed in :code:`/sys/bus/cxl/devices/`.
15+
16+
::
17+
18+
[
19+
{
20+
"bus":"root0",
21+
"provider":"ACPI.CXL",
22+
"nr_dports":4,
23+
"dports":[
24+
{
25+
"dport":"pci0000:00",
26+
"alias":"ACPI0016:01",
27+
"id":0
28+
},
29+
{
30+
"dport":"pci0000:a8",
31+
"alias":"ACPI0016:02",
32+
"id":4
33+
},
34+
{
35+
"dport":"pci0000:2a",
36+
"alias":"ACPI0016:03",
37+
"id":1
38+
},
39+
{
40+
"dport":"pci0000:d2",
41+
"alias":"ACPI0016:00",
42+
"id":5
43+
}
44+
],
45+
46+
This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to CXL
47+
Host Bridges. The `Root` can be considered the singular upstream port attached
48+
to the platform's memory controller - which routes memory requests to it.
49+
50+
The `ports:root0` section lays out how each of these downstream ports are
51+
configured. If a port is not configured (id's 0 and 1), they are omitted.
52+
53+
::
54+
55+
"ports:root0":[
56+
{
57+
"port":"port1",
58+
"host":"pci0000:d2",
59+
"depth":1,
60+
"nr_dports":3,
61+
"dports":[
62+
{
63+
"dport":"0000:d2:01.1",
64+
"alias":"device:02",
65+
"id":0
66+
},
67+
{
68+
"dport":"0000:d2:01.3",
69+
"alias":"device:05",
70+
"id":2
71+
},
72+
{
73+
"dport":"0000:d2:07.1",
74+
"alias":"device:0d",
75+
"id":113
76+
}
77+
],
78+
79+
This chunk shows the available downstream ports associated with the CXL Host
80+
Bridge :code:`port1`. In this case, :code:`port1` has 3 available downstream
81+
ports: :code:`dport1`, :code:`dport2`, and :code:`dport113`..
82+
83+
::
84+
85+
"endpoints:port1":[
86+
{
87+
"endpoint":"endpoint5",
88+
"host":"mem0",
89+
"parent_dport":"0000:d2:01.1",
90+
"depth":2,
91+
"memdev":{
92+
"memdev":"mem0",
93+
"ram_size":137438953472,
94+
"serial":0,
95+
"numa_node":0,
96+
"host":"0000:d3:00.0"
97+
},
98+
"decoders:endpoint5":[
99+
{
100+
"decoder":"decoder5.0",
101+
"resource":825975898112,
102+
"size":274877906944,
103+
"interleave_ways":2,
104+
"interleave_granularity":256,
105+
"region":"region0",
106+
"dpa_resource":0,
107+
"dpa_size":137438953472,
108+
"mode":"ram"
109+
}
110+
]
111+
}
112+
],
113+
114+
This chunk shows the endpoints attached to the host bridge :code:`port1`.
115+
116+
:code:`endpoint5` contains a single configured decoder :code:`decoder5.0`
117+
which has the same interleave configuration as :code:`region0` (shown later).
118+
119+
Next we have the decodesr belonging to the host bridge:
120+
121+
::
122+
123+
"decoders:port1":[
124+
{
125+
"decoder":"decoder1.0",
126+
"resource":825975898112,
127+
"size":274877906944,
128+
"interleave_ways":1,
129+
"region":"region0",
130+
"nr_targets":1,
131+
"targets":[
132+
{
133+
"target":"0000:d2:01.1",
134+
"alias":"device:02",
135+
"position":0,
136+
"id":0
137+
}
138+
]
139+
}
140+
]
141+
},
142+
143+
Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose only
144+
target is :code:`dport1` - which is attached to :code:`endpoint5`.
145+
146+
The following chunk shows a similar configuration for Host Bridge :code:`port3`,
147+
the second host bridge with a memory device attached.
148+
149+
::
150+
151+
{
152+
"port":"port3",
153+
"host":"pci0000:a8",
154+
"depth":1,
155+
"nr_dports":1,
156+
"dports":[
157+
{
158+
"dport":"0000:a8:01.1",
159+
"alias":"device:c3",
160+
"id":0
161+
}
162+
],
163+
"endpoints:port3":[
164+
{
165+
"endpoint":"endpoint6",
166+
"host":"mem1",
167+
"parent_dport":"0000:a8:01.1",
168+
"depth":2,
169+
"memdev":{
170+
"memdev":"mem1",
171+
"ram_size":137438953472,
172+
"serial":0,
173+
"numa_node":0,
174+
"host":"0000:a9:00.0"
175+
},
176+
"decoders:endpoint6":[
177+
{
178+
"decoder":"decoder6.0",
179+
"resource":825975898112,
180+
"size":274877906944,
181+
"interleave_ways":2,
182+
"interleave_granularity":256,
183+
"region":"region0",
184+
"dpa_resource":0,
185+
"dpa_size":137438953472,
186+
"mode":"ram"
187+
}
188+
]
189+
}
190+
],
191+
"decoders:port3":[
192+
{
193+
"decoder":"decoder3.0",
194+
"resource":825975898112,
195+
"size":274877906944,
196+
"interleave_ways":1,
197+
"region":"region0",
198+
"nr_targets":1,
199+
"targets":[
200+
{
201+
"target":"0000:a8:01.1",
202+
"alias":"device:c3",
203+
"position":0,
204+
"id":0
205+
}
206+
]
207+
}
208+
]
209+
},
210+
211+
212+
The next chunk shows the two CXL host bridges without attached endpoints.
213+
214+
::
215+
216+
{
217+
"port":"port2",
218+
"host":"pci0000:00",
219+
"depth":1,
220+
"nr_dports":2,
221+
"dports":[
222+
{
223+
"dport":"0000:00:01.3",
224+
"alias":"device:55",
225+
"id":2
226+
},
227+
{
228+
"dport":"0000:00:07.1",
229+
"alias":"device:5d",
230+
"id":113
231+
}
232+
]
233+
},
234+
{
235+
"port":"port4",
236+
"host":"pci0000:2a",
237+
"depth":1,
238+
"nr_dports":1,
239+
"dports":[
240+
{
241+
"dport":"0000:2a:01.1",
242+
"alias":"device:d0",
243+
"id":0
244+
}
245+
]
246+
}
247+
],
248+
249+
Next we have the `Root Decoders` belonging to :code:`root0`. This root decoder
250+
applies the interleave across the downstream ports :code:`port1` and
251+
:code:`port3` - with a granularity of 256 bytes.
252+
253+
This information is generated by the CXL driver reading the ACPI CEDT CMFWS.
254+
255+
::
256+
257+
"decoders:root0":[
258+
{
259+
"decoder":"decoder0.0",
260+
"resource":825975898112,
261+
"size":274877906944,
262+
"interleave_ways":2,
263+
"interleave_granularity":256,
264+
"max_available_extent":0,
265+
"volatile_capable":true,
266+
"nr_targets":2,
267+
"targets":[
268+
{
269+
"target":"pci0000:a8",
270+
"alias":"ACPI0016:02",
271+
"position":1,
272+
"id":4
273+
},
274+
{
275+
"target":"pci0000:d2",
276+
"alias":"ACPI0016:00",
277+
"position":0,
278+
"id":5
279+
}
280+
],
281+
282+
Finally we have the `Memory Region` associated with the `Root Decoder`
283+
:code:`decoder0.0`. This region describes the overall interleave configuration
284+
of the interleave set.
285+
286+
::
287+
288+
"regions:decoder0.0":[
289+
{
290+
"region":"region0",
291+
"resource":825975898112,
292+
"size":274877906944,
293+
"type":"ram",
294+
"interleave_ways":2,
295+
"interleave_granularity":256,
296+
"decode_state":"commit",
297+
"mappings":[
298+
{
299+
"position":1,
300+
"memdev":"mem1",
301+
"decoder":"decoder6.0"
302+
},
303+
{
304+
"position":0,
305+
"memdev":"mem0",
306+
"decoder":"decoder5.0"
307+
}
308+
]
309+
}
310+
]
311+
}
312+
]
313+
}
314+
]

0 commit comments

Comments
 (0)