1 2 QoS Management in OpenSM 3 4============================================================================== 5 Table of contents 6============================================================================== 7 81. Overview 92. Full QoS Policy File 103. Simplified QoS Policy Definition 114. Policy File Syntax Guidelines 125. Examples of Full Policy File 136. Simplified QoS Policy - Details and Examples 147. SL2VL Mapping and VL Arbitration 15 16 17============================================================================== 18 1. Overview 19============================================================================== 20 21When QoS in OpenSM is enabled (-Q or --qos), OpenSM looks for QoS Policy file. 22The default name of OpenSM QoS policy file is 23/usr/local/etc/opensm/qos-policy.conf. The default may be changed by using -Y 24or --qos_policy_file option with OpenSM. 25 26During fabric initialization and at every heavy sweep OpenSM parses the QoS 27policy file, applies its settings to the discovered fabric elements, and 28enforces the provided policy on client requests. The overall flow for such 29requests is: 30 - The request is matched against the defined matching rules such that the 31 QoS Level definition is found. 32 - Given the QoS Level, path(s) search is performed with the given 33 restrictions imposed by that level. 34 35There are two ways to define QoS policy: 36 - Full policy, where the policy file syntax provides an administrator 37 various ways to match PathRecord/MultiPathRecord (PR/MPR) request and 38 enforce various QoS constraints on the requested PR/MPR 39 - Simplified QoS policy definition, where an administrator would be able to 40 match PR/MPR requests by various ULPs and applications running on top of 41 these ULPs. 42 43While the full policy syntax is very flexible, in many cases the simplified 44policy definition would be sufficient. 45 46 47============================================================================== 48 2. Full QoS Policy File 49============================================================================== 50 51QoS policy file has the following sections: 52 53I) Port Groups (denoted by port-groups). 54This section defines zero or more port groups that can be referred later by 55matching rules (see below). Port group lists ports by: 56 - Port GUID 57 - Port name, which is a combination of NodeDescription and IB port number 58 - PKey, which means that all the ports in the subnet that belong to 59 partition with a given PKey belong to this port group 60 - Partition name, which means that all the ports in the subnet that belong 61 to partition with a given name belong to this port group 62 - Node type, where possible node types are: CA, SWITCH, ROUTER, ALL, and 63 SELF (SM's port). 64 65II) QoS Setup (denoted by qos-setup). 66This section describes how to set up SL2VL and VL Arbitration tables on 67various nodes in the fabric. 68However, this is not supported in OpenSM currently. 69SL2VL and VLArb tables should be configured in the OpenSM options file 70(default location - /usr/local/etc/opensm/opensm.conf). 71 72III) QoS Levels (denoted by qos-levels). 73Each QoS Level defines Service Level (SL) and a few optional fields: 74 - MTU limit 75 - Rate limit 76 - PKey 77 - Packet lifetime 78When path(s) search is performed, it is done with regards to restriction that 79these QoS Level parameters impose. 80One QoS level that is mandatory to define is a DEFAULT QoS level. It is 81applied to a PR/MPR query that does not match any existing match rule. 82Similar to any other QoS Level, it can also be explicitly referred by any 83match rule. 84 85IV) QoS Matching Rules (denoted by qos-match-rules). 86Each PathRecord/MultiPathRecord query that OpenSM receives is matched against 87the set of matching rules. Rules are scanned in order of appearance in the QoS 88policy file such as the first match takes precedence. 89Each rule has a name of QoS level that will be applied to the matching query. 90A default QoS level is applied to a query that did not match any rule. 91Queries can be matched by: 92 - Source port group (whether a source port is a member of a specified group) 93 - Destination port group (same as above, only for destination port) 94 - PKey 95 - QoS class 96 - Service ID 97To match a certain matching rule, PR/MPR query has to match ALL the rule's 98criteria. However, not all the fields of the PR/MPR query have to appear in 99the matching rule. 100For instance, if the rule has a single criterion - Service ID, it will match 101any query that has this Service ID, disregarding rest of the query fields. 102However, if a certain query has only Service ID (which means that this is the 103only bit in the PR/MPR component mask that is on), it will not match any rule 104that has other matching criteria besides Service ID. 105 106 107============================================================================== 108 3. Simplified QoS Policy Definition 109============================================================================== 110 111Simplified QoS policy definition comprises of a single section denoted by 112qos-ulps. Similar to the full QoS policy, it has a list of match rules and 113their QoS Level, but in this case a match rule has only one criterion - its 114goal is to match a certain ULP (or a certain application on top of this ULP) 115PR/MPR request, and QoS Level has only one constraint - Service Level (SL). 116The simplified policy section may appear in the policy file in combine with 117the full policy, or as a stand-alone policy definition. 118See more details and list of match rule criteria below. 119 120 121============================================================================== 122 4. Policy File Syntax Guidelines 123============================================================================== 124 125- Empty lines are ignored. 126- Leading and trailing blanks, as well as empty lines, are ignored, so 127 the indentation in the example is just for better readability. 128- Comments are started with the pound sign (#) and terminated by EOL. 129- Any keyword should be the first non-blank in the line, unless it's a 130 comment. 131- Keywords that denote section/subsection start have matching closing 132 keywords. 133- Having a QoS Level named "DEFAULT" is a must - it is applied to PR/MPR 134 requests that didn't match any of the matching rules. 135- Any section/subsection of the policy file is optional. 136 137 138============================================================================== 139 5. Examples of Full Policy File 140============================================================================== 141 142As mentioned earlier, any section of the policy file is optional, and 143the only mandatory part of the policy file is a default QoS Level. 144Here's an example of the shortest policy file: 145 146 qos-levels 147 qos-level 148 name: DEFAULT 149 sl: 0 150 end-qos-level 151 end-qos-levels 152 153Port groups section is missing because there are no match rules, which means 154that port groups are not referred anywhere, and there is no need defining 155them. And since this policy file doesn't have any matching rules, PR/MPR query 156won't match any rule, and OpenSM will enforce default QoS level. 157Essentially, the above example is equivalent to not having QoS policy file 158at all. 159 160The following example shows all the possible options and keywords in the 161policy file and their syntax: 162 163 # 164 # See the comments in the following example. 165 # They explain different keywords and their meaning. 166 # 167 port-groups 168 169 port-group # using port GUIDs 170 name: Storage 171 # "use" is just a description that is used for logging 172 # Other than that, it is just a comment 173 use: SRP Targets 174 port-guid: 0x10000000000001, 0x10000000000005-0x1000000000FFFA 175 port-guid: 0x1000000000FFFF 176 end-port-group 177 178 port-group 179 name: Virtual Servers 180 # The syntax of the port name is as follows: 181 # "node_description/Pnum". 182 # node_description is compared to the NodeDescription of the node, 183 # and "Pnum" is a port number on that node. 184 port-name: vs1 HCA-1/P1, vs2 HCA-1/P1 185 end-port-group 186 187 # using partitions defined in the partition policy 188 port-group 189 name: Partitions 190 partition: Part1 191 pkey: 0x1234 192 end-port-group 193 194 # using node types: CA, ROUTER, SWITCH, SELF (for node that runs SM) 195 # or ALL (for all the nodes in the subnet) 196 port-group 197 name: CAs and SM 198 node-type: CA, SELF 199 end-port-group 200 201 end-port-groups 202 203 qos-setup 204 # This section of the policy file describes how to set up SL2VL and VL 205 # Arbitration tables on various nodes in the fabric. 206 # However, this is not supported in OpenSM currently - the section is 207 # parsed and ignored. SL2VL and VLArb tables should be configured in the 208 # OpenSM options file (by default - /usr/local/etc/opensm/opensm.conf). 209 end-qos-setup 210 211 qos-levels 212 213 # Having a QoS Level named "DEFAULT" is a must - it is applied to 214 # PR/MPR requests that didn't match any of the matching rules. 215 qos-level 216 name: DEFAULT 217 use: default QoS Level 218 sl: 0 219 end-qos-level 220 221 # the whole set: SL, MTU-Limit, Rate-Limit, PKey, Packet Lifetime 222 qos-level 223 name: WholeSet 224 sl: 1 225 mtu-limit: 4 226 rate-limit: 5 227 pkey: 0x1234 228 packet-life: 8 229 end-qos-level 230 231 end-qos-levels 232 233 # Match rules are scanned in order of their apperance in the policy file. 234 # First matched rule takes precedence. 235 qos-match-rules 236 237 # matching by single criteria: QoS class 238 qos-match-rule 239 use: by QoS class 240 qos-class: 7-9,11 241 # Name of qos-level to apply to the matching PR/MPR 242 qos-level-name: WholeSet 243 end-qos-match-rule 244 245 # show matching by destination group and service id 246 qos-match-rule 247 use: Storage targets 248 destination: Storage 249 service-id: 0x10000000000001, 0x10000000000008-0x10000000000FFF 250 qos-level-name: WholeSet 251 end-qos-match-rule 252 253 qos-match-rule 254 source: Storage 255 use: match by source group only 256 qos-level-name: DEFAULT 257 end-qos-match-rule 258 259 qos-match-rule 260 use: match by all parameters 261 qos-class: 7-9,11 262 source: Virtual Servers 263 destination: Storage 264 service-id: 0x0000000000010000-0x000000000001FFFF 265 pkey: 0x0F00-0x0FFF 266 qos-level-name: WholeSet 267 end-qos-match-rule 268 269 end-qos-match-rules 270 271 272============================================================================== 273 6. Simplified QoS Policy - Details and Examples 274============================================================================== 275 276Simplified QoS policy match rules are tailored for matching ULPs (or some 277application on top of a ULP) PR/MPR requests. This section has a list of 278per-ULP (or per-application) match rules and the SL that should be enforced 279on the matched PR/MPR query. 280 281Match rules include: 282 - Default match rule that is applied to PR/MPR query that didn't match any 283 of the other match rules 284 - SDP 285 - SDP application with a specific target TCP/IP port range 286 - SRP with a specific target IB port GUID 287 - RDS 288 - iSER 289 - iSER application with a specific target TCP/IP port range 290 - IPoIB with a default PKey 291 - IPoIB with a specific PKey 292 - any ULP/application with a specific Service ID in the PR/MPR query 293 - any ULP/application with a specific PKey in the PR/MPR query 294 - any ULP/application with a specific target IB port GUID in the PR/MPR query 295 296Since any section of the policy file is optional, as long as basic rules of 297the file are kept (such as no referring to nonexisting port group, having 298default QoS Level, etc), the simplified policy section (qos-ulps) can serve 299as a complete QoS policy file. 300The shortest policy file in this case would be as follows: 301 302 qos-ulps 303 default : 0 #default SL 304 end-qos-ulps 305 306It is equivalent to the previous example of the shortest policy file, and it 307is also equivalent to not having policy file at all. 308 309Below is an example of simplified QoS policy with all the possible keywords: 310 311 qos-ulps 312 default : 0 # default SL 313 sdp, port-num 30000 : 0 # SL for application running on top 314 # of SDP when a destination 315 # TCP/IPport is 30000 316 sdp, port-num 10000-20000 : 0 317 sdp : 1 # default SL for any other 318 # application running on top of SDP 319 rds : 2 # SL for RDS traffic 320 iser, port-num 900 : 0 # SL for iSER with a specific target 321 # port 322 iser : 3 # default SL for iSER 323 ipoib, pkey 0x0001 : 0 # SL for IPoIB on partition with 324 # pkey 0x0001 325 ipoib : 4 # default IPoIB partition, 326 # pkey=0x7FFF 327 any, service-id 0x6234 : 6 # match any PR/MPR query with a 328 # specific Service ID 329 any, pkey 0x0ABC : 6 # match any PR/MPR query with a 330 # specific PKey 331 srp, target-port-guid 0x1234 : 5 # SRP when SRP Target is located on 332 # a specified IB port GUID 333 any, target-port-guid 0x0ABC-0xFFFFF : 6 # match any PR/MPR query with 334 # a specific target port GUID 335 end-qos-ulps 336 337 338Similar to the full policy definition, matching of PR/MPR queries is done in 339order of appearance in the QoS policy file such as the first match takes 340precedence, except for the "default" rule, which is applied only if the query 341didn't match any other rule. 342 343All other sections of the QoS policy file take precedence over the qos-ulps 344section. That is, if a policy file has both qos-match-rules and qos-ulps 345sections, then any query is matched first against the rules in the 346qos-match-rules section, and only if there was no match, the query is matched 347against the rules in qos-ulps section. 348 349Note that some of these match rules may overlap, so in order to use the 350simplified QoS definition effectively, it is important to understand how each 351of the ULPs is matched: 352 3536.1 IPoIB 354IPoIB query is matched by PKey. Default PKey for IPoIB partition is 0x7fff, so 355the following three match rules are equivalent: 356 357 ipoib : <SL> 358 ipoib, pkey 0x7fff : <SL> 359 any, pkey 0x7fff : <SL> 360 3616.2 SDP 362SDP PR query is matched by Service ID. The Service-ID for SDP is 3630x000000000001PPPP, where PPPP are 4 hex digits holding the remote TCP/IP Port 364Number to connect to. The following two match rules are equivalent: 365 366 sdp : <SL> 367 any, service-id 0x0000000000010000-0x000000000001ffff : <SL> 368 3696.3 RDS 370Similar to SDP, RDS PR query is matched by Service ID. The Service ID for RDS 371is 0x000000000106PPPP, where PPPP are 4 hex digits holding the remote TCP/IP 372Port Number to connect to. Default port number for RDS is 0x48CA, which makes 373a default Service-ID 0x00000000010648CA. The following two match rules are 374equivalent: 375 376 rds : <SL> 377 any, service-id 0x00000000010648CA : <SL> 378 3796.4 iSER 380Similar to RDS, iSER query is matched by Service ID, where the the Service ID 381is also 0x000000000106PPPP. Default port number for iSER is 0x0CBC, which makes 382a default Service-ID 0x0000000001060CBC. The following two match rules are 383equivalent: 384 385 iser : <SL> 386 any, service-id 0x0000000001060CBC : <SL> 387 3886.5 SRP 389Service ID for SRP varies from storage vendor to vendor, thus SRP query is 390matched by the target IB port GUID. The following two match rules are 391equivalent: 392 393 srp, target-port-guid 0x1234 : <SL> 394 any, target-port-guid 0x1234 : <SL> 395 396Note that any of the above ULPs might contain target port GUID in the PR 397query, so in order for these queries not to be recognized by the QoS manager 398as SRP, the SRP match rule (or any match rule that refers to the target port 399guid only) should be placed at the end of the qos-ulps match rules. 400 4016.6 MPI 402SL for MPI is manually configured by MPI admin. OpenSM is not forcing any SL 403on the MPI traffic, and that's why it is the only ULP that did not appear in 404the qos-ulps section. 405 406 407============================================================================== 408 7. SL2VL Mapping and VL Arbitration 409============================================================================== 410 411OpenSM cached options file has a set of QoS related configuration parameters, 412that are used to configure SL2VL mapping and VL arbitration on IB ports. 413These parameters are: 414 - Max VLs: the maximum number of VLs that will be on the subnet. 415 - High limit: the limit of High Priority component of VL Arbitration 416 table (IBA 7.6.9). 417 - VLArb low table: Low priority VL Arbitration table (IBA 7.6.9) template. 418 - VLArb high table: High priority VL Arbitration table (IBA 7.6.9) template. 419 - SL2VL: SL2VL Mapping table (IBA 7.6.6) template. It is a list of VLs 420 corresponding to SLs 0-15 (Note that VL15 used here means drop this SL). 421 422There are separate QoS configuration parameters sets for various target types: 423CAs, routers, switch external ports, and switch's enhanced port 0. The names 424of such parameters are prefixed by "qos_<type>_" string. Here is a full list 425of the currently supported sets: 426 427 qos_ca_ - QoS configuration parameters set for CAs. 428 qos_rtr_ - parameters set for routers. 429 qos_sw0_ - parameters set for switches' port 0. 430 qos_swe_ - parameters set for switches' external ports. 431 432Here's the example of typical default values for CAs and switches' external 433ports (hard-coded in OpenSM initialization): 434 435 qos_ca_max_vls 15 436 qos_ca_high_limit 0 437 qos_ca_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0 438 qos_ca_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4 439 qos_ca_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7 440 441 qos_swe_max_vls 15 442 qos_swe_high_limit 0 443 qos_swe_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0 444 qos_swe_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4 445 qos_swe_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7 446 447VL arbitration tables (both high and low) are lists of VL/Weight pairs. 448Each list entry contains a VL number (values from 0-14), and a weighting value 449(values 0-255), indicating the number of 64 byte units (credits) which may be 450transmitted from that VL when its turn in the arbitration occurs. A weight 451of 0 indicates that this entry should be skipped. If a list entry is 452programmed for VL15 or for a VL that is not supported or is not currently 453configured by the port, the port may either skip that entry or send from any 454supported VL for that entry. 455 456Note, that the same VLs may be listed multiple times in the High or Low 457priority arbitration tables, and, further, it can be listed in both tables. 458 459The limit of high-priority VLArb table (qos_<type>_high_limit) indicates the 460number of high-priority packets that can be transmitted without an opportunity 461to send a low-priority packet. Specifically, the number of bytes that can be 462sent is high_limit times 4K bytes. 463 464A high_limit value of 255 indicates that the byte limit is unbounded. 465Note: if the 255 value is used, the low priority VLs may be starved. 466A value of 0 indicates that only a single packet from the high-priority table 467may be sent before an opportunity is given to the low-priority table. 468 469Keep in mind that ports usually transmit packets of size equal to MTU. 470For instance, for 4KB MTU a single packet will require 64 credits, so in order 471to achieve effective VL arbitration for packets of 4KB MTU, the weighting 472values for each VL should be multiples of 64. 473 474Below is an example of SL2VL and VL Arbitration configuration on subnet: 475 476 qos_ca_max_vls 15 477 qos_ca_high_limit 6 478 qos_ca_vlarb_high 0:4 479 qos_ca_vlarb_low 0:0,1:64,2:128,3:192,4:0,5:64,6:64,7:64 480 qos_ca_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7 481 482 qos_swe_max_vls 15 483 qos_swe_high_limit 6 484 qos_swe_vlarb_high 0:4 485 qos_swe_vlarb_low 0:0,1:64,2:128,3:192,4:0,5:64,6:64,7:64 486 qos_swe_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7 487 488In this example, there are 8 VLs configured on subnet: VL0 to VL7. VL0 is 489defined as a high priority VL, and it is limited to 6 x 4KB = 24KB in a single 490transmission burst. Such configuration would suilt VL that needs low latency 491and uses small MTU when transmitting packets. Rest of VLs are defined as low 492priority VLs with different weights, while VL4 is effectively turned off. 493