SMB Direct with Emulex

A while a go i was involved in a project were the customer wanted to deploy a 2 node Scale out file server with storage spaces on Windows Server 2012 R2. Since this was my first actual storage spaces and scale out file server deployment (besides a training and testing it in a lab with vm’s) i ran in to some issue’s and problems.

First of all there is a lot of stuff on the internet that is not always entirely true for all scenarios. Settings or test results that are used and reported are not necessarily true in your environment. For example DCB/PFC/QOS settings with Mellanox RoCe adapters are different than with Emulex RoCe adapters. And even with Emulex and HP or Fujitsu branded Emulex cards configuration is not the same. This could be because the vendors don’t know it exactly either and are still struggling with their drivers, firmware and settings, I don’t know. But it’s makes it pretty tough to get it all right. And as I said before, this is the first implementation I did, so I am not an expert on this either, so feel very free to give feedback if needed.

This Technet article describes several does and don’ts with SMB and RDMA. A lot of info and configurations are also available on Jose Barreto’s blog for RDMA and Mellanox RoCe adapters. But since there is very little info with Emulex, I started this blog to write and share my findings.

 Setup

In my setup I was using Emulex OCe14102-U 10Gb CNA adapters which are SFP+ adapters for the SMB traffic. Since it was all Fujitsu hardware these card where Fujitsu branded to.

  • Drivers and Firmware 

Before you start, verify drivers and firmware. I was using Firmware version 10.6.193.15  and the driver version 10.6.126.0. Which are released in October 2015. My dear colleague Hans Vredevoort experienced some issue’s earlier this year with HP Branded Emulex cards drivers and firmware version 10.5.x.x. They had to roll back to  10.2.x.x drivers and firmware to make SMB Direct work. So look very closely to drivers and firmware.

  • Configure the switch
Again several flavours of configuration on switches. In this scenario we had Cisco Nexus switches, so i can give a Cisco configuration sample.
Configuring Virtual LAN (VLAN) on the switch, please refer to your switch vendor’s configuration guide to create VLANs on the switch and tagging or trunking poorts on the vlan's. 
Below is a sample configuration for a Cisco Nexus switch:

#Enter configuration mode by typing “configure terminal”
configure terminal
#Enter configuration mode for a connected port using the “interface <type> <slot/port>” command.
interface ethernet 1/1
#Select the PFC mode using the “priority-flow-control <auto/on>” command. CHOOSE “auto.” Then get back to base
configuration mode using the “exit” command.
priority-flow-control auto
exit
#Repeat steps for the other ports that you are going to use for SMB Direct to.

#Configure Priority Flow Control (PFC) on the switch
#Please refer to your switch vendor’s guide for creating priority group and enabling PFC. This step enlists the detailed steps for configuring
#PFC and QoS for RoCE traffic for a Cisco Nexus switch:
#Create a priority group for RoCE traffic with a priority of 5. There are several commands needed to accomplish this task. Type the
#following commands listed in the order shown below:
class-map type qos roce
match cos 5
exit
class-map type queuing roce
match qos-group 5
exit
class-map type network-qos roce
match qos-group 5
exit

#Assign the Quality of Service (QoS) group for the different types of traffic. Enter into QoS policy map configuration mode for
#RoCE using the “policy-map type <mode> <group>” command. Type the following commands listed in the order shown below:
policy-map type qos roce
class roce
set qos-group 5
exit
class class-default
exit
# If you are going to use fcoe toe add fcoe classes to. If not you can skip de fcoe class.
class class-fcoe
set qos-group 1
exit
exit

#Allocate the appropriate bandwidth for the types of traffic. Enter into queuing policy map configuration mode for RoCE using
the “policy-map type <mode> <group>” command. Type the following commands listed in the order shown below:
policy-map type queuing roce
class type queuing roce
bandwidth percent 80
exit
class type queuing class-default
bandwidth percent 10
exit
# Same here for fcoe, create it when you need it. If you don not need it configure 20% for the class-default.
class type queuing class-foce
bandwidth percent 10
exit
exit

#Set the Maximum Transition Unit (MTU) for the separate types of traffic. Enter into the network policy map configuration mode
for RoCE using the “policy-map type <mode> <group>” command. Type the following commands listed in the order shown
below:
policy-map type network-qos roce
class type network-qos roce
pause no-drop
mtu 5000
class type network-qos class-default
mtu 9216
# Again class-foce only needed when you are going to use foce.
class type network-qos class-foce
pause no-drop
mtu 2158
exit
exit

#Configure the switches service policies. Enter into the system QoS configuration mode for the switch using the “system
<mode>” command. Type the following commands listed in the order shown below:
system qos
service-policy type qos input roce
service-policy type queuing input roce
service-policy type queuing output roce
service-policy type network-qos roce
exit
#Save the running configuration.
copy running-config startup-config

Emulex best practice is Priority of 5 for the priority group.

  • Configure the Network cards
 Next you should verify the adapter configuration with the OneCommand manager utility of Emulex. Make sure the personality is set to NIC+RoCe. The NIC+Roce configuration should be set to “ROCE-2” which is optimized for SMB purposes.
AdapterConfig

 

 

 

 

 

Beware! RoCE-1 and RoCE-2 are Emulex performance profiles and not routable versus non routable RoCE.

On the ports of the Emulex card go to the DCB tab and make sure DCB-X is enabled (which is default). With DCB-X enabled the NIC looks at the switch for PFC and QOS configuration and adopts it. Below it is disabled.

EmulexDCBxDisabled

After DCB-X is enabled and the server is rebooted you will see that it has received PFC configuration that is configured in the switch (like we did earlier).

DCBX

Set you’re SMB Nics in tagged VLANs. This is because the PFC/QOS information is added to the vlan part of the packet. So make sure the switch ports are tagged on the SMB VLAN (1 vlan per SMB NIC because of Multi Channel SMB) or in Cisco language the ports are trunk ports.

VLANTag

Also make sure the NetworkDirect MTU value is 4096

NetworkDirectMTU

And basically that’s it as far as the configuration goes for SMB Direct with Emulex Adapters. No need to disable DCB-X with powershell commands because you need it (and emulex uses it’s own hardware DCB-X and does not look at the OS DCB-X Settings). In the Emulex 14000 manuals their is not a single word on NetQOSpolicies because Emulex does not used it. All PFC and QOS is done by the hardware of the Emulex cards. And that information is past on from the switch.

Verifying

Now that all settings are in place we need to run some command’s to verify if all is configured correctly.

Get-NetadapterRDMA to verify if you NICS can be used for RDMA. (Of course you already checked this in an earlier state, because there is no point in configuring al this is your network card don’t support RDMA 😉 )
getnetadapterrdma

 

 

 

Get-NetOffloadGlobalSettings to verify if SMB Direct (NetworkDirect) is enabled.
getnetoffloadglobalsettings

 

 

 

Get-SmbServerNetworkInterface to verify if the interfaces report “True” in the colum “RDMA capable”. It could report “False” if that is the case, their is probably something wrong with the settings, driver or firmware)
getsmbservernetworkinterface

 

 

 

If this is all good, you can start testing!

Testing

The first thing that really got me off track is various blogs and video’s throughout the internet that show fast file copies with RDMA and zero network throughput. Well it turns out, that is not the case with Emulex.

For example on a system with a HP InfiniBand 10Gb/40Gb 2-port 544+QSFP Ethernet Adapter from Mellanox their is no network throughput during a file copy from one host to another.
MellanoxSMBDirectCopy

 

 

 

 

 

As you can see, lots of RDMA Outbound bytes and a 190MB per second file copy, but no network traffic.

The same copy on a system with an Emulex OCe14102-U 10Gb CNA Adapters, lots of RDMA Outbound Bytes, 310MB per second and 2.8Gbit network throughput. And all this time I was thinking that my RDMA/SMB Direct configuration was wrong because I keeped seeing lots of network traffic…
Emulexcopysmbdirectenabled

 

 

 

 

 

It turns out that Emulex is bypassing the TCP/IP stack but is still using Ethernet so there is still network traffic. But it is definitely using RDMA!

When I disable SMB Direct and copy the same file again you can see it’s a bit slower but still 240MB per second, still lots of network throughput but no RDMA Outbound Bytes.
Emulexcopysmbdirectdisabled

 

 

 

 

 

So in the end, when configuring SMB Direct with RDMA make sure you do some descend research and check the vendor for support. And also don’t stop reading after the first blog you find (including this one 🙂 ).

Good luck!


7 thoughts on “SMB Direct with Emulex”

  1. Hi
    Thanks alot fot this post. Just wish i found it 3 days sooner 🙂
    That my HP 556 branded Emulex cards didn’t want to take the enable-NetAdapterQos command had me troubleshooting for a couple of days.

    Now my biggest problem is translating the cisco config in your post to HP comware. HP is also lacking documentation/examples there.

    This post should be ranked first on google when searching for unable to enable-netadapterqos …..

  2. Hi Patrick,

    Good to know my blog helped You out. Sorry You didn’t find it 3 days earlier 😉

    I can help you with your switch config samples but if you figure it out test your environment really good for several weeks. Microsoft requested Emulex to disable RDMA by default because it is still “flakey” as they say. I know several customers in Europe to have boot stability problems for vm’s with Emulex and Rdma.

    Good luck!

  3. Had a call with HP today and got a config that seems to be working. Will leave it here in case someone else finds this post in search for answers.
    The Emulex commander says rdma is disabled but perfmon shows the nic using rmda so don’t know whats up with that.

    #
    qos map-table dot1p-lp
    import 0 export 0
    import 2 export 0
    import 3 export 0
    import 4 export 0
    import 6 export 0
    import 7 export 0
    #
    traffic classifier DCBX_class operator or
    if-match acl 3001
    if-match acl 3002
    #
    traffic behavior DCBX
    remark dot1p 5
    #
    qos policy DCBX_policy
    classifier DCBX_class behavior DCBX mode dcbx
    #
    interface Ten-GigabitEthernet1/0/1
    port link-type trunk
    port trunk permit vlan 1 10 to 11
    priority-flow-control auto
    priority-flow-control no-drop dot1p 5
    jumboframe enable 5000
    lldp tlv-enable dot1-tlv dcbx
    qos trust dot1p
    qos apply policy DCBX_policy inbound
    #

  4. Hi Patrick,

    Sorry for the tipo, I was writing the comment on my phone in a traffic jam :).
    I can’t help you indeed.

    Good to see that you got the config from HP!
    Together we create a better RDMA world! 😉

  5. Hi
    Would you possibly be willing (if possible) to share a printscreen of your DCB tab on the emulexonecommander. Not the configure dialog but the one before that.

Leave a Comment