Configuring FCoE in Linux (RHEL) and HP FlexFabric

Actually it’s easy. Very easy indeed, like going 1, 2, 3.

  1. Collect information about MAC addresses to distinguish pure Ethernet NICs and CNA that will pass FCoE traffic. The latter have both MAC and WWN addresses.
  2. Power on a server and update /etc/udev/rules.d/70-persistent-net.rules if required.
  3. Activate new dev rules: 
    # udevadm trigger
  4. Install fcoe-utils and lldpad packages: 
    # yum install fcoe-utils.x86_64
  5. Rename /etc/fcoe/cfg-ethx file using the name of you CNAs. For example, if eth5 is your CNA interface, then: 
    # cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eth5
  6. Edit /etc/fcoe/cfg-ethX files and set DCB_REQUIRED=”yes” to DCB_REQUIRED=”no”
  7. Start FCoE and LLDPAD services and set adminStatus to disable for ALL Broadcom-based CNA interfaces as stated by HP. Please note, that

    …In a FlexFabric environment, LLPAD must be disabled on all network adapters…

    # chkconfig lldpad on
# chkconfig fcoe on
# service lldpad start
# service fcoe start
# for d in `ip link ls | grep mtu | awk -F \: '{print $2}'`; do lldptool set-lldp -i $d adminStatus=disabled; done
# cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eth5
  8. Create Ethernet configuration file for all CNA iterfaces to make sure they will be brought online after reboot: 
    DEVICE=eth5
ONBOOT=yes
BOOTPROTO=none
USERCTL=NO
MTU=9000
  9. Run ifup to bring FCoE interfaces up. If everything is OK reboot the server as a final test and start enjoying FCoE. 
    # ifup eth5
  10. Why MTU=9000? Because FC payload is 2,112 bytes jumbo frames must be turned on to avoid unnecessary IP fragmentation.
Posted on April 23, 2015 at 12:31 pm by sergeyt · Permalink
In: Linux

4 Responses

Subscribe to comments via RSS

  1. Written by stivesso
    on August 30, 2016 at 3:50 pm
    Reply · Permalink

    Great tuto, but I think there is a small typo on the line 6. I think you instead wanted to edit cfg-eth5 (instead of cfg-ethX)…

    6. Edit /etc/fcoe/cfg-ethX files and set DCB_REQUIRED=”yes” to DCB_REQUIRED=”no”

    • Written by sergeyt
      on August 31, 2016 at 12:36 am
      Reply · Permalink

      Hi Steve,

      Thank you for stopping by and leaving a helpful comment. Deeply appreciate that. To tell the truth, I left cfg-ethX on purpose with the intention to emphasise that there could be other cfg-ethX interfaces and not all of them are CNAs. Obviously my approach didn’t work out and I will update the post to avoid any ambiguity.

      Cheers.

  2. Written by John Doe
    on February 6, 2017 at 10:49 pm
    Reply · Permalink

    Do these settings apply to a Dell qlogic CNA?

    • Written by sergeyt
      on February 7, 2017 at 1:12 pm
      Reply · Permalink

      Hi.

      Wish I had a solid answer but unfortunately, due to the lack of Dell’s equipment in my lab, I don’t.
      But based on the information I gleaned from Redhat Configuring a Fibre-Channel Over Ethernet Interface none of the settings are vendor specific and hopefully they could be applied to Dell branded Qlogic CNA as well. Just pain attention to the warning:

      Do not run software-based DCB or LLDP on CNAs that implement DCB.

      Some Combined Network Adapters (CNAs) implement the Data Center Bridging (DCB) protocol in firmware.
      The DCB protocol assumes that there is just one originator of DCB on a particular network link.
      This means that any higher-level software implementation of DCB, or Link Layer Discovery Protocol (LLDP), must be disabled on CNAs that implement DCB.

      That was exactly the reason to set DCB_REQUIRED=”no” and to disable adminStatus with lldptool. Which is also mentioned in Installation Notes on Supermicro’s web site.

      With QLogic’s FCoE solution, LLDP packets are sent by the MCP firmware. However, the lldp frames are also sent by lldpad daemon.
      To disable the duplicate lldp packets, run the following command:

      lldptool set-lldp -i adminStatus=disabled

      Hope that helps.

Subscribe to comments via RSS

Leave a Reply