Configuring FCoE in Linux (RHEL) and HP FlexFabric

by sergeyt

Actually it’s easy. Very easy indeed, like going 1, 2, 3.

  1. Collect information about MAC addresses to distinguish pure Ethernet NICs and CNA that will pass FCoE traffic. The latter have both MAC and WWN addresses.
  2. Power on a server and update /etc/udev/rules.d/70-persistent-net.rules if required.
  3. Activate new dev rules: 
    # udevadm trigger
  4. Install fcoe-utils and lldpad packages: 
    # yum install fcoe-utils.x86_64
  5. Rename /etc/fcoe/cfg-ethx file using the name of you CNAs. For example, if eth5 is your CNA interface, then: 
    # cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eth5
  6. Edit /etc/fcoe/cfg-ethX files and set DCB_REQUIRED=”yes” to DCB_REQUIRED=”no”
  7. Start FCoE and LLDPAD services and set adminStatus to disable for ALL Broadcom-based CNA interfaces as stated by HP. Please note, that

    …In a FlexFabric environment, LLPAD must be disabled on all network adapters…

    # chkconfig lldpad on
# chkconfig fcoe on
# service lldpad start
# service fcoe start
# for d in `ip link ls | grep mtu | awk -F \: '{print $2}'`; do lldptool set-lldp -i $d adminStatus=disabled; done
# cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eth5
  8. Create Ethernet configuration file for all CNA iterfaces to make sure they will be brought online after reboot: 
    DEVICE=eth5
ONBOOT=yes
BOOTPROTO=none
USERCTL=NO
MTU=9000
  9. Run ifup to bring FCoE interfaces up. If everything is OK reboot the server as a final test and start enjoying FCoE. 
    # ifup eth5
  10. Why MTU=9000? Because FC payload is 2,112 bytes jumbo frames must be turned on to avoid unnecessary IP fragmentation.

4 thoughts on “Configuring FCoE in Linux (RHEL) and HP FlexFabric”

  1. Great tuto, but I think there is a small typo on the line 6. I think you instead wanted to edit cfg-eth5 (instead of cfg-ethX)…

    6. Edit /etc/fcoe/cfg-ethX files and set DCB_REQUIRED=”yes” to DCB_REQUIRED=”no”

    Reply

    • Hi Steve,

      Thank you for stopping by and leaving a helpful comment. Deeply appreciate that. To tell the truth, I left cfg-ethX on purpose with the intention to emphasise that there could be other cfg-ethX interfaces and not all of them are CNAs. Obviously my approach didn’t work out and I will update the post to avoid any ambiguity.

      Cheers.

      Reply

    • Hi.

      Wish I had a solid answer but unfortunately, due to the lack of Dell’s equipment in my lab, I don’t.
      But based on the information I gleaned from Redhat Configuring a Fibre-Channel Over Ethernet Interface none of the settings are vendor specific and hopefully they could be applied to Dell branded Qlogic CNA as well. Just pain attention to the warning:

      Do not run software-based DCB or LLDP on CNAs that implement DCB.

      Some Combined Network Adapters (CNAs) implement the Data Center Bridging (DCB) protocol in firmware.
      The DCB protocol assumes that there is just one originator of DCB on a particular network link.
      This means that any higher-level software implementation of DCB, or Link Layer Discovery Protocol (LLDP), must be disabled on CNAs that implement DCB.

      That was exactly the reason to set DCB_REQUIRED=”no” and to disable adminStatus with lldptool. Which is also mentioned in Installation Notes on Supermicro’s web site.

      With QLogic’s FCoE solution, LLDP packets are sent by the MCP firmware. However, the lldp frames are also sent by lldpad daemon.
      To disable the duplicate lldp packets, run the following command:

      lldptool set-lldp -i adminStatus=disabled

      Hope that helps.

      Reply

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.