Introduction to FCoE

• SCSI protocol: carried over a network transport via serial implementation. Two primary transports today, FC and IP.
• Fibre Channel provides high-speed transport for SCSI payload via HBA. FC overcomes many shortcomings of parallel I/O and combines best attributes of a channel and a network together.
• Storage Protocol Technologies:
• FCP
• iSCSI
• ….
• FC has many similarities to IP (TCP). FC is hop-by-hop flow controlled. No end-to-end flow control at FC level only at SCSI level. To maintain no drop packets. SCSI has timeout of 60s. You can imagine if you drop one packet, scsi operation gets corrupted and then transmits, no rapid retransmission.
• can run a lot of parallel connections.
• E_port: expansion port, ISL
• TE_port: 802.1q, ability to run multiple VSAN, trunking for VSANs
• N_Port: Node Port, server, HBA etc…they connect to F_port on the switch.
• NP_Port: It goes in an NPV mode switch, a switch that emulates a host or proxy. Emulates an N port, reduces a lot of management.
• WWN: burnt-in unique addresses assigned to fabric switches, ports, and nodes by manufacturer. That’s where the similarity ends with comparison to MAC addresses. FC packets, WWN is not there, only used in a few frames to uniquely identify the sender of that packet. Otherwise you’d see in the src/dst is a dynamic address (FC_ID). They’re unique and registered with IEEE.
• FC uses something similar to DHCP. It’s called FCID. Divided into “switch domain” (8bits), “Area (8bits)” and “Device” (8bits). Makes routing decision easy with it. Switch Topology Model. Switches assign FC_ID addresses to N_Ports.
• 32K Exchange frames,8K chunks is sequence and each of those 8K chunks is made of 2K frames. FC-2 Hierarchy. Makes it easy to fire multiple IO because each one has unique OX_ID (exchange ID) so we can load balance them on ISL
• Cisco is the only vendor that supports FC portchannels. Trunking capability, really allows Cisco differentiation.
• VSANs. Same reason we have VLANs, we have VSANs. Shared services that are running in FC environment, in order to reduce, we use VSAN.
• Used for storage tiering. 5K, 7K and UCS have this as well as MDS.
• Initiator has HBA. East-West seperation.
• Fabric Shortest Path First: Just like OSPF. FSPF routes traffic based on destination domain ID.
• Storage Security, major activity done daily. Zones are bi-dir ACL. Use WWN for ACL, so use those for the ACL. Fabric in hardware enforces who the initiator communicates with. Zone members can only see and talk to other members of the zone. Zones belong to a zoneset. Zoneset must be “active” to enforce zoning. Only one active zoneset per fabric or per VSAN.
• When you first physically join the fabric and negotiate speed, the initiator will do a FLOGI, it’ll start sending packets to the switch (not target). Tell it, I’m an initiator and I need to register to Name Server and I need to tell it what my WWN is, it’s going to grant me a FCID. Now I have an address that I can use to send frames out there. Src/Dst is FCID and NOT WWN.
• Talk to FC switch and figure out what devices I can communicate to, and FC db will determine from zone devices it can talk to. Then it does a P_LOGI which will do end to end communication. PLOGI is done end to end. Target would do the same steps as initiator at the same time.
• What is NPIV? Before we do NPV, we need to understand NPIV. N-Port ID Virtualization. Allows to allocate multiple FCIDs to a single port. Feature on core director. If we have a VMware server, we can assign a FCID to each different VMs, which allows tracking them differently for each fabric.
• Then what is NPV? Think of a 5K with 2K and lots of end devices. NPV mode allows to turn that switch into an initiator or into a host. So don’t have to run shared services, no ISL b/w MDS or NPIV switch. It logs as an N_Port rather than as an E_Port. It’s proxying all of the real-servers that are plugged into it. In TOR design, you have hundreds of UCS and 5K, going into my MDS…you can really reduce that by using NPV mode in TOR switches. Use with NPIV core directors, could be an MDS or 7K.

Advancements in Ethernet:

• adoption for 10G is a major driver. Ramping to 40GE. Puts a nail in the coffin of native FC speeds. Better than 8G or 4G FC since every DC now has 10G. Once 10G to the server happens, it’ll put a nail in the coffin for FC, since FC requires a PCI card which is power hungry device.
• Standards for FCoE: FC is made up of T11 (FC-BB-5, FC on other network media) standard and IEEE 802.1 DCB. DCB includes, PFC [Lossless ethernet-802.1Qbb] + ETS[prority grouping, 802.1Qaz] + DCBX[configuration verification, 802.1Qaz]
• PFC:Priority Flow Control (802.1Qbb), available on 5k, 2k, 7k, MDS. Able to pause FCoE traffic. Ability to accept pause frames.
• ETS: Enhanced Transmission selection: allows ability to create groups of protocol and bandwidth to protocol. I want to reserve 80% on the wire for FCoE traffic and rest for Ethernet. Down at L2.
• DCBX: 802.1Qaz, going to go through the DCBX process, that they support PFC, ETC and FCoE before they send out FCoE packet.
• It is a standard. They are all technically stable. term used by standing committee that it has passed a milestone of standards and vendors can start making products. So FCoE is a standard now.
• You can use twinax cable for FCoE. SFP+ CX-1 Copper (SFF 8431). Drives down the power and cost significantly. <10m. Only 0.1W per port. Cable and SFP are physically one component.
• CNA: HBAs that enable both FCoE and LAN traffic out of the same port. Single chip. FCoE in software can also be done with the software driver. You can run FCoE on intel or broadcom chip.

FCoE Technology/Unified Fabric:

• completely based on the FC model. WWNs, FC-IDs, Zoning, Nameserver, RSCN. Compare this with iSCSI, completely different model than FC. Very different management and tools.
• yet another overlay network.
• Products, 2k, 5k, N7K (32 port F-series), MDS 9500 (8-port FCoE card)
• FCoE is two different protocols: FCoE itself and FIP (FCoE initialization Protocol)–> control plane protocol.
• FIP is fairly shortlived protocol. It does VLAN Discovery, FCF discovery (fibre channel forwarder…fc switch inside of a ethernet switch), FLOGI/FDISC..need to login and get FCID and will be using that inside my FC packets. FIP will complete and will hand it off to FC.
• 2180 byte frame (baby jumbo frame in ethernet environment).