Introduction to FCoE

  • SCSI protocol: carried over a network transport via serial implementation. Two primary transports today, FC and IP.
  • Fibre Channel provides high-speed transport for SCSI payload via HBA. FC overcomes many shortcomings of parallel I/O and combines best attributes of a channel and a network together.
  • Storage Protocol Technologies:
    • FCP
    • iSCSI
    • ….
  • FC has many similarities to IP (TCP). FC is hop-by-hop flow controlled. No end-to-end flow control at FC level only at SCSI level. To maintain no drop packets. SCSI has timeout of 60s. You can imagine if you drop one packet, scsi operation gets corrupted and then transmits, no rapid retransmission.
  • can run a lot of parallel connections.
  • E_port: expansion port, ISL
  • TE_port: 802.1q, ability to run multiple VSAN, trunking for VSANs
  • N_Port: Node Port, server, HBA etc…they connect to F_port on the switch.
  • NP_Port: It goes in an NPV mode switch, a switch that emulates a host or proxy. Emulates an N port, reduces a lot of management.
  • WWN: burnt-in unique addresses assigned to fabric switches, ports, and nodes by manufacturer. That’s where the similarity ends with comparison to MAC addresses. FC packets, WWN is not there, only used in a few frames to uniquely identify the sender of that packet. Otherwise you’d see in the src/dst is a dynamic address (FC_ID). They’re unique and registered with IEEE.
  • FC uses something similar to DHCP. It’s called FCID. Divided into “switch domain” (8bits), “Area (8bits)” and “Device” (8bits). Makes routing decision easy with it. Switch Topology Model. Switches assign FC_ID addresses to N_Ports.
  • 32K Exchange frames,8K chunks is sequence and each of those 8K chunks is made of 2K frames. FC-2 Hierarchy. Makes it easy to fire multiple IO because each one has unique OX_ID (exchange ID) so we can load balance them on ISL
  • Cisco is the only vendor that supports FC portchannels. Trunking capability, really allows Cisco differentiation.
  • VSANs. Same reason we have VLANs, we have VSANs. Shared services that are running in FC environment, in order to reduce, we use VSAN.
  • Used for storage tiering. 5K, 7K and UCS have this as well as MDS.
  • Initiator has HBA. East-West seperation.
  • Fabric Shortest Path First: Just like OSPF. FSPF routes traffic based on destination domain ID.
  • Storage Security, major activity done daily. Zones are bi-dir ACL. Use WWN for ACL, so use those for the ACL. Fabric in hardware enforces who the initiator communicates with. Zone members can only see and talk to other members of the zone. Zones belong to a zoneset. Zoneset must be “active” to enforce zoning. Only one active zoneset per fabric or per VSAN.
  • When you first physically join the fabric and negotiate speed, the initiator will do a FLOGI, it’ll start sending packets to the switch (not target). Tell it, I’m an initiator and I need to register to Name Server and I need to tell it what my WWN is, it’s going to grant me a FCID. Now I have an address that I can use to send frames out there. Src/Dst is FCID and NOT WWN.
  • Talk to FC switch and figure out what devices I can communicate to, and FC db will determine from zone devices it can talk to. Then it does a P_LOGI which will do end to end communication. PLOGI is done end to end. Target would do the same steps as initiator at the same time.
  • What is NPIV? Before we do NPV, we need to understand NPIV. N-Port ID Virtualization. Allows to allocate multiple FCIDs to a single port. Feature on core director. If we have a VMware server, we can assign a FCID to each different VMs, which allows tracking them differently for each fabric.
  • Then what is NPV? Think of a 5K with 2K and lots of end devices. NPV mode allows to turn that switch into an initiator or into a host. So don’t have to run shared services, no ISL b/w MDS or NPIV switch. It logs as an N_Port rather than as an E_Port. It’s proxying all of the real-servers that are plugged into it. In TOR design, you have hundreds of UCS and 5K, going into my MDS…you can really reduce that by using NPV mode in TOR switches. Use with NPIV core directors, could be an MDS or 7K.

Advancements in Ethernet:

  • adoption for 10G is a major driver. Ramping to 40GE. Puts a nail in the coffin of native FC speeds. Better than 8G or 4G FC since every DC now has 10G. Once 10G to the server happens, it’ll put a nail in the coffin for FC, since FC requires a PCI card which is power hungry device.
  • Standards for FCoE: FC is made up of T11 (FC-BB-5, FC on other network media) standard and IEEE 802.1 DCB. DCB includes, PFC [Lossless ethernet-802.1Qbb] + ETS[prority grouping, 802.1Qaz] + DCBX[configuration verification, 802.1Qaz]
  • PFC:Priority Flow Control (802.1Qbb), available on 5k, 2k, 7k, MDS. Able to pause FCoE traffic. Ability to accept pause frames.
  • ETS: Enhanced Transmission selection: allows ability to create groups of protocol and bandwidth to protocol. I want to reserve 80% on the wire for FCoE traffic and rest for Ethernet. Down at L2.
  • DCBX: 802.1Qaz, going to go through the DCBX process, that they support PFC, ETC and FCoE before they send out FCoE packet.
  • It is a standard. They are all technically stable. term used by standing committee that it has passed a milestone of standards and vendors can start making products. So FCoE is a standard now.
  • You can use twinax cable for FCoE. SFP+ CX-1 Copper (SFF 8431). Drives down the power and cost significantly. <10m. Only 0.1W per port. Cable and SFP are physically one component.
  • CNA: HBAs that enable both FCoE and LAN traffic out of the same port. Single chip. FCoE in software can also be done with the software driver. You can run FCoE on intel or broadcom chip.

FCoE Technology/Unified Fabric:

  • completely based on the FC model. WWNs, FC-IDs, Zoning, Nameserver, RSCN. Compare this with iSCSI, completely different model than FC. Very different management and tools.
  • yet another overlay network.
  • Products, 2k, 5k, N7K (32 port F-series), MDS 9500 (8-port FCoE card)
  • FCoE is two different protocols: FCoE itself and FIP (FCoE initialization Protocol)–> control plane protocol.
  • FIP is fairly shortlived protocol. It does VLAN Discovery, FCF discovery (fibre channel forwarder…fc switch inside of a ethernet switch), FLOGI/FDISC..need to login and get FCID and will be using that inside my FC packets. FIP will complete and will hand it off to FC.
  • 2180 byte frame (baby jumbo frame in ethernet environment).

Narbik’s Advanced Lab Notes — OSPF (II)

Lab 4 (summarization):

  • External routes are summarized on ASBR using “summary-address” command.
  • when summarizing internal routes on ABRs, the “area xx range” command must be used with xx is the area id. The routes that are being summarized originated in area xx (1 in our case), the “area range” command MUST specify the area “area 1 range” followed by the summary network address.
  • In OPSF, the discard route is created automatically whenever summary route is configured. 2 types of summary routes: Internal and External. When internal summary routes are configured, OSPF will inject an internal discard route and same thing with external. This is to prevent loops.
  • to get rid of these discard routes, you do, “no discard-route internal

Lab 5 (Virtual Links and GRE tunnels)

  • ensuring all adv network are reachable by all routers. Use any IP addressing and NOT using virtual link. We will be using GRE tunnels.
  • so if R5 is not connected to area 0 and R4 is the ABR b/w Area 4 and Area 2. The GRE tunnel must be configured b/w R4 and R3.
  • assign any arbitrary network on both routers, and include them in area 0. (e.g This will be the ip address of the GRE tunnel interface “int tun1”. Then make tunnel source to be the link b/w R4 and R3 and the destination to be the other end. Do this on both sides.
  • Neighbor relationship would be formed.
  • The IP address of the tunnel interface MUST be advertised in area 0 or else the tunnel will not work.
  • Configuring authentication over virtual link (simple)
    • router ospf 1 (on R1)
      • area 1 virtual-link authentication
      • area 1 virtual-link authentication-key Cisco
    • router ospf 1 (on R2)
      • area 1 virtual-link auth
      • area 1 virtual-link auth-key Cisco
  • MD5:
    • on router 4
      • Rack1R4(config-router)#$ual-link authentication message-digest
      • Rack1R4(config-router)#$ual-link mess
      • Rack1R4(config-router)#$ual-link message-digest-key 1 md5 cisco
    • on R3:
      • same thing…

Lab 6: Stub, totally stubby and NSSA

  • Stub:
    • stops type 4 and 5. Issue “area xx stub” command on all router and the ABR for that area.
    • stub cannot be a transit area so no virtual link but a GRE tunnel can be used instead.
    • Stub area cannot have an ASBR.
    • backbone area cannot be a stub area.
    • no Lsa type 5 (E1 or E2) are allowed in STUB area, but routers can connect to external routes via default route that is injected in area by ABR.
    • by default, cost of default route is 1; verify using “sh ip ospf” and show ip route. This can be changed by “area xx default-cost yy”.
  • Totally Stubby:
    • does not receive types 3,4 and 5.
    • on all routers “area xx stub” and on the ABR issue “area xx stub no-summary”
    • all IA and E routes are filtered.
    • default route IS injected
  • NSSA:
    • the default 0/0 route will not be injected
  • NSSA with default route:
    • you have to use the command: area xx nssa default-information-originate (on the ABR)
    • will inject a type N2 default route into the area.
  • Totally Stubby NSSA with default route
    • area 1 nssa no-summary on ABR (allows 1,2,7 and default route)

Lab 7 (filtering routes in OSPF)

  • Filter all LSA from other areas to area 2, the other areas do receive routes from area 2. No distribute-list, acl, or any command under router ospf is allowed:
    • go to interface level and issue: “ip ospf database-filter all out”
    • this command ONLY works on point-to-multipoint interfaces.
      • neighbor <ip> database-filter all out
  • Configure R3 or R4 such that R4 does not have reachability to Network Using distribute-list.
    • This must be done on R4 since distribute-list out ONLY works on ASBR and R3 is not an ASBR. So we need to do a “distribute-list in” on R4.
  • Configure the routers such that they don’t have reachability to network (which was redistributed on R4). No global config command, neighbor command, int config command or ip ospf command allowed.
    • use “summary-address no-advertise”
    • can be used to filter an external network when configured on ASBR
    • not be used to filter internal networks.
  • Configure routers such that R3 and R4 don’t have reachability to network
    • use “area 1 range not-advertise” command on the ABR. This filters internal routes within an OSPF routing domain and can only be used on an ABR.
  • Configure R2 such that network (from area 1) is not advertised to area 2.
    • use prefix-list here
      • ip prefix-list net seq 5 deny
      • ip prefix-list net seq 10 per le 32
      • area 2 filter-list prefix NET in ( is coming from area 1, but we’re filtering it out of area 2 by using “in”, filtering prefixes advertised in LSA type 3 BETWEEN OSPF area of an ABR)
  • Ways of filtering:
    • ip ospf database-filter all out on interface level
    • neighbor <ip> database-filter all out (on point to multipoint int only)
    • using distribute-list in.
    • using distribute-list out on ASBR
    • using summary-address no-advertise on ASBR
    • using area 1 range not-advertise on ABR
    • using ip prefix list and area filter list to filter LSA type 3s.

Lab 8 (redirecting traffic)

  • R1 has two ways to reach network Ensure that R1 uses R2 instead of R3. R1 should go directly to R3 to reach network DO NOT USE, bandwidth, any global config command, OSPF cost or distance command.
    • max-metric router-lsa
    • other routers do not prefer the router as a transit hop in their path to given network.

Lab 9 (limiting number of OSPF redistributed routes)

  • use the command “redistribute maximum-prefix 10 70 (warning-only)”

Lab 10 (OSPF and NBMA)

  • On Non-Broadcast media (like FR) OSPF can run in two modes:
    • NBMA: simulates broadcast model. Two ways to simulate a broadcast model:
      • ip ospf network broadcast (interface sub-command) [on both ends]. Map statements need to have “broadcast”.
      • configure neighbor statements using router ospf. [use “ip ospf priority 2” to assign DR]. Frame relay map commands in this case do NOT need a broadcast parameter because OSPF packets are unicasted with neighbor statement.
    • Point-to-Multipoint: treats non-broadcast networks as a collection of point-to-point links by configuring “ip ospf network point-to-multipoint” command. Need Broadcast statement. No DR and BDR elected when NBMA network is configured Point-to-Multipoint.
  • you MUST define network type on non-broadcast networks to avoid configuring neighbor statements.
  • FR subinterfaces can run in two modes:
    • p2p: the subinterface emulates a p2p network and OSPF treats it as a p2p network type.
    • multipoint: OSPF treats this subinterface as NBMA network type.
  • One end configured as P2P, other as Multipoint Physicaly interface. How do we resolve? Configure the P2P interface on the hub as “ip ospf network non-broadcast” and assign a neighbor command under router ospf. In non-broadcast networks, the “neighbor” command in router config mode must be configured so the OSPF hello packets are exchanged via Unicast.
  • changing an interface from “non-broadcast” to “point-to-point” earlier required changing hello intervals. in the latest IOS releases, hello intervals automatically change when you do “ip ospf network point-to-point”

Narbik’s Advanced Lab Notes – OSPF (I)

Currently doing OSPF labs based on Narbik’s workbook:


Lab 1 (optimization):

  • turning off LSA type 6
    • go under router ospf –> ignore lsa mospf
  • changing hello timers such that it sends 4 hellos per second with hold time of one second
    • under interface –> ip ospf dead-interval minimal hello-mult 4
  • configuring ospf to display router id in it’s ospf show commands
    • global: ip ospf name
    • ip host <R1> <>
  • router ospf 1 –> timers pacing retransmission 60 (interpacket pacing b/w consecutive LSUs)

Lab 2 (auth):

  • creating virtual link, “area <area to create vl over> virtual-link <routerid of other router>”
  • configuring area auth:
    • area X authenticaton <message-digest>
    • int
      • ip ospf message-digest key 1 md5 cisco12

Lab 3 (cost):

  • cost of loopback is cost of serial + cost of lo = 100,000,000/1.54MB (64) + 100MB/8000MB = 1

Narbik’s Soup-to-Nuts [Switching] Lab 1

<begin lab1 interesting stuff>

LAB 1:

  • 3550 by default would negotiate ISL trunk and two ports negotiate to “n-isl”, since by default ports are configured in Desirable state.
  • With 3560, ports are not in desirable mode, so the trunk must be configured statically to trunk or negotiate a trunk.
  • extended range vlans can only be created when vtp is in transparent mode. This configuration can only be performed in the global config mode and not in the VLAN Database. And you would see these vlans appearing in running/startup config but NOT in the vlan database.
  • CAT switches can be configured such that the IP address of any of it’s interfaces can be used as the source of all VTP messages. Use command: #vtp interface lo0. Verify it by looking at the last line in: show vtp status
  • The priority of a given vlan for a STP is the combination of base priority and the Vlan number, e.g, 32768 + (vlan) 12 = 32780.
  • Configuring a cat switch to be root for vlan 12 and 34 using the “root” command: #spanning-tree vlan 12,34 root primary. The keyword ‘root’ is a MACRO that REDUCES the BID of the switch for a given vlan by a value of 8192. For e.g. 32768 + 12 -8192 = 24588 (Priority of the above value after root macro is applied)
  • The “spanning-tree portfast bpduguard default” command in global config mode will shut the port down in “err-disable” mode if any portfast enabled port receives BPDU packets
  • switchport trunk encap command can have following options: 1) dot1q, 2) isl, 3) negotiate (local interface negotiates with neighboring interface to become either dot1q or isl).
  • switchport mode trunk puts the interface in permanent trunking mode.
  • MONITOR session: (local monitoring)
    • cat1(config)#monitor session 1 source interface f0/14 both
    • cat1(config)#monitor session 1 destination interface f0/15
    • there can only be two monitor sessions per switch. Their direction can be RX, TX or Both. Vlans can ONLY be configured in Rx direction.
    • Verify: #show monitor session 1
  • Configuring SNMP:
    • Must have an IP address configured on a switch otherwise snmp server cannot be configured.
    • Setup SNMP server: (config)# snmp-server host private
    • Configure it to send mac-address traps to NMS: (config)# snmp-server enable traps mac-notific
    • Enable MAC-address notification: (config)#mac-address-table notification
    • Enable SNMP trap on interface Fa0/1 to send MAC notification traps whenever MAC-address is added. int fa0/1 –> #snmp trap mac-notification added
    • if we want to switch to report when MAC addresses that are learnt are expired then, “snmp trap mac-notification removed” needs to be added.
    • Verify: show mac-address-table notification interface f0/1
  • Regular and Smart Port Macro:
    • Define a port range (regular macro): (config)#define interface-range router-ports f0/1-6
    • Smartport Macro: (to configure port security)
      • config# macro name port-secur <– starts with a macro name, can be applied to int, int-range or a regular macro
      • Enter macro commands one per line. end with @
      • switchport mode access
      • switchport port-security
      • swithcport port-security mac-address sticky
      • switchport port-security max 1
      • switchport port-security violation shutdown.
      • @
    • applying the macro:
      • int range macro Router-ports
      • Macro apply port-secur
  • Configuring bandwidth utilization for broadcast traffic to 50%
    • int fa0/1 –> #storm-control broadcast level 50.00
    • value of 0.0 means that type of traffic is blocked permanently (could be unicast, broadcast or multicast)
    • on 3550 when the rate of MULTICAST traffic exceeds a predefined threshold, ALL incoming traffic (BC, MC, or Unicast) is dropped until the level of MCast traffic is dropped below the threshold level.
  • Protected ports:
    • You have two ports on the same vlan and you don’t want them to be able to talk to each other, issue this: int range fa0/15-16 –> switchport protected.
    • unknown Bcast and Ucast packets can be blocked (for security issues) by issuning –> switchport block unicast && switchport block multicast
  • Configuring Dot1x
    • config# dot1x system-auth-control (enables dot1x)
    • config#aaa new model (enable AAA services)
    • config#aaa authentication dot1x default group radius (specify auth method list)
    • config#radius server host <ip> key cisco (specify radius server and password)
    • Now configure dot1x on the port. MAKE SURE the port in not in dynamic mode otherwise dot1x won’t work. Make sure it’s in access mode: “switch mode access”
    • configure on interface fa0/15 –> dot1x port-control auto

</end of lab 1 interesting stuff>

How to configure SSH on your router

[taken from IE VOL1 workbook]

It’s best to only allow SSH access on your VTY ports. Here’s how you can do it:

1) Configure a domain name on your router:

Rack1R4(config)#ip domain-name

2) Generate the keys:

Rack1R4(config)#crypto key generate rsa general-keys modulus 512

3) Configure the VTY ports with SSH access only:

Rack1R4(config)#line vty 0 4

Rack1R4(config-line)#transport input ssh

4) enable local login for your VTY ports:

Rack1R4(config-line)#login local

5) Don’t forget to configure a local username/password

Rack1R4(config)#username CISCO password CISCO

[make sure you also have an enable password configured]



Going back to the “intermediate” router with term server

[hint taken from Brian Dennis’s CoD]

Suppose you are using a term server to login to all your routers. And you’re on R1. You SSH to R4. How do you go back to R1? See below:

[Resuming connection 1 to r1 … ]

Rack1R1#ssh -l CISCO


<—-now you want to go back to R1, how would you do it? If you press, CTRL+SHIFT+6+x, you’d go back to the term server, not to R1!

Rack1R4> [press CTRL+SHIFT+6+x]
So to go back to R1 (the intermediate router), you’d have to press, CTRL+SHIFT+6+6+x, this would get you back to R1

Rack1R4> [press CTRL+SHIFT+6+6+x]

Hope this helps!