1 Router, 2 expansion card, multiples SFP types and interfaces inheriting multiples atributes...

[ppcarvalhof]

Solution for (x)SFP on expansion board in modular equipment.

Using a Huawei NE800 M8 router as a reference.
Scenario: 1 Router with 8 slots for expansion cards, two expansion cards, some types of SFP+ modules of 3 different types (SFP+ twisted pair, SFP+ WDM Simplex and SFP+ duplex).

I have 2 problems.

1st Model the expansion cards so that the interfaces are identified according to the position of the slot where the card was inserted. For example, card X contains 10 SFP+ ports (0-9) and has been inserted into slot 7, the ports are identified in the system as Xgigabit 0/7/[0-9] (F/S/P). The Y card contains 2 QSFP28 ports and being inserted into slot 3, the ports are identified as 100Gigabit 0/3/[0-1]. In this case, the modules option is not useful because each interface of the board mentioned above is not a “complete” interface, since the type and number of connections is intrinsically linked to the SFP module installed in the port (it can be twisted pair, a single fiber , even two fibers and up to 12 fibers in an MPO connector) with the same interface designation in the system. And each board also doesn't work in isolation (like the example of blade servers in the documentation), so it would need to be a module that accepts adding other modules (inception). How to model the Router and the boards so that each board has the second digit of the interface identification according to the position in which the board was inserted in the router (F/S/P identified frame)?

2º Model the (x)SFP ports as modules so that these modules represent the interface according to the module. Example: in port Xgigabit 0/7/0 an SFP+ module for twisted pair was inserted, in port X 0/7/4 an SFP+ WDM module with a single fiber was inserted, in port X 0/7/9 a Module was inserted Duplex (two-fiber) SFP+ therefore uses 2 strands on a single interface. How to model these “variable” interfaces according to the used module?

Summary: I need the router to have slots for the card to be automatically associated with its position (Frame/Slot/Port) and for the interfaces associated with that card to be slots for the interface connection to be relative to the module inserted in it (twisted pair, one fiber, two fibers or more).

---

Original em português....

Solução para (x)SFP em placa de expansão em equipamento modular.

Usando como referência um roteador Huawei NE800 M8.
Cenário: 1 Router com 8 slots para placas de expansão, duas placas de expansão, alguns tipos de módulos SFP+ de 3 tipos diferentes (SFP+ par trançado, SFP+ WDM Simplex e SFP+ dúplex).

Tenho 2 problemas.

1º Modelar as placas de expansão para que as interfaces sejam identificadas de acordo com a posição do slot em que a placa foi inserida. Por exemplo, a placa X contém 10 portas SFP+ (0 a 9) e foi inserida no slot 7, as portas são identificadas no sistema como Xgigabit 0/7/[0-9] (F/S/P). A placa Y contém 2 portas QSFP28 e sendo inserida no slot 3, as portas são identificadas como 100Gigabit 0/3/[0-1]. Nesse caso a opção módulos não é útil porque, cada interface da placa acima citada não é uma interface “completa” pois, o tipo e a quantidade de conexões é intrinsecamente ligado ao módulo SFP instalado na porta (podendo ser par trançado, uma única fibra, mesmo duas fibras e até 12 fibras num conector MPO) sendo a mesma designação de interface no sistema. E cada placa também não funciona isoladamente (como o exemplo de servidores blade da documentação), logo precisaria ser um módulo que aceite adicionar outros módulos (inception). Como modelar o Router e as placas para que cada placa tenha o segundo dígito da identificação da interface de acordo com a posição em que a placa fora inserida no router (frame no identificado F/S/P)?

2º Modelar as portas (x)SFP como módulos que esses módulos representem a interface de acordo com o módulo. Exemplo: na porta Xgigabit 0/7/0 foi inserido um módulo SFP+ para par trançado, na porta X 0/7/4 foi inserido um módulo SFP+ WDM com uma única fibra, na porta X 0/7/9 foi inserido um Módulo SFP+ dúplex (duas fibras), portanto, usa 2 cordões em uma única interface. Como modelar essas interfaces “variáveis” de acordo com o módulo usado?

Resumo: preciso que o router tenha slots que a placa seja automaticamente associado à sua posição (Frame/Slot/Porta) e que, as interfaces associadas a essa placa sejam slots que a conexão da interface seja relativa ao módulo nela inserido (par trançado, uma fibra, duas fibras ou mais).

---

Imagens

The router NE800 M8 (fame + cards) / O roteador (frame + placas) NE8000 M8

Position and slot names / Posição e nomenclatura dos slots

SFPs

SFP twisted pair / Par trançado

SFP Simplex

SFP Duplex / Dúplex

SFP MPO

[pdenessen]

Hi, did you tried to use the {module} variable? This will take the position field of the module-bay in the device.

When I understand correct then you want to add SFPs as modules? But adding a module to an module is not possible.

[ppcarvalhof]

Yes, I tried using the module option for the slots. The module option works correctly if used for an interface where the ports are “physical” (like a twisted pair ports card for a switch) but in that case the ports are slots for SFP modules and those modules are part of the interface. A NetBox master here in BR suggested using the inventory expedient for these SFPs, however, it does not solve the question of the type and quantity of physical connections that the SFP can add to the interface. If modules allowed inside these (one module inside another) it would work like magic...

[candlerb]

A NetBox master here in BR suggested using the inventory expedient for these SFP

They are giving you very good advice.

however, it does not solve the question of the type and quantity of physical connections that the SFP can add to the interface.

That is true, but it is a fundamental limitation of the Netbox data model. Cables can only connect to interfaces, frontports, consoleports etc (Netbox collectively calls these "components"). These are added directly to the Device - although they can be cloned from Device Type or Module definitions.

A cable must connect directly to the component, which could be an SFP port for example. Netbox does not allow components to contain other components (e.g. SFP modules inside SFP port), nor to terminate a cable on anything which is not a component.

The presence of an SFP module inventory item does not have any effect on the type or number of cables which can be connected to the related SFP port: each component can be occupied by only one cable endpoint. But a single cable can connect to one or more components at each end, and the number at each end does not have to match.

[ppcarvalhof]

It really is a limitation that should be circumvented in the future (I think). This unfortunately disturbs the documentation a bit because NetBox is faithful to reality, but with your suggestion I should partially solve this limitation.

Thank you again candlerb!!!

[candlerb]

Netbox's model of 'interfaces' is simplistic, and does not handle the full detail you're looking for around (Q)SFPs.

What you should do is:

• Model the expansion cards as "modules" containing a number of "interfaces" equal to the number of QSFP ports. Use the {module} variable so that the interfaces automatically get the slot as part of their name when the module is instantiated, e.g. Xgigabit 0/{module}/[0-9]
• Model the QSFP modules themselves as "inventory items" which are associated with an interface
• Cables connect directly to the "interfaces". The QSFP modules do not terminate cables.

That is, whilst what you want is

Interface <---- QSFP module <---- cable ====> ODF ports

what Netbox provides is:

Interface <---- cable ====> ODF ports
      ^
      `--- QSFP module (inventory item)

For the case where one QSFP module has an MPO port with 8 fibres (say), but still represents one logical interface (e.g. 100G), then you use the ability of Netbox to have a variable number of connections at each end: e.g. you create a single "cable" which has one "interface" connection at the QSFP interface end, and 8 "frontport" connections at the ODF end.

For the case where one QSFP module is visible to the OS as multiple interfaces (e.g. 4 x 25G) with a breakout cable, then you need to create additional interfaces in Netbox to represent those 4 interfaces. You create them by hand, after inserting the "module", and you can associate them with the physical port by using the "parent" interface association. Again, a QSFP breakout cable would be modelled as connecting to one "interface" at the QSFP end, and either four "interfaces" or eight "frontports" at the other end.

A direct MPO connection between two QSFP interfaces, which doesn't go via an ODF, would be modeled as a single cable with a single interface connection at both ends - even though internally the cable carries 8 fibre strands, and even though in reality it plugs into the outer port on the QSFP module, not directly into the QSFP slot of the switch/router. (Unless it's a DAC of course).

This model certainly has limitations. It mixes the concepts of a physical socket (e.g. a QSFP port), an interface (e.g. four 25G interfaces sharing the same QSFP port) and a fibre strand (e.g. separate TX and RX strands on one interface); it doesn't allow accurate end-to-end tracing between interfaces in these more complex scenarios. However it is simple to understand, and good enough for many use cases. It works because Netbox does not perform any validation of connectors and matings: that is, a cable in Netbox does not have any physical connectors defined, and can mate with any type of interface or port.

You should not be afraid of manually adjusting the device by adding or removing interfaces to it after the modules have been inserted. After all, this is what you would do for virtual interfaces (e.g. loopback interfaces, vlan subinterfaces etc)

[ppcarvalhof]

Really this NetBox limitation of not faithfully representing the physical “thing” (in the case of SFP being able to “change” the connection interface with “the outside world”) breaks my legs. Your suggestion really brought me light in trying to get around this limitation. I'll try using the front-port for that. It won't be the best option but it really should solve my problem. When SFP is duplex and uses two fibers on a single interface.

Thanks for the idea candlerb!!!

[CADbloke]

Like many others, I am trying to get my head around this modeling problem. Cable-tracing is an important requirement for us.

One of the devices I am trying to model is https://evertz.com/products/NATX-32/64-100G - it has QSFP-DDs (at 200GbE, not 400GbE) which support 8x channels of 10/25GbE as well as 2x 100GbE. Each QSDFP-DD is managed as a group of 8 10/25GbE "ports".

From the QSFP-DDs the cabling variably

• splits to 2x MPOs
  • into Rack to ODF ties into the ODF
  • or directly to devices,
• or into an MPO-8x LC panel
• or a QSFP-DD DAC to other NATX Switches,

From the ODF they variably exit as MPO or 4x LC and back to racks via more ties and then often break out at the racks in MPO-LC panels. The rack panels and ODF are fairly-obviously RearPort - FrontPort models. So far @candlerb's solution seems to be the best one for the NATX QSFP-DDs but cable-tracing is an important requirement for us.

I saw #10037 (Interfaces table, dropdown menu: "Add child interface" option) and pondered if a 100GbE QSFP could own 4x virtual 25GbE Interfaces but I suppose a cable trace would not honor the Port Lanes, or even trace to a virtual connection in any case. That feels like "wring way, go back".

Is the best/only modeling solution for the QSFP-DD that breaks into, say 1x 100GbE + 4x 10/25GbE to pretend it is all of those connectors? Should it also go through a pseudo device that represents the physical QSFP-DD connection to the ties, ODF etc. ?

Edit: dunno why this disappeared from the Wiki but here's the long-gone FAQ on breakout cables. https://github.com/netbox-community/netbox/wiki/Data-Model-Limitations/_compare/1f7b6961b9e9093ebad8bcf611c073d978733ec1...07275f26a2a7dd71e7fb212387be14477ebb44e5?diff=unified - not applicable any more?

[candlerb]

Summary of the tradeoff: #11917 (comment)

dunno why this disappeared from the Wiki

Probably because the cable model now allows each end to have multiple terminations, which means the physical model of a breakout cable can now match the real world. However it cannot handle multiple interfaces on the same physical port, or correctly handle interface neighbors or cable tracing.

[CADbloke]

Thanks @candlerb

I guess this also answers my other question

Is the best/only modeling solution for the QSFP-DD that breaks into, say 1x 100GbE + 4x 10/25GbE to pretend it is all of those connectors? Should it also go through a pseudo device that represents the physical QSFP-DD connection to the ties, ODF etc. ?

Pondering if the device type definition should have all of the pseudo Interfaces on it or is that just noise? I suppose, also answered at

You should not be afraid of manually adjusting the device by adding or removing interfaces to it after the modules have been inserted.

Sigh, it feels like a hack.

[ppcarvalhof]

I also think cable tracing is very important. I think at this point NetBox fails to model xSFP interfaces and the like. As stated by CADbloke, ZxSFP and similar interfaces are premised on a slot to fit a module and this module determines some things about the connection itself (such as the type of connector or technology: twisted pair or optical. Whether it will be one or more channels on the same interface like the simplex/duplex ones I mentioned. If optical, whether it will be LC, SC, ST, MPO etc...). And we still have the case of QSFP interfaces, for example, which can work as 1 channel of 40G/100G or 4 channels of 10G/25G, as CADbloke also mentioned. At the system level the identification can change (in the little experience I have with these interfaces on a Huawei NE800 M8) and an interface that being configured as a single channel of 40G/100G would be designated in the system as 100G 0/3/0, when the "division" is done, this port changes to "ports" and they are identified as GE0/3/0:0 to GE0/3/0:3. That is, the identification has changed and the quantity has also changed. So, I THINK the module function in the interface could be used for that function gaining inheritance features like modules (when we use the macro {module} in the name for the name to fit the position the module was inserted) and, both the type of interface (physical connection and the name or part of it) especially when using the "modular" interface tag. Remembering that the type of physical connection and the number of these connections is determined by the modules and the name of the interface may vary according to the configuration.

Attached are two pictures showing 2 connections of 40G/100G, one with a 40G-40G cable, the other with a 40G-4*10G cable. In the other photo we can see several 10G connections. Some with SFP+ simplex modules (1 fiber and using WDM), other modules are duplex (2 fibers being one Tx and one Rx) and also a DAC cable.