diff options
Diffstat (limited to 'Documentation/networking')
18 files changed, 1500 insertions, 396 deletions
diff --git a/Documentation/networking/can.rst b/Documentation/networking/can.rst index f8dae662e454..f34cb0e4460e 100644 --- a/Documentation/networking/can.rst +++ b/Documentation/networking/can.rst @@ -608,6 +608,8 @@ demand: setsockopt(s, SOL_CAN_RAW, CAN_RAW_RECV_OWN_MSGS, &recv_own_msgs, sizeof(recv_own_msgs)); +Note that reception of a socket's own CAN frames are subject to the same +filtering as other CAN frames (see :ref:`socketcan-rawfilter`). .. _socketcan-rawfd: diff --git a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst index 1b7e32d8a61b..936a10f1942c 100644 --- a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst +++ b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst @@ -183,6 +183,40 @@ User command examples: values: cmode driverinit value true +esw_port_metadata: Eswitch port metadata state +---------------------------------------------- +When applicable, disabling Eswitch metadata can increase packet rate +up to 20% depending on the use case and packet sizes. + +Eswitch port metadata state controls whether to internally tag packets with +metadata. Metadata tagging must be enabled for multi-port RoCE, failover +between representors and stacked devices. +By default metadata is enabled on the supported devices in E-switch. +Metadata is applicable only for E-switch in switchdev mode and +users may disable it when NONE of the below use cases will be in use: +1. HCA is in Dual/multi-port RoCE mode. +2. VF/SF representor bonding (Usually used for Live migration) +3. Stacked devices + +When metadata is disabled, the above use cases will fail to initialize if +users try to enable them. + +- Show eswitch port metadata:: + + $ devlink dev param show pci/0000:06:00.0 name esw_port_metadata + pci/0000:06:00.0: + name esw_port_metadata type driver-specific + values: + cmode runtime value true + +- Disable eswitch port metadata:: + + $ devlink dev param set pci/0000:06:00.0 name esw_port_metadata value false cmode runtime + +- Change eswitch mode to switchdev mode where after choosing the metadata value:: + + $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev + mlx5 subfunction ================ mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface. diff --git a/Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst b/Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst index c3f51c672a68..fc5acd427a5d 100644 --- a/Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst +++ b/Documentation/networking/device_drivers/ethernet/microsoft/netvsc.rst @@ -87,11 +87,15 @@ Receive Buffer contain one or more packets. The number of receive sections may be changed via ethtool Rx ring parameters. - There is a similar send buffer which is used to aggregate packets for sending. - The send area is broken into chunks of 6144 bytes, each of section may - contain one or more packets. The send buffer is an optimization, the driver - will use slower method to handle very large packets or if the send buffer - area is exhausted. + There is a similar send buffer which is used to aggregate packets + for sending. The send area is broken into chunks, typically of 6144 + bytes, each of section may contain one or more packets. Small + packets are usually transmitted via copy to the send buffer. However, + if the buffer is temporarily exhausted, or the packet to be transmitted is + an LSO packet, the driver will provide the host with pointers to the data + from the SKB. This attempts to achieve a balance between the overhead of + data copy and the impact of remapping VM memory to be accessible by the + host. XDP support ----------- diff --git a/Documentation/networking/device_drivers/fddi/defza.rst b/Documentation/networking/device_drivers/fddi/defza.rst index 73c2f793ea26..7393f33ea705 100644 --- a/Documentation/networking/device_drivers/fddi/defza.rst +++ b/Documentation/networking/device_drivers/fddi/defza.rst @@ -60,4 +60,4 @@ To do: Both success and failure reports are welcome. -Maciej W. Rozycki <macro@linux-mips.org> +Maciej W. Rozycki <macro@orcam.me.uk> diff --git a/Documentation/networking/devlink/devlink-health.rst b/Documentation/networking/devlink/devlink-health.rst index 0c99b11f05f9..e37f77734b5b 100644 --- a/Documentation/networking/devlink/devlink-health.rst +++ b/Documentation/networking/devlink/devlink-health.rst @@ -24,7 +24,7 @@ attributes of the health reporting and recovery procedures. The ``devlink`` health reporter: Device driver creates a "health reporter" per each error/health type. -Error/Health type can be a known/generic (eg pci error, fw error, rx/tx error) +Error/Health type can be a known/generic (e.g. PCI error, fw error, rx/tx error) or unknown (driver specific). For each registered health reporter a driver can issue error/health reports asynchronously. All health reports handling is done by ``devlink``. @@ -48,6 +48,7 @@ Once an error is reported, devlink health will perform the following actions: * Object dump is being taken and saved at the reporter instance (as long as there is no other dump which is already stored) * Auto recovery attempt is being done. Depends on: + - Auto-recovery configuration - Grace period vs. time passed since last recover @@ -72,14 +73,18 @@ via ``devlink``, e.g per error type (per health reporter): * - ``DEVLINK_CMD_HEALTH_REPORTER_SET`` - Allows reporter-related configuration setting. * - ``DEVLINK_CMD_HEALTH_REPORTER_RECOVER`` - - Triggers a reporter's recovery procedure. + - Triggers reporter's recovery procedure. + * - ``DEVLINK_CMD_HEALTH_REPORTER_TEST`` + - Triggers a fake health event on the reporter. The effects of the test + event in terms of recovery flow should follow closely that of a real + event. * - ``DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE`` - - Retrieves diagnostics data from a reporter on a device. + - Retrieves current device state related to the reporter. * - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_GET`` - Retrieves the last stored dump. Devlink health - saves a single dump. If an dump is not already stored by the devlink + saves a single dump. If an dump is not already stored by devlink for this reporter, devlink generates a new dump. - dump output is defined by the reporter. + Dump output is defined by the reporter. * - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_CLEAR`` - Clears the last saved dump file for the specified reporter. @@ -93,7 +98,7 @@ The following diagram provides a general overview of ``devlink-health``:: +--------------------------+ |request for ops |(diagnose, - mlx5_core devlink |recover, + driver devlink |recover, |dump) +--------+ +--------------------------+ | | | reporter| | diff --git a/Documentation/networking/dsa/configuration.rst b/Documentation/networking/dsa/configuration.rst index 11bd5e6108c0..774f0e76c746 100644 --- a/Documentation/networking/dsa/configuration.rst +++ b/Documentation/networking/dsa/configuration.rst @@ -34,8 +34,15 @@ interface. The CPU port is the switch port connected to an Ethernet MAC chip. The corresponding linux Ethernet interface is called the master interface. All other corresponding linux interfaces are called slave interfaces. -The slave interfaces depend on the master interface. They can only brought up, -when the master interface is up. +The slave interfaces depend on the master interface being up in order for them +to send or receive traffic. Prior to kernel v5.12, the state of the master +interface had to be managed explicitly by the user. Starting with kernel v5.12, +the behavior is as follows: + +- when a DSA slave interface is brought up, the master interface is + automatically brought up. +- when the master interface is brought down, all DSA slave interfaces are + automatically brought down. In this documentation the following Ethernet interfaces are used: @@ -78,79 +85,76 @@ The tagging based configuration is desired and supported by the majority of DSA switches. These switches are capable to tag incoming and outgoing traffic without using a VLAN based configuration. -single port -~~~~~~~~~~~ - -.. code-block:: sh - - # configure each interface - ip addr add 192.0.2.1/30 dev lan1 - ip addr add 192.0.2.5/30 dev lan2 - ip addr add 192.0.2.9/30 dev lan3 - - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up +*single port* + .. code-block:: sh - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up + # configure each interface + ip addr add 192.0.2.1/30 dev lan1 + ip addr add 192.0.2.5/30 dev lan2 + ip addr add 192.0.2.9/30 dev lan3 -bridge -~~~~~~ + # For kernels earlier than v5.12, the master interface needs to be + # brought up manually before the slave ports. + ip link set eth0 up -.. code-block:: sh + # bring up the slave interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up +*bridge* + .. code-block:: sh - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up + # For kernels earlier than v5.12, the master interface needs to be + # brought up manually before the slave ports. + ip link set eth0 up - # create bridge - ip link add name br0 type bridge + # bring up the slave interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # add ports to bridge - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 - ip link set dev lan3 master br0 + # create bridge + ip link add name br0 type bridge - # configure the bridge - ip addr add 192.0.2.129/25 dev br0 + # add ports to bridge + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + ip link set dev lan3 master br0 - # bring up the bridge - ip link set dev br0 up + # configure the bridge + ip addr add 192.0.2.129/25 dev br0 -gateway -~~~~~~~ + # bring up the bridge + ip link set dev br0 up -.. code-block:: sh +*gateway* + .. code-block:: sh - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up + # For kernels earlier than v5.12, the master interface needs to be + # brought up manually before the slave ports. + ip link set eth0 up - # bring up the slave interfaces - ip link set wan up - ip link set lan1 up - ip link set lan2 up + # bring up the slave interfaces + ip link set wan up + ip link set lan1 up + ip link set lan2 up - # configure the upstream port - ip addr add 192.0.2.1/30 dev wan + # configure the upstream port + ip addr add 192.0.2.1/30 dev wan - # create bridge - ip link add name br0 type bridge + # create bridge + ip link add name br0 type bridge - # add ports to bridge - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 + # add ports to bridge + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 - # configure the bridge - ip addr add 192.0.2.129/25 dev br0 + # configure the bridge + ip addr add 192.0.2.129/25 dev br0 - # bring up the bridge - ip link set dev br0 up + # bring up the bridge + ip link set dev br0 up .. _dsa-vlan-configuration: @@ -161,132 +165,130 @@ A minority of switches are not capable to use a taging protocol (DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based configuration. -single port -~~~~~~~~~~~ -The configuration can only be set up via VLAN tagging and bridge setup. - -.. code-block:: sh - - # tag traffic on CPU port - ip link add link eth0 name eth0.1 type vlan id 1 - ip link add link eth0 name eth0.2 type vlan id 2 - ip link add link eth0 name eth0.3 type vlan id 3 - - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up - ip link set eth0.1 up - ip link set eth0.2 up - ip link set eth0.3 up - - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up - - # create bridge - ip link add name br0 type bridge - - # activate VLAN filtering - ip link set dev br0 type bridge vlan_filtering 1 - - # add ports to bridges - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 - ip link set dev lan3 master br0 - - # tag traffic on ports - bridge vlan add dev lan1 vid 1 pvid untagged - bridge vlan add dev lan2 vid 2 pvid untagged - bridge vlan add dev lan3 vid 3 pvid untagged - - # configure the VLANs - ip addr add 192.0.2.1/30 dev eth0.1 - ip addr add 192.0.2.5/30 dev eth0.2 - ip addr add 192.0.2.9/30 dev eth0.3 - - # bring up the bridge devices - ip link set br0 up - +*single port* + The configuration can only be set up via VLAN tagging and bridge setup. -bridge -~~~~~~ + .. code-block:: sh -.. code-block:: sh + # tag traffic on CPU port + ip link add link eth0 name eth0.1 type vlan id 1 + ip link add link eth0 name eth0.2 type vlan id 2 + ip link add link eth0 name eth0.3 type vlan id 3 - # tag traffic on CPU port - ip link add link eth0 name eth0.1 type vlan id 1 + # For kernels earlier than v5.12, the master interface needs to be + # brought up manually before the slave ports. + ip link set eth0 up + ip link set eth0.1 up + ip link set eth0.2 up + ip link set eth0.3 up - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up - ip link set eth0.1 up + # bring up the slave interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # bring up the slave interfaces - ip link set lan1 up - ip link set lan2 up - ip link set lan3 up + # create bridge + ip link add name br0 type bridge - # create bridge - ip link add name br0 type bridge + # activate VLAN filtering + ip link set dev br0 type bridge vlan_filtering 1 - # activate VLAN filtering - ip link set dev br0 type bridge vlan_filtering 1 + # add ports to bridges + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + ip link set dev lan3 master br0 - # add ports to bridge - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 - ip link set dev lan3 master br0 - ip link set eth0.1 master br0 + # tag traffic on ports + bridge vlan add dev lan1 vid 1 pvid untagged + bridge vlan add dev lan2 vid 2 pvid untagged + bridge vlan add dev lan3 vid 3 pvid untagged - # tag traffic on ports - bridge vlan add dev lan1 vid 1 pvid untagged - bridge vlan add dev lan2 vid 1 pvid untagged - bridge vlan add dev lan3 vid 1 pvid untagged + # configure the VLANs + ip addr add 192.0.2.1/30 dev eth0.1 + ip addr add 192.0.2.5/30 dev eth0.2 + ip addr add 192.0.2.9/30 dev eth0.3 - # configure the bridge - ip addr add 192.0.2.129/25 dev br0 + # bring up the bridge devices + ip link set br0 up - # bring up the bridge - ip link set dev br0 up -gateway -~~~~~~~ +*bridge* + .. code-block:: sh -.. code-block:: sh + # tag traffic on CPU port + ip link add link eth0 name eth0.1 type vlan id 1 - # tag traffic on CPU port - ip link add link eth0 name eth0.1 type vlan id 1 - ip link add link eth0 name eth0.2 type vlan id 2 + # For kernels earlier than v5.12, the master interface needs to be + # brought up manually before the slave ports. + ip link set eth0 up + ip link set eth0.1 up - # The master interface needs to be brought up before the slave ports. - ip link set eth0 up - ip link set eth0.1 up - ip link set eth0.2 up + # bring up the slave interfaces + ip link set lan1 up + ip link set lan2 up + ip link set lan3 up - # bring up the slave interfaces - ip link set wan up - ip link set lan1 up - ip link set lan2 up + # create bridge + ip link add name br0 type bridge - # create bridge - ip link add name br0 type bridge + # activate VLAN filtering + ip link set dev br0 type bridge vlan_filtering 1 - # activate VLAN filtering - ip link set dev br0 type bridge vlan_filtering 1 + # add ports to bridge + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + ip link set dev lan3 master br0 + ip link set eth0.1 master br0 - # add ports to bridges - ip link set dev wan master br0 - ip link set eth0.1 master br0 - ip link set dev lan1 master br0 - ip link set dev lan2 master br0 + # tag traffic on ports + bridge vlan add dev lan1 vid 1 pvid untagged + bridge vlan add dev lan2 vid 1 pvid untagged + bridge vlan add dev lan3 vid 1 pvid untagged - # tag traffic on ports - bridge vlan add dev lan1 vid 1 pvid untagged - bridge vlan add dev lan2 vid 1 pvid untagged - bridge vlan add dev wan vid 2 pvid untagged + # configure the bridge + ip addr add 192.0.2.129/25 dev br0 - # configure the VLANs - ip addr add 192.0.2.1/30 dev eth0.2 - ip addr add 192.0.2.129/25 dev br0 + # bring up the bridge + ip link set dev br0 up - # bring up the bridge devices - ip link set br0 up +*gateway* + .. code-block:: sh + + # tag traffic on CPU port + ip link add link eth0 name eth0.1 type vlan id 1 + ip link add link eth0 name eth0.2 type vlan id 2 + + # For kernels earlier than v5.12, the master interface needs to be + # brought up manually before the slave ports. + ip link set eth0 up + ip link set eth0.1 up + ip link set eth0.2 up + + # bring up the slave interfaces + ip link set wan up + ip link set lan1 up + ip link set lan2 up + + # create bridge + ip link add name br0 type bridge + + # activate VLAN filtering + ip link set dev br0 type bridge vlan_filtering 1 + + # add ports to bridges + ip link set dev wan master br0 + ip link set eth0.1 master br0 + ip link set dev lan1 master br0 + ip link set dev lan2 master br0 + + # tag traffic on ports + bridge vlan add dev lan1 vid 1 pvid untagged + bridge vlan add dev lan2 vid 1 pvid untagged + bridge vlan add dev wan vid 2 pvid untagged + + # configure the VLANs + ip addr add 192.0.2.1/30 dev eth0.2 + ip addr add 192.0.2.129/25 dev br0 + + # bring up the bridge devices + ip link set br0 up diff --git a/Documentation/networking/dsa/dsa.rst b/Documentation/networking/dsa/dsa.rst index e9517af5fe02..8688009514cc 100644 --- a/Documentation/networking/dsa/dsa.rst +++ b/Documentation/networking/dsa/dsa.rst @@ -65,14 +65,8 @@ Note that DSA does not currently create network interfaces for the "cpu" and Switch tagging protocols ------------------------ -DSA currently supports 5 different tagging protocols, and a tag-less mode as -well. The different protocols are implemented in: - -- ``net/dsa/tag_trailer.c``: Marvell's 4 trailer tag mode (legacy) -- ``net/dsa/tag_dsa.c``: Marvell's original DSA tag -- ``net/dsa/tag_edsa.c``: Marvell's enhanced DSA tag -- ``net/dsa/tag_brcm.c``: Broadcom's 4 bytes tag -- ``net/dsa/tag_qca.c``: Qualcomm's 2 bytes tag +DSA supports many vendor-specific tagging protocols, one software-defined +tagging protocol, and a tag-less mode as well (``DSA_TAG_PROTO_NONE``). The exact format of the tag protocol is vendor specific, but in general, they all contain something which: @@ -80,6 +74,144 @@ all contain something which: - identifies which port the Ethernet frame came from/should be sent to - provides a reason why this frame was forwarded to the management interface +All tagging protocols are in ``net/dsa/tag_*.c`` files and implement the +methods of the ``struct dsa_device_ops`` structure, which are detailed below. + +Tagging protocols generally fall in one of three categories: + +1. The switch-specific frame header is located before the Ethernet header, + shifting to the right (from the perspective of the DSA master's frame + parser) the MAC DA, MAC SA, EtherType and the entire L2 payload. +2. The switch-specific frame header is located before the EtherType, keeping + the MAC DA and MAC SA in place from the DSA master's perspective, but + shifting the 'real' EtherType and L2 payload to the right. +3. The switch-specific frame header is located at the tail of the packet, + keeping all frame headers in place and not altering the view of the packet + that the DSA master's frame parser has. + +A tagging protocol may tag all packets with switch tags of the same length, or +the tag length might vary (for example packets with PTP timestamps might +require an extended switch tag, or there might be one tag length on TX and a +different one on RX). Either way, the tagging protocol driver must populate the +``struct dsa_device_ops::overhead`` with the length in octets of the longest +switch frame header. The DSA framework will automatically adjust the MTU of the +master interface to accomodate for this extra size in order for DSA user ports +to support the standard MTU (L2 payload length) of 1500 octets. The ``overhead`` +is also used to request from the network stack, on a best-effort basis, the +allocation of packets with a ``needed_headroom`` or ``needed_tailroom`` +sufficient such that the act of pushing the switch tag on transmission of a +packet does not cause it to reallocate due to lack of memory. + +Even though applications are not expected to parse DSA-specific frame headers, +the format on the wire of the tagging protocol represents an Application Binary +Interface exposed by the kernel towards user space, for decoders such as +``libpcap``. The tagging protocol driver must populate the ``proto`` member of +``struct dsa_device_ops`` with a value that uniquely describes the +characteristics of the interaction required between the switch hardware and the +data path driver: the offset of each bit field within the frame header and any +stateful processing required to deal with the frames (as may be required for +PTP timestamping). + +From the perspective of the network stack, all switches within the same DSA +switch tree use the same tagging protocol. In case of a packet transiting a +fabric with more than one switch, the switch-specific frame header is inserted +by the first switch in the fabric that the packet was received on. This header +typically contains information regarding its type (whether it is a control +frame that must be trapped to the CPU, or a data frame to be forwarded). +Control frames should be decapsulated only by the software data path, whereas +data frames might also be autonomously forwarded towards other user ports of +other switches from the same fabric, and in this case, the outermost switch +ports must decapsulate the packet. + +Note that in certain cases, it might be the case that the tagging format used +by a leaf switch (not connected directly to the CPU) to not be the same as what +the network stack sees. This can be seen with Marvell switch trees, where the +CPU port can be configured to use either the DSA or the Ethertype DSA (EDSA) +format, but the DSA links are configured to use the shorter (without Ethertype) +DSA frame header, in order to reduce the autonomous packet forwarding overhead. +It still remains the case that, if the DSA switch tree is configured for the +EDSA tagging protocol, the operating system sees EDSA-tagged packets from the +leaf switches that tagged them with the shorter DSA header. This can be done +because the Marvell switch connected directly to the CPU is configured to +perform tag translation between DSA and EDSA (which is simply the operation of +adding or removing the ``ETH_P_EDSA`` EtherType and some padding octets). + +It is possible to construct cascaded setups of DSA switches even if their +tagging protocols are not compatible with one another. In this case, there are +no DSA links in this fabric, and each switch constitutes a disjoint DSA switch +tree. The DSA links are viewed as simply a pair of a DSA master (the out-facing +port of the upstream DSA switch) and a CPU port (the in-facing port of the +downstream DSA switch). + +The tagging protocol of the attached DSA switch tree can be viewed through the +``dsa/tagging`` sysfs attribute of the DSA master:: + + cat /sys/class/net/eth0/dsa/tagging + +If the hardware and driver are capable, the tagging protocol of the DSA switch +tree can be changed at runtime. This is done by writing the new tagging +protocol name to the same sysfs device attribute as above (the DSA master and +all attached switch ports must be down while doing this). + +It is desirable that all tagging protocols are testable with the ``dsa_loop`` +mockup driver, which can be attached to any network interface. The goal is that +any network interface should be capable of transmitting the same packet in the +same way, and the tagger should decode the same received packet in the same way +regardless of the driver used for the switch control path, and the driver used +for the DSA master. + +The transmission of a packet goes through the tagger's ``xmit`` function. +The passed ``struct sk_buff *skb`` has ``skb->data`` pointing at +``skb_mac_header(skb)``, i.e. at the destination MAC address, and the passed +``struct net_device *dev`` represents the virtual DSA user network interface +whose hardware counterpart the packet must be steered to (i.e. ``swp0``). +The job of this method is to prepare the skb in a way that the switch will +understand what egress port the packet is for (and not deliver it towards other +ports). Typically this is fulfilled by pushing a frame header. Checking for +insufficient size in the skb headroom or tailroom is unnecessary provided that +the ``overhead`` and ``tail_tag`` properties were filled out properly, because +DSA ensures there is enough space before calling this method. + +The reception of a packet goes through the tagger's ``rcv`` function. The +passed ``struct sk_buff *skb`` has ``skb->data`` pointing at +``skb_mac_header(skb) + ETH_ALEN`` octets, i.e. to where the first octet after +the EtherType would have been, were this frame not tagged. The role of this +method is to consume the frame header, adjust ``skb->data`` to really point at +the first octet after the EtherType, and to change ``skb->dev`` to point to the +virtual DSA user network interface corresponding to the physical front-facing +switch port that the packet was received on. + +Since tagging protocols in category 1 and 2 break software (and most often also +hardware) packet dissection on the DSA master, features such as RPS (Receive +Packet Steering) on the DSA master would be broken. The DSA framework deals +with this by hooking into the flow dissector and shifting the offset at which +the IP header is to be found in the tagged frame as seen by the DSA master. +This behavior is automatic based on the ``overhead`` value of the tagging +protocol. If not all packets are of equal size, the tagger can implement the +``flow_dissect`` method of the ``struct dsa_device_ops`` and override this +default behavior by specifying the correct offset incurred by each individual +RX packet. Tail taggers do not cause issues to the flow dissector. + +Due to various reasons (most common being category 1 taggers being associated +with DSA-unaware masters, mangling what the master perceives as MAC DA), the +tagging protocol may require the DSA master to operate in promiscuous mode, to +receive all frames regardless of the value of the MAC DA. This can be done by +setting the ``promisc_on_master`` property of the ``struct dsa_device_ops``. +Note that this assumes a DSA-unaware master driver, which is the norm. + +Hardware manufacturers are strongly discouraged to do this, but some tagging +protocols might not provide source port information on RX for all packets, but +e.g. only for control traffic (link-local PDUs). In this case, by implementing +the ``filter`` method of ``struct dsa_device_ops``, the tagger might select +which packets are to be redirected on RX towards the virtual DSA user network +interfaces, and which are to be left in the DSA master's RX data path. + +It might also happen (although silicon vendors are strongly discouraged to +produce hardware like this) that a tagging protocol splits the switch-specific +information into a header portion and a tail portion, therefore not falling +cleanly into any of the above 3 categories. DSA does not support this +configuration. + Master network devices ---------------------- @@ -172,23 +304,34 @@ Graphical representation Summarized, this is basically how DSA looks like from a network device perspective:: - - |--------------------------- - | CPU network device (eth0)| - ---------------------------- - | <tag added by switch | - | | - | | - | tag added by CPU> | - |--------------------------------------------| - | Switch driver | - |--------------------------------------------| - || || || - |-------| |-------| |-------| - | sw0p0 | | sw0p1 | | sw0p2 | - |-------| |-------| |-------| - - + Unaware application + opens and binds socket + | ^ + | | + +-----------v--|--------------------+ + |+------+ +------+ +------+ +------+| + || swp0 | | swp1 | | swp2 | | swp3 || + |+------+-+------+-+------+-+------+| + | DSA switch driver | + +-----------------------------------+ + | ^ + Tag added by | | Tag consumed by + switch driver | | switch driver + v | + +-----------------------------------+ + | Unmodified host interface driver | Software + --------+-----------------------------------+------------ + | Host interface (eth0) | Hardware + +-----------------------------------+ + | ^ + Tag consumed by | | Tag added by + switch hardware | | switch hardware + v | + +-----------------------------------+ + | Switch | + |+------+ +------+ +------+ +------+| + || swp0 | | swp1 | | swp2 | | swp3 || + ++------+-+------+-+------+-+------++ Slave MDIO bus -------------- @@ -239,14 +382,6 @@ DSA data structures are defined in ``include/net/dsa.h`` as well as Design limitations ================== -Limits on the number of devices and ports ------------------------------------------ - -DSA currently limits the number of maximum switches within a tree to 4 -(``DSA_MAX_SWITCHES``), and the number of ports per switch to 12 (``DSA_MAX_PORTS``). -These limits could be extended to support larger configurations would this need -arise. - Lack of CPU/DSA network devices ------------------------------- @@ -281,6 +416,7 @@ DSA currently leverages the following subsystems: - MDIO/PHY library: ``drivers/net/phy/phy.c``, ``mdio_bus.c`` - Switchdev:``net/switchdev/*`` - Device Tree for various of_* functions +- Devlink: ``net/core/devlink.c`` MDIO/PHY library ---------------- @@ -317,14 +453,39 @@ SWITCHDEV DSA directly utilizes SWITCHDEV when interfacing with the bridge layer, and more specifically with its VLAN filtering portion when configuring VLANs on top -of per-port slave network devices. Since DSA primarily deals with -MDIO-connected switches, although not exclusively, SWITCHDEV's -prepare/abort/commit phases are often simplified into a prepare phase which -checks whether the operation is supported by the DSA switch driver, and a commit -phase which applies the changes. - -As of today, the only SWITCHDEV objects supported by DSA are the FDB and VLAN -objects. +of per-port slave network devices. As of today, the only SWITCHDEV objects +supported by DSA are the FDB and VLAN objects. + +Devlink +------- + +DSA registers one devlink device per physical switch in the fabric. +For each devlink device, every physical port (i.e. user ports, CPU ports, DSA +links or unused ports) is exposed as a devlink port. + +DSA drivers can make use of the following devlink features: + +- Regions: debugging feature which allows user space to dump driver-defined + areas of hardware information in a low-level, binary format. Both global + regions as well as per-port regions are supported. It is possible to export + devlink regions even for pieces of data that are already exposed in some way + to the standard iproute2 user space programs (ip-link, bridge), like address + tables and VLAN tables. For example, this might be useful if the tables + contain additional hardware-specific details which are not visible through + the iproute2 abstraction, or it might be useful to inspect these tables on + the non-user ports too, which are invisible to iproute2 because no network + interface is registered for them. +- Params: a feature which enables user to configure certain low-level tunable + knobs pertaining to the device. Drivers may implement applicable generic + devlink params, or may add new device-specific devlink params. +- Resources: a monitoring feature which enables users to see the degree of + utilization of certain hardware tables in the device, such as FDB, VLAN, etc. +- Shared buffers: a QoS feature for adjusting and partitioning memory and frame + reservations per port and per traffic class, in the ingress and egress + directions, such that low-priority bulk traffic does not impede the + processing of high-priority critical traffic. + +For more details, consult ``Documentation/networking/devlink/``. Device Tree ----------- @@ -490,6 +651,17 @@ Bridge layer computing a STP state change based on current and asked parameters and perform the relevant ageing based on the intersection results +- ``port_bridge_flags``: bridge layer function invoked when a port must + configure its settings for e.g. flooding of unknown traffic or source address + learning. The switch driver is responsible for initial setup of the + standalone ports with address learning disabled and egress flooding of all + types of traffic, then the DSA core notifies of any change to the bridge port + flags when the port joins and leaves a bridge. DSA does not currently manage + the bridge port flags for the CPU port. The assumption is that address + learning should be statically enabled (if supported by the hardware) on the + CPU port, and flooding towards the CPU port should also be enabled, due to a + lack of an explicit address filtering mechanism in the DSA core. + Bridge VLAN filtering --------------------- @@ -503,14 +675,10 @@ Bridge VLAN filtering accept any 802.1Q frames irrespective of their VLAN ID, and untagged frames are allowed. -- ``port_vlan_prepare``: bridge layer function invoked when the bridge prepares the - configuration of a VLAN on the given port. If the operation is not supported - by the hardware, this function should return ``-EOPNOTSUPP`` to inform the bridge - code to fallback to a software implementation. No hardware setup must be done - in this function. See port_vlan_add for this and details. - - ``port_vlan_add``: bridge layer function invoked when a VLAN is configured - (tagged or untagged) for the given switch port + (tagged or untagged) for the given switch port. If the operation is not + supported by the hardware, this function should return ``-EOPNOTSUPP`` to + inform the bridge code to fallback to a software implementation. - ``port_vlan_del``: bridge layer function invoked when a VLAN is removed from the given switch port @@ -538,14 +706,10 @@ Bridge VLAN filtering function that the driver has to call for each MAC address known to be behind the given port. A switchdev object is used to carry the VID and FDB info. -- ``port_mdb_prepare``: bridge layer function invoked when the bridge prepares the - installation of a multicast database entry. If the operation is not supported, - this function should return ``-EOPNOTSUPP`` to inform the bridge code to fallback - to a software implementation. No hardware setup must be done in this function. - See ``port_fdb_add`` for this and details. - - ``port_mdb_add``: bridge layer function invoked when the bridge wants to install - a multicast database entry, the switch hardware should be programmed with the + a multicast database entry. If the operation is not supported, this function + should return ``-EOPNOTSUPP`` to inform the bridge code to fallback to a + software implementation. The switch hardware should be programmed with the specified address in the specified VLAN ID in the forwarding database associated with this VLAN ID. @@ -561,6 +725,101 @@ Bridge VLAN filtering function that the driver has to call for each MAC address known to be behind the given port. A switchdev object is used to carry the VID and MDB info. +Link aggregation +---------------- + +Link aggregation is implemented in the Linux networking stack by the bonding +and team drivers, which are modeled as virtual, stackable network interfaces. +DSA is capable of offloading a link aggregation group (LAG) to hardware that +supports the feature, and supports bridging between physical ports and LAGs, +as well as between LAGs. A bonding/team interface which holds multiple physical +ports constitutes a logical port, although DSA has no explicit concept of a +logical port at the moment. Due to this, events where a LAG joins/leaves a +bridge are treated as if all individual physical ports that are members of that +LAG join/leave the bridge. Switchdev port attributes (VLAN filtering, STP +state, etc) and objects (VLANs, MDB entries) offloaded to a LAG as bridge port +are treated similarly: DSA offloads the same switchdev object / port attribute +on all members of the LAG. Static bridge FDB entries on a LAG are not yet +supported, since the DSA driver API does not have the concept of a logical port +ID. + +- ``port_lag_join``: function invoked when a given switch port is added to a + LAG. The driver may return ``-EOPNOTSUPP``, and in this case, DSA will fall + back to a software implementation where all traffic from this port is sent to + the CPU. +- ``port_lag_leave``: function invoked when a given switch port leaves a LAG + and returns to operation as a standalone port. +- ``port_lag_change``: function invoked when the link state of any member of + the LAG changes, and the hashing function needs rebalancing to only make use + of the subset of physical LAG member ports that are up. + +Drivers that benefit from having an ID associated with each offloaded LAG +can optionally populate ``ds->num_lag_ids`` from the ``dsa_switch_ops::setup`` +method. The LAG ID associated with a bonding/team interface can then be +retrieved by a DSA switch driver using the ``dsa_lag_id`` function. + +IEC 62439-2 (MRP) +----------------- + +The Media Redundancy Protocol is a topology management protocol optimized for +fast fault recovery time for ring networks, which has some components +implemented as a function of the bridge driver. MRP uses management PDUs +(Test, Topology, LinkDown/Up, Option) sent at a multicast destination MAC +address range of 01:15:4e:00:00:0x and with an EtherType of 0x88e3. +Depending on the node's role in the ring (MRM: Media Redundancy Manager, +MRC: Media Redundancy Client, MRA: Media Redundancy Automanager), certain MRP +PDUs might need to be terminated locally and others might need to be forwarded. +An MRM might also benefit from offloading to hardware the creation and +transmission of certain MRP PDUs (Test). + +Normally an MRP instance can be created on top of any network interface, +however in the case of a device with an offloaded data path such as DSA, it is +necessary for the hardware, even if it is not MRP-aware, to be able to extract +the MRP PDUs from the fabric before the driver can proceed with the software +implementation. DSA today has no driver which is MRP-aware, therefore it only +listens for the bare minimum switchdev objects required for the software assist +to work properly. The operations are detailed below. + +- ``port_mrp_add`` and ``port_mrp_del``: notifies driver when an MRP instance + with a certain ring ID, priority, primary port and secondary port is + created/deleted. +- ``port_mrp_add_ring_role`` and ``port_mrp_del_ring_role``: function invoked + when an MRP instance changes ring roles between MRM or MRC. This affects + which MRP PDUs should be trapped to software and which should be autonomously + forwarded. + +IEC 62439-3 (HSR/PRP) +--------------------- + +The Parallel Redundancy Protocol (PRP) is a network redundancy protocol which +works by duplicating and sequence numbering packets through two independent L2 +networks (which are unaware of the PRP tail tags carried in the packets), and +eliminating the duplicates at the receiver. The High-availability Seamless +Redundancy (HSR) protocol is similar in concept, except all nodes that carry +the redundant traffic are aware of the fact that it is HSR-tagged (because HSR +uses a header with an EtherType of 0x892f) and are physically connected in a +ring topology. Both HSR and PRP use supervision frames for monitoring the +health of the network and for discovery of other nodes. + +In Linux, both HSR and PRP are implemented in the hsr driver, which +instantiates a virtual, stackable network interface with two member ports. +The driver only implements the basic roles of DANH (Doubly Attached Node +implementing HSR) and DANP (Doubly Attached Node implementing PRP); the roles +of RedBox and QuadBox are not implemented (therefore, bridging a hsr network +interface with a physical switch port does not produce the expected result). + +A driver which is able of offloading certain functions of a DANP or DANH should +declare the corresponding netdev features as indicated by the documentation at +``Documentation/networking/netdev-features.rst``. Additionally, the following +methods must be implemented: + +- ``port_hsr_join``: function invoked when a given switch port is added to a + DANP/DANH. The driver may return ``-EOPNOTSUPP`` and in this case, DSA will + fall back to a software implementation where all traffic from this port is + sent to the CPU. +- ``port_hsr_leave``: function invoked when a given switch port leaves a + DANP/DANH and returns to normal operation as a standalone port. + TODO ==== @@ -576,8 +835,5 @@ two subsystems and get the best of both worlds. Other hanging fruits -------------------- -- making the number of ports fully dynamic and not dependent on ``DSA_MAX_PORTS`` - allowing more than one CPU/management interface: http://comments.gmane.org/gmane.linux.network/365657 -- porting more drivers from other vendors: - http://comments.gmane.org/gmane.linux.network/365510 diff --git a/Documentation/networking/ethtool-netlink.rst b/Documentation/networking/ethtool-netlink.rst index dc03ff884541..25131df3c2bd 100644 --- a/Documentation/networking/ethtool-netlink.rst +++ b/Documentation/networking/ethtool-netlink.rst @@ -208,41 +208,49 @@ Userspace to kernel: ``ETHTOOL_MSG_CABLE_TEST_ACT`` action start cable test ``ETHTOOL_MSG_CABLE_TEST_TDR_ACT`` action start raw TDR cable test ``ETHTOOL_MSG_TUNNEL_INFO_GET`` get tunnel offload info + ``ETHTOOL_MSG_FEC_GET`` get FEC settings + ``ETHTOOL_MSG_FEC_SET`` set FEC settings + ``ETHTOOL_MSG_MODULE_EEPROM_GET`` read SFP module EEPROM + ``ETHTOOL_MSG_STATS_GET`` get standard statistics ===================================== ================================ Kernel to userspace: - ===================================== ================================= - ``ETHTOOL_MSG_STRSET_GET_REPLY`` string set contents - ``ETHTOOL_MSG_LINKINFO_GET_REPLY`` link settings - ``ETHTOOL_MSG_LINKINFO_NTF`` link settings notification - ``ETHTOOL_MSG_LINKMODES_GET_REPLY`` link modes info - ``ETHTOOL_MSG_LINKMODES_NTF`` link modes notification - ``ETHTOOL_MSG_LINKSTATE_GET_REPLY`` link state info - ``ETHTOOL_MSG_DEBUG_GET_REPLY`` debugging settings - ``ETHTOOL_MSG_DEBUG_NTF`` debugging settings notification - ``ETHTOOL_MSG_WOL_GET_REPLY`` wake-on-lan settings - ``ETHTOOL_MSG_WOL_NTF`` wake-on-lan settings notification - ``ETHTOOL_MSG_FEATURES_GET_REPLY`` device features - ``ETHTOOL_MSG_FEATURES_SET_REPLY`` optional reply to FEATURES_SET - ``ETHTOOL_MSG_FEATURES_NTF`` netdev features notification - ``ETHTOOL_MSG_PRIVFLAGS_GET_REPLY`` private flags - ``ETHTOOL_MSG_PRIVFLAGS_NTF`` private flags - ``ETHTOOL_MSG_RINGS_GET_REPLY`` ring sizes - ``ETHTOOL_MSG_RINGS_NTF`` ring sizes - ``ETHTOOL_MSG_CHANNELS_GET_REPLY`` channel counts - ``ETHTOOL_MSG_CHANNELS_NTF`` channel counts - ``ETHTOOL_MSG_COALESCE_GET_REPLY`` coalescing parameters - ``ETHTOOL_MSG_COALESCE_NTF`` coalescing parameters - ``ETHTOOL_MSG_PAUSE_GET_REPLY`` pause parameters - ``ETHTOOL_MSG_PAUSE_NTF`` pause parameters - ``ETHTOOL_MSG_EEE_GET_REPLY`` EEE settings - ``ETHTOOL_MSG_EEE_NTF`` EEE settings - ``ETHTOOL_MSG_TSINFO_GET_REPLY`` timestamping info - ``ETHTOOL_MSG_CABLE_TEST_NTF`` Cable test results - ``ETHTOOL_MSG_CABLE_TEST_TDR_NTF`` Cable test TDR results - ``ETHTOOL_MSG_TUNNEL_INFO_GET_REPLY`` tunnel offload info - ===================================== ================================= + ======================================== ================================= + ``ETHTOOL_MSG_STRSET_GET_REPLY`` string set contents + ``ETHTOOL_MSG_LINKINFO_GET_REPLY`` link settings + ``ETHTOOL_MSG_LINKINFO_NTF`` link settings notification + ``ETHTOOL_MSG_LINKMODES_GET_REPLY`` link modes info + ``ETHTOOL_MSG_LINKMODES_NTF`` link modes notification + ``ETHTOOL_MSG_LINKSTATE_GET_REPLY`` link state info + ``ETHTOOL_MSG_DEBUG_GET_REPLY`` debugging settings + ``ETHTOOL_MSG_DEBUG_NTF`` debugging settings notification + ``ETHTOOL_MSG_WOL_GET_REPLY`` wake-on-lan settings + ``ETHTOOL_MSG_WOL_NTF`` wake-on-lan settings notification + ``ETHTOOL_MSG_FEATURES_GET_REPLY`` device features + ``ETHTOOL_MSG_FEATURES_SET_REPLY`` optional reply to FEATURES_SET + ``ETHTOOL_MSG_FEATURES_NTF`` netdev features notification + ``ETHTOOL_MSG_PRIVFLAGS_GET_REPLY`` private flags + ``ETHTOOL_MSG_PRIVFLAGS_NTF`` private flags + ``ETHTOOL_MSG_RINGS_GET_REPLY`` ring sizes + ``ETHTOOL_MSG_RINGS_NTF`` ring sizes + ``ETHTOOL_MSG_CHANNELS_GET_REPLY`` channel counts + ``ETHTOOL_MSG_CHANNELS_NTF`` channel counts + ``ETHTOOL_MSG_COALESCE_GET_REPLY`` coalescing parameters + ``ETHTOOL_MSG_COALESCE_NTF`` coalescing parameters + ``ETHTOOL_MSG_PAUSE_GET_REPLY`` pause parameters + ``ETHTOOL_MSG_PAUSE_NTF`` pause parameters + ``ETHTOOL_MSG_EEE_GET_REPLY`` EEE settings + ``ETHTOOL_MSG_EEE_NTF`` EEE settings + ``ETHTOOL_MSG_TSINFO_GET_REPLY`` timestamping info + ``ETHTOOL_MSG_CABLE_TEST_NTF`` Cable test results + ``ETHTOOL_MSG_CABLE_TEST_TDR_NTF`` Cable test TDR results + ``ETHTOOL_MSG_TUNNEL_INFO_GET_REPLY`` tunnel offload info + ``ETHTOOL_MSG_FEC_GET_REPLY`` FEC settings + ``ETHTOOL_MSG_FEC_NTF`` FEC settings + ``ETHTOOL_MSG_MODULE_EEPROM_GET_REPLY`` read SFP module EEPROM + ``ETHTOOL_MSG_STATS_GET_REPLY`` standard statistics + ======================================== ================================= ``GET`` requests are sent by userspace applications to retrieve device information. They usually do not contain any message specific attributes. @@ -1280,6 +1288,193 @@ Kernel response contents: For UDP tunnel table empty ``ETHTOOL_A_TUNNEL_UDP_TABLE_TYPES`` indicates that the table contains static entries, hard-coded by the NIC. +FEC_GET +======= + +Gets FEC configuration and state like ``ETHTOOL_GFECPARAM`` ioctl request. + +Request contents: + + ===================================== ====== ========================== + ``ETHTOOL_A_FEC_HEADER`` nested request header + ===================================== ====== ========================== + +Kernel response contents: + + ===================================== ====== ========================== + ``ETHTOOL_A_FEC_HEADER`` nested request header + ``ETHTOOL_A_FEC_MODES`` bitset configured modes + ``ETHTOOL_A_FEC_AUTO`` bool FEC mode auto selection + ``ETHTOOL_A_FEC_ACTIVE`` u32 index of active FEC mode + ``ETHTOOL_A_FEC_STATS`` nested FEC statistics + ===================================== ====== ========================== + +``ETHTOOL_A_FEC_ACTIVE`` is the bit index of the FEC link mode currently +active on the interface. This attribute may not be present if device does +not support FEC. + +``ETHTOOL_A_FEC_MODES`` and ``ETHTOOL_A_FEC_AUTO`` are only meaningful when +autonegotiation is disabled. If ``ETHTOOL_A_FEC_AUTO`` is non-zero driver will +select the FEC mode automatically based on the parameters of the SFP module. +This is equivalent to the ``ETHTOOL_FEC_AUTO`` bit of the ioctl interface. +``ETHTOOL_A_FEC_MODES`` carry the current FEC configuration using link mode +bits (rather than old ``ETHTOOL_FEC_*`` bits). + +``ETHTOOL_A_FEC_STATS`` are reported if ``ETHTOOL_FLAG_STATS`` was set in +``ETHTOOL_A_HEADER_FLAGS``. +Each attribute carries an array of 64bit statistics. First entry in the array +contains the total number of events on the port, while the following entries +are counters corresponding to lanes/PCS instances. The number of entries in +the array will be: + ++--------------+---------------------------------------------+ +| `0` | device does not support FEC statistics | ++--------------+---------------------------------------------+ +| `1` | device does not support per-lane break down | ++--------------+---------------------------------------------+ +| `1 + #lanes` | device has full support for FEC stats | ++--------------+---------------------------------------------+ + +Drivers fill in the statistics in the following structure: + +.. kernel-doc:: include/linux/ethtool.h + :identifiers: ethtool_fec_stats + +FEC_SET +======= + +Sets FEC parameters like ``ETHTOOL_SFECPARAM`` ioctl request. + +Request contents: + + ===================================== ====== ========================== + ``ETHTOOL_A_FEC_HEADER`` nested request header + ``ETHTOOL_A_FEC_MODES`` bitset configured modes + ``ETHTOOL_A_FEC_AUTO`` bool FEC mode auto selection + ===================================== ====== ========================== + +``FEC_SET`` is only meaningful when autonegotiation is disabled. Otherwise +FEC mode is selected as part of autonegotiation. + +``ETHTOOL_A_FEC_MODES`` selects which FEC mode should be used. It's recommended +to set only one bit, if multiple bits are set driver may choose between them +in an implementation specific way. + +``ETHTOOL_A_FEC_AUTO`` requests the driver to choose FEC mode based on SFP +module parameters. This does not mean autonegotiation. + +MODULE_EEPROM +============= + +Fetch module EEPROM data dump. +This interface is designed to allow dumps of at most 1/2 page at once. This +means only dumps of 128 (or less) bytes are allowed, without crossing half page +boundary located at offset 128. For pages other than 0 only high 128 bytes are +accessible. + +Request contents: + + ======================================= ====== ========================== + ``ETHTOOL_A_MODULE_EEPROM_HEADER`` nested request header + ``ETHTOOL_A_MODULE_EEPROM_OFFSET`` u32 offset within a page + ``ETHTOOL_A_MODULE_EEPROM_LENGTH`` u32 amount of bytes to read + ``ETHTOOL_A_MODULE_EEPROM_PAGE`` u8 page number + ``ETHTOOL_A_MODULE_EEPROM_BANK`` u8 bank number + ``ETHTOOL_A_MODULE_EEPROM_I2C_ADDRESS`` u8 page I2C address + ======================================= ====== ========================== + +Kernel response contents: + + +---------------------------------------------+--------+---------------------+ + | ``ETHTOOL_A_MODULE_EEPROM_HEADER`` | nested | reply header | + +---------------------------------------------+--------+---------------------+ + | ``ETHTOOL_A_MODULE_EEPROM_DATA`` | nested | array of bytes from | + | | | module EEPROM | + +---------------------------------------------+--------+---------------------+ + +``ETHTOOL_A_MODULE_EEPROM_DATA`` has an attribute length equal to the amount of +bytes driver actually read. + +STATS_GET +========= + +Get standard statistics for the interface. Note that this is not +a re-implementation of ``ETHTOOL_GSTATS`` which exposed driver-defined +stats. + +Request contents: + + ======================================= ====== ========================== + ``ETHTOOL_A_STATS_HEADER`` nested request header + ``ETHTOOL_A_STATS_GROUPS`` bitset requested groups of stats + ======================================= ====== ========================== + +Kernel response contents: + + +-----------------------------------+--------+--------------------------------+ + | ``ETHTOOL_A_STATS_HEADER`` | nested | reply header | + +-----------------------------------+--------+--------------------------------+ + | ``ETHTOOL_A_STATS_GRP`` | nested | one or more group of stats | + +-+---------------------------------+--------+--------------------------------+ + | | ``ETHTOOL_A_STATS_GRP_ID`` | u32 | group ID - ``ETHTOOL_STATS_*`` | + +-+---------------------------------+--------+--------------------------------+ + | | ``ETHTOOL_A_STATS_GRP_SS_ID`` | u32 | string set ID for names | + +-+---------------------------------+--------+--------------------------------+ + | | ``ETHTOOL_A_STATS_GRP_STAT`` | nested | nest containing a statistic | + +-+---------------------------------+--------+--------------------------------+ + | | ``ETHTOOL_A_STATS_GRP_HIST_RX`` | nested | histogram statistic (Rx) | + +-+---------------------------------+--------+--------------------------------+ + | | ``ETHTOOL_A_STATS_GRP_HIST_TX`` | nested | histogram statistic (Tx) | + +-+---------------------------------+--------+--------------------------------+ + +Users specify which groups of statistics they are requesting via +the ``ETHTOOL_A_STATS_GROUPS`` bitset. Currently defined values are: + + ====================== ======== =============================================== + ETHTOOL_STATS_ETH_MAC eth-mac Basic IEEE 802.3 MAC statistics (30.3.1.1.*) + ETHTOOL_STATS_ETH_PHY eth-phy Basic IEEE 802.3 PHY statistics (30.3.2.1.*) + ETHTOOL_STATS_ETH_CTRL eth-ctrl Basic IEEE 802.3 MAC Ctrl statistics (30.3.3.*) + ETHTOOL_STATS_RMON rmon RMON (RFC 2819) statistics + ====================== ======== =============================================== + +Each group should have a corresponding ``ETHTOOL_A_STATS_GRP`` in the reply. +``ETHTOOL_A_STATS_GRP_ID`` identifies which group's statistics nest contains. +``ETHTOOL_A_STATS_GRP_SS_ID`` identifies the string set ID for the names of +the statistics in the group, if available. + +Statistics are added to the ``ETHTOOL_A_STATS_GRP`` nest under +``ETHTOOL_A_STATS_GRP_STAT``. ``ETHTOOL_A_STATS_GRP_STAT`` should contain +single 8 byte (u64) attribute inside - the type of that attribute is +the statistic ID and the value is the value of the statistic. +Each group has its own interpretation of statistic IDs. +Attribute IDs correspond to strings from the string set identified +by ``ETHTOOL_A_STATS_GRP_SS_ID``. Complex statistics (such as RMON histogram +entries) are also listed inside ``ETHTOOL_A_STATS_GRP`` and do not have +a string defined in the string set. + +RMON "histogram" counters count number of packets within given size range. +Because RFC does not specify the ranges beyond the standard 1518 MTU devices +differ in definition of buckets. For this reason the definition of packet ranges +is left to each driver. + +``ETHTOOL_A_STATS_GRP_HIST_RX`` and ``ETHTOOL_A_STATS_GRP_HIST_TX`` nests +contain the following attributes: + + ================================= ====== =================================== + ETHTOOL_A_STATS_RMON_HIST_BKT_LOW u32 low bound of the packet size bucket + ETHTOOL_A_STATS_RMON_HIST_BKT_HI u32 high bound of the bucket + ETHTOOL_A_STATS_RMON_HIST_VAL u64 packet counter + ================================= ====== =================================== + +Low and high bounds are inclusive, for example: + + ============================= ==== ==== + RFC statistic low high + ============================= ==== ==== + etherStatsPkts64Octets 0 64 + etherStatsPkts512to1023Octets 512 1023 + ============================= ==== ==== + Request translation =================== @@ -1357,8 +1552,8 @@ are netlink only. ``ETHTOOL_GET_DUMP_FLAG`` n/a ``ETHTOOL_GET_DUMP_DATA`` n/a ``ETHTOOL_GET_TS_INFO`` ``ETHTOOL_MSG_TSINFO_GET`` - ``ETHTOOL_GMODULEINFO`` n/a - ``ETHTOOL_GMODULEEEPROM`` n/a + ``ETHTOOL_GMODULEINFO`` ``ETHTOOL_MSG_MODULE_EEPROM_GET`` + ``ETHTOOL_GMODULEEEPROM`` ``ETHTOOL_MSG_MODULE_EEPROM_GET`` ``ETHTOOL_GEEE`` ``ETHTOOL_MSG_EEE_GET`` ``ETHTOOL_SEEE`` ``ETHTOOL_MSG_EEE_SET`` ``ETHTOOL_GRSSH`` n/a @@ -1373,9 +1568,9 @@ are netlink only. ``ETHTOOL_MSG_LINKMODES_SET`` ``ETHTOOL_PHY_GTUNABLE`` n/a ``ETHTOOL_PHY_STUNABLE`` n/a - ``ETHTOOL_GFECPARAM`` n/a - ``ETHTOOL_SFECPARAM`` n/a - n/a ''ETHTOOL_MSG_CABLE_TEST_ACT'' - n/a ''ETHTOOL_MSG_CABLE_TEST_TDR_ACT'' + ``ETHTOOL_GFECPARAM`` ``ETHTOOL_MSG_FEC_GET`` + ``ETHTOOL_SFECPARAM`` ``ETHTOOL_MSG_FEC_SET`` + n/a ``ETHTOOL_MSG_CABLE_TEST_ACT`` + n/a ``ETHTOOL_MSG_CABLE_TEST_TDR_ACT`` n/a ``ETHTOOL_MSG_TUNNEL_INFO_GET`` =================================== ===================================== diff --git a/Documentation/networking/filter.rst b/Documentation/networking/filter.rst index 251c6bd73d15..3e2221f4abe4 100644 --- a/Documentation/networking/filter.rst +++ b/Documentation/networking/filter.rst @@ -327,7 +327,7 @@ Examples for low-level BPF: ret #-1 drop: ret #0 -**icmp random packet sampling, 1 in 4**: +**icmp random packet sampling, 1 in 4**:: ldh [12] jne #0x800, drop diff --git a/Documentation/networking/index.rst b/Documentation/networking/index.rst index b8a29997d433..e9ce55992aa9 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -76,6 +76,7 @@ Contents: netdevices netfilter-sysctl netif-msg + nexthop-group-resilient nf_conntrack-sysctl nf_flowtable openvswitch diff --git a/Documentation/networking/ip-sysctl.rst b/Documentation/networking/ip-sysctl.rst index 3feb5e565b1a..c2ecc9894fd0 100644 --- a/Documentation/networking/ip-sysctl.rst +++ b/Documentation/networking/ip-sysctl.rst @@ -1073,7 +1073,9 @@ ip_local_reserved_ports - list of comma separated ranges although this is redundant. However such a setting is useful if later the port range is changed to a value that will - include the reserved ports. + include the reserved ports. Also keep in mind, that overlapping + of these ranges may affect probability of selecting ephemeral + ports which are right after block of reserved ports. Default: Empty @@ -1143,6 +1145,12 @@ icmp_echo_ignore_all - BOOLEAN Default: 0 +icmp_echo_enable_probe - BOOLEAN + If set to one, then the kernel will respond to RFC 8335 PROBE + requests sent to it. + + Default: 0 + icmp_echo_ignore_broadcasts - BOOLEAN If set non-zero, then the kernel will ignore all ICMP ECHO and TIMESTAMP requests sent to it via broadcast/multicast. diff --git a/Documentation/networking/nexthop-group-resilient.rst b/Documentation/networking/nexthop-group-resilient.rst new file mode 100644 index 000000000000..fabecee24d85 --- /dev/null +++ b/Documentation/networking/nexthop-group-resilient.rst @@ -0,0 +1,293 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================= +Resilient Next-hop Groups +========================= + +Resilient groups are a type of next-hop group that is aimed at minimizing +disruption in flow routing across changes to the group composition and +weights of constituent next hops. + +The idea behind resilient hashing groups is best explained in contrast to +the legacy multipath next-hop group, which uses the hash-threshold +algorithm, described in RFC 2992. + +To select a next hop, hash-threshold algorithm first assigns a range of +hashes to each next hop in the group, and then selects the next hop by +comparing the SKB hash with the individual ranges. When a next hop is +removed from the group, the ranges are recomputed, which leads to +reassignment of parts of hash space from one next hop to another. RFC 2992 +illustrates it thus:: + + +-------+-------+-------+-------+-------+ + | 1 | 2 | 3 | 4 | 5 | + +-------+-+-----+---+---+-----+-+-------+ + | 1 | 2 | 4 | 5 | + +---------+---------+---------+---------+ + + Before and after deletion of next hop 3 + under the hash-threshold algorithm. + +Note how next hop 2 gave up part of the hash space in favor of next hop 1, +and 4 in favor of 5. While there will usually be some overlap between the +previous and the new distribution, some traffic flows change the next hop +that they resolve to. + +If a multipath group is used for load-balancing between multiple servers, +this hash space reassignment causes an issue that packets from a single +flow suddenly end up arriving at a server that does not expect them. This +can result in TCP connections being reset. + +If a multipath group is used for load-balancing among available paths to +the same server, the issue is that different latencies and reordering along +the way causes the packets to arrive in the wrong order, resulting in +degraded application performance. + +To mitigate the above-mentioned flow redirection, resilient next-hop groups +insert another layer of indirection between the hash space and its +constituent next hops: a hash table. The selection algorithm uses SKB hash +to choose a hash table bucket, then reads the next hop that this bucket +contains, and forwards traffic there. + +This indirection brings an important feature. In the hash-threshold +algorithm, the range of hashes associated with a next hop must be +continuous. With a hash table, mapping between the hash table buckets and +the individual next hops is arbitrary. Therefore when a next hop is deleted +the buckets that held it are simply reassigned to other next hops:: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |1|1|1|1|2|2|2|2|3|3|3|3|4|4|4|4|5|5|5|5| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + v v v v + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |1|1|1|1|2|2|2|2|1|2|4|5|4|4|4|4|5|5|5|5| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Before and after deletion of next hop 3 + under the resilient hashing algorithm. + +When weights of next hops in a group are altered, it may be possible to +choose a subset of buckets that are currently not used for forwarding +traffic, and use those to satisfy the new next-hop distribution demands, +keeping the "busy" buckets intact. This way, established flows are ideally +kept being forwarded to the same endpoints through the same paths as before +the next-hop group change. + +Algorithm +--------- + +In a nutshell, the algorithm works as follows. Each next hop deserves a +certain number of buckets, according to its weight and the number of +buckets in the hash table. In accordance with the source code, we will call +this number a "wants count" of a next hop. In case of an event that might +cause bucket allocation change, the wants counts for individual next hops +are updated. + +Next hops that have fewer buckets than their wants count, are called +"underweight". Those that have more are "overweight". If there are no +overweight (and therefore no underweight) next hops in the group, it is +said to be "balanced". + +Each bucket maintains a last-used timer. Every time a packet is forwarded +through a bucket, this timer is updated to current jiffies value. One +attribute of a resilient group is then the "idle timer", which is the +amount of time that a bucket must not be hit by traffic in order for it to +be considered "idle". Buckets that are not idle are busy. + +After assigning wants counts to next hops, an "upkeep" algorithm runs. For +buckets: + +1) that have no assigned next hop, or +2) whose next hop has been removed, or +3) that are idle and their next hop is overweight, + +upkeep changes the next hop that the bucket references to one of the +underweight next hops. If, after considering all buckets in this manner, +there are still underweight next hops, another upkeep run is scheduled to a +future time. + +There may not be enough "idle" buckets to satisfy the updated wants counts +of all next hops. Another attribute of a resilient group is the "unbalanced +timer". This timer can be set to 0, in which case the table will stay out +of balance until idle buckets do appear, possibly never. If set to a +non-zero value, the value represents the period of time that the table is +permitted to stay out of balance. + +With this in mind, we update the above list of conditions with one more +item. Thus buckets: + +4) whose next hop is overweight, and the amount of time that the table has + been out of balance exceeds the unbalanced timer, if that is non-zero, + +\... are migrated as well. + +Offloading & Driver Feedback +---------------------------- + +When offloading resilient groups, the algorithm that distributes buckets +among next hops is still the one in SW. Drivers are notified of updates to +next hop groups in the following three ways: + +- Full group notification with the type + ``NH_NOTIFIER_INFO_TYPE_RES_TABLE``. This is used just after the group is + created and buckets populated for the first time. + +- Single-bucket notifications of the type + ``NH_NOTIFIER_INFO_TYPE_RES_BUCKET``, which is used for notifications of + individual migrations within an already-established group. + +- Pre-replace notification, ``NEXTHOP_EVENT_RES_TABLE_PRE_REPLACE``. This + is sent before the group is replaced, and is a way for the driver to veto + the group before committing anything to the HW. + +Some single-bucket notifications are forced, as indicated by the "force" +flag in the notification. Those are used for the cases where e.g. the next +hop associated with the bucket was removed, and the bucket really must be +migrated. + +Non-forced notifications can be overridden by the driver by returning an +error code. The use case for this is that the driver notifies the HW that a +bucket should be migrated, but the HW discovers that the bucket has in fact +been hit by traffic. + +A second way for the HW to report that a bucket is busy is through the +``nexthop_res_grp_activity_update()`` API. The buckets identified this way +as busy are treated as if traffic hit them. + +Offloaded buckets should be flagged as either "offload" or "trap". This is +done through the ``nexthop_bucket_set_hw_flags()`` API. + +Netlink UAPI +------------ + +Resilient Group Replacement +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Resilient groups are configured using the ``RTM_NEWNEXTHOP`` message in the +same manner as other multipath groups. The following changes apply to the +attributes passed in the netlink message: + + =================== ========================================================= + ``NHA_GROUP_TYPE`` Should be ``NEXTHOP_GRP_TYPE_RES`` for resilient group. + ``NHA_RES_GROUP`` A nest that contains attributes specific to resilient + groups. + =================== ========================================================= + +``NHA_RES_GROUP`` payload: + + =================================== ========================================= + ``NHA_RES_GROUP_BUCKETS`` Number of buckets in the hash table. + ``NHA_RES_GROUP_IDLE_TIMER`` Idle timer in units of clock_t. + ``NHA_RES_GROUP_UNBALANCED_TIMER`` Unbalanced timer in units of clock_t. + =================================== ========================================= + +Next Hop Get +^^^^^^^^^^^^ + +Requests to get resilient next-hop groups use the ``RTM_GETNEXTHOP`` +message in exactly the same way as other next hop get requests. The +response attributes match the replacement attributes cited above, except +``NHA_RES_GROUP`` payload will include the following attribute: + + =================================== ========================================= + ``NHA_RES_GROUP_UNBALANCED_TIME`` How long has the resilient group been out + of balance, in units of clock_t. + =================================== ========================================= + +Bucket Get +^^^^^^^^^^ + +The message ``RTM_GETNEXTHOPBUCKET`` without the ``NLM_F_DUMP`` flag is +used to request a single bucket. The attributes recognized at get requests +are: + + =================== ========================================================= + ``NHA_ID`` ID of the next-hop group that the bucket belongs to. + ``NHA_RES_BUCKET`` A nest that contains attributes specific to bucket. + =================== ========================================================= + +``NHA_RES_BUCKET`` payload: + + ======================== ==================================================== + ``NHA_RES_BUCKET_INDEX`` Index of bucket in the resilient table. + ======================== ==================================================== + +Bucket Dumps +^^^^^^^^^^^^ + +The message ``RTM_GETNEXTHOPBUCKET`` with the ``NLM_F_DUMP`` flag is used +to request a dump of matching buckets. The attributes recognized at dump +requests are: + + =================== ========================================================= + ``NHA_ID`` If specified, limits the dump to just the next-hop group + with this ID. + ``NHA_OIF`` If specified, limits the dump to buckets that contain + next hops that use the device with this ifindex. + ``NHA_MASTER`` If specified, limits the dump to buckets that contain + next hops that use a device in the VRF with this ifindex. + ``NHA_RES_BUCKET`` A nest that contains attributes specific to bucket. + =================== ========================================================= + +``NHA_RES_BUCKET`` payload: + + ======================== ==================================================== + ``NHA_RES_BUCKET_NH_ID`` If specified, limits the dump to just the buckets + that contain the next hop with this ID. + ======================== ==================================================== + +Usage +----- + +To illustrate the usage, consider the following commands:: + + # ip nexthop add id 1 via 192.0.2.2 dev eth0 + # ip nexthop add id 2 via 192.0.2.3 dev eth0 + # ip nexthop add id 10 group 1/2 type resilient \ + buckets 8 idle_timer 60 unbalanced_timer 300 + +The last command creates a resilient next-hop group. It will have 8 buckets +(which is unusually low number, and used here for demonstration purposes +only), each bucket will be considered idle when no traffic hits it for at +least 60 seconds, and if the table remains out of balance for 300 seconds, +it will be forcefully brought into balance. + +Changing next-hop weights leads to change in bucket allocation:: + + # ip nexthop replace id 10 group 1,3/2 type resilient + +This can be confirmed by looking at individual buckets:: + + # ip nexthop bucket show id 10 + id 10 index 0 idle_time 5.59 nhid 1 + id 10 index 1 idle_time 5.59 nhid 1 + id 10 index 2 idle_time 8.74 nhid 2 + id 10 index 3 idle_time 8.74 nhid 2 + id 10 index 4 idle_time 8.74 nhid 1 + id 10 index 5 idle_time 8.74 nhid 1 + id 10 index 6 idle_time 8.74 nhid 1 + id 10 index 7 idle_time 8.74 nhid 1 + +Note the two buckets that have a shorter idle time. Those are the ones that +were migrated after the next-hop replace command to satisfy the new demand +that next hop 1 be given 6 buckets instead of 4. + +Netdevsim +--------- + +The netdevsim driver implements a mock offload of resilient groups, and +exposes debugfs interface that allows marking individual buckets as busy. +For example, the following will mark bucket 23 in next-hop group 10 as +active:: + + # echo 10 23 > /sys/kernel/debug/netdevsim/netdevsim10/fib/nexthop_bucket_activity + +In addition, another debugfs interface can be used to configure that the +next attempt to migrate a bucket should fail:: + + # echo 1 > /sys/kernel/debug/netdevsim/netdevsim10/fib/fail_nexthop_bucket_replace + +Besides serving as an example, the interfaces that netdevsim exposes are +useful in automated testing, and +``tools/testing/selftests/drivers/net/netdevsim/nexthop.sh`` makes use of +them to test the algorithm. diff --git a/Documentation/networking/nf_flowtable.rst b/Documentation/networking/nf_flowtable.rst index 6cdf9a1724b6..d757c21c10f2 100644 --- a/Documentation/networking/nf_flowtable.rst +++ b/Documentation/networking/nf_flowtable.rst @@ -4,35 +4,38 @@ Netfilter's flowtable infrastructure ==================================== -This documentation describes the software flowtable infrastructure available in -Netfilter since Linux kernel 4.16. +This documentation describes the Netfilter flowtable infrastructure which allows +you to define a fastpath through the flowtable datapath. This infrastructure +also provides hardware offload support. The flowtable supports for the layer 3 +IPv4 and IPv6 and the layer 4 TCP and UDP protocols. Overview -------- -Initial packets follow the classic forwarding path, once the flow enters the -established state according to the conntrack semantics (ie. we have seen traffic -in both directions), then you can decide to offload the flow to the flowtable -from the forward chain via the 'flow offload' action available in nftables. +Once the first packet of the flow successfully goes through the IP forwarding +path, from the second packet on, you might decide to offload the flow to the +flowtable through your ruleset. The flowtable infrastructure provides a rule +action that allows you to specify when to add a flow to the flowtable. -Packets that find an entry in the flowtable (ie. flowtable hit) are sent to the -output netdevice via neigh_xmit(), hence, they bypass the classic forwarding -path (the visible effect is that you do not see these packets from any of the -netfilter hooks coming after the ingress). In case of flowtable miss, the packet -follows the classic forward path. +A packet that finds a matching entry in the flowtable (ie. flowtable hit) is +transmitted to the output netdevice via neigh_xmit(), hence, packets bypass the +classic IP forwarding path (the visible effect is that you do not see these +packets from any of the Netfilter hooks coming after ingress). In case that +there is no matching entry in the flowtable (ie. flowtable miss), the packet +follows the classic IP forwarding path. -The flowtable uses a resizable hashtable, lookups are based on the following -7-tuple selectors: source, destination, layer 3 and layer 4 protocols, source -and destination ports and the input interface (useful in case there are several -conntrack zones in place). +The flowtable uses a resizable hashtable. Lookups are based on the following +n-tuple selectors: layer 2 protocol encapsulation (VLAN and PPPoE), layer 3 +source and destination, layer 4 source and destination ports and the input +interface (useful in case there are several conntrack zones in place). -Flowtables are populated via the 'flow offload' nftables action, so the user can -selectively specify what flows are placed into the flow table. Hence, packets -follow the classic forwarding path unless the user explicitly instruct packets -to use this new alternative forwarding path via nftables policy. +The 'flow add' action allows you to populate the flowtable, the user selectively +specifies what flows are placed into the flowtable. Hence, packets follow the +classic IP forwarding path unless the user explicitly instruct flows to use this +new alternative forwarding path via policy. -This is represented in Fig.1, which describes the classic forwarding path -including the Netfilter hooks and the flowtable fastpath bypass. +The flowtable datapath is represented in Fig.1, which describes the classic IP +forwarding path including the Netfilter hooks and the flowtable fastpath bypass. :: @@ -67,11 +70,13 @@ including the Netfilter hooks and the flowtable fastpath bypass. Fig.1 Netfilter hooks and flowtable interactions The flowtable entry also stores the NAT configuration, so all packets are -mangled according to the NAT policy that matches the initial packets that went -through the classic forwarding path. The TTL is decremented before calling -neigh_xmit(). Fragmented traffic is passed up to follow the classic forwarding -path given that the transport selectors are missing, therefore flowtable lookup -is not possible. +mangled according to the NAT policy that is specified from the classic IP +forwarding path. The TTL is decremented before calling neigh_xmit(). Fragmented +traffic is passed up to follow the classic IP forwarding path given that the +transport header is missing, in this case, flowtable lookups are not possible. +TCP RST and FIN packets are also passed up to the classic IP forwarding path to +release the flow gracefully. Packets that exceed the MTU are also passed up to +the classic forwarding path to report packet-too-big ICMP errors to the sender. Example configuration --------------------- @@ -85,7 +90,7 @@ flowtable and add one rule to your forward chain:: } chain y { type filter hook forward priority 0; policy accept; - ip protocol tcp flow offload @f + ip protocol tcp flow add @f counter packets 0 bytes 0 } } @@ -103,6 +108,119 @@ flow is offloaded, you will observe that the counter rule in the example above does not get updated for the packets that are being forwarded through the forwarding bypass. +You can identify offloaded flows through the [OFFLOAD] tag when listing your +connection tracking table. + +:: + + # conntrack -L + tcp 6 src=10.141.10.2 dst=192.168.10.2 sport=52728 dport=5201 src=192.168.10.2 dst=192.168.10.1 sport=5201 dport=52728 [OFFLOAD] mark=0 use=2 + + +Layer 2 encapsulation +--------------------- + +Since Linux kernel 5.13, the flowtable infrastructure discovers the real +netdevice behind VLAN and PPPoE netdevices. The flowtable software datapath +parses the VLAN and PPPoE layer 2 headers to extract the ethertype and the +VLAN ID / PPPoE session ID which are used for the flowtable lookups. The +flowtable datapath also deals with layer 2 decapsulation. + +You do not need to add the PPPoE and the VLAN devices to your flowtable, +instead the real device is sufficient for the flowtable to track your flows. + +Bridge and IP forwarding +------------------------ + +Since Linux kernel 5.13, you can add bridge ports to the flowtable. The +flowtable infrastructure discovers the topology behind the bridge device. This +allows the flowtable to define a fastpath bypass between the bridge ports +(represented as eth1 and eth2 in the example figure below) and the gateway +device (represented as eth0) in your switch/router. + +:: + + fastpath bypass + .-------------------------. + / \ + | IP forwarding | + | / \ \/ + | br0 eth0 ..... eth0 + . / \ *host B* + -> eth1 eth2 + . *switch/router* + . + . + eth0 + *host A* + +The flowtable infrastructure also supports for bridge VLAN filtering actions +such as PVID and untagged. You can also stack a classic VLAN device on top of +your bridge port. + +If you would like that your flowtable defines a fastpath between your bridge +ports and your IP forwarding path, you have to add your bridge ports (as +represented by the real netdevice) to your flowtable definition. + +Counters +-------- + +The flowtable can synchronize packet and byte counters with the existing +connection tracking entry by specifying the counter statement in your flowtable +definition, e.g. + +:: + + table inet x { + flowtable f { + hook ingress priority 0; devices = { eth0, eth1 }; + counter + } + } + +Counter support is available since Linux kernel 5.7. + +Hardware offload +---------------- + +If your network device provides hardware offload support, you can turn it on by +means of the 'offload' flag in your flowtable definition, e.g. + +:: + + table inet x { + flowtable f { + hook ingress priority 0; devices = { eth0, eth1 }; + flags offload; + } + } + +There is a workqueue that adds the flows to the hardware. Note that a few +packets might still run over the flowtable software path until the workqueue has +a chance to offload the flow to the network device. + +You can identify hardware offloaded flows through the [HW_OFFLOAD] tag when +listing your connection tracking table. Please, note that the [OFFLOAD] tag +refers to the software offload mode, so there is a distinction between [OFFLOAD] +which refers to the software flowtable fastpath and [HW_OFFLOAD] which refers +to the hardware offload datapath being used by the flow. + +The flowtable hardware offload infrastructure also supports for the DSA +(Distributed Switch Architecture). + +Limitations +----------- + +The flowtable behaves like a cache. The flowtable entries might get stale if +either the destination MAC address or the egress netdevice that is used for +transmission changes. + +This might be a problem if: + +- You run the flowtable in software mode and you combine bridge and IP + forwarding in your setup. +- Hardware offload is enabled. + More reading ------------ diff --git a/Documentation/networking/phy.rst b/Documentation/networking/phy.rst index 06adfc2afcf0..3f05d50ecd6e 100644 --- a/Documentation/networking/phy.rst +++ b/Documentation/networking/phy.rst @@ -80,8 +80,8 @@ values of phy_interface_t must be understood from the perspective of the PHY device itself, leading to the following: * PHY_INTERFACE_MODE_RGMII: the PHY is not responsible for inserting any - internal delay by itself, it assumes that either the Ethernet MAC (if capable - or the PCB traces) insert the correct 1.5-2ns delay + internal delay by itself, it assumes that either the Ethernet MAC (if capable) + or the PCB traces insert the correct 1.5-2ns delay * PHY_INTERFACE_MODE_RGMII_TXID: the PHY should insert an internal delay for the transmit data lines (TXD[3:0]) processed by the PHY device diff --git a/Documentation/networking/statistics.rst b/Documentation/networking/statistics.rst index 234abedc29b2..c9aeb70dafa2 100644 --- a/Documentation/networking/statistics.rst +++ b/Documentation/networking/statistics.rst @@ -44,8 +44,27 @@ If `-s` is specified once the detailed errors won't be shown. Protocol-specific statistics ---------------------------- -Some of the interfaces used for configuring devices are also able -to report related statistics. For example ethtool interface used +Protocol-specific statistics are exposed via relevant interfaces, +the same interfaces as are used to configure them. + +ethtool +~~~~~~~ + +Ethtool exposes common low-level statistics. +All the standard statistics are expected to be maintained +by the device, not the driver (as opposed to driver-defined stats +described in the next section which mix software and hardware stats). +For devices which contain unmanaged +switches (e.g. legacy SR-IOV or multi-host NICs) the events counted +may not pertain exclusively to the packets destined to +the local host interface. In other words the events may +be counted at the network port (MAC/PHY blocks) without separation +for different host side (PCIe) devices. Such ambiguity must not +be present when internal switch is managed by Linux (so called +switchdev mode for NICs). + +Standard ethtool statistics can be accessed via the interfaces used +for configuration. For example ethtool interface used to configure pause frames can report corresponding hardware counters:: $ ethtool --include-statistics -a eth0 @@ -57,6 +76,27 @@ to configure pause frames can report corresponding hardware counters:: tx_pause_frames: 1 rx_pause_frames: 1 +General Ethernet statistics not associated with any particular +functionality are exposed via ``ethtool -S $ifc`` by specifying +the ``--groups`` parameter:: + + $ ethtool -S eth0 --groups eth-phy eth-mac eth-ctrl rmon + Stats for eth0: + eth-phy-SymbolErrorDuringCarrier: 0 + eth-mac-FramesTransmittedOK: 1 + eth-mac-FrameTooLongErrors: 1 + eth-ctrl-MACControlFramesTransmitted: 1 + eth-ctrl-MACControlFramesReceived: 0 + eth-ctrl-UnsupportedOpcodesReceived: 1 + rmon-etherStatsUndersizePkts: 1 + rmon-etherStatsJabbers: 0 + rmon-rx-etherStatsPkts64Octets: 1 + rmon-rx-etherStatsPkts65to127Octets: 0 + rmon-rx-etherStatsPkts128to255Octets: 0 + rmon-tx-etherStatsPkts64Octets: 2 + rmon-tx-etherStatsPkts65to127Octets: 3 + rmon-tx-etherStatsPkts128to255Octets: 0 + Driver-defined statistics ------------------------- @@ -130,6 +170,7 @@ the `ETHTOOL_FLAG_STATS` flag in `ETHTOOL_A_HEADER_FLAGS`. Currently statistics are supported in the following commands: - `ETHTOOL_MSG_PAUSE_GET` + - `ETHTOOL_MSG_FEC_GET` debugfs ------- @@ -176,3 +217,4 @@ translated to netlink attributes when dumped. Drivers must not overwrite the statistics they don't report with 0. - ethtool_pause_stats() +- ethtool_fec_stats() diff --git a/Documentation/networking/switchdev.rst b/Documentation/networking/switchdev.rst index ddc3f35775dc..f1f4e6a85a29 100644 --- a/Documentation/networking/switchdev.rst +++ b/Documentation/networking/switchdev.rst @@ -181,18 +181,41 @@ To offloading L2 bridging, the switchdev driver/device should support: Static FDB Entries ^^^^^^^^^^^^^^^^^^ -The switchdev driver should implement ndo_fdb_add, ndo_fdb_del and ndo_fdb_dump -to support static FDB entries installed to the device. Static bridge FDB -entries are installed, for example, using iproute2 bridge cmd:: - - bridge fdb add ADDR dev DEV [vlan VID] [self] - -The driver should use the helper switchdev_port_fdb_xxx ops for ndo_fdb_xxx -ops, and handle add/delete/dump of SWITCHDEV_OBJ_ID_PORT_FDB object using -switchdev_port_obj_xxx ops. - -XXX: what should be done if offloading this rule to hardware fails (for -example, due to full capacity in hardware tables) ? +A driver which implements the ``ndo_fdb_add``, ``ndo_fdb_del`` and +``ndo_fdb_dump`` operations is able to support the command below, which adds a +static bridge FDB entry:: + + bridge fdb add dev DEV ADDRESS [vlan VID] [self] static + +(the "static" keyword is non-optional: if not specified, the entry defaults to +being "local", which means that it should not be forwarded) + +The "self" keyword (optional because it is implicit) has the role of +instructing the kernel to fulfill the operation through the ``ndo_fdb_add`` +implementation of the ``DEV`` device itself. If ``DEV`` is a bridge port, this +will bypass the bridge and therefore leave the software database out of sync +with the hardware one. + +To avoid this, the "master" keyword can be used:: + + bridge fdb add dev DEV ADDRESS [vlan VID] master static + +The above command instructs the kernel to search for a master interface of +``DEV`` and fulfill the operation through the ``ndo_fdb_add`` method of that. +This time, the bridge generates a ``SWITCHDEV_FDB_ADD_TO_DEVICE`` notification +which the port driver can handle and use it to program its hardware table. This +way, the software and the hardware database will both contain this static FDB +entry. + +Note: for new switchdev drivers that offload the Linux bridge, implementing the +``ndo_fdb_add`` and ``ndo_fdb_del`` bridge bypass methods is strongly +discouraged: all static FDB entries should be added on a bridge port using the +"master" flag. The ``ndo_fdb_dump`` is an exception and can be implemented to +visualize the hardware tables, if the device does not have an interrupt for +notifying the operating system of newly learned/forgotten dynamic FDB +addresses. In that case, the hardware FDB might end up having entries that the +software FDB does not, and implementing ``ndo_fdb_dump`` is the only way to see +them. Note: by default, the bridge does not filter on VLAN and only bridges untagged traffic. To enable VLAN support, turn on VLAN filtering:: @@ -385,3 +408,156 @@ The driver can monitor for updates to arp_tbl using the netevent notifier NETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops for the routes as arp_tbl updates. The driver implements ndo_neigh_destroy to know when arp_tbl neighbor entries are purged from the port. + +Device driver expected behavior +------------------------------- + +Below is a set of defined behavior that switchdev enabled network devices must +adhere to. + +Configuration-less state +^^^^^^^^^^^^^^^^^^^^^^^^ + +Upon driver bring up, the network devices must be fully operational, and the +backing driver must configure the network device such that it is possible to +send and receive traffic to this network device and it is properly separated +from other network devices/ports (e.g.: as is frequent with a switch ASIC). How +this is achieved is heavily hardware dependent, but a simple solution can be to +use per-port VLAN identifiers unless a better mechanism is available +(proprietary metadata for each network port for instance). + +The network device must be capable of running a full IP protocol stack +including multicast, DHCP, IPv4/6, etc. If necessary, it should program the +appropriate filters for VLAN, multicast, unicast etc. The underlying device +driver must effectively be configured in a similar fashion to what it would do +when IGMP snooping is enabled for IP multicast over these switchdev network +devices and unsolicited multicast must be filtered as early as possible in +the hardware. + +When configuring VLANs on top of the network device, all VLANs must be working, +irrespective of the state of other network devices (e.g.: other ports being part +of a VLAN-aware bridge doing ingress VID checking). See below for details. + +If the device implements e.g.: VLAN filtering, putting the interface in +promiscuous mode should allow the reception of all VLAN tags (including those +not present in the filter(s)). + +Bridged switch ports +^^^^^^^^^^^^^^^^^^^^ + +When a switchdev enabled network device is added as a bridge member, it should +not disrupt any functionality of non-bridged network devices and they +should continue to behave as normal network devices. Depending on the bridge +configuration knobs below, the expected behavior is documented. + +Bridge VLAN filtering +^^^^^^^^^^^^^^^^^^^^^ + +The Linux bridge allows the configuration of a VLAN filtering mode (statically, +at device creation time, and dynamically, during run time) which must be +observed by the underlying switchdev network device/hardware: + +- with VLAN filtering turned off: the bridge is strictly VLAN unaware and its + data path will process all Ethernet frames as if they are VLAN-untagged. + The bridge VLAN database can still be modified, but the modifications should + have no effect while VLAN filtering is turned off. Frames ingressing the + device with a VID that is not programmed into the bridge/switch's VLAN table + must be forwarded and may be processed using a VLAN device (see below). + +- with VLAN filtering turned on: the bridge is VLAN-aware and frames ingressing + the device with a VID that is not programmed into the bridges/switch's VLAN + table must be dropped (strict VID checking). + +When there is a VLAN device (e.g: sw0p1.100) configured on top of a switchdev +network device which is a bridge port member, the behavior of the software +network stack must be preserved, or the configuration must be refused if that +is not possible. + +- with VLAN filtering turned off, the bridge will process all ingress traffic + for the port, except for the traffic tagged with a VLAN ID destined for a + VLAN upper. The VLAN upper interface (which consumes the VLAN tag) can even + be added to a second bridge, which includes other switch ports or software + interfaces. Some approaches to ensure that the forwarding domain for traffic + belonging to the VLAN upper interfaces are managed properly: + + * If forwarding destinations can be managed per VLAN, the hardware could be + configured to map all traffic, except the packets tagged with a VID + belonging to a VLAN upper interface, to an internal VID corresponding to + untagged packets. This internal VID spans all ports of the VLAN-unaware + bridge. The VID corresponding to the VLAN upper interface spans the + physical port of that VLAN interface, as well as the other ports that + might be bridged with it. + * Treat bridge ports with VLAN upper interfaces as standalone, and let + forwarding be handled in the software data path. + +- with VLAN filtering turned on, these VLAN devices can be created as long as + the bridge does not have an existing VLAN entry with the same VID on any + bridge port. These VLAN devices cannot be enslaved into the bridge since they + duplicate functionality/use case with the bridge's VLAN data path processing. + +Non-bridged network ports of the same switch fabric must not be disturbed in any +way by the enabling of VLAN filtering on the bridge device(s). If the VLAN +filtering setting is global to the entire chip, then the standalone ports +should indicate to the network stack that VLAN filtering is required by setting +'rx-vlan-filter: on [fixed]' in the ethtool features. + +Because VLAN filtering can be turned on/off at runtime, the switchdev driver +must be able to reconfigure the underlying hardware on the fly to honor the +toggling of that option and behave appropriately. If that is not possible, the +switchdev driver can also refuse to support dynamic toggling of the VLAN +filtering knob at runtime and require a destruction of the bridge device(s) and +creation of new bridge device(s) with a different VLAN filtering value to +ensure VLAN awareness is pushed down to the hardware. + +Even when VLAN filtering in the bridge is turned off, the underlying switch +hardware and driver may still configure itself in a VLAN-aware mode provided +that the behavior described above is observed. + +The VLAN protocol of the bridge plays a role in deciding whether a packet is +treated as tagged or not: a bridge using the 802.1ad protocol must treat both +VLAN-untagged packets, as well as packets tagged with 802.1Q headers, as +untagged. + +The 802.1p (VID 0) tagged packets must be treated in the same way by the device +as untagged packets, since the bridge device does not allow the manipulation of +VID 0 in its database. + +When the bridge has VLAN filtering enabled and a PVID is not configured on the +ingress port, untagged and 802.1p tagged packets must be dropped. When the bridge +has VLAN filtering enabled and a PVID exists on the ingress port, untagged and +priority-tagged packets must be accepted and forwarded according to the +bridge's port membership of the PVID VLAN. When the bridge has VLAN filtering +disabled, the presence/lack of a PVID should not influence the packet +forwarding decision. + +Bridge IGMP snooping +^^^^^^^^^^^^^^^^^^^^ + +The Linux bridge allows the configuration of IGMP snooping (statically, at +interface creation time, or dynamically, during runtime) which must be observed +by the underlying switchdev network device/hardware in the following way: + +- when IGMP snooping is turned off, multicast traffic must be flooded to all + ports within the same bridge that have mcast_flood=true. The CPU/management + port should ideally not be flooded (unless the ingress interface has + IFF_ALLMULTI or IFF_PROMISC) and continue to learn multicast traffic through + the network stack notifications. If the hardware is not capable of doing that + then the CPU/management port must also be flooded and multicast filtering + happens in software. + +- when IGMP snooping is turned on, multicast traffic must selectively flow + to the appropriate network ports (including CPU/management port). Flooding of + unknown multicast should be only towards the ports connected to a multicast + router (the local device may also act as a multicast router). + +The switch must adhere to RFC 4541 and flood multicast traffic accordingly +since that is what the Linux bridge implementation does. + +Because IGMP snooping can be turned on/off at runtime, the switchdev driver +must be able to reconfigure the underlying hardware on the fly to honor the +toggling of that option and behave appropriately. + +A switchdev driver can also refuse to support dynamic toggling of the multicast +snooping knob at runtime and require the destruction of the bridge device(s) +and creation of a new bridge device(s) with a different multicast snooping +value. diff --git a/Documentation/networking/timestamping.rst b/Documentation/networking/timestamping.rst index f682e88fa87e..7db3985359bc 100644 --- a/Documentation/networking/timestamping.rst +++ b/Documentation/networking/timestamping.rst @@ -630,30 +630,45 @@ hardware timestamping on it. This is because the SO_TIMESTAMPING API does not allow the delivery of multiple hardware timestamps for the same packet, so anybody else except for the DSA switch port must be prevented from doing so. -In code, DSA provides for most of the infrastructure for timestamping already, -in generic code: a BPF classifier (``ptp_classify_raw``) is used to identify -PTP event messages (any other packets, including PTP general messages, are not -timestamped), and provides two hooks to drivers: - -- ``.port_txtstamp()``: The driver is passed a clone of the timestampable skb - to be transmitted, before actually transmitting it. Typically, a switch will - have a PTP TX timestamp register (or sometimes a FIFO) where the timestamp - becomes available. There may be an IRQ that is raised upon this timestamp's - availability, or the driver might have to poll after invoking - ``dev_queue_xmit()`` towards the host interface. Either way, in the - ``.port_txtstamp()`` method, the driver only needs to save the clone for - later use (when the timestamp becomes available). Each skb is annotated with - a pointer to its clone, in ``DSA_SKB_CB(skb)->clone``, to ease the driver's - job of keeping track of which clone belongs to which skb. - -- ``.port_rxtstamp()``: The original (and only) timestampable skb is provided - to the driver, for it to annotate it with a timestamp, if that is immediately - available, or defer to later. On reception, timestamps might either be - available in-band (through metadata in the DSA header, or attached in other - ways to the packet), or out-of-band (through another RX timestamping FIFO). - Deferral on RX is typically necessary when retrieving the timestamp needs a - sleepable context. In that case, it is the responsibility of the DSA driver - to call ``netif_rx_ni()`` on the freshly timestamped skb. +In the generic layer, DSA provides the following infrastructure for PTP +timestamping: + +- ``.port_txtstamp()``: a hook called prior to the transmission of + packets with a hardware TX timestamping request from user space. + This is required for two-step timestamping, since the hardware + timestamp becomes available after the actual MAC transmission, so the + driver must be prepared to correlate the timestamp with the original + packet so that it can re-enqueue the packet back into the socket's + error queue. To save the packet for when the timestamp becomes + available, the driver can call ``skb_clone_sk`` , save the clone pointer + in skb->cb and enqueue a tx skb queue. Typically, a switch will have a + PTP TX timestamp register (or sometimes a FIFO) where the timestamp + becomes available. In case of a FIFO, the hardware might store + key-value pairs of PTP sequence ID/message type/domain number and the + actual timestamp. To perform the correlation correctly between the + packets in a queue waiting for timestamping and the actual timestamps, + drivers can use a BPF classifier (``ptp_classify_raw``) to identify + the PTP transport type, and ``ptp_parse_header`` to interpret the PTP + header fields. There may be an IRQ that is raised upon this + timestamp's availability, or the driver might have to poll after + invoking ``dev_queue_xmit()`` towards the host interface. + One-step TX timestamping do not require packet cloning, since there is + no follow-up message required by the PTP protocol (because the + TX timestamp is embedded into the packet by the MAC), and therefore + user space does not expect the packet annotated with the TX timestamp + to be re-enqueued into its socket's error queue. + +- ``.port_rxtstamp()``: On RX, the BPF classifier is run by DSA to + identify PTP event messages (any other packets, including PTP general + messages, are not timestamped). The original (and only) timestampable + skb is provided to the driver, for it to annotate it with a timestamp, + if that is immediately available, or defer to later. On reception, + timestamps might either be available in-band (through metadata in the + DSA header, or attached in other ways to the packet), or out-of-band + (through another RX timestamping FIFO). Deferral on RX is typically + necessary when retrieving the timestamp needs a sleepable context. In + that case, it is the responsibility of the DSA driver to call + ``netif_rx_ni()`` on the freshly timestamped skb. 3.2.2 Ethernet PHYs ^^^^^^^^^^^^^^^^^^^ diff --git a/Documentation/networking/x25-iface.rst b/Documentation/networking/x25-iface.rst index df401891dce6..f34e9ec64937 100644 --- a/Documentation/networking/x25-iface.rst +++ b/Documentation/networking/x25-iface.rst @@ -70,60 +70,13 @@ First Byte = 0x03 (X25_IFACE_PARAMS) LAPB parameters. To be defined. +Requirements for the device driver +---------------------------------- -Possible Problems -================= - -(Henner Eisen, 2000-10-28) - -The X.25 packet layer protocol depends on a reliable datalink service. -The LAPB protocol provides such reliable service. But this reliability -is not preserved by the Linux network device driver interface: - -- With Linux 2.4.x (and above) SMP kernels, packet ordering is not - preserved. Even if a device driver calls netif_rx(skb1) and later - netif_rx(skb2), skb2 might be delivered to the network layer - earlier that skb1. -- Data passed upstream by means of netif_rx() might be dropped by the - kernel if the backlog queue is congested. - -The X.25 packet layer protocol will detect this and reset the virtual -call in question. But many upper layer protocols are not designed to -handle such N-Reset events gracefully. And frequent N-Reset events -will always degrade performance. - -Thus, driver authors should make netif_rx() as reliable as possible: - -SMP re-ordering will not occur if the driver's interrupt handler is -always executed on the same CPU. Thus, - -- Driver authors should use irq affinity for the interrupt handler. - -The probability of packet loss due to backlog congestion can be -reduced by the following measures or a combination thereof: - -(1) Drivers for kernel versions 2.4.x and above should always check the - return value of netif_rx(). If it returns NET_RX_DROP, the - driver's LAPB protocol must not confirm reception of the frame - to the peer. - This will reliably suppress packet loss. The LAPB protocol will - automatically cause the peer to re-transmit the dropped packet - later. - The lapb module interface was modified to support this. Its - data_indication() method should now transparently pass the - netif_rx() return value to the (lapb module) caller. -(2) Drivers for kernel versions 2.2.x should always check the global - variable netdev_dropping when a new frame is received. The driver - should only call netif_rx() if netdev_dropping is zero. Otherwise - the driver should not confirm delivery of the frame and drop it. - Alternatively, the driver can queue the frame internally and call - netif_rx() later when netif_dropping is 0 again. In that case, delivery - confirmation should also be deferred such that the internal queue - cannot grow to much. - This will not reliably avoid packet loss, but the probability - of packet loss in netif_rx() path will be significantly reduced. -(3) Additionally, driver authors might consider to support - CONFIG_NET_HW_FLOWCONTROL. This allows the driver to be woken up - when a previously congested backlog queue becomes empty again. - The driver could uses this for flow-controlling the peer by means - of the LAPB protocol's flow-control service. +Packets should not be reordered or dropped when delivering between the +Packet Layer and the device driver. + +To avoid packets from being reordered or dropped when delivering from +the device driver to the Packet Layer, the device driver should not +call "netif_rx" to deliver the received packets. Instead, it should +call "netif_receive_skb_core" from softirq context to deliver them. |