As per Relevance of the word forwarding, we have this rfc below:
Network Working Group G.
Request for Comments: 2121
Category: Informational March 1997
Issues affecting MARS Cluster
Status of this
This memo provides information for the Internet community. This
does not specify an Internet standard of any kind. Distribution
this memo is unlimited
IP multicast over ATM currently uses the MARS model [1] to manage
use of ATM pt-mpt SVCs for IP multicast packet forwarding. The
of any given MARS services is the MARS Cluster - typically the
as an IPv4 Logical IP Subnet (LIS). Current IP/ATM networks
usually architected with unicast routing and forwarding
dictating the sizes of individual LISes. However, as IP multicast
deployed as a service, the size of a LIS will only be as big as
MARS Cluster can be. This document provides a qualitative look at
issues constraining a MARS Cluster's size, including the impact of
limits in switches and NICs, geographical distribution of
members, and the use of VC Mesh or MCS modes to support
groups
1.
A MARS Cluster is the set of IP/ATM interfaces that are willing
engage in direct, ATM level pt-mpt SVCs to perform IP
packet forwarding [1]. Each IP/ATM interface (a MARS Client)
keep state information regarding the ATM addresses of each leaf
(recipient) of each pt-mpt SVC it has open. In addition, each
Client receives MARS_JOIN and MARS_LEAVE messages from the
whenever there is a requirement that Clients around the Cluster
to update their pt-mpt SVCs for a given IP multicast group
The definition of Cluster 'size' can mean two things - the number
MARS Clients using a given MARS, and the geographic distribution
MARS Clients. The number of MARS Clients in a Cluster impacts on
amount of state information any given client may need to store
managing outgoing pt- mpt SVCs. It also impacts on the average
of JOIN/LEAVE traffic that is propagated by the MARS
ClusterControlVC, and the number of pt-mpt VCs that may
modification each time a MARS_JOIN or MARS_LEAVE appears
ClusterControlVC
Armitage Informational [Page 1]
RFC 2121 Issues affecting MARS Cluster Size March 1997
The geographic distribution of clients affects the latency between
client issuing a MARS_JOIN, and it finally being added onto the pt
mpt VCs of the other MARS Clients transmitting to the
multicast group. (This latency is made up of both the time
propagate the MARS_JOIN, and the delay in the underlying ATM cloud'
reaction to the subsequent ADD_PARTY messages.)
When architecting an IP/ATM network it is important to understand
worst case scaling limits applicable to your Clusters. This
provides a primarily qualitative look at the design choices
impose the most dramatic constraints on Cluster size. Since the
is on worst-case scenarios, most of the analysis will
multicast groups that are VC Mesh based and have all cluster
as sources and receivers. Engineering using the worst-case
conditions, then applying optimisations such as Multicast
(MCS), provides the Cluster with a margin of safety. It is
that more detailed quantitative analysis of Cluster sizing
will be prompted by this document
Section 2 comments on the VC state requirements of the MARS model
while Sections 3 and 4 identify the group change processing load
latency characteristics of a cluster as a function of its size
Section 5 looks at how Multicast Routers (both conventional
combination router/switch architectures) increase the scale of
multicast capable IP/ATM network. Finally, Section 6 discusses
the use of Multicast Servers (MCS) might impact on the worst
Cluster size limits
2. VC state limitations
Two characteristics of ATM NICs and switches will limit the number
members a Cluster may contain. They are
The maximum number of VCs that can be originated from,
terminate on, a port (VCmax).
The maximum number of leaf nodes supportable by a root
(LEAFmax).
We'll assume that the MARS node has similar VCmax and LEAFmax
as Cluster members. VCmax affects the Cluster size because of
following
The MARS terminates a pt-pt control VC from each cluster member
and originates a VC for ClusterControlVC and ServerControlVC
Armitage Informational [Page 2]
RFC 2121 Issues affecting MARS Cluster Size March 1997
When a multicast group is VC Mesh based, a group member
a VC from every sender to the group, per group
When a multicast group is MCS based, the MCS terminates a VC
every sender to the group
LEAFmax affects the Cluster size because of the following
ClusterControlVC from the MARS. It has a leaf node per
member (MARS Client).
Packet forwarding SVCs out of each MARS Client for each
multicast group being sent to. It has a leaf node for each
member when a group is VC Mesh based
Packet forwarding SVCs out of each MCS for each IP multicast
being sent to. It has a leaf node for each group member when
group is MCS based
If we have N cluster members, and M multicast groups active (using
Mesh mode, and densely populated - all receivers are senders),
following observations may be made
ClusterControlVC has N leaf nodes,
N <= LEAFmax
The MARS terminates a pt-pt VC from each cluster member,
originates ClusterControlVC and ServerControlVC,
(N+2) <= VCmax
Each Cluster Member sources 1 VC per group, terminates (N-1)
per group, originates a pt-pt VC to the MARS, and terminates 1
as a leaf on ClusterControlVC,
(M*N) + 2 <= VCmax
The VC sourced by each Cluster member per group goes to all
cluster members,
(N-1) <= LEAFmax
Armitage Informational [Page 3]
RFC 2121 Issues affecting MARS Cluster Size March 1997
Since all the above conditions must be simultaneously true, we
see that the most constraining requirement is either
(M*N) + 2 <= VCmax
N <= LEAFmax
The limit involving VCmax is fundamentally controlled by the
consumption of group members using a VC Mesh for data forwarding
rather than the termination of pt-pt control VCs on the MARS. (It
in practice going to be very dependent on the multicast
membership distributions within the cluster.)
The LEAFmax limit comes from ClusterControlVC, and is independent
the density of group members (or the ratios of senders to receivers
for active multicast groups within the cluster
Under UNI 3.0/3.1 the most obvious limit on LEAFmax is 2^15 (the
node ID is 15 bits wide). However, the signaling driver software
most ATM NICs may impose a limit much lower than this - a function
how much per-leaf node state information they need to store (and
capable of storing) for pt-mpt SVCs
VCmax is constrained by the ATM NIC hardware (for
segmentation or reassembly instances), or by the VC capacity of
switch port that the NIC is attached to. VCmax will be the
of the two
A MARS Client may impose its own state storage limitations, such
the combined memory consumption of a MARS Client and the ATM NIC'
driver in a given host limits both LEAFmax and VCmax to values
than the ATM NIC alone might have been able to support
It may be possible to work around LEAFmax limits by distributing
leaf nodes across multiple pt-mpt SVCs operating in parallel
However, such an approach requires further study, and doesn't
the VCmax limitation associated with a node terminating too many VCs
A related observation can also be made that the number of
Clients in a Cluster may be limited by the memory constraints of
MARS itself. It is required to keep state on all the groups
every one of its MARS Clients have joined. For a given memory limit
the maximum number of MARS Clients must drop if the average number
groups joined per Client rises. Depending on the level of
memberships, this limitation may be more severe than LEAFmax
Armitage Informational [Page 4]
RFC 2121 Issues affecting MARS Cluster Size March 1997
3. Signaling load
In any given cluster there will be an 'ambient' level
MARS_JOIN/LEAVE activity. The dynamic characteristics of
activity will depend on the types of multicast applications
within the cluster. For a constant relative distribution of
applications we can assume that, as the number of MARS Clients in
given cluster rises, so does the ambient level of MARS_JOIN/
activity. This increases the average frequency with which the
processes and propagates MARS_JOIN/LEAVE messages
The existence of MARS_JOIN/LEAVE traffic also has a
impact on signaling activity at the ATM level (across the UNI
{P}NNI boundaries). For groups that are VC Mesh supported,
MARS_JOIN or MARS_LEAVE propagated on ClusterControlVC will result
an ADD_PARTY or DROP_PARTY message sent across the UNIs of all
Clients that are transmitting to a given group. As a cluster'
membership increases, so does the average number of MARS Clients
trigger ATM signaling activity in response to MARS_JOIN/LEAVEs
The size of a cluster needs to be chosen to provide some level
containment to this ambient level of MARS and UNI/NNI signaling
Some refinements to the MARS Client behaviour may also be explored
smooth out UNI signaling transients. MARS Clients are
required to initiate revalidation of group memberships only when
Client next sends a packet to an invalidated group SVC. A
could apply a similar algorithm to decide when it should
ADD_PARTYs. For example, after seeing a MARS_JOIN, wait until
actually has a packet to send, send the packet, then initiate
ADD_PARTY. As a result actively transmitting Clients would
their SVCs sooner than intermittently transmitting Clients
4. Group change
The group change latency can be defined as the time it takes for
the senders to a group to have correctly updated their
SVCs after a MARS_JOIN or MARS_LEAVE is received from the MARS.
is affected by both the number of Cluster members and
geographical distribution of Cluster members. (Groups that are
based create the lowest impact when new members join or leave,
only the MCS needs to update its forwarding SVC.) Under
circumstances, especially modelling or simulation environments,
change latencies within a cluster may be an important
to control
Armitage Informational [Page 5]
RFC 2121 Issues affecting MARS Cluster Size March 1997
As noted in the previous section, the ADD_PARTY/DROP_PARTY
load created by membership changes in VC Mesh based groups goes up
the number of cluster members rises (assuming worst case scenario
each cluster member being a sender to the group). As the UNI
rises, the ATM network itself may start delivering slower
of the requested events
Wide geographic distribution of Cluster members also delays
propagation of MARS_JOIN/LEAVE and ATM UNI/NNI messages. The
apart various members are, the longer it takes for them to
MARS_JOIN/LEAVE traffic on ClusterControlVC, and the longer it
for the ATM network to react to ADD_PARTY and DROP_PARTY requests.
the long distance paths are populated by many ATM switches
propagation delays due to per-switch processing will
substantially to delays due to the speed of light
(Unfortunately, mechanisms for smoothing out the transient
signaling load described in section 3 have a consequence
increasing the group change latency, since the goal is for some
the senders to deliberately delay updating their forwarding SVCs
This is an area where the system architect needs to make
situation-specific trade-off.)
It is not clear what affect the internal processing of the
itself has on group change latency, and how this might be impacted
cluster size. A component of the MARS processing latency will
on the specific database implementation and search algorithms as
as on the number of group members for the group being modified at
instant. Since the maximum number of group members for a given
is equal to the number of cluster members, there will be an
(even if small) relationship between worst case MARS
latencies and cluster size
5. Large IP/ATM networks using
Building a large scale, multicast capable IP over ATM network is
tradeoff between Cluster sizes and numbers of Mrouters. For a
number of hosts, the number of clusters goes up as
clusters shrink. Since Mrouters are the topological
between clusters, the number of Mrouters rises as the size
individual clusters shrinks. (The actual number of Mrouters
largely on the logical IP topology you choose to implement, since
single physical Mrouter may interconnect more than two Clusters
once.) It is a local deployment question as to what the optimal
of Clusters and Mrouters will be
Armitage Informational [Page 6]
RFC 2121 Issues affecting MARS Cluster Size March 1997
Currently two broad classes of Mrouters may be identified
Those that originate unique VCs into target Clusters,
forward/interleave data at the IP packet level (the
Mrouter).
Those that originate unique VCs into target Clusters, but
internal, cell level 'cut through' paths between VCs
different Clusters (e.g. the Cell Switch Router).
How these Mrouters establish and manage the associations of VCs to
traffic flows is beyond the scope of this document. However, it
worth looking briefly at their impact on VC consumption and
signaling load
5.1 Impact of the Conventional
A conventional Mrouter acts as an aggregation point for
signaling and data plane loads. It hides host specific
membership changes in one cluster from senders within other clusters
and protects group members (receivers) in one cluster from having
be leaf nodes on SVCs from senders in other Clusters
When acting as an ingress point into a cluster, a
Mrouter establishes a single forwarding SVC for IP packets.
single SVC carries data from other clusters interleaved at the
packet level. Only this single SVC needs to be modified in
to group memberships changes within the target cluster. As
consequence, there is no need for sources in other clusters to
aware of, or react to, MARS_JOIN/LEAVE traffic in the target cluster
(The consequential UNI signaling load identified in section 3 is
localized within the target Cluster.)
MARS Clients within the target cluster also benefit from this
path aggregation because they terminate only one SVC from the
(per group), rather than multiple SVCs originating from
senders in other Clusters
Conventional Mrouters help control the limiting factors described
sections 2, 3, and 4. A hypothetical 10000 node Cluster could
broken into two 5000 node Clusters, or four 2500 node Clusters, etc
to reduce VC consumption. Or you might have 200 nodes of the
10000 that are known to join and leave groups rapidly, whilst
other 9800 are fairly steady - so you deploy clusters of 200, 2500,
2500, 2500, 2300 hosts respectively
Armitage Informational [Page 7]
RFC 2121 Issues affecting MARS Cluster Size March 1997
5.2. Impact of the Cell Switch Router (CSR).
Another class of Mrouter, the Cell Switch Router (CSR) attempts
utilize IP level flow information to dynamically manage the
of data through the device below the IP level. Once the CSR
identified a flow of IP traffic, and associated it with an
and outbound SVC, it begins to function as an ATM cell level
rather than a packet level device
Even when operating in this mode the CSR isolates attached
from each other's MARS_JOIN/LEAVE activities, in the same manner as
conventional Mrouter. This occurs because the CSR manages
forwarding SVCs just like a normal MARS Client - responding
MARS_JOIN/LEAVE messages within the target cluster by updating
pt-mpt trees rooted on its own ATM ports
However, since AAL5 AAL_SDUs cannot be interleaved at the cell
on a single SVC, a CSR cannot simultaneously perform cell level cut
through and aggregate the IP packet flows from multiple senders
a single SVC into a target Cluster. As a result, the CSR
construct a separate forwarding SVC into a target cluster for
SVC it is a leaf of in a source Cluster (to to ensure that cells
individual sources are not interleaved prior to reaching the re
assembly engines of the group members in the target cluster).
Interestingly, the UNI signaling load offered within the
Cluster by the CSR is potentially greater than that of a
Mrouter. If there are N senders in the source Cluster, the CSR
have built N identical pt-mpt SVCs out to the group members
the target Cluster. If a new MARS_JOIN is issued within the
Cluster, the CSR must issue N ADD_PARTYs to update the N SVCs
the target Cluster. (Under similar circumstances a
Mrouter would have issued only one ADD_PARTY for its single SVC
the target Cluster.)
Thus, without the ability to provide internal cut-through
with AAL_SDU boundaries intact, the CSR only provides for
isolation of MARS_JOIN/LEAVE traffic within clusters. It
provide the data path aggregation of a conventional Mrouter
Armitage Informational [Page 8]
RFC 2121 Issues affecting MARS Cluster Size March 1997
6. The impact of Multicast Servers (MCSs
Since the focus of this document is on worst-case scenarios, most
the analysis has assumed multicast groups that are VC Mesh based
have all cluster members as sources and receivers. The impact
using an MCS to support a multicast group can be dramatic in
context of the group's resource consumption, but less so in
over-all context of cluster size limits
The intra-cluster, per group impact of an MCS is somewhat
to the inter-cluster impact of a conventional Mrouter. The
aggregates the data flows (only 1 SVC terminates on each
member, independent of the number of senders), and
MARS_JOIN/LEAVE traffic (which is shifted to ServerControlVC
than ClusterControlVC). The resulting UNI signaling traffic and
is reduced too, as only the forwarding SVC out of the MCS needs to
modified for every membership change in the MCS supported group
Deploying a mixture of MCS and VC Mesh based groups will
improve resource utilization. However, the actual extent of
improvements (and consequently how large the cluster can be made
will depend greatly on the dynamics of your typical applications
which characteristics from sections 2, 3, and 4 are your
limitations
For example, if VCmax or LEAFmax (section 2) are primary limitations
one must keep in mind that each MCS itself suffers the same
limits as the MARS and MARS Clients. Even though using an
dramatically reduces the number of VCs per MARS Client per group
each MCS still needs to terminate 1 SVC per sender - potentially
to 1 SVC from each Cluster member. (This may become 1 SVC per
per group if the MCS supports multiple groups simultaneously.)
Assume we have a Cluster where every group is MCS based, each
supports only one group, and both VCmax and LEAFmax apply equally
MCS nodes as MARS and MARS Clients nodes. If we have N
members, M groups, and all receivers are senders for a given
supported group, the following observations may be made
Each MCS forwarding SVC has N leaf nodes,
N <= LEAFmax
Each MCS terminates an SVC from N senders, originates 1
forwarding path, originates a pt-pt control SVC to the MARS,
terminates 1 SVC as a leaf on ServerControlVC,
N + 3 <= VCmax
Armitage Informational [Page 9]
RFC 2121 Issues affecting MARS Cluster Size March 1997
MARS ClusterControlVC has N leaf nodes,
N <= LEAFmax
MARS ServerControlVC has M leaf nodes,
M <= LEAFmax
The MARS terminates a pt-pt VC from each cluster member, a pt-
VC from each MCS, originates ClusterControlVC, and
ServerControlVC,
N + M + 2 <= VCmax
Each Cluster Member sources 1 VC per group, terminates 1 VC
group, originates a pt-pt VC to the MARS, and terminates 1 VC as
leaf on ClusterControlVC,
2*M + 2 <= VCmax
Since all the above conditions must be simultaneously true, we
see that the most constraining requirements are
N + M + 2 <= VCmax (if M <= N
2*M + 2 <= VCmax (if M >= N
N <= LEAFmax
(Assuming that in general M+2 > 3, so the VCmax constraint at
MCS is not a limiting factor.)
We can get a feel for the relative impacts of VC Mesh groups vs
based groups by considering a cluster where M1 represents the
of VC Mesh based groups, and M2 represents the number of MCS
groups. Again we assume worst case group density (all N
members are group members, all receivers are also senders).
As noted in section 2, the VCmax constraint in VC Mesh mode
from each MARS Client, and is
N*M1 <= VCmax - 2
For the MCS case we have two scenarios, M2 <= N and M2 >= N
If M2 <= N we can see the VC consumption by VC Mesh based groups
become the applicable constraint on cluster size N when
N + M2 <= N*M
i.e
M1 >= 1 + (M2/N
Armitage Informational [Page 10]
RFC 2121 Issues affecting MARS Cluster Size March 1997
Thus, if there is more than 1 VC Mesh based group, and less MCS
groups than cluster members (M2 < N), the constraint on cluster
is dictated by the VC Mesh characteristics: N*M1 <= VCmax - 2. (If M
== N, then there may be 2 VC Mesh based groups before the VC
characteristics are the dictating factor.)
Now, if M2 > N (more MCS based groups, and hence MCSes, than
members) the calculation is more complex since in this case VCmax
the MARS Client is the limiting parameter for both VC Mesh and
cases. The limit becomes
N*M1 + 2*M2 <= VCmax - 2
However, on face value this is an odd situation anyway, since
implies more MCS entities than hosts or router interfaces into
cluster (given the assumption of one group per MCS).
The impact of MCS entities that simultaneously support
groups is left for future study
7. Open
There is a wide range of qualitative analysis that can be
from typical MARS deployment scenarios. This document does
attempt to develop any numerical models for VC consumptions, end
end latencies, etc
8.
This document has provided a high level, qualitative overview of
parameters affecting the size of MARS Clusters. Limitations on
number of leaf nodes a pt-mpt SVC may support, sizes of the
database, propagation delays of MARS and UNI messages, and
frequency of MARS and UNI control messages are all identified
issues that will constrain Clusters. Conventional Mrouters
identified as useful aggregators of IP multicast traffic
signaling information. Cell Switch Routers are noted to offer
some of the aggregation attributes of conventional Mrouters.
scale IP multicasting over ATM requires a combination of Mrouters
appropriately sized MARS Clusters. Finally, it has been shown that
a simple cluster where there are less MCS based groups than
members, two or more VC Mesh based groups are sufficient to
the use of Multicast Servers irrelevant to the worst case
size limit
Armitage Informational [Page 11]
RFC 2121 Issues affecting MARS Cluster Size March 1997
Security
Security issues are not discussed in this memo
Thanks must go to Rajesh Talpade (Georgia Tech) for specific input
aspects of the VC Mesh vs MCS tradeoffs, and Joel Halpern (Newbridge
for general input on the document's focus
Author's
Grenville
Bellcore, 445 South
Morristown, NJ, 07960
EMail: gja@thumper.bellcore.
Phone +1 201 829 2635
[1] Armitage, G., "Support for Multicast over UNI 3.0/3.1 based
Networks.", Bellcore, RFC 2022, November 1996.
Armitage Informational [Page 12]
if you see any problems within the linking, don't worry be happy,
this is version 0.1 of the Relevance System and you gotta expect some crappy subroutines sometimes,
just be content we did not write this in Java, which would have made this "bigger and better" HAHAHHA.
RFC documents can be found at I.E.T.F.
Relevance System Copyright © 2002 Spectrum WorldResearch
other technical nosh by ServerMasters Corporation
collaboration of BobX
