Computer Science Technical Report Archive

USC Computer Science Technical Reports, no. 631 (1996)

(USC DC Other)

USC Computer Science Technical Reports, no. 631 (1996)

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Persistent Route Oscillations in Inter-Domain Routing
Kannan Varadhan
Ramesh Govindan
Deborah Estrin
USC/Information Sciences Institute
Marina Del Rey, CA 90292.
{kannan,govindan,estrin}@isi.edu
February 6, 1996
Abstract
Hop-by-hop inter-domain routing protocols, such as BGP and IDRP, use independent route selection to realize domains’
local policies. A domain chooses its routes based on path attributes present in a route. It is widely believed that these inter-
domain routing protocols always converge. We show that there exist domain policies that cause BGP/IDRP to exhibit persistent
oscillations. In these oscillations, each domain repeatedly chooses a sequence of routes to a destination. Complex oscillation
patterns can occur even in very simple topologies. We analyze the conditions for persistent route oscillations in a simple class
of inter-domain topologies and policies. Using this analysis, we evaluate ways to prevent or avoid persistent oscillations in
general topologies.
We conclude that if a hop-by-hop inter-domain routing protocol allows unconstrained route selection at a domain, the
protocol may be susceptible to route oscillations. Constraining route selection to a provably “safe” procedure (such as shortest
path) can reduce the number of realizable policies. Alternatively, a routing policy registry can help detect unsafe policies.
Explicit routing can then be used to realize desired policies (i.e., make routes available) that hop by hop routing cannot safely
advertise.
1 Introduction
Internet resources, such as hosts, routers and transmission facilities, are partitioned into different administrative domains.In
general, domains fall into two categories: subscribers and providers. A university campus network or a corporate internal
network is an example of a subscriber domain. Provider domains facilitate data exchange between subscriber domains. For
economic reasons, a provider domain may wish to allow only certain classes of transit traffic to traverse its facilities. Similarly,
a subscriber domain may prefer to route its traffic through a designated provider (e.g, a national backbone).
In a hop-by-hop routing infrastructure, such policies can be realized by selective dissemination of routing information.
Both the Border Gateway Protocol version 4 (BGP [21], the widely deployed Internet standard for inter-domain routing) and
the Inter-Domain Routing Protocol (IDRP [13]) provide this functionality. BGP and IDRP are sometimes called path-vector
protocols, after the routing loop suppression mechanism they use.
BGP and IDRP also share another characteristic—they both use a similar distributed routing algorithm for hop-by-hop rout-
ing. We have coined the phrase path-attribute based, independent, route selection (PAIRS)
1
to describe this type of distributed
This work was supported by the National Science Foundation under Cooperative Agreement NCR-9321043. The work of K. Varadhan and D. Estrin was
supported by the National Science Foundation under contract number NCR-92-06418. Systems research at USC is supported through NSF infrastructure grant,
award number CDA-9216321. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the National Science Foundation.
1
Even though BGP and IDRP are path-vector protocols, the routing behavior described in this paper is not caused by the loop suppression mechanism they
1
route computation. A simplified description of PAIRS follows. Each domain receives one or more routes from each of its
neighbors. A route indicates its sender’s reachability to an address prefix (a network-layer address aggregate). Each route also
contains one or more path attributes. For each received loop-free route to a given address prefix, the domain first computes an
integer preference, then selects the route with the highest preference. Route preference assignment reflects domains’ policies.
The preference function takes as input a route’s path attributes. However, domains can independently choose their preference
functions.
It is widely believed that this distributed route computation converges, regardless of the preference functions at participating
domains [19]. In this paper, we demonstrate the contrary. Specifically, we show that there exist domain preference functions
for which PAIRS exhibits persistent route oscillations, even in the absence of topology changes. In these oscillations, each
domain in a cycle of domains repeatedly selects the same sequence of routes, never converging on a single route.
The rest of the paper is organized as follows. In Section 2, we describe inter-domain topologies and preference functions
for which PAIRS exhibits persistent route oscillations. We show that these oscillations may be attributed to route “feedback”,
caused by inter-dependent domain preference functions. By appropriately configuring a public-domain BGP implementation
with such preference functions, we have re-created these oscillations in our laboratory testbed. However, despite the widespread
deployment of BGP in the Internet, there is no anecdotal evidence of observed route oscillations of the form discussed in this
paper. Existing provider policies are safe probably because the commercial Internet infrastructure is still in its infancy—
therefore, the range of policies currently expressed is still limited. We think conditions for route oscillations are more likely to
occur as the commercial Internet matures, and as the Internet transitions to the more expressive IDRP.
In Section 3, we study these oscillations in a restricted class of inter-domain topologies. For these topologies, we describe
a representation of domain preference functions that we call return graphs. Using this representation, we derive necessary and
sufficient conditions for the existence of route oscillations in these topologies. Our derivation shows that these oscillations can
happen in relatively complex ways even in simple topologies.
The existence of route oscillations in inter-domain routing points to a routing protocol design failure. In Section 4, we
show that constraining PAIRS to consider only a small “safe” subset of path attributes can significantly reduce the number of
policies realizable in those protocols. Not surprisingly, perhaps, realizing richer policies through independent route selection
can adversely affect route convergence in PAIRS.
However, in the existing commercial Internet infrastructure, mechanisms to realize policy through independent route selec-
tion are already widely deployed. In this situation, a combination of the following two approaches can be adopted (Section 5).
The first approach analyzes domain policies a priori to detect the likelihood of route oscillations; one or more domain policies
can then be modified to avoid oscillations. A routing policy registry (such as the Internet Routing Registry [3, 1]) is useful for
this. The second approach introduces additional protocol mechanisms that detect the existence of an oscillation, and modify
one or more domains’ policies to suppress the oscillation. Unsafe policies can then be realized using explicit routes [8, 10].
use. If a routing protocol were to use a different loop suppression mechanism [6], but allow independent route selection, it would also be susceptible to the
routing behavior we describe here. To focus on the independent route selection aspect of these protocols, we classify them as PAIRS protocols.
2
2 Examples and Motivation
In this section, we show how persistent route oscillations can result from PAIRS route computation. We first introduce a simple
model of PAIRS route computation at each participating domain D. In this paper, we assume that all references to routes
pertain to address prefix x, unless otherwise stated. Domain D maintains the last route advertisement to x heard from each of
its neighbors. D also maintains the last route r to x that it advertised. Suppose D hears a new route advertisement for x from its
neighbor. It assigns a preference to this route and recomputes the most preferred route to x. If this route is different from r, D
propagates this route. PAIRS route computation is said to converge if no further route advertisements are heard at participating
domains.
Even in a relatively small inter-domain topology, PAIRS can exhibit persistent route oscillations (i.e., non-convergence)
for routes to x. Consider three domains D
, D
, and D
connected together as shown in Figure 1. Suppose that domain D
(respectively D
and D
) has a “direct” route r
(respectively r
and r
) to the destination. With the preference functions shown
in Figure 1, the PAIRS algorithm exhibits persistent route oscillations (Figure 2). Intuitively, this is because the policies of the
three domains are not simultaneously satisfiable.
D
D
D
In this figure, each domain is represented by a circle. The table below represents the route selection
functions for each of the domains D , D , and D . The entries in each column are the routes that a
domain will select when it receives a route corresponding to that column.
r
r
r
D
r
r
D
r
r
D
r
r
Notice that the preference functions are inter-dependent: D ’s most preferred route is r , D ’s most
preferred route is r , and D ’s most preferred route is r . Also, D will never select r , D will never
select r , and D will never select r .
Figure 1: Example of a cyclic domain policy that leads to non-convergence in PAIRS
The preference functions of Figure 1 are not simply based on the identity of the domain that advertises a route; for instance,
even though D
hears r
and r
from D
, it selects one and not the other. D
’s preference functions express its policies
regarding D
and D
. It is not unusual for a provider to specify such a policy in the existing Internet. We think that inter-
dependent policies similar to that in Figure 1 are not unlikely in the future Internet.
In Figure 1, each domain alternates between two routes—its own “direct” route and that of its anti-clockwise neighbor. At
3
D
D
D
r
r
r
(a) State 0
D
D
D
r
r
r
(b) State 1
D
D
D
r
r
r
(c) State 2
D
D
D
r
r
r
(d) State 3
D
D
D
r
r
r
(e) State 1
An arrow indicates the selected route to x at a domain.
State 0 In the initial state, each of the three domains has a route to x.
Without loss of generality, assume that D advertises r initially.
State 1 D selects r ;but D prefers r .
State 2 When D advertises r , D selects r ;but D prefers r .
State 3 When D advertises r , D selects r ;but D prefers r .
State 1 D advertises r , and the sequence repeats.
Figure 2: States of route oscillations for the topology of Figure 1
some instant t, a domain can have selected exactly one of these routes. The selected route defines that domain’s route state (or
r-state)at t. If a domain is in an r-state r at t, it must have last advertised r. When a domain receives a route advertisement,
its r-state may change. At different times, a domain can be in different r-states. The r-states at a domain are determined by its
neighbors’ r-states, and its own preference functions. For example, D
has exactly two states: r
and r
. D
never selects r
,
since the direct route r
is always available.
From Figure 1, we can make the following observations. (1) These persistent route oscillations occur in the absence of
topology changes. (2) This topology oscillates regardless of route processing times and route propagation delays at the three
domains. (3) The lack of a global metric in PAIRS causes each domain to oscillate between loop-free paths. (4) If packet
forwarding is synchronized with route exchange, packets could loop indefinitely. For example, in state 1 of Figure 2, D
could
forward a packet towards D
. D
could be in state 2 when it forwards that packet towards D
, and D
could switch to state
3 before forwarding the packet back to D
. (5) Independent of the initial r-states of the domains, PAIRS always exhibits
persistent oscillations in this topology.
There exist preference functions that cause PAIRS to behave differently for different initial r-states. Figure 3 shows prefer-
ence functions for a cycle of four domains. PAIRS converges in this topology for a particular assignment of initial states. With
other initial r-states, the topology exhibits two different kinds of oscillations. In one of them, D
repeatedly selects r
and r
.
In the other, D
repeatedly oscillates among r
, r
, and r
. In this example, PAIRS can exhibit a persistent route oscillation
despite the existence of a stable route assignment.
4
D
D
D
D
r
r
r
r
D
r
r
r
D
r
r
r
D
r
r
r
D
r
r
r
(a) Preference functions
D
D
D
D
r
r
r
r
(b) Initial r-states for
D
to oscillate as
hr
r
r
i
D
D
D
D
r
r
r
r
(c) A possible stable
assignment
D
D
D
D
r
r
r
r
(d) Initial r-states for
D
to oscillate as
hr
r
r
r
r
i
3(a) This figure shows preference functions at four domains. Each domain has a direct route.
3(b) These initial r-states can be realized, for example, if D ’s advertisement reaches D , D , and
D before those domains have processed their direct route.
3(c) If D ’s advertisement for r reaches all other domains first, these r-states result.
3(d) If D and D select their direct route, and the advertisement for r reaches D and D , these
initial r-states are achieved.
Figure 3: PAIRS behavior with four nodes
What causes the oscillations described in Figures 1 and 3? In Figure 1, observe that a domain’s r-state can “feedback” into
another, possibly different, r-state. Informally, when D
advertises r
, D
transitions to r-state r
, as its preference function
dictates. Then, D
’s advertisement of r
causes D
to select and advertise r
. This advertisement causes D
to select r
.We
say that r-state r
returnsto r
at D
. Intuitively, route oscillations happen in Figure 1 because there exists a cycle of returnsat
D
: r
returns to r
, and r
returns to r
. In Figure 3, D
has two such cycles, in one of which r
returns to itself.
3 Characterizing Route Oscillations in Simple Topologies
In this section, we attempt to analyze persistent route oscillations in a particular class of inter-domain topologies. In these
topologies, domain preference functions can be represented as return graphs, based on the notion of return states. We derive
necessary and sufficient conditions on return graphs for the existence of route oscillations in these topologies.
Table 1 summarizes the various terms introduced in this paper.
5
D A cyclic inter-domain topology in which no domain selects routes from its clockwise neighbor.
D i A domain in D. Domains in D are numbered with integer subscripts. D i is D i’s clockwise neigh-
bor.
r i D i ’s direct route. We assume that this route is always present as a fall-back route.
pref
i
A function that takes two routes, and returns the route that has a higher preference at D i .
r-state At any instant t, the route selected by a domain. At different instants, a domain can select different
routes.
R i The collection of possible r-statesof D i.
return relation We say r a returnsto r
b
at D i , if when D i advertises r a, route feedback causes D i to select r
b
.
return graph For each D i , the directed graph whose nodes are the r-statesin R i and whose arcs express the return
relationships between those states.
G The return graph at D . G i is the return graph for any D i in D.
return cycle A cycle in a component of the return graph. Every component in D has exactly one cycle.
C A single cycle in G . C i denotes the return cycle isomorphic to C in G i .
Table 1: Glossary of terminology and notation
D
n D
n D
D
D
r
r
r
... r
n r
n D
r
r
r
n r
n D
r
r
r
n r
n D
r
r
r
n r
n .
.
.
.
.
.
D
n r
r
r
r
n D
n r
n r
r
r
n This figure shows n domains, each with a direct route, and the table of domain preference functions.
Each domain prefers its anti-clockwise neighbor’s direct route more than its own. D oscillates as
hr r n r n r r i, if some domain initially advertises its direct route.
Figure 4: A persistent route oscillation among n domains
3.1 Assumptions and Problem Statement
Suppose that the three domains of Figure 1 were part of a larger inter-domain topology. Depending on the policies of adjacent
domains, the oscillations at these three domains could affect a number of other domains, perhaps triggering other “sympathetic”
oscillations. Visualizing, and reasoning about, these complex oscillation patterns in general topologies is difficult.
For this reason, we consider a more restricted class of topologies which exhibit route oscillations. Informally, the kind
of route feedback described in the previous section cannot happen in acyclic topologies. So, we consider a class of simple
cyclic topologies, which we call D. D contains n domains D
, D
, ..., D
n . Each domain D
i
peers with D
i mo d n
and
D
i mo d n
(respectively notated D
i and D
i ). Without loss of generality, assume D
i is D
i
’s clockwise neighbor.
Assume further that, each domain D
i
has a direct route r
i
that is always available as a fall-back. Figure 4 describes preference
functions for a persistent route oscillation in D. In this oscillation, each domain repeatedly selects n r-states.
6
In this paper, we study those route oscillations in D that occur in the absence of topology changes and are independent
of route computation times and route processing speeds. We say a D
i
oscillates if it repeatedly selects a sequence of r-states
r
a
r
b
r
x
. One of these r-states can be r
i
. The other r-states must correspond to routes heard from D
i , D
i , or both.
Here, we consider those oscillations in which D
i
’s r-states correspond either to r
i
or to routes heard from D
i . That is, we
restrict the class of preference functions in D to those in which a D
i
never selects a route from D
i . Our analysis also applies
to oscillations in which D
i
selects either r
i
or routes from D
i . Section 3.5 discusses the likelihood of oscillations in which
D
i
’s r-states include routes from both D
i and D
i .
With these assumptions, we attempt to answer the following two questions:
Among the class of preference functions we consider, which ones can cause route oscillations in D?
For given preference functions in D, what are the different ways (if any) in which D can oscillate?
One possible answer to these questions is suggested by the following approach. If we represent the current state of D
by a vector of r-states, we can represent the next state of D as a product of a state transformation matrix and the current
state of D. This transformation matrix is determined by the given preference functions. Conditions on the eigenvalues of this
matrix determine whether D can oscillate or not [18]. In this paper, we describe an alternative representation of D’s preference
functions, which we call a return graph. The choice of this term was motivated by the roughly analogous control-theoretic
notion of a first return map (sometimes also called a Poincaré map) [11].
3.2 Return Graphs
In Section 2, we informally introduced the notion of a domain’s r-state. At any instant t, the r-state of a domain is the route it
has selected. In D, the r-statesof D
i
can include r
i
and some other domains’ direct routes heard from D
i . A direct route r
j
is an r-state of D
i
if and only if, when D
j
advertises r
j
, all domains between D
j
and D
i
(going clockwise in D, D
i
inclusive),
select that route. Denote the preference function at D
i
by pref
i
; pref
i
r
a
r
b
is the more preferred of r
a
and r
b
at D
i
. Formally,
r
j
is an r-state of D
i
if and only if: pref
k
r
j
r
k
is r
j
, for all k in j i i . Thus, the set of possible r-statesof D
i
(denoted by R
i
) can be determined entirely from domain preference functions.
In Section 2, we also introduced the returns relation between two states. We said that, at D
i
, r
a
returnsto r
b
if, when
D
i
advertises r
a
, route feedback causes D
i
to transition to r
b
. Equivalently, the returns relation is can be defined in terms of
domain preference functions in D. Suppose that r
a
and r
b
are two r-statesat D
i
. Then, r
a
returnsto r
b
at D
i
if and only if:
r
b
pref
i
pref
i pref
i pref
i r
a
r
i r
i r
i r
i
(1)
That is, when D
i
selects r
a
, D
i selects pref
i r
a
r
i , D
i selects pref
i pref
i r
a
r
i r
i , and so on, exactly
once around D.
7
Given the preference functions in D, we can define at D
i
a directed return graph G
i
, whose nodes are the r-statesinR
i
. G
i
has a directed arc from r
a
to r
b
if and only if r
a
returnsto r
b
. Figure 5 shows the return graph for the example in Figure 3.
This collection of return graphs is an alternative representation of the preference functions shown in Figure 3.
D
D
D
D
r
r
r
r
r
r
r
r
r
r
r
r
In this topology, the return graph corresponding to each domain is shown adjacent to that domain. Each
return graph has two components. One component is a cycle consisting of two nodes. The second
component is a cycle consisting of one node.
Figure 5: Return graphs for the topology of Figure 3(a)
3.3 Properties of Return Graphs
We can make several general observations about return graphs. Equation 1 implies that each node in a return graph has exactly
one outgoing arc. Such a directed graph has a well-defined structure; it may be disconnected, and each connected component
generally contains one or more chains “leading into” exactly one cycle. (For, if a connected component contained two cycles,
some node in that component must have two outgoing arcs). A cycle in a return graph may have one or more nodes (Figure 5).
A one-node cycle corresponds to an r-state that returns to itself.
Cycles in a return graph G
i
have several interesting properties.
1. Since every node in a return graph has exactly one outgoing arc, a node in G
i
can be in at most one cycle. Moreover, the
directed path leading out from any node in a return graph eventually leads into a cycle.
2. A one-node cycle corresponds to a stable route assignment. That is, if r
a
returnsto r
a
at D
i
, then the following is a stable
route assignment in D: r
a
at D
i
, pref
i rr
i at D
i , and so on.
3. From the previous property, it is trivially true that for a one-node cycle in G
i
, there exists a “corresponding” one-node
cycle in G
i . If there exists a two-node cycle in G
i
, there exists a “corresponding” two-node cycle in G
i . To see this,
suppose that r
a
and r
b
constitute the two-node cycle in G
i
. Clearly the states pref
i r
a
r
i and pref
i r
b
r
i (say r
c
and r
d
respectively) must be in R
i .If r
a
returnsto r
b
in G
i
, then r
c
returnsto r
d
in G
i . Conversely, if r
b
8
returnsto r
a
in G
i
, then r
d
returns to r
c
in G
i . Finally, r
c
and r
d
cannot be identical; if they were, r
a
and r
b
must
also be identical. Extending this argument, if there exists a k-node cycle in G
i
, there exists a k-node cycle in every other
domain’s return graph. Since a node in G
i
can be in at most one cycle, the cycles in other domains’ return graphs are
isomorphic to the cycles in G
i
.
From Property 3, cycles in G
are representative of cycles in all G
i
.
In D, one of the r-states of every domain D
i
is its direct route r
i
. This r-state must lead into some cycle in G
i
(from
Property 1). That cycle corresponds to a cycle (call it C)in G
. We say that r
i
can activate C. Thus, in Figure 3(c), r
can
activate the one-node cycle. Intuitively, if only D
i
were to advertise r
i
initially, cycle C would eventually be realized. More
than one direct route can activate the same cycle. In Figure 3(b), any one of r
, r
,or r
can activate the two-node cycle. The
collection of initially activated cycles defines the initial r-states of domains in D.
3.4 Persistent Route Oscillations in D
In this section, we describe necessary and sufficient conditions on cycles in G
for the existence of persistent route oscillations
in D. Earlier, we said D
i
oscillates if it repeatedly visits the same sequence of k r-states, for some k. We say an oscillation
exists in D if at least one domain in D oscillates. In D,if D
i
oscillates among k r-states, then D
i must oscillate among at least
k r-states (any state transition in D
i
can only be triggered by a state transition in D
i , since r
i
is always available and, by our
assumption, D
i
never selects a route advertised by D
i ). Actually, D
i must oscillate among exactly k routes. Otherwise,
D
i and all other domains in D, including D
i must oscillate among more than k routes. This is a contradiction. We call the
smallest repeated sequence of r-statesat D
i
its period.
Intuitively, if G
has a multi-node cycle C, D can exhibit persistent route oscillations. This happens if r
i
can activate C, and
D
i
initially advertises r
i
. Thus, there exists an oscillation in the topology of Figure 1 if D
initially advertises r
.
Less obviously, two (or more) one-node cycles can also cause oscillations in D. For example, Figure 6 shows a topology in
D. In this topology, G
has three one-node cycles. There exists an oscillation in this topology if D
and D
initially advertise
r
and r
respectively.
The following theorem formalizes these two observations.
Theorem 1 D can exhibit persistent route oscillations if and only if either G
has at least one k-node cycle (k ), or G
has
more than one one-node cycle.
Proof We now sketch a proof for Theorem 1. The proof sketch has two parts. In the first part, we show that if G
has exactly
one one-node cycle, D cannot oscillate. In the second part, we show initial conditions for an oscillation in D if G
has one
k-node cycle (k ), or G
has more than one one-node cycle.
1. Suppose G
has exactly one one-node cycle. Then, since the cycles in G
are representative of cycles in other domains’
9
D
D
D
r
r
r
r
r
r
D
r
r
D
r
r
D
r
r
D
D
D
r
r
r
(b) State 1
D
D
D
r
r
r
(c) State 2
D
D
D
r
r
r
(d) State 3
This figure shows the preference functions at D . The figure also shows G , which contains three one-
node cycles. There exists an oscillation in this topology if at least two of these three cycles are initially
activated. The figure also shows the three states of the oscillation, when r and r are initially activated.
Figure 6: Multiple one-node cycles can cause an oscillation
return graphs, all other G
i
must have exactly one one-node cycle. Assume to the contrary that D exhibits a persistent
route oscillation. Suppose that D
i
’s period has two routes r
a
and r
b
(the proof for the case when D
i
’s period has k routes
is similar). Now, in G
i
, the directed path out of r
a
must lead into a cycle (from Property 1). The same is true for r
b
.
Suppose r
c
constitutes the one-node cycle in G
i
.
If r
c
is different from both r
a
and r
b
, then when D
i
advertises either r
a
or r
b
, it will eventually attain r-state r
c
. But that
is a contradiction, since we assume that D
i
repeatedly selects only r
a
and r
b
.
If r
c
is r
b
, then advertisement of both r
a
and r
b
by D
i
results in an eventual transition into r
b
(i.e., r
a
cannot recur in the
sequence), a contradiction.
A similar contradiction occurs if r
c
is r
a
.
2. Suppose G
has one k-node cycle. Without loss of generality, assume that r
j
can activate this cycle. Then, a start state
in which only r
j
is initially advertised will result in persistent route oscillations. A period of the oscillation at each D
i
contains the r-states of the k-node cycle. This follows from the definition of the returns relation. Suppose G
has two
one-node cycles. Without loss of generality, assume that r
i
and r
j
can activate these two cycles. An initial state in which
only r
i
and r
j
’s advertisements initially traverse D will result in an oscillation whose period contains the r-states in their
two one-node cycles.
10
D
D
D
D
D
D
r
r
r
r
r
r
D
r
r
r
r
D
r
r
r
r
D
r
r
r
r
D
r
r
r
r
D
r
r
r
r
D
r
r
r
r
r
r
r
r
This figure shows a six domain topology in D. There are two two-node cycles in G .If r and r are
initially advertised, the resulting oscillation has the following period at D : hr r r r i. However, if
r and r are initially advertised, the resulting oscillation has the following period at D : hr r r r i.
Figure 7: Different oscillation periods for different initial conditions
3.5 Discussion
In this section, we discuss the implications of the conditions for the existence of an oscillation in D. We also consider the effect
of relaxing some of our assumptions about the topology and the preference functions.
Given a set of preference functions in D, we can use Theorem 1 to determine the different ways in which domains in D
can oscillate. The theorem describes two ways: when either a single multi-node cycle, or two one-node cycles are initially
activated. Oscillations with more complex periods are possible. Suppose r
a
and r
b
activate two different cycles C
a
and C
b
.
The period of the resulting oscillation contains the r-statesof C
a
and C
b
. The oscillation of Figure 3(d) is an example of this.
However, if r
a
and r
b
activate C, it is possible for the period of the oscillation to contain two instances of each r-state in C.
When two or more cycles are initially activated, the order of the r-states in a period of the oscillation depends on the routes
used to activate the cycles. Figure 7 demonstrates this.
If G
has one multi-node cycle, only one cycle need be activated to cause route oscillations. If G
has only multi-node
cycles, it follows from Property 1 above that any initial state leads to route oscillations. This is the case with Figure 1; there
exists no stable route assignment in D. We call such return graphs unsatisfiable. PAIRS admits preference functions which can
result in unsatisfiable return graphs.
We considered a particular kind of oscillation, one in which route advertisements “flow” clockwise around D.In D, can D
i
’s
r-states include routes advertised both by D
i and D
i ? We have not been able to construct examples of such oscillations
without assuming some temporal ordering on each domain’s route selection policies. Intuitively, if a period of the oscillation
at D
i
includes routes from both D
i and D
i , then varying route propagation delays can perturb the order of r-states within
a period of the oscillation. For this reason, we believe that if D oscillates independent of route processing times and route
propagation delays, the oscillations must either be clockwise or anti-clockwise.
11
We also believe that if a more general topology oscillates independent of topology changes, there must exist at least one
cycle of domains that oscillates in a clockwise or anti-clockwise manner. As we have said before, in a more general topology,
other domains may exhibit sympathetic oscillations.
To analytically examine route oscillations, we considered a constrained class of topologies. Are return graphs applicable
in more general topologies? Obviously, our analysis applies to those sub-graphs of the more general topologies that satisfy the
requirements for D. However, in D an r-state r’s return state was uniquely defined. In a general topology, more than one return
state is possible for a given r at D
i
. Whether it is possible to derive conditions for the existence of an oscillation in these more
general return graphs is left for future study.
4 Constraining PAIRS
In the previous section, we showed preference functions for which PAIRS can exhibit persistent route oscillations. We now
consider constraining PAIRS to allow preference functions expressed in terms of a path attributeX. Such preference functions
are safe if they do not cause oscillations in a general topology. We consider the question: Do there exist safe preference
functions on X? If there exists an X such that preference functions on X allow “interesting” policies, constraining PAIRS to
these preference functions is an acceptable solution to the oscillation problem.
We believe that if preference functions onX are safe in D, then they are safe in more general topologies. Put differently,
if preference functions based on X can result in a cycle of oscillating domains in a general topology, we can construct an
oscillation in D caused by preference functions based onX. Intuitively, this construction simply “extracts” the cycle of domains
and their preference functions from the more general topology.
To show that preference functions based onX are safe in D, it suffices to show that the equivalent return graphs can contain
exactly one one-node cycle. As we have discussed earlier, routes in BGP and IDRP carry aPATH attribute—this is a sequence
of domains that the route has traversed. In this section, we consider two possible preference functions based on the PATH
attribute. We show that if all domains are constrained to selecting the shortestPATH route, oscillations cannot happen in D.We
also show that if domains were allowed to independently select routes based on the first element of thePATH only (next-hop),
multi-node cycles cannot form in a domain’s return graph.
4.1 ShortestPATH
In Figure 1, at least one domain’s r-states contains a route with aPATH longer than its direct route. Denote by l r
thePATH
length of r
at D
. If each domain always selected its shortest path route, then l r
i
l r
i , i.e., l r
i
l r
i . Putting
these inequalities together, we arrive at a contradiction.
This observation motivates considering shortest path route selection to realize safe policies. We can show that this preference
function is always safe in D.If r
i
is D
i
’s shortest path route, then any route that D
i
selects and advertises will return to r
i
.
12
Therefore there is only one one-node cycle at G
i
, and by extension, there is only one one-node cycle in G
. Therefore, D will
not oscillate if every domain uses shortest path route selection. We believe that shortest path route selection will not cause
oscillations in other more general topologies.
This is not a new result. We know from [24, 15] that a distance vector hop-by-hop algorithm augmented with a loop
suppression mechanism always converges, i.e., never oscillates. This algorithm is similar to a PAIRS algorithm constrained to
only select shortestPATH routes [22, 6].
4.2 Next-hop
D
in Figure 1 advertises r
and r
to D
. By looking at the entirePATH of those routes, D
only selects r
and not r
.If
D
’s preference functions are based only on the first element of thePATH, i.e., the next-hop, then D
cannot assign different
preferences to r
and r
.
This observation motivates considering next-hop based preference functions to realize safe policies. However, domain
preference functions in Figure 6 are based on next-hop; we have shown that this topology is susceptible to a route oscillation
for certain initial r-states.
Next-hop based functions cannot result in multi-node cycles in D. Consider route preference functions expressed only on
the next-hop. Each D
i
has two possible choices: it prefers no route advertised by D
i , or it prefers every route advertised
by D
i . If any one D
i
always chooses r
i
regardless of any route advertised by its neighbor, i.e., r
i
returnsto r
i
in G
, G
contains exactly one one-node cycle. If all domains D
i
prefer routes advertised by their neighbors D
i , G
has n one-node
cycles, one corresponding to each r
i
.
Next-hop based preference functions have a possible stable assignment in D. In the next section, we show how next-hop
based policies may be relatively safely realized when used in conjunction with other mechanisms.
Existing Internet provider policies are largely next-hop based [3]. However, for next-hop based preference functions to
cause oscillations, there must exist a cycle of domains in which every D
i
prefers D
i over their fall-back route. The relatively
small likelihood of this configuration probably explains why route oscillations have not been observed in the current Internet.
4.3 Discussion
Preference functions based on shortest PATH and next-hop restrict the kinds of policies that can be realized. A domain that
has multiple routes to a given destination can choose any of those routes in PAIRS; but with shortest PATH route selection
the domain can only choose among those routes that have the shortest path length. With next-hop based preference functions,
a domain cannot express policies about providers that are not directly adjacent; domains may desire such expressivity in a
commercial Internet.
Other preference functions on the PATH are likely unsafe. All of our examples of topologies in earlier sections can be
13
created using arbitrary preference functions on thePATH attribute. We have also found that preference functions on most other
BGP and IDRP path attributes are unsafe (e.g.,DIST_LIST_INCL). We conclude that in hop-by-hop inter-domain routing
protocols, such as BGP/IDRP, constraining PAIRS to preference functions based on safe attributes allows only relatively
“uninteresting” policies.
5 Other Approaches
The previous section indicates that there do not seem to exist path attributes that are simultaneously safe and interesting. To
realize richer policy through independent route selection, yet avoid or minimize the impact of route oscillations, two other
approaches are possible: 1) Require domains to coordinate among themselves for specifying policy. This coordination can
allow interesting yet safe preference functions to be realized. 2) Allow domains to independently specify their policies, and
deploy mechanisms to detect and suppress oscillations.
5.1 Coordination
Given global knowledge of the policies for all domains, it may be possible to analyze those policies for the likelihood of route
oscillations. One or more domains could then modify their policies based on the results of this analysis.
One way of doing such an analysis may be to extend the return graph representation to more general topologies. We are
considering this for future study. An alternative approach might be to simulate the effect of these policies off-line. Such
a simulation would capture those oscillations that occur independent of initial conditions, e.g., Figure 1. More extensive
simulations might be necessary to capture those oscillations that depend on initial conditions, for example, the oscillations in
Figure 3.
For analysis to be possible, each domain’s policy must be available to all other domains at all times. One mechanism for
making policies available is a route registry. Such a route registry currently exists in the Internet for inter-provider route co-
ordination [1, 3]. This seems to be a reasonable approach for safely realizing richer policies through hop-by-hop inter-domain
routing in the Internet.
5.2 Detecting and Suppressing Oscillations
Global analysis detects the likelihood of oscillations a priori. It may be acceptable to allow domains to realize their policies
independently and suppress oscillations when they occur. In order to suppress an oscillation in a cycle of domains, at least one
of the domains in that cycle must modify its policies. In Figure 1, if D
modifies its preference function to assign a higher
preference value to r
than r
, the oscillation will cease. This suggests the following general rule: when a domain detects that
it is oscillating, it should assign the highest preference value to its fall-back route.
14
It is possible to conceive of a variety of detection schemes that indicate the likelihood of oscillations. We describe two
schemes that maintain some history of route transitions at a domain. The first scheme maintains all the r-states seen at a domain
over some time period T . If this history contains a repeated sequence of routes and the domain is in a cycle of oscillating
domains, then the rule described above will suppress the oscillation. To reliably detect oscillations, this scheme will need to
keep a significant amount of history. Alternatively, a domain can maintain a time-decayed count of the route advertisements
seen from each neighbor domain. If this “instability” count exceeds an empirically derived threshold, the domain may assume
the likelihood of an oscillation. This scheme is currently deployed in the Internet [25] to suppress route advertisements caused
by frequent topology changes.
The above detection schemes can generate false positives. In either scheme, the domain that maintains the history may not
contribute to the oscillation, but may be sympathetically oscillating with some other domain that does. Instability counts cannot
distinguish between oscillations and route advertisements caused by frequent topology changes. Therefore, it is desirable to
apply our general rule to modify policies only temporarily.
Instability based suppression [25] modifies policies temporarily. The original policies are restored after the instability count
decays below another empirically derived threshold. Depending on the decay rate and the thresholds, this scheme may suppress
oscillations in cases where a stable route assignment exists (for example, with next-hop based preference functions, Figure 3(c)).
For other kinds of oscillations, such as in Figure 1, domains do not oscillate when modified policies are in effect; but when the
original policies are restored, they oscillate briefly until the instability based suppresion is re-established. This approach only
reduces the impact of persistent route oscillations on the routing infrastructure.
Finally, other detection schemes are also possible. For example, in Figure 1, D
sees an oscillation with period hr
r
i.
The transition from r
to r
is a “negative transition” [20] because r
has a lower preference than r
at D
. D
’s advertisement
of r
causes D
to make a positive transition. A negative transition followed by a positive transition could be used to indicate
the likelihood of an oscillation.
6 Related Work
IGRP [12] is a hop-by-hop protocol in which the metric is a weighted sum of a traffic sensitive component, and a distance
sensitive component. Route oscillations can occur in IGRP [16]. When a domain chooses a route with a lower traffic sensitive
component, and forwards traffic along that route, that route’s metric increases. Intuitively, this “traffic feedback” causes route
oscillations in IGRP.
The original ARPANET link state based SPF algorithms [17, 14] used delay as a metric. Route oscillations were observed
on the ARPANET when portions of the network were heavily congested. These route oscillations are also due to similar
traffic feedback [4]. Several solutions to delay based oscillations have been proposed. These solutions use approaches such as
constraining route selection [14], co-ordination [26], and explicit routing [5].
15
Path vector protocols were developed [22, 23] to suppress counting-to-infinity problems that occur in distance vector al-
gorithms. These protocols use the shortest path route selection function and are therefore not susceptible to oscillations. BGP
[21] and IDRP [13, 19] add independent route selection to path vector algorithms. In our paper, we have shown that eliminating
the monotonically increasing metric can introduce route oscillations.
Link state based routing protocols, such as Viewservers [2], and map state based protocols, such as Nimrod [7], compute
explicit routes from a global information base on demand. These protocols are not susceptible to route oscillations. However,
PAIRS protocols have two advantages [9]: 1) Destination based hop-by-hop routing aggregates state across all sources whereas
explicit-route based protocols do not, and 2) PAIRS protocols support selective reachability advertisement, i.e., a domain can
unilaterally choose to not advertise routes to particular destinations.
The Unified routing architecture [9] uses a combination of PAIRS based hop-by-hop routing and explicit routing to realize
policies. Our work provides additional justification for an explicit routing component to complement PAIRS based hop-by-hop
routing.
7 Conclusions and Future Work
We have shown that independent route selection can result in persistent route oscillations in hop-by-hop inter-domain routing.
We believe that only shortest path route selection is provably safe. This significantly reduces the policies that can be realized
using inter-domain routing.
Given the existence of a widely deployed commercial Internet infrastructure, a combination of policy analysis, and insta-
bility based route suppression can be used to deal with route oscillations. The former can detect most route oscillations caused
by inter-dependent policies. The latter mitigates the impact, on the infrastructure, of route oscillations not detected by analysis.
Further research is necessary to develop techniques to analytically determine the existence of route oscillations in more gen-
eral topologies. Future work may also focus on simulation-based methodologies to determine the existence of route oscillations.
Another promising area is the investigation of protocol mechanisms for detecting oscillations.
8 Acknowledgements
The authors would like to thank Cengiz Alaettino˘ glu, Lee Breslau, Ram Gurumoorthy, Shai Herzog, Steve Hotz, Bill Manning,
Yakov Rekhter, and Daniel Zappala, for their suggestions and contributions, both to the problem itself, and in their review of
this paper.
16
References
[1] C. Alaettinoglu and J. Yu. Autonomous System Path Expression Extension to Ripe-181. Internet Draft: RPS working
group, November 21, 1995. Work in progress.
[2] C. Alaettino˘ glu and A.U. Shankar. The Viewserver Hierarchy for Inter-Domain Routing: Protocols and Evaluation. IEEE
Journal on Selected Areas in Communications, 13(8):1396–1410, October 1995.
[3] T. Bates, E. Gerich, L. Joncheray, J-M. Jouanigot, D. Karrenberg, M. Terpstra, and J. Yu. Representation of IP Routing
Policies in a Routing Registry, RIPE-181 edition, October 3, 1994. Available at ftp://ftp.ripe.net/ripe/docs.
[4] D.P. Bertsekas. Dynamic Behavior of Shortest Path Routing Algorithms for Communication Networks. IEEE Transactions
on Automatic Control, AC-27(1):60–74, February 1982.
[5] L. M. Breslau. Adaptive Source Routing of Real-Time Traffic in Integrated Services Networks. PhD thesis, University of
Southern California, December 1995.
[6] C. Cheng, Riley R., Srikanta P.R. Kumar, and J.J. Garcia Aceves. A Loop-Free Extended Bellman-Ford Routing Protocol
without Bouncing Effect. In Proceedings of the ACM SIGCOMM, pages 224–236, 1989.
[7] J. N. Chiappa. A New IP Routing and Addressing Architecture. Internet Draft: NIMROD Working Group, April 1994.
Work in progress.
[8] D. Estrin, T. Li, Y . Rekhter, et al. Source Demand Routing: Packet Format and Forwarding Specification (Version 1).
Internet Draft: SDR working group, October 30, 1995. Work in progress.
[9] D. Estrin, Y . Rekhter, and S. Hotz. Scalable Inter-Domain Routing Architecture. In Proceedings of the ACM SIGCOMM,
pages 40–52, Baltimore MD, U.S.A., August 1992.
[10] P. Ford and Y . Rekhter. Explicit Route Protocol (ERP) for IPv6. Internet Draft: SDR Working Group, January 1995. Work
in progress.
[11] J. Guckenheimer and P. Holmes. Nonlinear Oscillations, Dynamic Systems, and Bifurcations of Vector Fields, chapter 1,
pages 22–27. Springer-Verlag New York Inc., 1983.
[12] C.L. Hedrick. An Introduction to IGRP. Technical report, Rutgers University, August 1991.
[13] ISO. Protocol for Exchange of Inter-domain Routeing Information among Intermediate Systems to Support Forwarding
of ISO 8473 PDUs, ISO/IEC/JTC1/SC6 IS10747 edition, 1993.
[14] A. Khanna and J. Zinky. The Revised ARPANET Metric. In Proceedings of the ACM SIGCOMM, pages 45–56, September
1989.
17
[15] L. Lamport. An Assertional Correctness Proof of a Distributed Algorithm. Science of Computer Programming, 2:175–
206, 1982.
[16] S. Low and P. Varaiya. Stability of a Class of Dynamic Routing Protocols (IGRP). In IEEE Proceedings of the INFOCOM,
volume 2, pages 610–616, March 1993.
[17] J. McQuillan, I. Richer, and E.C. Rosen. The New Routing Algorithm for the ARPANET. IEEE Transactions on Com-
munications, COM-28(5):711–719, May 1980.
[18] K. Ogata. Discrete-Time Control Systems, chapter 4, pages 351–353. Prentice-Hall Inc., Englewood Cliffs, NJ 07632,
USA, 1987.
[19] Y . Rekhter. Inter-Domain Routing Protocol (IDRP). Internetworking: Research and Experience, 4:61–80, 1993.
[20] Y . Rekhter. Private communication, January 1996.
[21] Y . Rekhter and T. Li. A Border Gateway Protocol 4 (BGP-4), RFC 1771 edition, 1995. (Obsoletes RFC1654).
[22] K. Shin and M. Chen. Performance Analysis of Distributed Routing Strategies Free of Ping-Pong-Type Looping. IEEE
Transactions on Computers, C-36(2):129–137, February 1987.
[23] K. Shin and M. Chen. Minimal Order Loop-Free Routing Strategy. IEEE Transactions on Computers, 39(7):870–881,
July 1990.
[24] W.D. Tajibnapis. A correctness proof of a topology information maintainence protocol for a distributed computer network.
Communications of the ACM, 20(7), 1977.
[25] C. Villamizar and R. Govindan. Controlling BGP route processing overhead. Internet Draft: IDR Working Group, October
1993. Work in progress.
[26] Z. Wang and J. Crowcroft. Shortest Path First with Emergency Exits. In Proceedings of the ACM SIGCOMM, pages
166–176, September 1990.
18

Abstract (if available)

Linked assets

Computer Science Technical Report Archive

Conceptually similar

PDF

USC Computer Science Technical Reports, no. 603 (1995)

PDF

USC Computer Science Technical Reports, no. 642 (1996)

PDF

USC Computer Science Technical Reports, no. 706 (1999)

PDF

USC Computer Science Technical Reports, no. 605 (1995)

PDF

USC Computer Science Technical Reports, no. 723 (2000)

PDF

USC Computer Science Technical Reports, no. 745 (2001)

PDF

USC Computer Science Technical Reports, no. 672 (1998)

PDF

USC Computer Science Technical Reports, no. 677 (1998)

PDF

USC Computer Science Technical Reports, no. 774 (2002)

PDF

USC Computer Science Technical Reports, no. 692 (1999)

PDF

USC Computer Science Technical Reports, no. 669 (1998)

PDF

USC Computer Science Technical Reports, no. 750 (2001)

PDF

USC Computer Science Technical Reports, no. 697 (1999)

PDF

USC Computer Science Technical Reports, no. 704 (1999)

PDF

USC Computer Science Technical Reports, no. 731 (2000)

PDF

USC Computer Science Technical Reports, no. 732 (2000)

PDF

USC Computer Science Technical Reports, no. 702 (1999)

PDF

USC Computer Science Technical Reports, no. 678 (1998)

PDF

USC Computer Science Technical Reports, no. 613 (1995)

PDF

USC Computer Science Technical Reports, no. 640 (1996)

Description

Kannan Varadhan, Ramesh Govindan, Deborah Estrin. "Persistent route oscillations in inter-domain routing." Computer Science Technical Reports (Los Angeles, California, USA: University of Southern California. Department of Computer Science) no. 631 (1996).

Asset Metadata

Creator Estrin, Deborah (author), Govindan, Ramesh (author), Varadhan, Kannan (author)

Core Title USC Computer Science Technical Reports, no. 631 (1996)

Alternative Title Persistent route oscillations in inter-domain routing (title)

Publisher Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA (publisher)

Tag OAI-PMH Harvest

Format 18 pages (extent), technical reports (aat)

Language English

Unique identifier UC16269489

Identifier 96-631 Persistent route Oscillations in Inter-Domain Routing (filename)

Legacy Identifier usc-cstr-96-631

Format 18 pages (extent),technical reports (aat)

Rights Department of Computer Science (University of Southern California) and the author(s).

Internet Media Type application/pdf

Source 20180426-rozan-cstechreports-shoaf (batch), Computer Science Technical Report Archive (collection), University of Southern California. Department of Computer Science. Technical Reports (series)

Access Conditions The author(s) retain rights to their work according to U.S. copyright law. Electronic access is being provided by the USC Libraries, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name USC Viterbi School of Engineering Department of Computer Science

Repository Location Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089

Repository Email csdept@usc.edu

Inherited Values

Title Computer Science Technical Report Archive

Coverage Temporal 1991/2017

Repository Email csdept@usc.edu

Repository Name USC Viterbi School of Engineering Department of Computer Science

Repository Location Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089

Publisher Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA (publisher)