Computer Science Technical Report Archive

USC Computer Science Technical Reports, no. 642 (1996)

(USC DC Other)

USC Computer Science Technical Reports, no. 642 (1996)

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content An Analysis of In ternet In terDomain T op ology and Route Stabilit y
R amesh Govindan
USCInformation Sciences Institute
Admiralt yW a y Suite Marina del Rey CA Phone ext F ax govindanisiedu
A no op R e ddy
USCInformation Sciences Institute
Admiralt yW a y Suite Marina del Rey CA Phone ext F ax areddyisiedu
Abstract
The In ternet routing fabric is partitioned in to sev eral domains Eac h domain represen ts a
region of the fabric administered b y a single commercial en tit yOv er the past t woy ears the
routing fabric has exp erienced signican t gro wth F rom more than a y ears w orth of in terdomain
routing traces w e analyze the In ternet in terdomain top ology its route stabilit y b eha vior and
the eect of gro wth on these c haracteristics Suc h an analysis is imp ortan t b ecause in terdomain
routing pro vides the foundation for In ternet widearea comm unication
Despite gro wth the degree distribution and the diameter of the in terdomain top ology ha vere mained relativ ely unc hanged F urthermore there exists a fourlev el hierarc hyof In ternet domains
classied b y degree Ho w ev er connectivit ybet w een domains is signican tly nonhierarc hical De
spite increased connectivit y at higher lev els in the top ology the distribution of paths to prexes
from the bac kb one remained relativ ely unc hanged There is evidence that b oth route a v ailabili t y
and the mean time to unreac habilityha v e degraded with In ternet gro wth
Keyw ords In ternet In terdomain Routing BGP T op ology Route Stabilit y
In tro duction
In the past t woy ears w eha v e witnessed the emergence of a commercial In ternet infrastructure T oda y the
In ternet routing fabric is partitioned in to dieren t administrativ e domains A campus or in ternal corp orate
net w ork is an example of a domain Data exc hange b et w een campus or corp orate domains is facilitated b y
one or more pr ovider domains these domains oer as a service transmission and switc hing facilities for
data exc hange b et w een their customers Pro viders usually in terconnect at In ternet exchange p oints and can
v ary in the geographical scop e of their op erations from regional to national and in ternational
Domains ma y need to exercize trac access restrictions and express transit preferences Curren tly domains realize these p olicies b y selectiv ely disseminating routing information The Border Gatew a y Proto col
BGP v ersion an interdomain hopb yhop routing proto col is used for this purp ose In BGP a
domain ma y originate one or more r outes Routes adv ertise reac habilitytoIP addr ess pr exes within a
domain A domain realizes its trac p olicies b y indep enden tly selecting and selectiv ely propagating routes
heard from its neigh bors or p e ers Asso ciated with eac h route is a list of iden tiers for domains tra v ersed b y
the route This list is the routes ASPATH The collection of domains their p olicies and p eering relationships and the address prexes they adv ertise
denes the In ternet interdomain r outing systemThe in terdomain routing system is large In Decem ber
there w ere more than domains and nearly address prexes P aralleling the gro wth of the
In ternet the routing system has also rapidly increased in size Bet w een mid and late domains
and address prexes ha v e eac h nearly doubled
The in terdomain routing system pro vides the foundation for In ternet widearea comm unicatio n Tw o
c haracteristics of the routing system can impact the p erformance of suc h comm unication
Interdomain top olo gy This is the graph of domains and the in terdomain p eering relationships A link in
this graph signies route exc hangeand presumably IP trac exc hangeb et w een the corresp onding
domains
R oute stability The routing system exp eriences transientc hanges in routes caused b y router and link failures
or router misconguration
With the gro wth of the routing system it is reasonable to exp ect signican tc hanges in these c haracteristics
With the emergence of the W orldWide W eb and Mb onebased applications widearea comm unication
is b ecoming more prev alent Understanding the top ology and route stabilit y b eha vior of the routing
system is imp ortan t for endtoend proto col design and ev aluation Ho w ev er the geographic exten t of the
routing system precludes an accurate analysis of these c haracteristics

F rom traces of routing up dates seen at a large In ternet service pro vider and at a p opular In ternet exc hange
p oin t this pap er appr oximately c haracterizes the routing system and its gro wth This appro ximation is
p ossible b ecause most large service pro viders carry routes to almost all IPconnected destinations and BGPs
ASPATH s con tain some in terdomain top ology information
Our analysis rev eals sev eral in teresting results ab out the in terdomain top ology Despite the gro wth
of the routing system the diameter and degree distribution in the in terdomain top ology ha v e remained
relativ ely constan t The path redundancythe n um b er of distinct paths to a prexhas also not c hanged
signican tly despite increased connectivit y A closer analysis of the degree distribution rev eals the existence
of a fourlev el hierarc h y of domains Connectivit ybet w een domains ho w ev er app ears to b e signican tly
nonhierarc hical
While routes to prexes w ere highly a v ailable more than of the address prexes w ere a v ailable
for more than of the time there w as some evidence of degradation of a v ailabili t y with gro wth Not
surprisingly more than of the prexes w ere reac hable for more than of the time b y a single primary
path The mean reac habilit y duration for a prex sho ws wide v ariation but a noticeable degradation with
gro wth This reac habilit y duration ho w ev er app ears to b e correlated with the length of the primary path
Section motiv ates the need for a study of routing system gro wth Section describ es some simple
measures of growthofthe in terdomain routing system Section studies the in terdomain top ology and the
eect of gro wth on the top ology Section analyzes route stabilit y and its v ariation with gro wth Finally Section describ es related w ork and Section presen ts our conclusions
Bac kground and Motiv ation
In this section w e brie y describ e In ternet in terdomain routing W e also motiv ate the need for an analysis
of the in terdomain routing system
The In ternet In terDomain T op ology
A domain is an autonomously administered p ortion of the In ternets routing fabric Domains v aryinsize geographic exten t and function Backb ones pro vide national or in tercon tinen tal transit r e gionals servea
metrop olitan area or a collection of suc h areas and stubs pro vide campus lev el IP connectivit y Domains
in terconnect in sev eral w a ys The larger pro viders usually exc hange trac at neutral exchange p ointssuc h
as MAEEast in W ashington DC and the NSFsp onsored NAPs Stubs usually connect to their pro viders
using direct leased lines

By abstracting out the in ternal top ology of domains w e can describ e the In ternet as an undirected graph
whose no des are the domains and whose links are the in terdomain in terconnects In the early In ternet this
interdomain top olo gy w as appro ximately treestructured A single national b ackb one pro vided crosscoun try
transit to sev eral regional pro viders Campus net w orks obtained global IP connectivityb y attac hing to their
resp ectiv e regionals With the adv en t of commercial IP service the structure of this graph has b ecome
more complex Sev eral bac kb ones nowin terconnect in a partial mesh A regional pro vider mayin terconnect
with other regionals serving the same or adjacen t geographical area as w ell as with one or more bac kb ones
Finally a stub ma y connect to one or more regionals or bac kb ones
The Border Gatew a y Proto col
The Border Gatew a y Proto col BGP is used to ac hievein terdomain routing information exc hange The
unit of information exc hanged b y BGPsp eak ers is an up date An up date con tains a collection of address
prexes Address prex X represen ts the top ological region of the In ternet in whic h hosts and routers
net w orkla y er addresses ha v e X as a prex a route from a p eer con taining prex X adv ertises that p eers
reac habilit y to some en tit y within that top ological region Address prexes are usually represen ted byan IP
address and a prex length
F or eac h address prex it con tains an up date denes a r outing tr ansitionW e distinguish b et w een t w o
kinds of transitions unr e achables and routes An unreac hable for prex X adv ertises the senders inabilit y
to reac h destinations in X Con v ersely a route for prex X adv ertises the senders reac habilit y to destinations
in X Asso ciated with eac h route is an ASPATH The ASPATH for prex X lists the domainlev el path to
destinations in X In to da ys In ternet eac h domain originates routes to all address prexes within its b orders Usually the
domains b order routers learn of these address prexes through a separate intr adomain routing proto col
BGP allo ws domains to realize access restrictions and transit preferencesdomain trac p olicyb y selec
tiv ely propagating routing information A domain that receiv es t w o or more routes to an address prex can
indep enden tly select one of the routes thereb y exercising its transit preference A domain can also indep en
den tly restrict the adv ertisemen t of a selected route to a neigh b or thereb y realizing its access restrictions
Characterizing the In terDomain Routing System
The in terdomain routing system determines the p erceiv ed qualit y of widearea comm uni cation in the In
ternet Suc h widearea comm unicatio n is b ecoming increasingly prev alen t studies at one campus indicate
that widearea TCP connections originating at that campus domain grew exp onen tially Sev eral factors

including the rapid growthofthe W orldWide W eb and the increasing use of the Mb one accoun t for this
increase Another measure of the gro wth in widearea comm unication is the gro wth in aggregated bac kb one
trac the utilization of the MAEEast Gigaswitc h nearly doubled in the v emon ths b et w een Mayand
Septem b er T o understand the impact of the routing system on widearea comm uni cation w e fo cus on t woc har
acteristics of the routing system the in terdomain top ology and route stabilit y What factors in uence
in terdomain top ology and route stabilit y! In to da ys In ternet the in terdomain top ology is determined b y
bilateral transit agreemen ts b et w een pro viders The presence of sev eral exc hange p oin ts enables increased do
main connectivit y Router failure or temp orary router o v erload is one reason for route instabilit y Loss of link
connectivit y either within a domain or b et w een domain b order routers is another Routing misconguration
is a thirdcurren tly most in terdomain route exc hange is eected b yman ual p olicy conguration
When analyzing top ology and route stabilit y it is imp ortan t to understand ho w these c haracteristics
c hange with gr owth A unique feature of the In ternet is the relativ ely rapid gro wth in the top ology and the
n um b er of routed address prexes caused b y the rapid increase in the n um b er of connected hosts and
the deplo ymen t of a commercial In ternet infrastructure In June the routing system consisted of ab out
domains and prexes By No v em b er it had gro wn to domains and prexes
Wh y migh t an analysis of routing system top ology and route stabilit y b e useful! First suc h an analysis
enables a b etter understanding of ho w these c haracteristics impact endtoend comm unicatio n p erformance
Greater domain connectivit y caused b ygro wth can increase the redundancy of endtoend connectivit y Ho w ev er a larger In ternet migh t increase the op erational range of dela ys and throughput encoun tered b y
distributed applications Changes in prex reac habilit y can aect the dela y loss pac k et reordering and
throughput c haracteristics observ ed b y longliv ed connections Suc hv ariations can aect proto col b eha vior
and application arc hitectures P ath delayv ariation can aect TCP roundtrip estimates and adv ersely
impact adaptiv e audio and video applications Increased pac k et losses or pac k et reordering ma y reduce the
eciency of the net w ork as a whole F urther route c hanges can also reduce the ecacy of o w cac hing
sc hemes
Second suc h an analysis can impact the design of routing proto cols and route distribution mec hanisms as
w ell as the implemen tati on and pro visioning of routers F or example frequen tc hanges in prex reac habilit y
can increase route computation o v erhead on routers and ha v e anecdotally b een kno wn reduce the ecacy
of forw arding table cac hing sc hemes Gro wth aects the amoun t of information routing proto cols need to
propagate and the storage and computational requiremen ts of routers Greater domainlev el connectivit y and the a v ailabilit y of alternate paths ma y argue for the deplo ymen t of explicit routing mec hanisms
Finally a b etter understanding of the routing system can impro v e mo deling of in ternet w ork top ologies
and of endtoend dela y and loss c haracteristics Accurately analyzing the routing system and its gro wth is extremely dicult Because the In ternet is
geographically distributed no c omplete map of the in terdomain top ology exists to da y Eorts are underw a y
to establish In ternet p olicy registries con taining suc h information Because the In ternet is adminis
trativ ely decen tralized it is dicult to obtain a comp osite picture of the in terdomain top ology and route
stabilit y In this pap er w e appr oximately c haracterize routing system top ology and route stabilit y and the impact
of gro wth on the routing system using t woc hronological traces of routing transitions
D
A
con tains nearly milli on BGP up dates heard from a bac kb one router of a ma jor IP service
pro vider b et w een June and June D
M
con tains nearly milli on BGP up dates heard from more than t w en t y large and small IP service
pro viders attac hed to a large In ternet exc hange p oin t MAEEast in W ashington DC b et w een August
to No v em b er D
M
w as collected at an exp erimen tal facilit y the MAEEast Route Serv er
These traces are incomplete in parts caused b y failures at the trace collection p oin ts In our analysis w e
ha v e b een careful to eliminate the eect of these trace a ws
T o simplify our analysis of top ology and route stabilit yw e fo cused on three dieren t snapshots Section of these traces separated b y nearly six mon ths Eac h snapshot con tains some in terdomain connectivit y infor
mation namely the ASPATH s asso ciated with up dates Using this w e deriveanappro ximate c haracterization
of the in terdomain top ology This enables us to quan titativ ely analyze the gro wth of sev eral c haracteristics
of this top ology the n um b er and size of domains the exten tof in terdomain connectivit ythe n um ber of
dieren t domainlev el paths to address prexes and so on F rom the traces w e also obtain distributions for
t w o measures of route stabilit y a v ailabilit y of a path to a prex and the mean time to unreac habilit y for a
prex
Routing System Gro wth
Before understanding the eect of gro wth on top ology and route stabilit y it is imp ortan ttoquan tify this
gro wth A simple measure of the gro wth of the routing system is the arriv al pattern of new address prexes
domains and links
Adv anced Net w orks and Services who op erated the no w defunct NSFNET bac kb one service

Jun94 Mid−Dec94 Jul95
0
500
1000
1500
Ordinal Number
Link Arrivals
0
10000
20000
30000
40000
Ordinal Number
Prefix Arrivals
0
200
400
600
800
1000
Ordinal Number
Domain Arrivals
Figure A rrivals In D A the arriv als of prexes domains and links is appro ximately linear In eac h graph the
horizon tal segmen ts corresp ond to gaps in D A W e dene the arriv al time of a new address prex as the rst instan t at whichan updatecon taining that
prex is seen in our traces Figure plots successiv e new prex arriv als in D
A
Ov er the duration of D
A
the
arriv als of new prexes is appro ximately linear the initial step in the gure corresp onds to prexes presen t
in the routing system prior to June and the horizon tal segmen ts represen t gaps in D
A
The slop e of
the curv e corresp onds to one new prex every minutes Considering that a prex usually represen ts an
en tit y the size of a campus departmen tal net w ork this represen ts a signican t rate of address prex gro wth
The linearit y of the gro wth curv e oers some evidence of the abilit yof CIDR to curtail the gro wth of the
bac kb one routing table in the face of exp onen tial In ternet gro wth The arriv al patterns in D
M
are similar
There is some evidence in D
A
of t w o or more new prexes arriving together W e see nearly suc h
prex arrival clusters The median cluster size is and nearly of the clusters con tain less than prexes This corresp onds to domains turning on connectivit y to small n um b ers of their customers at a
time Some clusters of h undreds of prexes are also visible but these ma y b e an artifact of the gaps in D
A
Ev en though the traces in D
A
w ere collected at a single bac kb one router w e exp ect the prex arriv al
patterns to b e represen tativ e of the In ternet as a whole This is b ecause bac kb one routers to da y con tain
reac habilit y information to a signican t fraction of the In ternet reac habilit y to a new prex is lik elytobe
heard at a bac kb one router

W e dene the arriv al time of a new domain as the rst instan tat whic h that domains iden tier w as seen
in an up dates ASPATH Figure also sho ws the arriv al pattern of new domains Ov er the duration of D
A
the arriv al of new domains is also appro ximately linear The slop e of the arriv al curv e corresp onds to one
new domain every hours Considering that a domain corresp onds to an IP service pro vider or a campus
net w ork this statistic oers more evidence of the signican t recentgro wth of the In ternet Domain arriv als
in D
M
sho w a similar trend Our traces ma y underestimate the arriv als of new domains particularly when
the p olicies at our trace collection lo cation exclude routes from a particular domain Ho w ev er w e b eliev e
that the exten t of this underestimation is small
W e dene the arriv al time of a new in terdomain link as the rst instan tatwhic h the endp oin ts of the
link w ere observ ed as successiv e elemen ts of an up dates ASPATH Figure sho ws the arriv al pattern of new
links Ov er the duration of D
A
the arriv al of new links is appro ximately linear The slop e of this curv e
corresp onds to a new in terdomain link ev ery hours Ho w ev er our tec hnique of coun ting new links can
can signican tly underestimate their actual arriv al pattern If for p olicy reasons routes with a particular
ASPATH w ere nev er visible at our trace collection lo cation our tec hnique w ould not record the corresp onding
links F or example w e migh t miss bac kdo or links b et w een service pro viders
T op ology
The arriv al patterns of new domains and in terdomain links do not pro vide information ab out the in ter
domain top ology In this section w e obtain appro ximate c haracterizations called snapshots of the in ter
domain top ology from three dieren t segmen ts of our traces Using these snapshots w e study the follo wing
prop erties of the top ology and their v ariation o v er time domain degree distribution diameter and connec
tivit y Snapshots
Bac kb one routers from whic h D
A
and D
M
w ere obtained carry routing information ab out most of the
routed address prexes in the In ternet It is generally b eliev ed that bac kb one routers for larger pro viders
con tain up w ards of of the en tire routed IP address space Th us from D
A
and D
M
w e can dra w fairly
reliable conclusions ab out macroscopic trends in arriv al patterns of domains in terdomain links and prexes
Of in terest to us ho w ev er are more detailed c haracteristics of the routing system its top ology and its
reliabilit y and a v ailabili t yTw o artifacts of the traces preclude our using the en tire traces for analyzing these
c haracteristics First the traces con tain sev eral gaps primarily caused b y outages at the trace collection

Domains Links Address Prexes
S
a
S
b
S
c
T able Snapshots W e obtain an appro ximate represen tation of the routing system top ology from da y segmen ts
of our traces This table sho ws the n um b er of domains links and address prexes in the top ologies obtained from
our three snapshots
routers Second D
M
w as collected at an exp erimen tal facilit y the MaeEast Route Serv er and is incomplete
in parts
Nev ertheless there exist segmen ts of these traces that are dev oid of these artifacts F rom these segmen ts
w e can deriv e an appro ximate instan taneous c haracterizations snapshots of the routing system By
comparing widely separated snapshots w e can observethe v ariation of routing system c haracteristics with
gro wth This is the metho dology w e use in this and the next section F or our analysis w ec hose three
snapshots S
a
corresp onding to the routing system in No v em b er S
b
corresp onding to the routing
system in Ma y and S
c
corresp onding to the routing system in No v em ber T able giv es the sizes
of the top ology and address prexes obtained from these snapshots
F rom the ASPATH s in a snapshot w e can derivethe in terdomain top ology Eac h elemen t of an up dates
ASPATH denes a no de in the top ology and eac h successiv e pair of domains in the ASPATH represen ts the end
p oin ts of a link F urthermore the collection of unique address prexes seen in the snapshot appro ximately
represen ts the routed destinations at that instan t
The longer the segmen t of the trace c hosen the more complete our snapshot is lik ely to b e Ho w ev er
a longer duration trace segmentma y represen t signican t gro wth and therebysk ew our snapshot W e
empirically c hose da y segmen ts of our traces In eac h da ysegmen t the n um b er of domains links
and prexes gro ws b y no more than F urther the routing activit y in the In ternet is suc h that in
eac h segmen t at least t w o transitions are seen for more than of all prexes seen in the windo w F or
this reason w e b eliev e that our c hoice of segmen t length is lik ely to capture most of the routed address
prexes in the routing system at the time W ev alidated this b y comparing the address prex coun t in our
snapshots with an indep enden tly collected history of the size of bac kb one routing tables In eac h case
our appro ximation w as within of the size of the routing table at the corresp onding instan t
This tec hnique pro vides a ric her view of the top ology than an instan taneous dump of the routing table
This is b ecause wema y see more than one ASPATH to an address prex o v er the da y trace segmen t

0 5 10 15
0.85
0.9
0.95
1
Domain Degree
Fraction of Domains
(b)
0 5 10 15 20
0.9
0.92
0.94
0.96
0.98
1
Domain Degree
Fraction of Domains
(c)
5 10 15 20 25 30 35 40
0.95
0.96
0.97
0.98
0.99
1
Domain Degree
Fraction of Domains
(d)

0 20 40 60 80 100
0
0.2
0.4
0.6
0.8
1
Domain Degree
Fraction of Domains
(a)
S
a
S
b
S
c
e
Av erage Degree
S
a
S
b
S
c
Figure De gr e e Distribution The cum ulativ e fraction of domains plotted against degree a The distributio ns
are nearly iden tical for the three snapshots A closer lo ok at the distributio n b c and d sho ws that domains
can b e divided in to dieren t classes based on frequency of o ccurrence The gra y lines in eac h of b c and d
demarcate dieren t classes of domains see text The a v erage degree has remained relativ ely constan t
Ho w ev er there is still a lik eliho o d of missing some of the in terdomain links and a smaller lik eliho o d of
missing some domains as w ell Section In general w e exp ect that this tec hnique giv es a fairly go o d
picture of the top ology closer to the trace collection lo cations ie in the North American p ortion of the
In ternet The fuzziness of our snapshots is lik ely to increase with with increasing top ological distance
from the trace collection lo cations
Weno w describ e the diameter degree distribution and connectivit yc haracteristics of the in terdomain
top ology Degree Distribution
A link in the in terdomain top ology represen ts BGP p eering b et w een the endp oin ts of the link Usually suc h p eering is based on bilateral data trac exc hange agreemen ts b et w een domains The n um ber of suc h
p eerings the domains degree is therefore an appro ximate measure of its size The distribution of domain
degree illustrates the range of domain sizes in the In ternet

Class Degree Range Appro ximate F raction T yp es of domains
of Domains in Class
C National or in ternational bac kb ones
C Large North American regional pro viders
and Europ ean national net w orks
C Smaller regional pro viders and large
metrop olitan area pro viders
C Smaller metrop olitan area pro viders and
m ulticampus corp orate or academic net w orks
T able De gr e e Classic ation The degree distribution can b e w ell appro ximated b y a piecewise linear function with
four comp onen ts This suggests a classication of domains b y degree range Within a degree range the frequency of
o ccurrence of domains of a certain degree is similar
Figure e sho ws that the a v erage degree of domains in the top ology is relativ ely small This indicates
that the in terdomain top ology is sparse This observ ation is illustrated b etter b y plotting the cum ulativ e
fraction of domains against no de degree Figure a Nearly of the domains ha v e a degree of or Figure e also sho ws that the a v erage degree has increased only sligh tly across the dieren t snapshots
ev en though the n um b er of domains and links has doubled b et w een S
a
and S
c
This ma y indicate that the
rate of gro wth has b een appro ximately uniform across all sizes of domains The degree distribution conrms
thisthe distributions for the three snapshots app ear identic al Figure a This suggests that as the
In ternet has gro wn the fraction of domains ha ving a giv en degree has remained appro ximately constan t
The degree distribution is w ell appro ximated b y a piecewise linear function with four comp onen ts Fig
ure b c and d Eac h comp onen t denes a degree range Within a degree range the fraction of no des
with a giv en degree is similar W e b eliev e that these four degree ranges represen t a natural hierarc hyof
domains classied b y function T able describ es this hierarc h y The constancy of the degree distribution
implies that the In ternet has b een gro wing laterally adding domains to the dieren tlev els of the hierarc h y
in prop ortion to gro wth
Diameter
T able sho ws the diameter of the in terdomain top ologyF rom this w e see that an y domain is at most
domainlev el hops a w a y in a shortestpath sense from an y other domain The path tak en b y pac k ets
bet w een t w o domains is a function of domain p olicies and can exceed this gure This gure is consisten t
with our domain classication ab o v e Bet w een t w o stub domains there are at most or domainlev el hops
eachto C
pro viders As w eshowbelo w connectivit y within C
is rather high so the shortest path b et w een

Diameter
S
a
S
b
S
c
T able Diameter of InterDomain T op olo gy Despite the rapid gro wth of the In ternet the diameter of the
in terdomain top ology has remained relativ ely constan t
C
pro viders should b e b et w een and More imp ortan tly despite the near doubling in the n umberofdomains andlinks bet w een S
a
and S
c
the
domainlev el diameter of the Internet has r emainedr elatively c onstant Despite this ho w ev er it is lik ely that
the r outerlevel diameter has increased with gro wth our traces do not con tain the information necessary to
conrm this
ConnectivityBet w een Domain Classes
The degree distribution rev ealed a hierarc h y of domains b y size and function Are in terdomain links also
largely hierarc hical! That is is a domain primarily connected to domains in classes imm ediately ab o veand
b elo w its o wn! In this section w ein v estigate domain connectivit y within and b et w een classes
Figure depicts the subgraph of the in terdomain top ology con taining domains in C
and C
for the three
snapshots This gure reinforces the uniform gro wth in the n umberofdomains in C
and C
In addition
this visual represen tation of the top ology rev eals sev eral imp ortan t features of domainlev el connectivityin
C
and C
Bet w een S
a
and S
c
the connectivit y within C
has increased signican tly A similar observ ation
ma y b e made ab out the connectivit y within C
Bet w een S
a
and S
c
the connectivit ybet w een C
and C
has also increased Eac h C
no de app ears to p eer directly with more C
no des on a v erage W e attribute
these trends to the establishmentof sev eral tens of In ternet exc hange p oin ts whic h facilitate in terpro vider
connectivit y These trends are quan tied in the bar graph sho wn in Figure The fraction of links within
C
has nearly quadrupled the fraction of links b et w een C
and C
has nearly doubled and the fraction of
links within C
has increased b y nearly a half
The fraction of links within and b et w een other classes remain appro ximately constan t Ho w ev er it is
in teresting to note that connectivitybet w een domains is signican tly nonhierarc hical T able sho ws the
fraction of links b et w een dieren t classes in S
c
Nearly a quarter of the total n um b er of links is b et w een C
domains and C
domains There app ear to also b e some evidence of bac kdo or links b et w een domains in

0
0.01
0.02
0.03
0.04
0.05
0.06
Fraction of links
C 1−C 1 links C 1−C 2 links C 2−C 2 links
S
a
S
b
S
c
Figure C and C T op olo gy The three graphs describ e the subgraph of the in terdomain top ology con taining C and C no des for S a S band S c resp ectiv ely from left to righ t The larger circles represen t C no des and the smaller
circles C no des The solid blac k lines indicate links b et w een C no des the fain t lines indicate links b et w een C and
C no des and the gra y lines links b et w een C no des The bar graph plots links within C bet w een C and C and
within C as a fraction of the total n um b er of links for the three snapshots
the same class ab out of all links are eac h within C
and C
Redundancy in DomainLev el P aths
If the in terdomain connectivit y has b ecome increasingly meshed the n um b er of domainlev el paths b et w een
anyt w o domains m ust also ha v e increased There are sev eral p ossible measures of this p ath r e dundancy in
general graphs In the in terdomain top ology ho w ev er not all feasible domainlev el paths are usefulW e
dene a useful path as one whose length diers from the shortest domainlev el path byatmost A simple measure of path redundancy in the in terdomain top ology is the a v erage n um b er of useful paths
from a xed p oin t in the bac kb one mesh T able This a v erage indicates a considerable path redundancy

C
C
C
C
C
C
C
C
T able Conne ctivity b etwe en classes An elemen t of this matrix indicates the fraction of the total n um b er of links
in S c that exist b et w een the corresp onding classes Links are signican tly nonhierarc hic al noticeable nearly half of
all the links are b et w een domains in C and domains in C and C in the In ternet in S
c
on a v erage there w ere nearly paths to a domain from a xed bac kb one pro vider
F urther this n um b er has more than doubled from S
a
to S
c
Recall that the routeand implicitly the pathc hosen to a destination is a function of domain p olicy F or p olicy reasons man y of these redundan t paths ma y not b e a v ailable at bac kb one pro vider What is the
lev el of redundancy in the n um b er of paths to destinations! The ASPATH asso ciated with a route up date
indicates the domainlev el path to the address prexes in the up date W euse then um b er of distinct ASPATH s
seen for an address prex as a measure of the redundancy to destinations within that address prex
Figure plots the fraction of address prexes in eac h of the snapshots for whic h n dieren t ASPATH s
are seen for dieren tv alues of nT o more than half the destinations w e see exactly one domainlev el path
T o ab out a fourth w e see exactly t w o domainlev el paths This is not surprising Route distribution and
hence a routes ASPATH is go v erned b y routing p olicy Routing p olicy is usually set b y bilateral transit
agreemen ts In most cases a domain negotiates a primary and p ossibly one bac kup transit to a collection
of destinations
Surprisingly this distribution has not change d signic antly from S
a
to S
c
This indicates that the routing
p olicies expressed in the In ternet ha v e remained relativ ely simple despite the gro wth of the top ologyIt
is conceiv able that the lac k of to ols to congure and analyze in terdomain routing has prev en ted pro viders
from realizing more complicated p olicies Summary and Discussion
In this section w e describ ed sev eral c haracteristics of the In ternet in terdomain routing system and its
gro wth
Adv anced Net w orks and Services

Useful P aths
S
a
S
b
S
c
T able Useful Paths This gure sho ws the a v erage n um b er of useful paths those whose length diers from the
shortest path byat most bet w een a xed p oin tinthe In ternets bac kb one mesh and other domains
The in terdomain top ology is relativ ely sparse Nearly of the domains ha v e a degree less than Despite the rapid gro wth of the In ternet there app ears to b e a common natural classication of domains
b y size These four classes roughly corresp ond to national and in ternational bac kb one pro viders large
regional pro viders smaller regional or large metrop olitan area pro viders and smaller dialin pro viders
or m ulticam pus corp orate net w orks
The connectivitybet w een domains in the upp er t wola y ers of the hierarc h y has noticeably increased
p ossibly due to the app earance of sev eral exc hange p oin ts
The in terdomain connectivit y do es not follo w the domain hierarc h y In particular nearly a fourth of
the links in S
c
connect a C
domain directly to a bac kb one pro vider
The diameter of the In ternet has remained relativ ely constan t
The increased connectivitybet w een pro viders has not resulted in a greater redundancy of connectivit y
to destinations
Our snapshot tec hnique is lik ely to underestimate the top ology This is particularly true of p ortions
of the top ology distan t from our observ ation p oin ts on the North American pro vider mesh Ho w ev er it is
lik ely that eac h snapshot equally underestimates the instan taneous top ology W e b eliev e therefore that
these resultsobtained b y c omp aring snapshotsare reliable estimators for the v ariation of the in terdomain
top ology with gro wth
Route Stabilit y
In this section w e describ e in terdomain route stabilit ybeha vior Route stabilit y aects reac habilityto
prexes F rom the p ersp ectiv e of routes to a single prex t w o stabilit y measures are of in terest prex

0 1 2 3 4 5 6 7 8 9 10
0
0.1
0.2
0.3
0.4
0.5
0.6
Number of AS_PATHs
Fraction of total prefixes
S
a
S
b
S
c
Figure Path r e dundancy The fraction of all address prexes in a snapshot for whic h a giv en n um b er of paths
are seen has not c hanged signican tly o v er the three snapshots
availability the fraction of time that a prex is reac hable and prex ste adiness the a v erage time that a
prex is con tin uously reac hable W e study the distribution of these measures
Route stabilitycan v ary with the size of the top ology It is imp ortan t therefore to factor out the
rapid gro wth of the In ternet when analyzing our traces F or this reason w e use the same trace snapshot
metho dology describ ed in the previous section W e assume that the in terdomain top ology has remained
static within a snapshot Recall that a snapshot is long enough to capture at least t w o transitions for more
than of the prexes seen On a v erage ho w ev er eac h snapshot con tains more than a h undred transitions
p er prex
T o study the eect of gro wth on route stabilit yw e compare the distributions of these stabilit y measures
across snapshots F or this purp ose w e consider only snapshots S
a
and S
b
and not S
c
S
c
is extracted from
D
M
and this trace w as obtained at an exp erimen tal facilit y the MAEEast Route Serv er In these traces w e
observ e frequentdo wn times for p eering sessions with clien ts of the Route Serv er These trace a ws prev enta
reliable estimation of route stabilitybeha vior from D
M
in general On a v erage ho w ev er route serv er clien ts
remained highly a v ailable and w eha v e no reason to susp ect the collection of ASPATH s seen in S
c
orour
conclusions ab out the gro wth of top ology Section

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Availability
Cumulative fraction of total prefixes
S
a
S
b
Figure Pr ex A vailability Prex a v ailabili t y is dened as the fraction of time that the prex w as reac hable in
the duration b et w een when the prex w as rst seen and the end of the snapshot More than of the prexes are
a v ailable for more than of the time
Prex Av ailabilit y
W e dene the availability of a prex as the fraction of time for whic hit w as reac hable in the in terv al b et w een
when the prex w as rst seen and the end of the snapshot Figure plots the cum ulativ e fraction of prexes
in a snapshot against prex a v ailabili t y In general prex a v ailabilit y is high ab out of the prexes w ere
a v ailable for less than of the time
Ho w ev er b et w een S
a
and S
b
there is some evidence of degradation in a v ailabili t y In S
a
of
the prexes are a v ailable for more than of the time In S
b
that a v ailabili tyis There are t w o
p ossible explanations for this degradation First the n um b er of routers and links in paths to prexes could
ha v e increased b et w een the t w o snapshots This increases the lik eliho o d of router or link failure in these
paths Second the errors due to routing misconguration could ha v e increased o wing to an increase in the
complexit y of the in terdomain top ology T o nearly half the prexes in eac h snapshot w esa w more than one ASPATH Section Giv en that
routing on the In ternet is based on relativ ely static metrics w e mighth yp othesize that one of these paths
is used for a signican t fraction of the time that a prex is reac hable Call this most frequen tly used path
to a prex its primary path W e can then dene the primary availability of a prex as that fraction of
its reac hable time for whic h the prex w as reac hable b y its primary path Figure sho ws the cum ulativ e
distribution of the fraction of prexes against primary a v ailabilit yF or more than of the prexes the
primary a v ailabili t y is greater than This indicates that the outages in the primary most preferred
path are relativ ely shortliv ed for most prexes The gure also sho ws impro v ed primary a v ailabil ityin S
b

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Primary Availability
Cumulative fraction of total prefixes
S
a
b
S
Figure Pr ex Primary A vailability Prex primary a v ailabil it y is dened as the fraction of a prexs reac hable
time that it is reac hable b y its primary path More than of the prexes are reac hable through a primary path
for more than of the time
despite decreased o v erall a v ailabili t y Prex Steadiness
W e dene the ste adiness of a prex as the mean duration of all in terv als in a snapshot o v er whic h the prex
w as con tin uously reac hable Figure plots the cum ulativ e distribution of the fraction of prexes seen in eac h
snapshot against steadiness There is a wide v ariation in prex steadiness Remark able ho w ev er is the fact
that almost of the prexes ha v e steadiness v alues of more than da y" There is some evidence that prex
steadiness decreases with gro wth in the In ternet S
b
s distribution of prex steadiness is almost completely
to the left of that of S
a
F urthermore the steadiness distribution sho ws sev eral steps indicating that
there exist clusters of prexes with similar steadiness b eha vior A closer examination of our traces rev eals
that prexes in a cluster app ear to share a common primary path
Weha v e already sho wn that to most prexes there exists a highly a v ailable primary path W e migh t
h yp othesize that a prexs steadiness is determined b ythe n um b er of routers and links in its primary path
T o a rst appro ximation w e study a prexs steadiness as a function of the n um b er of domains in its primary
path Figure plots the cum ulativ e distribution of the fraction of prexes seen in eac h snapshot against
steadiness for dieren t path lengths In a distribution sense the steadiness of prexes app ears to generally
decrease with path length There is one exception In S
b
prexes whose primary path length is three app ear
steadier than prexes whose primary path length is t w o A p ossible explanation for this is the existence of
other factors suc h as routing misconguration that aect steadiness

4 days 8 days 12 days 16 days 20 days
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Steadiness
Cumulative fraction of total prefixes
S
a
b
S
Figure Pr ex Ste adiness This gure plots the cum ulativ e distribution of prex steadiness There is a wide
v ariation in prex steadiness but nearly of the prexes ha v e a steadiness greater than da y Discussion
The ab o v e sections describ e the distributions of pr ex a v ailabilit y and steadiness Prexes dier in the
n um b er of destinations they co v er Little is kno wn ab out the address space utilization of IP prexes A
rst order appro ximation is that this utilization is prop ortional to the address space co v ered b y a prex Using
this appro ximation w e can estimate the distribution of host a v ailabilit y and steadiness These distributions
are sho wn in Figure The distribution of host a v ailabilit y Figure sho ws that in S
a
nearly of hosts
are a v ailable for more than of the duration of our snapshot in terv al In S
b
only of the hosts are
a v ailable for more than of the time The steadiness curv es are similar to those in Figure Ho w ev er
the disparit y in steadiness b et w een S
a
and S
b
is greater in Figure Our prex stabilit y b eha vior is as observ ed from a single p ointonthe bac kb one mesh The b eha vior of
individual prexes is lik ely to dier when observ ed at dieren tpoin ts on this mesh Ho w ev er w e b eliev e
that the distribution of prex a v ailabil it y and steadiness is lik ely to b e similar at other bac kb one pro viders
Related W ork
T o our kno wledge no prior w ork has attempted to c haracterize the In ternet in terdomain top ology and its
gro wth Sev eral researc hers ha v e prop osed in terdomain top ology mo dels These mo dels p osit a
three or fourlev el pro vider hierarc h y Some of these mo dels also assume that domains in a class primarily
connect to domains in classes ab o veorbelo w their o wn Our analysis conrms the existence of a fourlev el
hierarc h yHo w ev er our classication exp oses a signican tlev el of connectivit y within classes as w ell as

5 days 10 days 15 days 20 days
0
0.2
0.4
0.6
0.8
1
Steadiness
Cumulative frac. of prefixes
S
a
Path lengths
5 days 10 days 15 days 20 days
0
0.2
0.4
0.6
0.8
1
Steadiness
Cumulative frac. of prefixes
S
b
1
2
3
4
Figure Pr ex ste adiness for dier ent p ath lengths This gure plots the cum ulativ e distribution of prex
steadiness for dieren t primary path lengths for the t w o snapshots The distributions are consisten tly w orse for
longer path lengths with one exception in S b for path lengths and connectivitybet w een nonadjacen t classes
Tw o other studies ha v e attempted to c haracterize In ternet route stabilit y One analyzes a hour
trace of routing trac on the no w defunct NSFNET bac kb one net w ork The In ternet top ology has
c hanged signican tly since then This study analyzes routing proto col b eha vior the frequency and size of
up dates routing proto col con v ergence times and so on It do es not analyze prex a v ailabili t y and steadiness
c haracteristics
A second more recen t study uses the traceroute to ol to sample a v ailabilit y and steadiness c har
acteristics of paths b et w een a small set of host pairs Their endtoend path a v ailabili t y estimates of or
higher app ear to matc h our estimated host a v ailabili t y distribution they to o observ e a degradation in host
a v ailabili t y with In ternet gro wth Their distribution of the path with the highest a v ailabili t yobserv ed at
AS gran ularit y b et w een host pairs closely matc hes our primary prex a v ailabili t y distribution
Conclusions
There are t w o ma jor conclusions ab out In ternet in terdomain routing that emerge from our ndings First
the In ternet has gro wn laterally Second route stabilit ybeha vior has degraded with gro wth A t a high
lev el these conclusions are not v ery surprising The ubiquit yof pro vider connectivit y aorded b yIn ternet
exc hange p oin ts encourages lateral gro wth man ypro viders at dieren tlev els and constan tn um ber of
lev els rather than v ertical gro wth The widespread use of man ual routing conguration on the In ternet
to da y and the lac k of sophisticated routing analysis and debugging to ols probably accoun ts for the decreased

0 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Availability
Cumulative frac. of addr. space
5 days 10 days 15 days 20 days
0
0.2
0.4
0.6
0.8
1
Steadiness
Cumulative frac. of addr. space
S
a
S
b
S
a
S
b
Figure A vailabilit y and ste adiness of p aths to hosts These graphs plot the estimated distribution of host
a v ailabil it y and steadiness
route stabilit y W e see no reason to b eliev e that the pattern of gro wth in the in terdomain top ology will c hange signif
ican tly in the near future Ho w ev er an increasingly commercial and comp etitiv e infrastructure is lik ely to
spur the dev elopmen t of automated routing analysis and engineering to ols resulting in b etter route stabilit y References
C Alaettino# glu Scalable Router Conguration for the In ternet Pr o c e e dings of the International
Confer enc e on Networking Pr oto c ols Octob er C Alaettino# glu T Bates E Geric h D Karren b erg M T erpstra and C Villami zar Routing P olicy
Sp ecication Language RPSL In ternet Draft draftietfrpsrpsltxt Net w ork Information Cen ter
Marc h Revised June C Alaettino# glu and AUShank ar The Viewserv er Hierarc h y for In terdomain Routing Proto cols and
Ev aluation Journal on Sele ctedA r e as in Communic ations Octob er T Bates E Geric h L Jonc hera y JM Jouanigot D Karren b erg M T erpstra and J Y u Repre
sen tation of ip routing p olicies in a routing registryT ec hnical Rep ort rip e RIPE RIPE NCC
Amsterdam Netherlands Octob er
J C Bolot Endtoend P ac k et Dela y and Loss Beha vior in the In ternet In Pr o c e e dings of the A CM
SIGCOMM Symp osium on Communic ation A r chite ctur es and Pr oto c ols pages San F rancisco
CA Sep E J Bos Routing T able Gro wth httpnicsurf netn lnet mana geme ntip cidr B Chino y Dynamics of In ternet Routing Information In Pr o c e e dings of the A CM SIGCOMM Symp o
sium on Communic ation A r chite ctur es and Pr oto c ols pages San F rancisco CA Sep MFS Datanet MaeEast T rac Statistics httpwwwmfsda tane tcom MAE east stats html D Estrin Y Rekh ter and S Hotz A Scalable In terDomain Routing Arc hitecture In Pr o c e e dings of the
A CM SIGCOMM Symp osium on Communic ation A r chite ctur es and Pr oto c ols pages Baltimore
MD August National Science F oundation Net w ork Access P oin t Manager Routing Arbiter Regional Net w ork
Pro viders and V ery HighSp eed Services Pro vider for NSFNET and NRENSM Programs Program
Solicitation Ma y V F uller T Li J Y u and K V aradhan Classless In terDomain Routing CIDR An Address Assign
men t and Aggregation Strategy Request for Commen ts In ternic Directory Services Septem ber
S M Hotz R outing Information Or ganization T o Supp ort Sc alable Inter domain R outing With Heter o
gene ous Path R e quir ements PhD thesis Univ ersit y of Southern California M Lottor In ternet Domain Surv ey T ec hnical rep ort Net w ork Information Systems Cen ter SRI
In ternational Octob er V P axson Gro wth T rends in WideArea TCP Connections IEEE Network pages JulyAugust
V P axson Endtoend Routing Beha vior in the In ternet In Pr o c e e dings of the A CM SIGCOMM
Symp osium on Communic ation A r chite ctur es and Pr oto c olsSan F rancisco CA Septem ber Y Rekh ter and T Li A Border Gatew a y Proto col BGP Request for Commen ts In ternic
Directory Services Marc h E W Zegura K L Calv ert and Samrat Bhattac harjee Ho wto Model an In ternet w ork In Pr o c e e dings
of IEEE INF OCOMM pages San F rancisco CA Septem b er

Abstract (if available)

Linked assets

Computer Science Technical Report Archive

Conceptually similar

PDF

USC Computer Science Technical Reports, no. 706 (1999)

PDF

USC Computer Science Technical Reports, no. 723 (2000)

PDF

USC Computer Science Technical Reports, no. 631 (1996)

PDF

USC Computer Science Technical Reports, no. 603 (1995)

PDF

USC Computer Science Technical Reports, no. 760 (2002)

PDF

USC Computer Science Technical Reports, no. 822 (2004)

PDF

USC Computer Science Technical Reports, no. 746 (2001)

PDF

USC Computer Science Technical Reports, no. 731 (2000)

PDF

USC Computer Science Technical Reports, no. 841 (2005)

PDF

USC Computer Science Technical Reports, no. 704 (1999)

PDF

USC Computer Science Technical Reports, no. 741 (2001)

PDF

USC Computer Science Technical Reports, no. 773 (2002)

PDF

USC Computer Science Technical Reports, no. 777 (2002)

PDF

USC Computer Science Technical Reports, no. 782 (2003)

PDF

USC Computer Science Technical Reports, no. 902 (2009)

PDF

USC Computer Science Technical Reports, no. 961 (2015)

PDF

USC Computer Science Technical Reports, no. 931 (2012)

PDF

USC Computer Science Technical Reports, no. 774 (2002)

PDF

USC Computer Science Technical Reports, no. 745 (2001)

PDF

USC Computer Science Technical Reports, no. 921 (2011)

Description

Ramesh Govindan, Anoop Reddy. "An analysis of internet inter-domain topology and route stability." Computer Science Technical Reports (Los Angeles, California, USA: University of Southern California. Department of Computer Science) no. 642 (1996).

Asset Metadata

Creator Govindan, Ramesh (author), Reddy, Anoop (author)

Core Title USC Computer Science Technical Reports, no. 642 (1996)

Alternative Title An analysis of internet inter-domain topology and route stability (title)

Publisher Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA (publisher)

Tag OAI-PMH Harvest

Format 22 pages (extent), technical reports (aat)

Language English

Unique identifier UC16270318

Identifier 96-642 An Analysis of Internet Inter-Domain Topology and Route Stability (filename)

Legacy Identifier usc-cstr-96-642

Format 22 pages (extent),technical reports (aat)

Rights Department of Computer Science (University of Southern California) and the author(s).

Internet Media Type application/pdf

Source 20180426-rozan-cstechreports-shoaf (batch), Computer Science Technical Report Archive (collection), University of Southern California. Department of Computer Science. Technical Reports (series)

Access Conditions The author(s) retain rights to their work according to U.S. copyright law. Electronic access is being provided by the USC Libraries, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name USC Viterbi School of Engineering Department of Computer Science

Repository Location Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089

Repository Email csdept@usc.edu

Inherited Values

Title Computer Science Technical Report Archive

Coverage Temporal 1991/2017

Repository Email csdept@usc.edu

Repository Name USC Viterbi School of Engineering Department of Computer Science

Repository Location Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089

Publisher Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA (publisher)