University of Southern California Dissertations and Theses

The extraction and complexity limits of graphical models for linear codes

(USC Thesis Other)

The extraction and complexity limits of graphical models for linear codes

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content THE EXTRACTION AND COMPLEXITY LIMITS OF GRAPHICAL MODELS FOR LINEAR CODES by Thomas Rhys Halford A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (ELECTRICAL ENGINEERING) May 2007 Copyright 2007 Thomas Rhys Halford Acknowledgements I am indebted first and foremost to my advisor Keith Chugg. I couldn’t have asked for a better mentor. I thank you Keith for the countless hours we spent discussing research problems and for your constant advocacy for my academic career. A number of faculty members, both inside and outside of USC, have also assisted my research and contributed to my professional development. Thanks are due to Richard Arratia, Todd Brun, Jim Cavers, Solomon Golomb, Vijay Kumar, Urbashi Mitra, Susan Montgomery, Armand Tanguay, Chuck Weber, Lloyd Welch, and Zhen Zhang. Special thanks are due to Dave Forney, Alex Grant, Frank Kschischang, and Robert Scholtz for their assistance in my academic job search. Special thanks go to the many members of the Viterbi School staff. Thanks are due to Margery Berti, Tim Boston, Diane Demetras, and Marilee Reynolds for administering myfellowshipandassistantships. ThanksareduetoMillyMontenegro,GerrielynRamos, and Mayumi Thrasher for making the CSI a great place to spend five years. Finally, I have been extremely fortunate to have the support of my parents, my brother, Marcie, and Marcie’s parents during these past five years. I dedicate this disser- tation to Marcie and thank you for your support and love. ii Table of Contents Acknowledgements ii List Of Tables v List Of Figures vi Abstract xi Chapter 1: Introduction 1 1.1 Systems and Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Graphical Models for Inference . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Model Construction and Extraction . . . . . . . . . . . . . . . . . . . . . 3 1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 2: Graphical Models for Linear Codes: One Code, Many Models 6 2.1 Review of Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Graphical Models of Codes . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Seven Models for the Extended Hamming Code . . . . . . . . . . . . . . . 12 2.4 From Graphical Model to Decoding Algorithm . . . . . . . . . . . . . . . 19 2.5 Extraction: The Coding Perspective . . . . . . . . . . . . . . . . . . . . . 22 Chapter 3: The Complexity Limits of Graphical Models for Linear Codes 24 3.1 Summary of Known Results . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2 Which Codes Have 4-Cycle-Free Tanner Graphs? . . . . . . . . . . . . . . 30 3.3 Which Codes Have Cycle-Free Generalized Tanner Graphs? . . . . . . . . 43 3.4 A Complexity Measure for Graphical Models . . . . . . . . . . . . . . . . 48 3.5 The Tradeoff Between Cyclic Topology and Complexity . . . . . . . . . . 53 3.6 Appendix: Graphical Model Properties by Example . . . . . . . . . . . . 61 Chapter 4: The Extraction of Graphical Models for Linear Codes I 72 4.1 Cycle-Free Models for Reed-Solomon Codes . . . . . . . . . . . . . . . . . 74 4.2 Extensions of the Reed-Solomon Code Models . . . . . . . . . . . . . . . . 84 4.3 Suboptimal Soft-In Soft-Out Decoding of Reed-Solomon Codes . . . . . . 89 4.4 Cycle-Free Models for First-Order Reed-Muller Codes . . . . . . . . . . . 99 4.5 Application: High-Rate Serially Concatenated Codes . . . . . . . . . . . . 106 4.6 Appendix: K-Shortest Paths Algorithm . . . . . . . . . . . . . . . . . . . 113 iii Chapter 5: The Extraction of Graphical Models for Linear Codes II 115 5.1 Graphical Model Transformation . . . . . . . . . . . . . . . . . . . . . . . 116 5.2 Extraction via Transformation . . . . . . . . . . . . . . . . . . . . . . . . 126 5.3 Greedy Heuristics for Cyclic Model Extraction . . . . . . . . . . . . . . . 130 5.4 Simulation Results and Discussion . . . . . . . . . . . . . . . . . . . . . . 139 5.5 The Complexity of Graphical Model Extraction . . . . . . . . . . . . . . . 143 Chapter 6: Random Redundant Iterative Decoding 145 6.1 Review of Redundant Tanner Graph Decoding . . . . . . . . . . . . . . . 147 6.2 Practical Redundant Tanner Graph Decoding . . . . . . . . . . . . . . . . 151 6.3 Empirical Performance Study . . . . . . . . . . . . . . . . . . . . . . . . . 154 6.4 Analytical Performance Study . . . . . . . . . . . . . . . . . . . . . . . . . 159 Chapter 7: Conclusions 172 7.1 Future Directions: Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 7.2 Future Directions: Beyond Coding . . . . . . . . . . . . . . . . . . . . . . 176 References 177 Appendix: Counting Short Cycles in Bipartite Graphs 189 A.1 Review of Graph Theory and Notation . . . . . . . . . . . . . . . . . . . . 190 A.2 The Lollipop Recursion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 A.3 Lollipop Matrix Proof Structure . . . . . . . . . . . . . . . . . . . . . . . 196 A.4 Summary of the Lollipop Matrix Equations . . . . . . . . . . . . . . . . . 197 A.5 Detailed Proofs for the Lollipop Matrix Equations . . . . . . . . . . . . . 199 A.6 Short Cycle Counting Examples . . . . . . . . . . . . . . . . . . . . . . . . 213 A.7 Complexity of Counting Short Cycles. . . . . . . . . . . . . . . . . . . . . 216 iv List Of Tables 3.1 Application of Theorem 3.7 to theC 0 S(2,2;v) family of codes for 2≤v≤10. 37 3.2 Application of Theorem 3.7 to Reed-Muller codes. . . . . . . . . . . . . . 39 3.3 Application of Theorem 3.7 to primitive BCH codes. . . . . . . . . . . . . 40 4.1 ComplexitycomparisonofproposedoptimalSISOandWolftrellisdecoders for various Reed-Solomon codes. . . . . . . . . . . . . . . . . . . . . . . . 83 4.2 Complexity of trellis decoding vs. the proposed optimal SISO decoding algorithm for first-order Reed-Muller codes. . . . . . . . . . . . . . . . . . 105 4.3 Average number of operations per input bit, per decoding iteration, for optimal SISO decoding of the mixed extended Hamming codes. . . . . . . 112 5.1 Generalizedcodeextensiondegreescorrespondingtothe4-cycle-freeGTGs obtained via the SV, KM, and HC heuristics. . . . . . . . . . . . . . . . . 135 5.2 Short cycle structure of the initial and extracted graphical models for the [32,21,6] extended BCH code.. . . . . . . . . . . . . . . . . . . . . . . . . 141 5.3 Short cycle structure of the initial and extracted graphical models for the [64,51,6] extended BCH code.. . . . . . . . . . . . . . . . . . . . . . . . . 142 A.1 Short cycle structure of the Code A and B Tanner graphs. . . . . . . . . . 214 A.2 Short cycle structure of the Code C and D Tanner graphs. . . . . . . . . . 215 A.3 Short cycle statistics of the Code C and D Tanner graphs. . . . . . . . . . 216 v List Of Figures 1.1 A generic systemS mapping inputs{a i } k i=1 to outputs{x j } n j=1 . . . . . . . 1 2.1 Star graph for the [8,4,4] extended Hamming codeC H . . . . . . . . . . . 13 2.2 Tanner graph forC H implied by (2.12). . . . . . . . . . . . . . . . . . . . . 14 2.3 Dualized Tanner graph forC H implied by (2.11). . . . . . . . . . . . . . . 14 2.4 Redundant Tanner graph forC H implied by (2.13). . . . . . . . . . . . . . 15 2.5 Generalized Tanner graph forC H implied by (2.14).. . . . . . . . . . . . . 16 2.6 Tail-biting trellis diagram forC H . Transitions corresponding to 0 and 1 inputs are represented by solid and dashed lines, respectively. . . . . . . . 17 2.7 Tail-biting trellis graphical model forC H . . . . . . . . . . . . . . . . . . . 17 2.8 Conventional trellis diagram forC H . Transitions corresponding to 0 and 1 inputs are represented by solid and dashed lines, respectively. . . . . . . . 18 2.9 Conventional trellis graphical model forC H . . . . . . . . . . . . . . . . . . 18 3.1 Girth 6 generalized Tanner graph for the [8,4,4] extended Hamming code. 44 3.2 Girth 10 generalized Tanner graph for the [8,4,4] extended Hamming code. 45 3.3 Removal of a degree-2 single parity-check constraint in a cycle-free gener- alized Tanner graph. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.4 The q s(C) -ary graphical model representation of a trellis section. . . . . . . 51 3.5 Minimum number of cycles required for 2 m -ary graphical models of the binary image of the [255,223,33] Reed-Solomon code. . . . . . . . . . . . 60 vi 3.6 Tail-bitingtrellisgraphicalmodelforthelength8extendedHammingcode C H . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.7 Insertion of hidden variable S 9 and internal local constraintC 9 into the tail-biting trellis forC H . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.8 Removal of internal local constraintC 9 from the tail-biting trellis forC H . . 65 3.9 Cycle-free binary graphical model forC \9 H . The minimal spanning tree containing the interface constraints which involve V 1 , V 2 , V 5 , and V 8 , re- spectively, is highlighted.. . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.1 Structure of the generator matrix of the binary image ofC RS . . . . . . . . 75 4.2 Cycle-free graphical model for Reed-Solomon codes based on Vardy and Be’ery’s decomposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3 Generic glue code list generation algorithm. . . . . . . . . . . . . . . . . . 90 4.4 List generation for [n,n−2,3] RS codes algorithm. . . . . . . . . . . . . . 92 4.5 Structure of the generator matrix of the binary image ofT for the [n,n− 4,5] Reed-Solomon codes where bits from each syndrome are grouped to- gether. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.6 List generation for [n,n−4,5] RS codes algorithm. . . . . . . . . . . . . . 95 4.7 Codeword error rate performance comparison of the proposed optimal SISO, proposed suboptimal SISO, and standard list-based SISO for the [15,13,3] and [15,11,5] RS codes. List sizes appear in parentheses. . . . . 97 4.8 Bit error rate performance comparison of the proposed optimal SISO, proposed suboptimal SISO, and standard list-based SISO for the rate 13 17 [15,13,3]×[15,13,3] RS turbo product code. Ten decoding iterations were performed. List sizes appear in parentheses. . . . . . . . . . . . . . . . . . 98 4.9 Generalized Tanner graph forC H implied by (4.43).. . . . . . . . . . . . . 99 4.10 Conditionally cycle-free generalized Tanner graph forC H implied by (4.43). 100 4.11 Generalized Tanner graph forC H implied by a squaring construction. . . . 102 4.12 Generalized Tanner graph for the [2 m ,m+1,2 m−1 ] Reed-Muller code im- plied by a squaring construction. . . . . . . . . . . . . . . . . . . . . . . . 104 vii 4.13 Potential structures for high-rate, low-floor codes for use in systems with recursive channels. Random-like interleavers are labeled π. . . . . . . . . . 108 4.14 Bit error rate performance of the respective rate 8/9 and 11/12 serially concatenated codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.15 Bit error rate performance of the respective rate 9/10 and 16/17 serially concatenated codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.16 Next shortest path algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.1 Local constraint merging notation. The local constraintsC c 1 andC c 2 are common. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.2 Cycle-free 4-ary graphical model forC H . . . . . . . . . . . . . . . . . . . . 119 5.3 The merging of constraintsC 9 andC 10 in a 4-ary graphical model forC H . The resulting graphical model is 8-ary. . . . . . . . . . . . . . . . . . . . . 120 5.4 Insertion and removal of degree-2 repetition constraints. . . . . . . . . . . 123 5.5 The insertion/removal of an isolated partial parity-check constraint on V 7 and V 8 in a Tanner graph forC H . . . . . . . . . . . . . . . . . . . . . . . . 125 5.6 The transformation ofG C into e G C via five sub-transformations. . . . . . . 127 5.7 Transformation of the q m -ary hidden variable S j into q-ary hidden variables.128 5.8 Greedy heuristic for the reduction of short cycles in Tanner graphs for binary codes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 5.9 Bit error rate performance of three GTG decoding algorithms for the [31,21,5] BCH code. One-hundred iterations of a flooding schedule were performed. Binary antipodal signaling over an additive white Gaussian noise (AWGN) channel is assumed. . . . . . . . . . . . . . . . . . . . . . . 135 5.10 Heuristic for generating candidate partial parity symbols. . . . . . . . . . 137 5.11 Greedy heuristic for the removal of 4-cycles in binary generalized Tanner graphs.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 5.12 Greedy heuristic for the extraction of 2 m -ary graphical models. . . . . . . 138 viii 5.13 Biterrorrateperformanceofdifferentdecodingalgorithmsforthe[32,21,6] extendedBCHcode. Fiftyiterationsofafloodingschedulewereperformed for all of the suboptimal SISO decoding algorithms. . . . . . . . . . . . . 141 5.14 Biterrorrateperformanceofdifferentdecodingalgorithmsforthe[64,51,6] extendedBCHcode. Fiftyiterationsofafloodingschedulewereperformed for all of the suboptimal SISO decoding algorithms. . . . . . . . . . . . . 142 6.1 Degree-2 redundant Tanner graph for the [8,4,4] extended Hamming code corresponding to the concatenation of the rows of H 1 and H 2 . . . . . . . . 148 6.2 Random redundant iterative decoding. . . . . . . . . . . . . . . . . . . . . 152 6.3 Bit error rate performance comparison of different decoding algorithms for the [24,12,8] extended Golay code. . . . . . . . . . . . . . . . . . . . . . . 156 6.4 Bit error rate performance comparison of different decoding algorithms for the [63,39,9] BCH code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 6.5 Pseudo-weight spectra of three graphical models for the [8,4,4] extended Hamming code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 6.6 Bit error rate performance comparison of different decoding algorithms for the [8,4,4] extended Hamming code. . . . . . . . . . . . . . . . . . . . . . 171 6.7 Bit error rate performance comparison of different decoding algorithms for the [24,12,8] extended Golay code. . . . . . . . . . . . . . . . . . . . . . . 171 A.1 A (4,2)-lollipop walk where α 7 =α 5 while α 1 ,α 2 ,...,α 6 are distinct. . . . 191 A.2 A (2,4)-lollipop walk where α 7 =α 3 while α 1 ,α 2 ,...,α 6 are distinct. . . . 191 A.3 An (m,n−m)-lollipop walk from α 1 to α n+1 =α m+1 where m is even.. . 196 A.4 A (2,6)-lollipop walk from α 1 to α 9 =α 3 . . . . . . . . . . . . . . . . . . . 197 A.5 A (1,2)-lollipop walk from α 1 to α 4 =α 2 . . . . . . . . . . . . . . . . . . . 199 A.6 A (2,2)-lollipop walk from α 1 to α 5 =α 3 . . . . . . . . . . . . . . . . . . . 200 A.7 A (2k−1,2)-lollipop walk from α 1 to α 2k+2 =α 2k . . . . . . . . . . . . . . 200 A.8 A (2k,2)-lollipop walk from α 1 to α 2k+3 =α 2k+1 . . . . . . . . . . . . . . . 201 A.9 A (g−1,2)-lollipop walk from α 1 to α g+2 =α g . . . . . . . . . . . . . . . . 202 ix A.10A (g,2)-lollipop walk from α 1 to α g+3 =α g+1 . . . . . . . . . . . . . . . . 203 A.11A (g+1,2)-lollipop walk from α 1 to α g+4 =α g+2 . . . . . . . . . . . . . . 204 A.12The case where α 1 =α g+1 and α 2 =α g+2 . . . . . . . . . . . . . . . . . . . 206 A.13The case where α 1 =α 5 and α 3 =α 7 for g =4. . . . . . . . . . . . . . . . 206 A.14A (1,g)-lollipop walk from α 1 to α g+2 =α 2 . . . . . . . . . . . . . . . . . . 207 A.15A (2,g)-lollipop walk from α 1 to α g+3 =α 3 . . . . . . . . . . . . . . . . . . 208 A.16A (3,g)-lollipop walk from α 1 to α g+4 =α 4 . . . . . . . . . . . . . . . . . . 209 A.17The case where α 1 =α 3 and α 2 =α 6 for g =4. . . . . . . . . . . . . . . . 210 A.18A (1,g+2)-lollipop walk from α 1 to α g+4 =α 2 . . . . . . . . . . . . . . . . 211 x Abstract A decade ago, the introduction of turbo codes and iterative message passing algorithms revolutionized the theory and practice of coding. In the ensuing years, the coding theory community has become adept at designing codes from good graphical models - that is, models which imply low-complexity, near-optimal iterative message passing algorithms. Specifically, modern codes are constructed by connecting a large number of simple local codes together via a rich, random-like, cyclic interconnection network. A key observation from this work is that the introduction of cycles to graphical models can enable massive complexity reductions in model, and thus decoding, complexity. Whereas constructive graphical modeling problems (e.g. code design) have been widely addressed by the coding theory community, less is understood about the inverse problem of model extraction. Specifically, can good graphical models be obtained for existing algebraic codes, or more generally, for arbitrary systems? What tradeoffs exist betweenmodelcomplexityandcyclictopologyforagivencode? Ifgoodmodelscanexist, how can they be obtained, or extracted? This dissertation presents a theoretical framework for the study of extractive graph- ical modeling problems. The limits of extraction are first considered and a number of new results are presented on the space of graphical models for a given (fixed) code. Most xi significantly, a new characterization of the tradeoff between cyclic topology and complex- ity in graphical models for linear codes is provided. Inasmuch as the cyclic topology of a graphical model is related to the performance of the decoding algorithms it implies, this tree-inducing cut-set bound provides insight into the fundamental limits of graphical model extraction. Extraction is then treated formally using the language of combina- torial optimization and a number of novel heuristics for both defining and solving this optimization problem are presented. The results of a number of related problems that arose in the aforementioned study of graphical model extraction are also reported. Novel optimal soft-in soft-out (SISO) decodingalgorithmsaredescribedforReed-Solomoncodesandforfirst-orderReed-Muller codes. A practically realizable - yet remarkably successful - suboptimal SISO decoding algorithm for arbitrary linear block codes based on massively redundant Tanner graphs is also developed. Finally, an efficient algorithm for counting short cycles in bipartite graphs is described. xii Chapter 1 Introduction 1.1 Systems and Inference x n x 1 x 2 a 2 a k a 1 S Figure 1.1: A generic systemS mapping inputs{a i } k i=1 to outputs{x j } n j=1 . Consider the systemS pictured in Figure 1.1 that maps a set of k inputs{a i } k i=1 to a set of n outputs {x j } n j=1 . Let the inputs be discrete random variables drawn from the alphabets{A i } k i=1 and let a specific realization of the random vector a(ζ) = (a 1 (ζ),a 2 (ζ),...,a k (ζ)) (where ζ is the sample space variable) be denoted ˆ a: ˆ a=(ˆ a 1 ,...,ˆ a k )∈A 1 ×···×A k =A. (1.1) The outputs are random variables that may be discrete or continuous and are described by similar notation. 1 It is assumed that the probability mass function of a(ζ) p a(ζ) (ˆ a)=Pr{a(ζ)= ˆ a}, (1.2) is known and thatS is specified by the density of the outputs conditioned on a given input: p x(ζ) (ˆ x|a(ζ)= ˆ a). (1.3) The goal of inference (in this most general formulation) is to determine the input density conditioned on an observation ˆ x of the outputs: p a(ζ) (ˆ a|x(ζ)= ˆ x)= p x(ζ) (ˆ x|a(ζ)= ˆ a)p a(ζ) (ˆ a) P a 0 ∈A p x(ζ) (ˆ x|a(ζ)=a 0 )p a(ζ) (a 0 ) . (1.4) The inference problem can thus be solved exhaustively by evaluating (1.3) for all ˆ a∈A and applying Bayes’ rule. However, such an approach is generally prohibitively complex due to the enormity of|A|. Tractable solutions of the inference problem thus require efficient methods for evaluating (1.3) for all ˆ a∈A. 1.2 Graphical Models for Inference In the language of behavioral system modeling [153, 154], the system pictured in Figure 1.1 defines a global constraint on configurations of the input and output variables. The inference problem can be greatly simplified by factoring the global constraint into a set of local constraints where each local constraint acts on subsets of the input and output variable sets. There exist well-known graphical models of the relationships between these 2 local constraints in a number of fields, e.g. Markov random fields [81, 90, 126], Bayesian belief networks [82, 101, 120], junction trees [2], and factor graphs [97]. Such graphical models for inference have been used in a wide range of fields including statistical physics [58], genetics [28], research medicine [41], machine learning [55], image processing [30], data compression [55], and communications [30]. Given a graphical model of a system there exists a well-known algorithm for inference which operates by passing messages between vertices representing local constraints and those representing input and output variables according to a specified schedule (cf. [30] and the references therein for a detailed treatment). If the graphical model is cycle-free then there exists a message passing schedule that solves the inference problem exactly. If the graphical model contains cycles then message passing algorithms yield solutions that are approximate, yet often substantially less complex. 1.3 Model Construction and Extraction The map between graphical models and inference algorithms is well-understood. In cer- tain contexts such as code design (as will be discussed in Chapter 2), systems can be designed with graphical model-based inference algorithms in mind. The vast majority of inference problems, however, involve fixed systems for which models must be sought. Two broad classes of graphical modeling problems can thus be identified: • Constructive problems: Given a set of design requirements, design a suitable sys- tem by constructing a good graphical model (i.e. a model which implies a low- complexity, near-optimal inference algorithm). 3 • Extractive problems: Given a specific (fixed) system, extract a graphical model for that system with desired complexity and performance characteristics. This dissertation focuses on extractive graphical modeling problems. A number of such problems have been solved successfully. Good cyclic graphical models of sparse intersymbol interference (S-ISI) channels were presented in [25]. Good cyclic graphical models for two-dimensional (2D) data detection were presented in [26, 31, 32]. Within the coding community, a good single-cycle model for the Golay code (i.e. a tail-biting trellis) is known [22]. Sparse graphical models for certain pseudo-noise (PN) sequences are also known [157]. Theredoesnotasyetexistatheoreticalframeworkforunderstandinggraphicalmodel extraction. Nor do there exist systematic procedures for extraction. Successful model ex- traction solutions have thus far depended on the exploitation of system-specific algebraic properties, as in the case of tail-biting trellises and 2D data detection, and on ingenuity on the part of the designer, as in the case of S-ISI channels and PN sequences. In order to fully realize the potential of cyclic graphical models for inference, a more formal understanding of extraction is clearly required. To this end, the primary aim of thisdissertationistolayoutsomeofthefoundationsofthetheoryofextractivegraphical modelingproblems. Aspecificclassofsystems-linearerror-correctingcodes-isthefocus of this work. Cyclic graphical modes have seen their most celebrated use in the context of constructing modern codes and coding thus provides a logical place to begin a formal treatment of model extraction. 4 1.4 Outline The remainder of this dissertation is organized as follows. Chapter 2 provides the neces- sary background on graphical models for codes as well as a reformulation of the problem of cyclic graphical model extraction in the context of codes. Chapter 3 studies the space of graphical models for a given (fixed) code and presents a number of new results on the tradeoff between the cyclic topology and complexity of graphical models for linear codes, the most significant of which is the tree-inducing cut-set bound (TI-CSB). Inasmuch as the cyclic topology of a graphical model is related to the performance of the decoding algorithms it implies, the TI-CSB provides insight into the limits of graphical model extraction. Chapters 4 and 5 study the extraction of novel graphical models for codes. Chapter 4 focuses on the extraction of cycle-free graphical models from algebraic struc- ture while Chapter 5 focuses on the extraction of cyclic graphical models by formulating extraction as an optimization problem. Chapter 6 presents an alternate approach to the development of practically realizable decoding algorithms and in turn demonstrates that thereremainmanyinterestingopenproblemsconcerningtheextractionofgraphicalmod- els for codes. Directions for future work and concluding remarks are given in Chapter 7. Finally, a complete derivation of the algorithm for counting short cycles in bipartite graphs which is utilized in Chapters 5 and 6 is appended. 5 Chapter 2 Graphical Models for Linear Codes: One Code, Many Models Chapter 1 described graphical model extraction in the context of abstract systems. In this chapter, graphical models for the specific class of systems on which this dissertation focuses - linear codes - are examined in detail. Section 2.1 provides a brief introduction to linear codes and also describes the notation used throughout this dissertation. 1 The graphical modeling convention adopted in this dissertation is detailed in Section 2.2. Sec- tion 2.3 gives an example of the multitude of models that even a very simple code can provide. The decoding algorithms implied by graphical models for codes are described in Section 2.4. Section 2.4 also summarizes the known results on understanding the per- formance of iterative decoding algorithms for linear codes. Finally, the constructive and extractiveproblemsthatweredescribedforgeneralsystemsinChapter1arereformulated and elaborated upon in the context of linear codes in Section 2.5. 1 A brief review of graph theory is provided separately in the appendix. 6 2.1 Review of Codes 2.1.1 Notation ThesetofintegersandfieldofrealnumbersaredenotedbyZandR,respectively. Subsets of consecutive integers are denoted [a,b]={a,a+1,...,b−1,b} (2.1) where a < b∈ Z. The binomial coefficient is denoted a b where a,b∈ Z are integers. Vectors and matrices are denoted by bold-faced lower-case letters (e.g. v v v) and upper-case letters (e.g. M), respectively. With the exception of Chapter 6, vectors and matrices are indexed from 1 throughout. The finite field with q elements is denotedF q . Given a finite index set I, the vector space overF q defined on I is the set of vectors F I q ={f f f =(f i ∈F q ,i∈I)}. (2.2) Suppose that J⊆I is some subset of the index set I. The projection of a vector f f f∈F I q onto J is denoted f f f |J =(f i ,i∈J). (2.3) 2.1.2 Codes, Projections, and Subcodes Given a finite index set I, a linear code over F q defined on I is some vector subspace C⊆F I q . The block length and dimension ofC are denoted n(C) =|I| and k(C) = dimC, respectively. If known, the minimum Hamming distance ofC is denoted d(C) andC may 7 bedescribedbythetriplet[n(C),k(C),d(C)]. Thisdissertationconsidersonlylinearcodes and the terms code and linear code are used interchangeably. A codeC can be described by an r G ×n(C), r G ≥k(C), generator matrix G C overF q , the rows of which spanC. An r G ×n(C) generator matrix is redundant if r G is strictly greater than k(C). A codeC can also be described by an r H ×n(C), r H ≥ n(C)−k(C), parity-check matrix H C over F q , the rows of which span the null space ofC (i.e. the dual codeC ⊥ ). Each row of H C defines a q-ary single parity-check equation which every codewordinC mustsatisfy. Anr H ×n(C)parity-checkmatrixisredundant ifr H isstrictly greater than k(C ⊥ )=n(C)−k(C). (2.4) Given a subset J⊆ I of the index set I, the projection ofC onto J is the set of all codeword projections: C |J = c c c |J ,c c c∈C . (2.5) Closely related toC |J is the subcodeC J : the projection onto J of the subset of codewords satisfying c i =0 for i∈I\J. BothC |J andC J are linear codes. Suppose thatC 1 andC 2 are two codes overF q defined on the same index set I. The intersectionC 1 ∩C 2 ofC 1 andC 2 is a linear code defined on I comprising the vectors in F I q that are contained in bothC 1 andC 2 . Finally, suppose thatC a andC b are two codes defined on the disjoint index sets J a and J b , respectively. The Cartesian productC =C a ×C b is the code defined on the index set J ={J a ,J b } such thatC |Ja =C Ja =C a andC |J b =C J b =C b . 8 2.1.3 Generalized Extension Codes LetC be a linear code overF q defined on the index set I. Let J⊆I be some subset of I and let β β β =(β j 6=0∈F q ,j∈J) (2.6) be a vector of non-zero elements ofF q . A generalized extension ofC is formed by adding a q-ary parity-check on the codeword coordinates indexed by J toC (i.e. a q-ary partial parity symbol). The generalized extension code e C is defined on the index set e I =I∪{p} such that ifc c c=(c i ,i∈I)∈C thene c c c=(e c i ,i∈ e I)∈ e C wheree c i =c i if i∈I and e c p = X j∈J β j c j . (2.7) Thelengthanddimensionof e C aren( e C)=n(C)+1andk( e C)=k(C), respectively, andthe minimum distance of e C satisfies d( e C)∈{d(C),d(C)+1}. Note that if J = I and β j = 1 for all j∈ J, then e C is simply a classically defined extended code [111]. More generally, a degree-g generalized extension ofC is formed by adding g q-ary partial parity symbols toC and is defined on the index set I∪{p 1 ,p 2 ,...,p g }. The j th partial parity symbol c p j in such an extension is defined as a partial parity on some subset of I∪{p 1 ,...,p j−1 }. 9 2.2 Graphical Models of Codes 2.2.1 Normal Realizations of Codes Graphical models for codes have been described by a number of different authors using a widevarietyofnotation(e.g. [2,30,50,97,140,151]). Thisdissertationusesthenotation described below which was established by Forney in his Codes on Graphs papers [50, 51]. A linear behavioral realization of a linear codeC⊆F I q comprises three sets: • A set of visible (or symbol) variables V i ,i∈ I corresponding to the codeword coordinates 2 with alphabets F i q ,i∈I . • A set of hidden (or state) variables S i ,i∈I S with alphabets F T i q ,i∈I S . • A set of linear local constraint codes C i ,i∈I C . Each visible variable is q-ary while the hidden variable S i with alphabetF T i q is q |T i | -ary. Thehiddenvariablealphabetindexsets T i ,i∈I S aredisjointandunrelatedtoI. Each local constraint codeC i involves a certain subset of the visible, I V (i)⊆ I, and hidden, I S (i)⊆I S , variables and defines a subspace of the local configuration space: C i ⊆ Y j∈I V (i) F j q ! × Y j∈I S (i) F T j q ! . (2.8) Each local constraint codeC i thus has a well-defined block length n(C i )=|I V (i)|+ X j∈I S (i) |T j | (2.9) 2 Observe that this definition is slightly different than that proposed in [51] which permitted the use of q r -ary visible variables corresponding to r codeword coordinates. By appropriately introducing equality constraints and q-ary hidden variables, it can be seen that these two definitions are essentially equivalent. 10 anddimensionk(C i )=dimC i overF q . Localconstraintsthatinvolveonlyhiddenvariables are internal constraints while those involving visible variables are interface constraints. The full behavior of the realization is the setB of all visible and hidden variable config- urations which simultaneously satisfy all local constraint codes: B⊆ Y i∈I F i q ! × Y j∈I S F T j q ! =F I q × Y j∈I S F T j q ! . (2.10) The projection of the linear codeB onto I is preciselyC. Forney demonstrated in [50] that it is sufficient to consider only those realizations in which all visible variables are involved in a single local constraint and all hidden variables are involved in two local constraints. Such normal realizations have a natural graphical representation in which local constraints are represented by vertices, visible variables by half-edges and hidden variables by edges. The half-edge corresponding to the visible variable V i is incident on the vertex corresponding to the single local constraint which involves V i . The edge corresponding to the hidden variable S j is incident on the vertices corresponding to the two local constraints which involve S j . The notationG C and term graphical model is used throughout this dissertation to denote both a normal realization of a codeC and its associated graphical representation. It is assumed throughout that the graphical models considered are connected. Equiv- alently, it is assumed throughout that the codes studied cannot be decomposed into Cartesian products of shorter codes [50]. Note that this restriction will apply only to the global code considered and not to the local constraints in a given graphical model. 11 2.2.2 Tanner Graphs and Generalized Tanner Graphs Following Tanner’s pioneering work [140], the term Tanner graph has been used to de- scribe different classes of graphical models by different authors. Tanner graphs denote thosegraphicalmodelscorrespondingtoparity-checkmatricesinthisdissertation. Specif- ically, let H C be an r H ×n(C) parity-check matrix for the codeC overF q defined on the index set I. The Tanner graph corresponding to H C contains r H +n(C) local constraints of which n(C) are interface repetition constraints, one corresponding to each codeword coordinate, and r H are internal q-ary single parity-check constraints, one corresponding to each row of H C . An edge (hidden variable) connects a repetition constraintC i to a single parity-check constraintC j if and only if the codeword coordinate corresponding to C i is involved in the single parity-check equation defined by the row corresponding toC j . A Tanner graph forC is redundant if it corresponds to a redundant parity-check matrix. A degree-g generalized Tanner graph forC is simply a Tanner graph corresponding to some degree-g generalized extension ofC in which the visible variables corresponding to the partial parity symbols have been removed. Generalized Tanner graphs have been studied previously in the literature under the rubric of generalized parity-check matrices [108, 156]. 2.3 Seven Models for the Extended Hamming Code The [8,4,4] extended Hamming codeC H is self-dual (i.e.C ⊥ H =C H ) and generated by G H =    1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 0   . (2.11) 12 In this section, seven graphical models for this simple code are described. 2.3.1 Star Graph Conceptually, the simplest graphical model forC H contains one constraint which enforces C H . Wiberg discussed this trivial model in [151]. Figure 2.1 illustrates this model, for which the term star graph is introduced. V 1 V 2 V 3 V 4 V 5 V 6 V 7 C H V 8 Figure 2.1: Star graph for the [8,4,4] extended Hamming codeC H . 2.3.2 Tanner Graph The extended Hamming code is self-dual and (2.11) thus also defines a valid parity-check matrix forC H : H H =    1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 0   . (2.12) TheTannergraphcorrespondingto(2.12)isillustratedinFigure2.2. Notethatrepetition and single parity-check constraints are labeled by “=” and “+” symbols, respectively, throughout. 13 V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 2.2: Tanner graph forC H implied by (2.12). 2.3.3 Dualized Tanner Graph In his first Codes on Graphs paper [50], Forney described a dualizing procedure for ob- taining a graphical model forC ⊥ from one forC. This procedure simply replaces local constraints by their duals and variable alphabets by their duals (i.e. character groups). SinceC H is self-dual, applying Forney’s dualizing procedure to any graphical model for C H yields another graphical model forC H . Figure 2.3 illustrates a dualized Tanner graph forC H . Observe that this model can be equivalently derived from (2.11) as described in Section 2.7 of [30]. V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 2.3: Dualized Tanner graph forC H implied by (2.11). 14 2.3.4 Redundant Tanner Graph Aredundantparity-checkmatrixforC H canbeobtainedbyadding4additionalcodewords ofC ⊥ H to the parity-check matrix defined by (2.12): H H,R =            1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 0 0 1 0 1 0 1 0 1 1 1 0 0 0 0 1 1 1 0 1 0 1 0 1 0 0 1 0 1 1 0 1 0            . (2.13) The Tanner graph corresponding to (2.13) is redundant and is illustrated in Figure 2.4. V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 2.4: Redundant Tanner graph forC H implied by (2.13). 2.3.5 Generalized Tanner Graph Adegree-1generalizedextensionforC H canbeformedbyaddingthepartialparitysymbol p 1 =V 4 +V 8 . One parity-check matrix for this extension is e H H =      1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 0 1 1      . (2.14) The generalized Tanner graph corresponding to (2.14) is illustrated in Figure 2.5. 15 p 1 V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 2.5: Generalized Tanner graph forC H implied by (2.14). 2.3.6 Tail-biting Trellis A tail-biting trellis is a single cycle graphical model. In order to describe a tail-biting trellisforC H ,thefollowinggeneratormatrix(whichcorrespondstoadifferentbitordering than the standard bit ordering adopted in (2.11)) in considered in the place of (2.11): G H,TB =    1 1 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 0 0 0 1 1   . (2.15) Figure 2.6 illustrates the trellis diagram corresponding to the minimal tail-biting trellis impliedby(2.15)[22],whileFigure2.7illustratesthecorrespondinggraphicalmodel. The pathswhichbeginandendinthesamestateinFigure2.6areinone-to-onecorrespondence with the codewords generated by (2.15). The appendix to Chapter 3 provides definitions of the hidden variables and local constraints in the model illustrated in Figure 2.7. 16 Figure2.6: Tail-bitingtrellisdiagramforC H . Transitionscorrespondingto0and1inputs are represented by solid and dashed lines, respectively. V 8 V 1 V 2 V 3 V 4 V 5 V 6 V 7 Figure 2.7: Tail-biting trellis graphical model forC H . 2.3.7 Conventional Trellis Figure2.8illustratesthetrellisdiagramcorrespondingtotheminimalconventionaltrellis implied by (2.11) [104], while Figure 2.9 illustrates the corresponding graphical model. NotethatthelocalconstraintsinFigure2.9simplyenforcethetrellisnextstatefunctions which are shown explicitly in Figure 2.8. 17 Figure 2.8: Conventional trellis diagram forC H . Transitions corresponding to 0 and 1 inputs are represented by solid and dashed lines, respectively. V 8 V 3 V 4 V 5 V 6 V 7 V 1 V 2 Figure 2.9: Conventional trellis graphical model forC H . 18 2.4 From Graphical Model to Decoding Algorithm 2.4.1 Optimal Soft-In Soft-Out Decoding Optimal soft-in soft-out (SISO) decoding in the metric domain 3 can be formalized as follows. LetC be a block length n code over F q defined on an index set I. The (soft) inputs to SISO decoding are the input metrics SI[i][f j ]=−log(Pr[c i =f j ]) (2.16) for each i∈I and f j ∈F q . Associated with every codewordc c c∈C is the codeword metric M[c c c]= X i∈I SI[i][c i ], (2.17) which is simply the sum of the input metrics corresponding to the symbols in that code- word. Optimal SISO decoding then entails the computation of output metrics SO[i][f j ]= min c c c∈C|c i =f j M[c c c]−SI[i][f j ] (2.18) for each i∈ I and f j ∈F q . Since the number of codewords inC is exponential in n, the number of terms in this minimization is exponential in n and optimal SISO decoding is intractable in general. 3 Metrics are negative logarithms of probabilities. In this section, min-sum decoding is assumed for the purpose of illustration; however, any operations which form semi-ring can be used in place of min and sum (e.g. min ? and sum) [2, 30]. 19 2.4.2 Suboptimal SISO Decoding via Message Passing A number of authors have described the suboptimal decoding algorithms implied by graphical models (cf. [2, 30, 50, 97, 151]) and such iterative message passing algorithms (iMPAs) are now well-understood. Briefly, graphical models imply decoding algorithms which operate by iteratively passing messages between locally optimal SISO decoders for thelocalconstraints. Notethatgivenagraphicalmodel,amessagepassingschedulemust be specified in order to define a decoding algorithm. The complexity of the suboptimal SISO decoding algorithms implied by a graphical model are thus a function of both the message passing schedule (e.g. the number of iterations) and the complexity of optimal SISO decoding of the local constraints. 2.4.3 Iterative Message Passing Decoding Performance It is well-known that cycle-free graphical models imply optimal decoding algorithms whereas models with cycles imply decoding algorithms that are suboptimal in general. With the notable exception of single-cycle models (cf. [1, 5, 52, 151]), the development of tools for the analysis of iMPAs on the additive white Gaussian noise (AWGN) channel is not fully mature and there, as yet, exists no known precise functional mapping from graphical models to decoding performance. In the absence of such a precise tool, the coding community has taken two major approaches to approximating this mapping. The first approach is to precisely characterize the performance of a related decoding problem and then abstract the results to iterative decoding on the AWGN channel. For example, the performance of the iMPAs implied by Tanner graphs on the binary erasure 20 channel can be completely characterized by stopping sets [36]. Stopping sets are com- binatorial objects that have been empirically observed to successfully predict (but not precisely characterize) the performance of low-density parity-check codes on the AWGN channel. Vontobel and Koetter noted structural similarities between the linear program- ming (LP) decoder 4 implied by a parity-check matrix H and the iMPA implied by the corresponding Tanner graph TG(H) [149], and thus introduced the study of the perfor- mance of LP decoding, which can be completely characterized by pseudo-codewords, as a proxy for iterative decoding. Thesecondapproachischaracterizethosecombinatorialobjectswhichareempirically observed to be detrimental to the performance of iMPAs. One such combinatorial object is short cycle structure. Specifically, the presence of short cycles has been observed to be particularly detrimental to decoding performance (cf. [7, 50, 57, 66, 112, 140, 143, 151]). It was further observed in [66] that it is not only the number of short cycles but the distribution of those cycles which can affect decoding performance. Other such combinatorial objects which have received attention in the literature are near-codewords [110], trapping sets [130], extrinsic message degree [127, 143], and instantons [27, 139]. For a performance proxy to be useful in the context of extraction, its evaluation must be tractable. There exists a known low-complexity (i.e. polynomial-time) algorithm for the evaluation of only one of the aforementioned performance proxies: short cycle structure. An efficient algorithm for determining the short cycle structure of a graphical 4 LP decoding, which was introduced by Feldman [44], is discussed further in Chapter 6. 21 model was described in [66]. 5 Accordingly, this dissertation focuses primarily on short cycle structure as a measure of graphical model quality. 2.5 Extraction: The Coding Perspective The description of constructive and extractive graphical modeling problems provided for abstract systems in Chapter 1 can now be made more precise in the context of codes: • Constructive problems: Givenasetofdesignrequirements,designasuitablecodeby constructing a good graphical model (i.e. a model which implies a low-complexity, near-optimal decoding algorithm). • Extractive problems: Given a specific (fixed) code, extract a graphical model for that code which implies a decoding algorithm with desired complexity and perfor- mance characteristics. Constructivegraphicalmodelingproblemshavebeenwidelyaddressedbythecodingthe- orycommunity. Capacityapproachinglow-densityparity-checkcodeshavebeendesigned for both the additive white Gaussian noise channel (cf. [34, 131]) and the binary erasure channel (cf. [119, 121]). Other classes of modern codes have been successfully designed forawiderangeofpracticallymotivatedblocklengthsandrates(cf. [15,18,33,40,141]). Less is understood about extractive graphical modeling problems for codes, how- ever. The extractive problems that have received the most attention are those con- cerning Tanner graph [140] and trellis representations of block codes. Tanner graphs imply low-complexity decoding algorithms; however, the Tanner graphs corresponding 5 This short cycle counting algorithm is detailed in the appendix. 22 to many block codes of practical interest, e.g. high-rate Reed-Muller, Reed-Solomon, and Bose-Chaudhuri-Hocquenghem codes, necessarily contain many short cycles [71] and thus imply poorly performing decoding algorithms. There is a well-developed theory of conventional trellises [145] and tail-biting trellises [22, 92] for linear block codes. Con- ventional and tail-biting trellises imply optimal and, respectively, near-optimal decoding algorithms; however, for many block codes of practical interest these decoding algorithms are prohibitively complex thus motivating the study of more general graphical models (i.e. models with a richer cyclic topology than a single cycle). It is the study of such models upon which much of the remainder of this dissertation focuses. 23 Chapter 3 The Complexity Limits of Graphical Models for Linear Codes A primary aim of this dissertation is the characterization of the tradeoff between cyclic topology and complexity in graphical models for linear codes. This chapter begins with a review in Section 3.1 of the known results concerning this fundamental tradeoff, the mostprofoundofwhichareduetoWiberg[151]. Followingthisreview,constraintsonthe spaceofsimplemodelsforcodesarestudiedinSections3.2and3.3. ItisshowninSection 3.2 that the Tanner graphs for many classical block codes necessarily contain cycles of length four thus quantifying “folk-knowledge” regarding the high density of parity-check matrices for these codes. It is shown in Section 3.3 that cycle-free generalized Tanner graphs cannot support good codes. Themostsignificantcontributionsofthisdissertationconcerningthecomplexitylimits of graphical models for linear codes are developed in Sections 3.4-3.6. A new complexity measureforgraphicalmodelsisintroducedinSection3.4. Theproposedmeasurecaptures a cyclic graphical model analog of the familiar notions of state and branch complexity for trellises [145]. The minimal tree complexity of a code, which is a natural generalization of 24 the well-understood minimal trellis complexity of a code to arbitrary cycle-free models, is then defined using this measure. The tradeoff betweencyclic topologyandcomplexityingraphicalmodels is studiedin Section 3.5. Wiberg’s Cut-Set Bound (CSB) is the existing tool that best characterizes thistradeoff[151]. WhiletheCSBcanbeusedtoestablishthesquare-rootboundfortail- biting trellises [22] and thus provides a precise characterization of the potential tradeoff between cyclic topology and complexity for single-cycle models, as was first noted by Wiberg et al. [152], it is very challenging to use the CSB to characterize this tradeoff for graphical models with cyclic topologies richer than a single cycle. In order to provide a more precise characterization of this tradeoff than that offered by the CSB alone, a new bound is introduced in Section 3.5 - the tree-inducing cut-set bound - which may be viewed as a generalization of the square-root bound to graphical models with arbitrary cyclic topologies. Specifically, it is shown that an r th -root complexity reduction (with respecttotheminimaltreecomplexityasdefinedinSection3.4)requirestheintroduction of at least r(r−1)/2 cycles. The proposed bound can thus be viewed as an extension of the square-root bound to graphical models with arbitrary cyclic topologies. Finally, Section 3.6 serves as appendix to Sections 3.4 and 3.5. The results of Section 3.2 were presented first at the 2006 International Symposium on Information Theory in Seattle, WA [72] and later appeared in the IEEE Transactions on Information Theory [71]. The results of Section 3.3 were reported previously in [67]. The results of Sections 3.4-3.6 were presented first at the 2006 Allerton Conference on Communication, Control, and Computing in Monticello, IL [70] and were submitted to the IEEE Transactions on Information Theory in November of 2006 [68]. 25 3.1 Summary of Known Results 3.1.1 Which Codes Have Cycle-Free Tanner Graphs? Tanner graphs constitute the least complex class of graphical models. Codes for which there exist cycle-free Tanner graphs thus have particularly simple optimal soft-in soft-out (SISO) decoding algorithms. Etzion, Trachtenberg, and Vardy studied the class of codes for which cycle-free Tanner graphs can exist by formulating an equivalent combinatorial identity problem [43]. Specifically, they proved the following result and thus concluded that cycle-free Tanner graphs cannot support good codes. Theorem 3.1 ([43]). LetC be an [n,k,d] linear code for which there exists a cycle-free Tanner graph. Then d≤ n k+1 + n+1 k+1 . (3.1) Specifically, if k/n≥1/2 then d≤2. 3.1.2 The Minimal Trellis Complexity of a Linear Code Trellises constitute the most complex class of graphical models. Since trellises imply optimal decoding algorithms, characterizing the simplest possible trellis for a given code provides insight into the important, and difficult, problem of determining the minimum possible complexity of optimal SISO decoding for that code. Considerable attention has thus been paid to this characterization in the literature (see, for example, [17, 49, 87, 89, 99, 117]). LetC be a linear code over F q and let π be a ordering for the (unordered) index set I. Given this choice of coordinate ordering, there exist well-defined procedures for 26 determining a conventional (i.e. unsectionalized) trellis forC that contains the smallest possible maximum hidden variable size (cf. [99]). Denote by s π (C) the base-q logarithm of the maximum hidden variable size in such a minimal trellis. The minimal trellis complexity s(C) ofC is simply the smallest possible s π (C) over all coordinate orderings: s(C)=min π s π (C). (3.2) Evaluating s(C) thus requires the determination of a (possibly non-unique) optimal co- ordinate ordering π opt (C) forC. While such an ordering is known for the family of Reed-Muller codes [89], determining π opt (C) is intractable in general [76]. 3.1.3 The Cut-Set and Square-Root Bounds Theorem3.1requiresthatverysimplegraphicalmodelsforgoodcodesnecessarilycontain cycles while the minimal trellis complexity literature provides bounds on the complexity of one class of cycle-free graphical models for codes. These results scratch the surface of the fundamental tradeoff between the cyclic topology of graphical models for a given linear codeC. Three questions arise in a deeper exploration of this tradeoff: 1. What is the least complex cycle-free graphical model forC? 2. How many cycles must be contained in a model forC with some prescribed com- plexity? 3. What is the least complex cyclic graphical model forC containing some prescribed number of cycles? 27 Wiberg’s Cut-Set Bound (CSB) [151, 152], which is stated below without proof in the language of Chapter 2, is the best existing tool for studying these questions. Theorem 3.2 (Cut-Set Bound, [151]). LetC be a linear code over F q defined on the index set I. LetG C be a graphical model forC containing a cutX corresponding to the hidden variables S i , i∈ I S (X), which partitions the index set into J 1 ⊂ I and J 2 ⊂ I. Let the base-q logarithm of the midpoint hidden variable alphabet size of the minimal two- section trellis forC on the two-section time axis{J 1 ,J 2 } be s X,min (C). The sum of the hidden variable alphabet sizes corresponding to the cutX is lower-bounded by X i∈I S (X) |T i |≥s X,min (C). (3.3) Forney provided the following description of the implications of the Cut-Set Bound in his first Codes on Graphs paper ([50], Sec. IV.G): On cycle-free graphs, each edge is a cut set, and the size of the state space associated with that edge is lower-bounded by the Cut-Set Bound to be at least as great as that of a state space in some conventional state realization (trellis). Therefore, no dramatic reduction in complexity over conventional realizations can be expected by using more general cycle-free realizations. On the other hand, on graphs with cycles, cut sets generally comprise multiple state variables, and the minimum complexity implied by the Cut-Set Bound may be spread across these variables; therefore, dramatic reductions in state complexity may be obtained. 28 The CSB thus addresses the first question posed above. The CSB can also shed light on the second and third questions. Specifically, for a fixed cyclic topology, the CSB can be simultaneously applied to all cuts yielding a linear programming lower bound on the hiddenvariablealphabetsizes[152]. Forthespecialcaseofasingle-cyclegraphicalmodel (i.e. a tail-biting trellis), this technique yields a simple solution [22]: Theorem 3.3 (Square-Root Bound, [22]). LetC be a linear code over F q of even length and let s mid,min (C) be the base-q logarithm of the minimum possible hidden variable alphabet size of a conventional trellis forC at its midpoint over all coordinate orderings. The base-q logarithm of the minimum possible hidden variable alphabet size s TB (C) of a tail-biting trellis forC is lower-bounded by s TB (C)≥s mid,min (C)/2. (3.4) The square-root bound can thus be used to answer the questions posed above for a specific class of single-cycle graphical models. For topologies richer than a single cycle, however, the aforementioned linear programming technique quickly becomes intractable. Specifically, there are 2 n(C)−1 −1 (3.5) ways to partition a size n(C) visible variable index set into two non-empty, disjoint, subsets. The number of cuts to be considered by the linear programming technique for a given cyclic topology thus grows exponentially with block length and a different minimal two-stage trellis must be constructed in order to bound the size of each of those cuts. 29 3.2 Which Codes Have 4-Cycle-Free Tanner Graphs? Inspired by the work of Etzion, Trachtenberg, and Vardy concerning codes with cycle- free Tanner graphs described in Section 3.1.1, this section addresses the question: which codes have 4-cycle-free Tanner graphs? ItisprovedinSection3.2.2thatan[n,k,d]binary linear code with rate R = k/n and minimum dual distance d ⊥ can be represented by a 4-cycle-free Tanner graph only if pd ⊥ ≤ $r np(p−1)+ n 2 4 + n 2 % (3.6) where p = n−k. Equation (3.6) follows directly from results in graph theory which are reviewed in Section 3.2.1. In Section 3.2.3, the tightness of this result is considered. In Section 3.2.4, (3.6) is applied to a number of classical linear block codes and it is shown that the following binary codes do not have 4-cycle-free Tanner graphs: 1. The [23,12,7] binary Golay and [24,12,8] extended binary Golay codes. 2. Those [n = 2 m ,k,d = 2 m−r ] Reed-Muller (RM) codes with rate R ≥ 1/2 and minimum distance d>2 for 3≤m≤9. 3. Those[n=2 m −1,k,d]primitiveBose-Chaudhuri-Hocquenghem(BCH)codeswith rate R≥1/2 for 3≤m≤8. 4. The binary image of those [n=2 m −1,k,d=n−k+1] Reed-Solomon (RS) codes with rate 1/2≤R≤1−2/n for 3≤m≤5. 5. The binary image of those [n = 2 m − 1,k,d = n− k + 1] RS codes with rate 1/3≤R≤1−2/n for m≥6. 30 3.2.1 Preliminaries from Graph Theory LetG =(U∪W,E) be a bipartite graph such that|U|=n u and|W|=n w . Without loss of generality, assume that n w ≥ n u . 1 The size ofG is n e =|E|. The degree of a vertex v∈U∪W is denoted d(v). Propositions 3.4 and 3.5 are well-known; the proofs presented below are due to Neuwirth [118] and are given for completeness. Proposition 3.4. LetG =(U∪W,E) be a 4-cycle-free bipartite graph. Then nw X j=1 d(w j ) 2 ≤ n u 2 (3.7) where x y is the binomial coefficient. Proof ([118]). Define e G =(U, e E) as the graph with vertex setU and edge set e E = n {x,z}∈U×U ∃ y∈W so that{x,y}∈E and{z,y}∈E o . (3.8) BecauseG is 4-cycle-free, there is at most one y∈W such that{x,y}∈E and{z,y}∈E for each{x,z} ∈ U×U. Thus, e G contains no multiple edges and| e E| ≤ nu 2 . The proposition then follows by noting that there is a bijection between edges in e G and pairs of vertices incident on vertices w j ∈W inG. 1 In the context of Tanner graphs, W thus corresponds to the variable node set and U to the check node set. 31 Proposition 3.5. LetG = (U∪W,E) be a 4-cycle-free bipartite graph such that n w > nu 2 . Then there are at least n w − nu 2 vertices inW with degree 0 or 1. Proof ([118]). LetW 2 ={w j ∈W|d(w j )≥2}. By Proposition 3.4, since 0 2 = 1 2 = 0 and d(w j ) 2 ≥1∀ w j ∈W 2 , |W 2 |≤ nw X j=1 d(w j ) 2 ≤ n u 2 . (3.9) Because n w > nu 2 , there must be at least n w − nu 2 vertices inW\W 2 . Theorem 3.6 is well-known as Reiman’s inequality [129]; the proof presented here is adopted from Bollobas [21]. Theorem 3.6 ([129]). LetG =(U∪W,E) be a 4-cycle-free bipartite graph with n u ≤n w . Then the size ofG satisfies n e ≤ r n w n u (n u −1)+ n 2 w 4 + n w 2 . (3.10) Proof ([21]). Let z = q n w n u (n u −1)+ n 2 w 4 + nw 2 and note that z(z−n w ) 2n w = n u (n u −1) 2 = n u 2 . (3.11) Suppose thatG has size strictly greater than z so that nw X j=1 d(w j )=n e >z. (3.12) 32 Proposition 3.4 implies the following series of inequalities: n u 2 ≥ nw X j=1 d(w j ) 2 (3.13) = 1 2 nw X j=1 (d(w j )) 2 − 1 2 nw X j=1 d(w j ) (3.14) ≥ 1 2n w n 2 e − n e 2 (3.15) > z(z−n w ) 2n w (3.16) = n u 2 . (3.17) Notethat(3.15)followsfrom(3.14)viatheCauchy-Schwarzinequality. Since nu 2 ≯ nu 2 , n e ≯z proving the Theorem. Neuwirth noted that equality holds in Theorem 3.6 if and only ifG is the incidence graph of a Steiner system S(2,k;n u ) (see, for example, [19]) on n u points with block degree k satisfying n w k(k−1)=n u (n u −1) [118]. 3.2.2 Proof of the Main Result Theorem 3.7. LetC be an [n,k,d] binary linear block code with dualC ⊥ . ThenC can be represented by a 4-cycle-free Tanner graph only if pd ⊥ ≤ $r np(p−1)+ n 2 4 + n 2 % (3.18) where p=n−k and d ⊥ is the minimum distance ofC ⊥ . 33 Proof. Let H C be a p×n parity-check matrix forC and let wt(H C ) denote the number of 1’s in H C . The Tanner graph TG(H C ) corresponding to H C is bipartite with n w = n, n u = p, and n e = wt(H C ). By Theorem 3.6, in order for TG(H C ) to be 4-cycle-free its size must satisfy wt(H C )≤ $r np(p−1)+ n 2 4 + n 2 % . (3.19) Since H C generatesC ⊥ , any row of H C is a nonzero codeword inC ⊥ and contains at least d ⊥ 1’s. Thus, any parity-check matrix forC must satisfy wt(H C )≥ pd ⊥ completing the proof. The following remark follows immediately from Theorem 3.7 upon replacing p by (1−R)n. Remark 3.8. Consider a sequence of rate R < 1 codes of increasing length n for which there exist 4-cycle-free Tanner graphs. The dual distances of this sequence of codes grow at most as √ n. Since the addition of any edge beyond Reiman’s inequality must introduce a 4-cycle to a bipartite graph, Theorem 3.7 immediately implies the following corollary which is given without proof. Corollary 3.9. LetC be an [n,k,d] binary linear block code for which a 4-cycle-free Tan- ner graph does not exist. Then the number of 4-cycles in any Tanner graph representing C is lower-bounded by pd ⊥ − $r np(p−1)+ n 2 4 + n 2 % (3.20) where p=n−k and d ⊥ is the minimum distance ofC ⊥ . 34 As an example, consider the [7,4,3] Hamming codeC [7,4,3] with d ⊥ = 4. Any parity- check matrix forC [7,4,3] contains as columns the 7 nonzero binary vectors of length 3 and is thus isomorphic (under permutation of columns) to [111] H [7,4,3] =   0 0 0 1 1 1 1 0 1 1 0 0 1 1 1 0 1 0 1 0 1   . (3.21) The Tanner graph corresponding to H [7,4,3] clearly contains 3 4-cycles. Indeed, for this code pd ⊥ =12 while $r np(p−1)+ n 2 4 + n 2 % =10 (3.22) precluding the existence of a 4-cycle-free Tanner graph while requiring that any Tanner graph for H [7,4,3] contains at least 2 4-cycles. 3.2.3 Remarks of the Main Result Two questions arise naturally from the bound provided by Theorem 3.7: 1. How tight is the bound? 2. Do there exist codes with 4-cycle-free Tanner graphs which meet the bound with equality? HoorynotedthatReiman’sinequalityisthetightestknownboundonthesizeofa4-cycle- free bipartite graph [75]. However, Theorem 3.6 provides only a necessary condition for the existence of a 4-cycle-free bipartite graph - it is not clear that a 4-cycle-free Tanner graph can be found for any code that satisfies Theorem 3.7. 35 Neuwirth noted that graphs which meet the bound of Theorem 3.6 with equality are necessarily the incidence graphs of certain Steiner systems [118]. A number of authors have used Steiner systems as a tool for designing algebraically constructed LDPC codes (see, for example, [86]). A search for codes which meet the bound of Theorem 3.7 thus begins by examining codes with duals generated by the incidence matrices of Steiner systems. LetC S(2,3;9) be the code with parity-check matrix H S(2,3;9) =              1 0 0 1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 1 0 1 0 0 1 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 1 0 1 0 0 0 0 1 1 0 0 0 1 0 0 0 1 0 0 1 0 1 0 0 0 1 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0              . (3.23) Equation (3.23) defines the incidence matrix of the Steiner system S(2,3;9) and the Tanner graph corresponding toC S(2,3;9) meets the bound of Theorem 3.6 with equality. However, theminimumdistanceofthecodegeneratedbyH S(2,3;9) is2, whichisnotequal to the minimum row weight of H S(2,3;9) . Therefore,C S(2,3;9) does not meet the bound of Theorem 3.7 with equality. In general, the codes generated by the incidence matrices of Steiner systems do not have minimum distance equal to the minimum row weight of those matrices and an alternate approach is required to meet the bound of Theorem 3.7 with equality. LetC 0 S(2,2;v) be the code with parity-check matrix H 0 S(2,2;v) = I H S(2,2;v) (3.24) 36 v pd ⊥ q np(p−1)+ n 2 4 + n 2 2 4 4 3 9 9 4 16 17 5 25 26 6 36 37 7 49 51 8 64 66 9 81 83 10 100 103 Table 3.1: Application of Theorem 3.7 to theC 0 S(2,2;v) family of codes for 2≤v≤10. whereH S(2,2;v) is thev× v 2 incidence matrix of theS(2,2;v) Steiner system andI is the v×v identity matrix. It is readily verified thatC 0 S(2,2;v) has lengthn=v+ v 2 , dimension k = v 2 , and minimum dual distance d ⊥ = v. Table 3.1 summarizes the application of Theorem 3.7 to this family of codes for 2≤ v≤ 10. Note theC 0 S(2,2;2) andC 0 S(2,2;3) meet Theorem 3.7 with equality while the remaining codes nearly meet the bound.C 0 S(2,2;2) is the length 3 repetition code with parity-check matrix H 0 S(2,2;2) = 1 0 1 0 1 1 (3.25) whileC 0 S(2,2;3) is a [6,3,3] code with parity-check matrix H 0 S(2,2;3) =   1 0 0 1 0 1 0 1 0 1 1 0 0 0 1 0 1 1   . (3.26) 37 Finally, Hoory provided an upper bound on the size of a bipartite graph with given girth g which reduces to that of Theorem 3.6 when g =6. Hoory’s bound thus provides a recipeforthedevelopmentofanecessaryconditionfortheexistenceofa(g−2)-cycle-free Tanner graph for a given code. The application of Hoory’s bound to the Tanner graphs of low-density parity-check codes was explored in [115, 137]. 3.2.4 Application of the Main Result 3.2.4.1 The Golay Code The duals of both the [23,12,7] binary Golay code and the [24,12,8] extended binary Golay code have minimum distance 8 [111]. Neither code satisfies Theorem 3.7 and thus any Tanner graph for either code must contain 4-cycles. 3.2.4.2 Reed-Muller Codes An [n = 2 m ,k,d = 2 m−r ] RM code has dimension k = P r i=0 m i and its dual has minimum distance 2 r+1 [111]. Table 3.2 summarizes the application of Theorem 3.7 to those Reed-Muller codes with rate R≥1/2 and minimum distance d>2 for 3≤m≤9. Note that the existence of a 4-cycle-free Tanner graph is precluded by Theorem 3.7 for all of these codes. Note also that the [256,93,32] code is an example of an R < 1/2 RM code for which a 4-cycle-free Tanner graph cannot exist. 3.2.4.3 Primitive BCH Codes Table 3.3 summarizes the application of Theorem 3.7 to those [n=2 m −1,k,d] primitve BCH codes with rate R≥ 1/2 for 3≤ m≤ 8 [111]. Determining d ⊥ is difficult when 38 Code pd ⊥ q np(p−1)+ n 2 4 + n 2 [8,4,4] 16 14 [16,11,4] 40 27 [32,26,4] 96 50 [32,16,8] 128 105 [64,57,4] 224 92 [64,42,8] 352 206 [128,120,4] 512 170 [128,99,8] 928 392 [128,64,16] 1024 785 [256,219,8] 2368 725 [256,136,16] 5952 1613 [256,93,32] 5216 2731 [512,502,4] 2048 562 [512,466,8] 5888 1316 [512,382,16] 8320 3197 [512,256,32] 8192 6042 Table 3.2: Application of Theorem 3.7 to Reed-Muller codes. m≥5 and lower bounds have been used. The lower bounds labeled ? , 0 , and † correspond to Sikelnikov’s bound [111], Theorem 5 of [9], and Schaub’s bound [133] (as reported in [9]), respectively. Note that the existence of 4-cycle-free Tanner graphs is precluded by Theorem 3.7 for all of the codes in Table 3.3. Note also that there exist a number of length 127 and 255 BCH codes with R<1/2 without 4-cycle-free Tanner graphs. 39 Code d ⊥ pd ⊥ q np(p−1)+ n 2 4 + n 2 [7,4,3] 4 12 10 [15,11,3] 8 32 22 [31,26,3] ≥16 ? ≥ 80 44 [31,21,5] ≥8 ? ≥ 80 70 [31,16,7] ≥8 ? ≥ 120 97 [63,57,3] ≥32 ? ≥ 192 85 [63,51,5] ≥16 ? ≥ 192 127 [63,45,7] ≥16 ? ≥ 288 173 [63,39,9] ≥12 0 ≥ 288 220 [63,36,11] ≥12 0 ≥ 324 244 [127,120,3] ≥64 † ≥ 448 160 [127,113,5] ≥56 † ≥ 784 228 [127,106,7] ≥48 † ≥ 1008 303 [127,99,9] ≥40 † ≥ 1120 379 [127,92,11] ≥32 † ≥ 1120 457 [127,85,13] ≥30 † ≥ 1260 535 [127,78,15] ≥28 † ≥ 1372 613 [127,71,19] ≥22 † ≥ 1232 692 [127,64,21] ≥20 † ≥ 1260 770 [127,57,23] ≥16 † ≥ 1120 849 [127,50,27] ≥14 † ≥ 1078 927 [127,43,31] ≥12 † ≥ 1008 1006 [255,247,3] ≥128 † ≥ 1024 302 [255,239,5] ≥112 † ≥ 1792 405 [255,231,7] ≥96 † ≥ 2304 523 [255,223,9] ≥88 † ≥ 2816 646 [255,215,11] ≥64 † ≥ 2560 770 [255,207,13] ≥64 † ≥ 3072 896 [255,199,15] ≥60 † ≥ 3360 1022 [255,191,17] ≥42 † ≥ 2688 1149 [255,187,19] ≥42 † ≥ 2856 1212 [255,179,21] ≥40 † ≥ 3040 1339 [255,171,23] ≥32 † ≥ 2688 1466 [255,163,25] ≥32 † ≥ 2944 1594 [255,155,27] ≥32 † ≥ 3200 1721 [255,147,29] ≥28 † ≥ 3024 1848 [255,139,31] ≥26 † ≥ 3016 1976 [255,131,37] ≥22 † ≥ 2728 2103 [255,123,39] ≥22 † ≥ 2904 2231 [255,115,43] ≥20 † ≥ 2800 2358 [255,107,45] ≥20 † ≥ 2960 2486 Table 3.3: Application of Theorem 3.7 to primitive BCH codes. 40 3.2.4.4 Binary Images of Reed-Solomon Codes Corollary 3.10 follows from Theorem 3.7. Corollary 3.10. LetC be the binary image of an [n=2 m −1,k,d=n−k+1] RS code with rate 1/2≤R≤1−2/n for m≥4. There exists no 4-cycle-free Tanner graph forC. Proof.C is an [mn,mk,d 0 ≥ d] binary code whileC ⊥ is an [mn,m(n−k),d ⊥ ≥ k +1] binary code. It thus suffices to show that k+1> s mn 1− 1 m(n−k) + n 2 4(n−k) 2 + n 2(n−k) (3.27) or, equivalently g(n,R)=(Rn+1) 2 (1−R)−Rn>mn(1−R) (3.28) where R =k/n. It is readily verified that ∂ 2 g(n,R)/∂R 2 ≤0 for 1/2≤R≤1−2/n and it thus suffices to verify only that g(n,R) > mn(1−R) at R = 1/2 and R = 1−2/n. When m=4, n=15, and the lowest possible code rate that is greater or equal to 1/2 is 8/15 so that g(15,8/15)=149/5>28. (3.29) More generally, since n/4>m when m≥5, g(n,1/2)= n 2 8 + 1 2 > mn 2 . (3.30) 41 Finally, it is readily verified that for m≥4 g(n,1−2/n)=n−2+ 2 n >2m. (3.31) The proof technique of Corollary 3.10 can be readily extended to show that if m≥6, then any Tanner graph corresponding to the binary image of a rate 1/3≤ R≤ 1−2/n, [n=2 m −1,k,d=n−k+1] RS code contains 4-cycles. Corollary 3.10 does not immediately extend to the binary image of length 7 Reed- Solomon codes. Specifically, evaluating Theorem 3.7 for the binary image of the [7,5,3] RS codeC [7,5,3] with d ⊥ ≥6 yields m(n−k)d ⊥ ≥36 and $r m 2 n(n−k)(m(n−k)−1)+ m 2 n 2 4 + mn 2 % =37. (3.32) The following alternate argument establishes that any Tanner graph representing the binary image of theC [7,5,3] must indeed contain a 4-cycle. Suppose a 4-cycle-free Tanner graph does exist forC [7,5,3] . The parity-check matrix H [21,15,3] corresponding to this graph has 21 columns and 6 rows. By Proposition 3.5, H [21,15,3] mustcontainatleast6weight1columns. Sincetheminimumdistanceofthedual ofC [7,5,3] isatleast3,H [21,15,3] mustcontainexactly6weight1columns. Nowconsiderthe bipartite graph corresponding to the remaining 15 columns. Since the minimum distance of the dual ofC [7,5,3] is at least 6, this graph must contain at least 30 edges. On the other hand, Theorem 3.6 states that this graph contains at most 30 edges with equality if only 42 if the graph corresponds to the incidence matrix of the Steiner system S(2,2;6). It has thus been shown that if a 4-cycle-Tanner graph exists forC [7,5,3] , thenC [7,5,3] must be isomorphic toC 0 S(2,2;6) with parity check matrix H 0 S(2,2;6) =        1 0 0 0 0 0 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 1 1 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 0 1 1        . (3.33) MacWilliams and Sloane show that the code with parity check matrix H 0 S(2,2;6) can be interpreted as the binary image of a [7,5,3] maximum distance separable (MDS) code overF 8 which is not an RS code ([111], Ch. 10.5). Thus,C 0 S(2,2;6) is not isomorphic to a binary image of the [7,5,3] RS code and no 4-cycle-free Tanner graph exists forC [7,5,3] . 3.3 Which Codes Have Cycle-Free Generalized Tanner Graphs? It was shown in Section 3.2 that many well-known classical codes cannot be represented by 4-cycle-free Tanner graphs. This result, however, does not preclude the existence of other simple 4-cycle-free graphical models for these codes. Sankaranarayanan and Vasi´ c demonstrated that there exist 4-cycle-free generalized Tanner graphs for all binary linear block codes in [132] and stated further in a concluding remark that: A logical extension of this work is to develop a general algorithm to remove cycles of any specified length. 43 The extension of Sankaranarayanan and Vasi´ c’s work is explored in this section. Specif- ically, it is first shown that cycles of any specified finite length can be trivially removed from generalized Tanner graphs. It is then shown that cycle-free generalized Tanner graphs do not support good codes. In order to demonstrate that cycles of any specified finite length can be trivially removed from generalized Tanner graphs (GTGs), consider again the 4-cycle-free GTG for the [8,4,4] extended Hamming codeC H that was studied in Chapter 2. This GTG is re-illustrated in Figure 3.1 and corresponds a degree-1 generalized extension ofC H with parity-check matrix e H (2) H =      1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 0 1 1      . (3.34) V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 3.1: Girth 6 generalized Tanner graph for the [8,4,4] extended Hamming code. A GTG forC H with girth 10 can be obtained by appropriately inserting (trivial) degree- 2 single parity-check and repetition constraints to the model illustrated in Figure 3.1. 44 The resulting GTG is illustrated in Figure 3.2 and corresponds a degree-5 generalized extension ofC H with parity-check matrix e H (3) H =              1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 1 1              . (3.35) V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 3.2: Girth 10 generalized Tanner graph for the [8,4,4] extended Hamming code. The procedure described by example above can clearly be used to remove all cycles of any specified finite length from any GTG. This procedure cannot, however, be used to obtain a cycle-free GTGs as will be shown below. Before proceeding, the following proposition is required. 45 Proposition 3.11. Without loss of generality, a cycle-free generalized Tanner graph for a binary codeC can be assumed to contain no degree-2 single parity-check constraints that are adjacent to at most one visible repetition constraint. Figure3.3: Removalofadegree-2singleparity-checkconstraintinacycle-freegeneralized Tanner graph. Proof. Suppose such a constraint exists is a cycle-free generalized Tanner graph. As illustrated in Figure 3.3, this constraint can be removed yielding a GTG which is also cycle-free. Note that this removal is valid since degree-2 single parity-check codes are also degree-2 repetition codes. Note also that this removal cannot be performed if the degree-2 single parity-check constraint is incident on two visible repetition constraints based on the graphical model definition proposed in Chapter 2. The following establishes that, under the reasonable assumption that a binary code contains no repeated bits, Etzion et al.’s result concerning cycle-free Tanner graphs can be extended to cycle-free generalized Tanner graphs. That is, cycle-free GTGs cannot support good codes. Theorem 3.12. LetC be an [n,k,d] binary code containing no repeated bits and suppose that there exists a cycle-free generalized Tanner graph corresponding toC. Then d≤2. (3.36) 46 Proof. Let the cycle-free generalized Tanner graph GTG(H e C ) correspond to a degree-g generalizedextension e C ofC withparameters [n+g,k, ˜ d], minimumdual distance ˜ d ⊥ , and parity-check matrix H e C . Combining the restriction thatC contains no repeated bits with Proposition 3.11 implies that ˜ d ⊥ ≥3. (3.37) There are thus V =2n+2g−k (3.38) vertices and at least E≥3(n−k+g) (3.39) edges in GTG(H e C ). Since a connected graph is cycle-free if and only if E≤ V−1 (cf. [37]), n+g <2k (3.40) and e C is a rate ˜ R =k/(n+g)>1/2 (3.41) code for which there exists a cycle-free Tanner graph. Theorem 3.1 thus requires that ˜ d≤2. (3.42) The result then follows from the observation that d≤ ˜ d. 47 3.4 A Complexity Measure for Graphical Models Following the study of simple graphical models in Sections 3.2 and 3.3, attention is now turned to models with arbitrary complexity. 3.4.1 q m -ary Graphical Models This dissertation introduces the term q m -ary graphical model to denote a normal realiza- tion of a linear codeC overF q that satisfies the following constraints: • The alphabet index size of every hidden variable S i , i∈I S , satisfies|T i |≤m. • Every local constraintC i , i∈I C , either satifies min(k(C i ),n(C i )−k(C i ))≤m (3.43) or can be decomposed as a Cartesian product of codes, each of which satisfies this condition. The complexity measure m simultaneously captures a cyclic graphical model analog of the familiar notions of state and branch complexity for trellises [145]. From the above definition, it is clear that Tanner graphs and generalized Tanner graphs for codes overF q are q-ary graphical models. The efficacy of this complexity measure is discussed further in Section 3.5.3. 48 3.4.2 Properties of q m -ary Graphical Models The following three properties of q m -ary graphical models will be used in the proof of Theorem 3.19 in Section 3.5: 1. Internal Local Constraint Involvement Property: Any hidden variable in a q m -ary graphical model can be made to be incident on an internal local constraintC i which satisfiesn(C i )−k(C i )≤m without fundamentally altering the complexity or cyclic topology of that graphical model. 2. Internal Local Constraint Removal Property: The removal of an internal local constraint from a q m -ary graphical model results in a q m -ary graphical model for a new code defined on same index set. 3. Internal Local Constraint Redefinition Property: Any internal local constraintC i in a q m -ary graphical model satisfying n(C i )− k(C i ) = m 0 ≤ m can be equivalently represented by m 0 q-ary single parity-check equations over the visible variable index set. These properties, which are defined in detail in Section 3.6, are particularly useful in con- cert. Specifically, letG C be aq m -ary graphical model for the linear codeC overF q defined on an index set I. Suppose that the internal constraintC r satisfying n(C r )−k(C r ) = m 0 ≤m is removed fromG C resulting in the new codeC \r . Denote byC (1) r ,...,C (m 0 ) r the set of m 0 q-ary single parity-check equations that result whenC r is redefined over I. A 49 vector in F I q is a codeword inC if and only if it is contained inC \r and satisfies each of these m 0 single parity-check equations so that C =C \r ∩C (1) r ∩···∩C (m 0 ) r . (3.44) 3.4.3 The Minimal Tree Complexity of a Linear Code This dissertation introduces the minimal tree complexity of a linear code as a generaliza- tion of minimal trellis complexity to arbitrary cycle-free graphical model topologies. Definition 3.13. The minimal tree complexity of a linear codeC overF q is the smallest integer t(C) such that there exists a cycle-free q t(C) -ary graphical model forC. Much as s(C)=s(C ⊥ ), the minimal tree complexity of a codeC is equal to that of its dual. Proposition 3.14. LetC be a linear code over F q with dualC ⊥ . Then t(C)=t(C ⊥ ). (3.45) Proof. The dualizing procedure described by Forney [50] can be applied to a q t(C) -ary graphical model forC in order to obtain a graphical model forC ⊥ which is readily shown to be q t(C) -ary. Since a trellis is a cycle-free graphical model, t(C) ≤ s(C), and all known upper bounds on s(C) extend to t(C). Specifically, consider the section of a minimal trellis for C illustrated in Figure 3.4. The hidden (state) variables have alphabet sizes|T i |≤ s(C) 50 V i S i S i−1 C i Figure 3.4: The q s(C) -ary graphical model representation of a trellis section. and|T i−1 |≤s(C), respectively. The local constraintC i has length n(C i )=|T i |+|T i−1 |+1 (3.46) and dimension k(C i )=1+min(|T i |,|T i−1 |) (3.47) so that n(C i )−k(C i )=max(|T i |,|T i−1 |)≤s(C). (3.48) Lower bounds on s(C) do not, however, necessarily extend to t(C). The Cut-Set Bound, however, precludes t(C) from being significantly smaller than s(C) [50, 51]. The following lemma concerning minimal tree complexity will be used in the proof of Theorem 3.19 in Section 3.5. 51 Lemma 3.15. LetC andC SPC be linear codes over F q defined on the index sets I and J⊆ I, respectively, such thatC SPC is a q-ary single parity-check code. Define by e C the intersection ofC andC SPC : e C =C∩C SPC . (3.49) The minimal tree complexity of e C is upper-bounded by t( e C)≤t(C)+1. (3.50) Proof. By explicit construction of a q t(C)+1 -ary graphical model for e C. LetG C be some q t(C) -ary cycle-free graphical model forC and letT be a minimal connected subtree ofG C containing the set of|J| interface constraints which involve the visible variables inJ. De- note by I S (T)⊆I S and I C (T)⊆I C the subset of hidden variables and local constraints, respectively, contained inT. Choose some local constraint vertexC Λ , Λ∈ I C (T), as a root forT. Observe that the choice ofC Λ , while arbitrary, induces a directionality inT: downstream toward the root vertex or upstream away from the root vertex. For every S i , i∈I S (T), denote by J i,↑ ⊆J the subset of visible variables in J which are upstream from that hidden variable edge. A q t(C)+1 -ary graphical model for e C is then constructed fromG C by updating each hidden variable S i , i∈ I S (T), to also contain the q-ary partial parity of the upstream visible variables in J i,↑ ⊆ J. The local constraintsC j , j ∈ I C (T)\ Λ, are updated accordingly. Finally,C Λ is updated to enforce the q-ary single parity constraint defined byC SPC . This updating procedure increases the alphabet size of each hidden variable S i , i∈ I S (T), by at most one and adds at most one single parity-check (or repetition) 52 constraint to the definition of eachC j , j∈ I C (T), and the resulting cycle-free graphical model is thus at most q t(C)+1 -ary. The proof of Lemma 3.15 is detailed further by example in Section 3.6. 3.5 TheTradeoffBetweenCyclicTopologyandComplexity InlightofthecomplexitymeasureintroducedinSection3.5,thesecondandthirdquestion regarding the tradeoff between cyclic topology and complexity in graphical models for a given linear codeC overF q posed in Section 3.1.3 can now be made more precise: 1. Foragivencomplexitym,howmanycyclesmustbecontainedinaq m -arygraphical model forC? 2. For a given number of cycles N, what is the smallest m such that a q m -ary model containing N cycles forC can exist? In this section, a new bound - the tree-inducing cut-set bound - is introduced in order to address these questions. 3.5.1 Tree-Inducing Cuts Recall that a cut in a graphG is some subset of the edgesX⊆E the removal of which yields a disconnected graph. A cut is thus defined without regard to the cyclic topology of the disconnected components which remain after its removal. In order to provide a characterization of the tradeoff between cyclic topology and complexity which is more precise than that provided by the CSB alone, this dissertation focuses on a specific type 53 of cut which is defined below. Two useful properties of such cuts are established by Propositions 3.17 and 3.18. Definition3.16. LetG be a connected graph. A tree-inducing cut is some subset of edges X T ⊆E the removal of which yields a tree with precisely two components. Proposition 3.17. LetG = (V,E,H) be a connected graph. The size X T of any tree- inducing cutX T inG is precisely X T =|E|−|V|+2. (3.51) Proof. It is well-known that a connected graph is a tree if and only if (cf. [37]) |E|=|V|−1. (3.52) Similarly, a graph composed of two cycle-free components satisfies |E|=|V|−2. (3.53) The result then follows from the observation that the size of a tree-inducing cut is the number of edges which must be removed in order to satisfy (3.53). Proposition 3.18. LetG be a connected graph with tree-inducing cut size X T . The number of cycles N G inG is lower-bounded by N G ≥ X T 2 . (3.54) 54 Proof. Lettheremovalofatree-inducingcutX T intheconnectedgraphG yieldthecycle- free componentsG 1 andG 2 and let e i 6=e j ∈X T . SinceG 1 (G 2 ) is a tree, there is a unique path inG 1 (G 2 ) connecting e i and e j . There is thus a unique cycle inG corresponding to the edge pair{e i ,e j }. There are X T 2 such distinct edge pairs which yields the lower bound. Note that this is a lower bound because for certain graphs, there can exist cycles which contain more than two edges from a tree-inducing cut. 3.5.2 The Tree-Inducing Cut-Set Bound With tree-inducing cuts defined, the required properties of q m -ary graphical models de- scribed, and Lemma 3.15 established, the main result concerning the tradeoff between cyclic topology and graphical model complexity can now be stated and proved. Theorem 3.19. LetC be a linear code over F q defined on the index set I and suppose thatG C is a q m -ary graphical model forC with tree-inducing cut size X T . The minimal tree complexity ofC is upper-bounded by t(C)≤mX T . (3.55) Proof. By induction on X T . Let X T = 1 and suppose that e∈X T is the sole edge in some tree-inducing cutX T inG C . Since the removal of e partitionsG C into disconnected cycle-free components,G C must be cycle-free and t(C)≤m by construction. Now suppose that X T = x > 1 and let e∈X T be an edge in some tree-inducing cut X T inG C . By the first q m -ary graphical model property of Section 3.4.2, e is incident on some internal local constraintC i satisfying n(C i )−k(C i ) = m 0 ≤ m. Denote byG C \i 55 the q m -ary graphical model that results whenC i is removed fromG C , and byC \i the corresponding code over I. The tree-inducing cut size ofG C \i is at most x−1 since the removal ofC i fromG C results in the removal a single vertex and at least two edges. By the induction hypothesis, the minimal tree complexity ofC \i is upper-bounded by t(C \i )≤m(x−1). (3.56) From the discussion of Section 3.4.2, it is clear thatC i can be redefined as m 0 ≤ m single parity check equations,C (j) i for j∈[1,m 0 ], overF q on I such that C =C \i ∩C (1) i ∩···∩C (m 0 ) i . (3.57) It follows from Lemma 3.15 that t(C)≤t(C \i )+m 0 ≤mx (3.58) completing the proof. An immediate corollary to Theorem 3.19 results when Proposition 3.18 is applied in conjunction with the main result: Corollary 3.20. LetC be a linear code over F q with minimal tree complexity t(C). The number of cycles N m in any q m -ary graphical model forC is lower-bounded by N m ≥ bt(C)/mc 2 . (3.59) 56 3.5.3 Interpretation of the TI-CSB Provided t(C) is known or can be lower-bounded, the tree-inducing cut-set bound (TI- CSB) (and more specifically Corollary 3.20) can be used to answer the questions posed at the beginning of this section. The TI-CSB is further discussed below. 3.5.3.1 The TI-CSB and the CSB Onthesurface,theTI-CSBandtheCSBaresimilarinstatement;however,therearethree important differences between the two. First, the CSB does not explicitly address the complexityofthelocalconstraintsoneithersideofagivencut. Forneyprovidedanumber of illustrative examples in [51] that stress the importance of characterizing graphical model complexity in terms of both hidden variable size and local constraint complexity. Second, the CSB does not explicitly address the cyclic topology of the graphical model that results when the edges in a cut are removed. The removal of a tree-inducing cut results in two cycle-free disconnected components and the size of a tree-inducing cut can thus be used to make statements about the complexity of optimal SISO decoding using variable conditioning in a cyclic graphical model (cf. [1, 5, 50, 67, 74, 150]). Finally, and most fundamentally, the TI-CSB addresses the aforementioned intractability of applying the CSB to graphical models with rich cyclic topologies. 3.5.3.2 The TI-CSB and the Square-Root Bound Theorem 3.19 can be used to make a statement similar to Theorem 3.3 which is valid for all graphical models containing a single cycle. 57 Corollary 3.21. LetC be a linear code over F q with minimal tree complexity t(C) and let m 1 be the smallest integer such that there exists a q m 1 -ary graphical model forC which contains at most one cycle. Then m 1 ≥t(C)/2. (3.60) More generally, Theorem 3.19 can be used to establish the following generalization of the square-root bound to graphical models with arbitrary cyclic topologies. Corollary 3.22. LetC be a linear code over F q with minimal tree complexity t(C) and let m ( r 2 ) be the smallest integer such that there exists a q m ( r 2 ) -ary graphical model forC which contains at most r 2 cycles. Then m ( r 2 ) ≥t(C)/r. (3.61) AlinearinterpretationofthelogarithmiccomplexitystatementofCorollary3.22yields thedesiredgeneralizationofthesquare-rootbound: anr th -rootcomplexityreductionwith respect to the minimal tree complexity requires the introduction of at least r(r−1)/2 cycles. There are few known examples of classical linear block codes which meet the square- root bound with equality. Shany and Be’ery proved that many Reed-Muller codes cannot meet this bound under any bit ordering [136]. There does, however, exist a tail-biting trellis for the extended binary Golay codeC G which meets the square-root bound with equality [22] so that s mid,min (C G )=8 and s TB (C G )=4. Given that this tail-biting trellis 58 is a 2 4 -ary single-cycle graphical model forC G , the minimal tree complexity of the the extended binary Golay code can be upper-bounded by Corollary 3.21 as t(C G )≤8. (3.62) Note that the minimal bit-level conventional trellis forC G contains (non-central) state variables with alphabet size 512 and is thus a 2 9 -ary graphical model [117]. The proof of Lemma 3.15 provides a recipe for the construction of a 2 8 -ary cycle-free graphical model forC G fromitstail-bitingtrellis. Itremainsopenastowheretheminimaltreecomplexity ofC G is precisely 8, however. 3.5.3.3 Aymptotics of the TI-CSB Denote by N m the minimum number of cycles in any q m -ary graphical model for a linear codeC overF q with minimal tree complexity t(C). For large values of t(C)/m, the lower bound on N m established by Corollary 3.20 becomes N m ≥ bt(C)/mc 2 ≈ t(C) 2 2m 2 . (3.63) The ratio of the minimal complexity of a cycle-free model forC to that of an q m -ary graphical model is thus upper-bounded by q t(C) q m /q 2m √ Nm . (3.64) 59 Inordertofurtherexploretheasymptoticsofthetree-inducingcut-setbound,consider a code of particular practical interest: the binary imageC 255|F 2 of the [255,223,33] Reed- Solomon codeC 255 . SinceC 255 is maximum distance separable, a reasonable estimate for the minimal tree complexity of this code is obtained from Wolf’s bound [155] t(C 255|F 2 )≈8(n(C 255 )−k(C 255 ))=256. (3.65) Figure 3.5 plots N m as a function of m forC 255|F 2 assuming (3.65). Note that since the complexity of the decoding algorithms implied by 2 m -ary graphical models grow roughly as 2 m , logm is roughly a loglog decoding complexity measure. 1 10 100 1000 10000 0 1 2 3 4 5 6 7 Minimum Number of Cycles log m Figure3.5: Minimumnumberofcyclesrequiredfor2 m -arygraphicalmodelsofthebinary image of the [255,223,33] Reed-Solomon code. 60 3.5.3.4 On Complexity Measures Much as there are many valid complexity measures for conventional trellises, there are manyreasonablemetricsforthemeasurementofcyclicgraphicalmodelcomplexity. While there exists a unique minimal trellis for any linear block code which simultaneously min- imizes all reasonable measures of complexity [114], even for the class of cyclic graphical models with the most basic cyclic topology - tail-biting trellises - minimal models are not unique [92]. The complexity measure introduced in Section 3.4 was motivated by the de- sire to have a metric which simultaneously captures hidden variable complexity and local constraint complexity thus disallowing local constraints from “hiding” complexity. There are many conceivable measures of local constraint complexity: one could upper-bound the state complexity of the local constraints or even their minimal tree complexity (thus defining minimal tree complexity recursively). The local constraint complexity measure used in this work is essentially Wolf’s bound [155] and is thus a potentially conservative upper bound on any reasonable measure of local constraint decoding complexity. 3.6 Appendix: Graphical Model Properties by Example This Section provides detailed definitions of the q m -ary graphical model properties de- scribed in Section 3.4.2. The proof of Lemma 3.15 is also further illustrated by example. In order to elucidate these properties and definitions, a single-cycle graphical model for the extended Hamming code is studied throughout. 61 3.6.1 Single-Cycle Model for the Extended Hamming Code Figure 3.6 re-illustrates the single-cycle graphical model (i.e. tail-biting trellis) for the length 8 extended Hamming codeC H that was studied previously in Chapter 2. S 1 S 2 S 3 S 4 S 5 S 6 S 7 S 8 V 8 V 1 V 2 V 3 V 4 V 5 V 6 V 7 C 7 C 8 C 1 C 2 C 3 C 4 C 5 C 6 Figure 3.6: Tail-biting trellis graphical model for the length 8 extended Hamming code C H . The hidden variables S 1 and S 5 are binary while S 2 , S 3 , S 4 , S 6 , S 7 , and S 8 are 4- ary. Equations (3.66) and (3.67) define the local constraint codes via generator matrices (where G i generatesC i ): S 1 V 1 S 2 S 2 V 2 S 3 S 3 V 3 S 4 S 4 V 4 S 5 G 1 = 1 0 10 0 1 01 , G 2 = 10 1 10 01 1 01 , G 3 =   10 0 01 01 0 11 00 1 01   , G 4 = 10 1 0 01 1 1 , (3.66) S 5 V 5 S 6 S 6 V 6 S 7 S 7 V 7 S 8 S 8 V 8 S 9 G 5 = 1 0 10 0 1 01 , G 6 = 10 1 10 01 1 01 , G 7 =   10 0 01 01 0 11 00 1 01   , G 8 = 10 1 0 01 1 1 . (3.67) 62 The graphical model forC H illustrated in Figure 3.6 is 4-ary (i.e. q = 2, m = 2): the maximum hidden variable alphabet index set size is 2 and all local constraints satisfy min(k(C i ),n(C i )−k(C i ))≤2. The behaviorB H of this graphical model is generated by S 1 V 1 S 2 V 2 S 3 V 3 S 4 V 4 S 5 V 5 S 6 V 6 S 7 V 7 S 8 V 8 G B H =    0 1 01 1 01 1 10 1 0 0 00 0 00 0 00 0 0 0 00 0 00 1 01 1 1 0 10 1 10 1 00 0 0 0 00 0 00 0 00 0 0 1 01 1 01 1 10 1 1 0 10 1 10 1 00 0 0 0 00 0 00 1 01 1   . (3.68) The projection ofB H onto the visible variable index set I,B H|I , is thus generated by V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 G B H|I =    1 1 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 0 0 0 1 1    (3.69) which coincides precisely with a generator matrix forC H . 3.6.2 q m -ary Graphical Model Properties The three properties ofq m -ary graphical models introduced in Section 3.4.2 are discussed in detail in the following where it is assumed that a q m -ary graphical modelG C with behaviorB for a linear codeC overF q defined on an index set I is given. 3.6.2.1 Internal Local Constraint Involvement Property SupposethereexistssomehiddenvariableS j (involvedinthelocalconstraintsC j 1 andC j 2 ) that does not satisfy the local constraint involvement property. A new hidden variable S i that is a copy of S j is introduced toG C by first redefiningC j 2 over S i and then inserting 63 C 1 C 1 C 8 C 8 C 9 S 1 S 1 S 2 S 2 S 8 S 8 S 9 V 1 V 1 V 8 V 8 Figure 3.7: Insertion of hidden variable S 9 and internal local constraintC 9 into the tail- biting trellis forC H . a local repetition constraintC i that enforces S j =S i . The insertion of S i andC i does not fundamentally alter the complexity ofG C sincen(C i )−k(C i )=|T j |≤m and since degree- 2 repetition constraints are trivial from a decoding complexity viewpoint. Furthermore, the insertion of S i andC i does not fundamentally alter the cyclic topology ofG C since no new cycles can be introduced by this procedure. As an example, consider the binary hidden variable S 1 in Figure 3.6 which is incident on the interface constraintsC 1 andC 8 . By introducing the new binary hidden variable S 9 and binary repetition constraintC 9 , as illustrated in Figure 3.7, S 1 can be made to be incident on the internal constraintC 9 . The insertion of S 9 andC 9 redefinesC 8 over S 9 resulting in the generator matrices S 8 V 8 S 9 S 9 S 1 G 8 = 10 1 0 01 1 1 , G 9 = 1 1 . (3.70) Clearly, the modified local constraintsC 8 andC 9 satisfy the condition for inclusion in a 4-ary graphical model. 64 3.6.2.2 Internal Local Constraint Removal Property The removal of the internal constraintC r fromG C in order to define the new codeC \r proceeds as follows. Each hidden variable S i , i∈I S (r), is first disconnected fromC r and connected to a new degree-1 internal constraintC i 0 which does not impose any constraint on the value of S i (since it is degree-1). The local constraintC r is then removed from the resulting graphical model yieldingG C \r with behaviorB \r . The new codeC \r is the projection ofB \r onto I. C 1 C 1 C 8 C 8 C 9 C 10 C 11 V 1 V 1 V 8 V 8 S 1 S 1 S 2 S 2 S 8 S 8 S 9 S 9 Figure 3.8: Removal of internal local constraintC 9 from the tail-biting trellis forC H . Asanexample,considertheremovaloftheinternallocalconstraintC 9 fromthegraph- ical model forC H described above; the resulting graphical model update is illustrated in Figure3.8. ThenewcodesC 10 andC 11 arelength1, dimension1codeswhichthusimpose no constraints on S 1 and S 9 , respectively. It is readily verified that the codeC \9 H which results from the removal ofC 9 fromC H has dimension 5 and is generated by V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 G H \9 =      1 1 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 1 1 1 1 0 1 1 0 0 0 1 1 1 0 1 0 0 1 1 0      . (3.71) 65 Note thatC \9 H corresponds to all paths in the tail-biting trellis representation ofC H , not just those paths which begin and end in the same state. The removal of an internal local constraintC r results in the introduction of|I S (r)| new degree-1 local constraints. Forney described such constraints as “useless” in [51] and they can indeed be removed fromG C \r since they impose no constraints on the variables theyinvolve. Specifically,foreachhiddenvariableS i ,i∈I S (r),involvedinthe(removed) local constraintC r , denote byC j the other constraint involving S i inG C . The constraint C j can be redefined as its projection ontoI V (j)∪{I S (j)\i}. It is readily verified that the resulting constraintC \r j satisfies the condition for inclusion in a q m -ary graphical model. Continuing with the above example,C 10 ,C 11 , S 1 , and S 9 can be removed from the graphical model illustrated in Figure 3.8 by redefiningC 1 andC 8 with generator matrices V 1 S 2 S 8 V 8 G 1 = 1 01 0 10 , G 8 = 10 1 01 1 . (3.72) 3.6.2.3 Internal Local Constraint Redefinition Property LetC i satisfy n(C i )−k(C i ) = m 0 ≤ m and consider a hidden variable S j involved inC i (i.e. j∈ I S (i)) with alphabet index set T j . Each of the|T j | coordinates of S j can be redefined as a q-ary sum of some subset of the visible variable set as follows. Consider the behaviorB \i and corresponding codeC \i which result whenC i is removed fromG C (before S j is discarded). The projection ofB \i onto T j ∪I,B \i |T j ∪I , has length n(C)+|T j | (3.73) 66 and dimension k(C \i )≥k(C) (3.74) overF q . There exists a generator matrix forB \i |T j ∪I that is systematic in some size k(C \i ) subset of the index set I [111]. A parity-check matrix H j that is systematic in the|T j | positions corresponding to the coordinates of S j can thus be found for this projection; each coordinate of S j is defined as a q-ary sum of some subset of the visible variables by H j . Following this procedure, the internal local constraintC i is redefined over I by substituting the definitions of S j implied by H j for each j∈ I S (i) into each of the m 0 q-ary single parity-check equations which determineC i . Returning to the example of the tail-biting trellis forC H , the internal local constraint C 9 is redefined over the visible variable set as follows. The projection ofB \9 H onto T 1 ∪I is generated by S 1 V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 G B \9 H|T 1 ∪I =      0 1 1 1 1 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 0 1 1 1 1 1 0 1 1 0 0 0 1 1 1 1 0 1 0 0 1 1 0      . (3.75) A valid parity-check matrix for this projection which is systematic in the position corre- sponding to S 1 is S 1 V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 H 1 =    1 1 1 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 1 1 0 0 0 1 1   , (3.76) 67 which defines the binary hidden variable S 1 as S 1 =V 1 +V 2 (3.77) where addition is overF 2 . A similar development defines S 9 as S 9 =V 5 +V 8 . (3.78) The local constraintC 9 thus can be redefined to enforce the single parity-check equation V 1 +V 2 +V 5 +V 8 =0. (3.79) Finally, in order to illustrate the use of the q m -ary graphical model properties in concert, denote byC (1) 9 the single parity-check constraint enforcing (3.79). It is readily verified that only the first four rows of G H \9 (as defined in (3.71)) satisfyC (1) 9 . It is precisely these four rows which generateC H proving that C H =C \9 H ∩C (1) 9 . (3.80) 68 3.6.3 Illustration of Proof of Lemma 3.15 Inthefollowing, theproofofLemma3.15isillustratedbyupdatingacycle-freemodelfor C \9 H (as generated by (3.71)) with the single parity-check constraint defined by (3.79) in order to obtain a cycle-free graphical model forC H . A cycle-free binary graphical model forC \9 H is illustrated in Figure 3.9 2 . All hidden variables in Figure 3.9 are binary and the C 12 C 13 C 14 C 15 C 16 C 17 C 18 C 19 C 20 C 21 C 22 C 23 C 24 S 23 S 22 S 21 S 20 S 19 S 18 S 17 S 16 S 15 S 14 S 12 S 13 V 1 V 4 V 2 V 3 V 8 V 7 V 6 V 5 Figure3.9: Cycle-freebinarygraphicalmodelforC \9 H . Theminimalspanningtreecontain- ing the interface constraints which involve V 1 , V 2 , V 5 , and V 8 , respectively, is highlighted. local constraints labeledC 14 ,C 17 ,C 20 , andC 23 are binary single parity-check constraints while the remaining local constraints are repetition codes. By construction, it has thus been shown that t(C \9 H )=1. (3.81) 2 In order to emphasize that the code and hidden variable labels in Figure 3.9 are in no way related to those labels used previously, the labeling of hidden variables and local constraints begin at S12 and C12, respectively. 69 In light of (3.79) and (3.80), a 4-ary graphical model forC H can be constructed by updating the graphical model illustrated in Figure 3.9 to enforce a single parity-check constraint on V 1 , V 2 , V 5 , and V 8 . A natural choice for the root of the minimal spanning tree containing the interface constraints incident on these variables isC 24 . The updating of the local constraints and hidden variables contained in this spanning tree proceeds as follows. First note that sinceC 12 ,C 15 ,C 18 , andC 22 simply enforce equality, neither these constraints, nor the hidden variables incident on these constraints, need updating. The hidden variables S 14 , S 17 , S 20 , and S 23 are updated to be 4-ary so that they send downstream toC 24 the values of V 1 , V 2 , V 8 , and V 5 , respectively. These hidden variable updates are accomplished by redefining the local constraintsC 14 ,C 17 ,C 20 , andC 23 ; the respective generator matrices for the redefined codes are S 12 S 13 S 14 S 15 S 16 S 17 G 14 = 1 0 11 0 1 10 , G 17 = 1 0 11 0 1 10 , (3.82) S 18 S 19 S 20 S 21 S 22 S 23 G 20 = 1 0 11 0 1 10 , G 23 = 1 0 10 0 1 11 . (3.83) Finally,C 24 is updated to enforce both the original repetition constraint on the respective firstcoordinatesofS 14 ,S 17 ,S 20 ,andS 23 andtheadditionalsingleparity-checkconstraint 70 on V 1 , V 2 , V 5 , and V 8 (which correspond to the respective second coordinates of S 14 , S 17 , S 20 , and S 23 ). The generator matrix for the redefinedC 24 is S 14 S 17 S 20 S 23 G 24 =    10 10 10 10 01 00 00 01 00 01 00 01 00 00 01 01   . (3.84) The updated constraints all satisfy the condition for inclusion in a 4-ary graphical model. Specifically,C 24 can be decomposed into the Cartesian product of a length 4 binary repetition code and a length 4 binary single parity-check code. The updated graphical model is 4-ary and it has thus been shown by construction that t(C H )≤t(C \9 H )+1=2. (3.85) 71 Chapter 4 The Extraction of Graphical Models for Linear Codes I: Cycle-Free Models from Algebraic Structure This chapter is the first of two focusing on the extraction of graphical models for linear codes. This chapter focuses on cycle-free graphical models with the aim of extracting models which imply less complex optimal soft-in soft-out (SISO) decoding algorithms thanthose impliedbytrellises. Whilethegraphicalmodels introducedinthischapterare novel, the search for such models is not new to this dissertation. Forney’s Coset Codes papers [47, 48] sparked a great deal of interest in trellis representations in the coding community. Inparticular,anumberofauthorsinvestigatedtrellissectionalizationinorder to derive decoding algorithms with complexity less than those implied by unsectionalized trellises (cf. [53, 56, 88, 89, 100, 138, 147, 148]). Furthermore, in his second Codes on Graphs paper [51], Forney investigated cycle-free graphical models for Reed-Muller (RM) codes. 72 Sections 4.1-4.3 study cycle-free graphical models for the binary image of Reed- Solomon 1 (RS) codes based on a decomposition due to Vardy and Be’ery [146]. Vardy and Be’ery [146] and Ponnampalam and Vucetic [125] developed optimal soft-in hard- out (SIHO) decoding algorithms based on this decomposition. Later, it was recognized independently by Ponnampalam and Grant [124] and in [60] that Vardy and Be’ery’s decomposition implies a cycle-free graphical model and thus an optimal SISO decoding algorithm. Section4.1describesthiscycle-freemodelwhileSection4.2describesanumber of extensions. In Section, 4.3, suboptimal SISO decoding algorithms based on Vardy and Be’ery’s decomposition are described. These decoding algorithms utilize list-based SISO decoding techniques and a generic list generation algorithm is appended in Section 4.6. The results of Sections 4.1 and 4.3 appeared in the IEEE Transactions on Information Theory [73] while those of Section 4.2 appeared in [61]. Section 4.4 describes novel cycle-free graphical models for the family of first-order Reed-Muller codes and their duals - the extended Hamming codes - based on variable conditioning in generalized Tanner graphs. The complexity of the decoding algorithms implied by the minimal trellises for these codes are known to grow with blocklength n as O n 2 . The novel cycle-free models presented in Section 4.4 imply decoding algorithms with complexity that grow asO(nlogn). Section 4.5 considers the application of the models described in Section 4.4 to a coding problem of particular practical interest: the design of high-rate, low-floor modern codes for use on recursive channels. Specifically, it is shown that extended Hamming codes offer an attractive alternative to high-rate 1 ThischapterassumesacertainfamiliaritywithReed-Solomoncodes. Theunfamiliarreaderisreferred to [20] for the necessary background material. 73 convolutional codes - in terms of both performance and complexity - for use in very high- rate, very low-floor serially concatenated coding schemes. The results of Sections 4.4 and 4.5 appeared first in [62] and were subsequently submitted to the 2007 International Symposium on Information Theory [63]. 4.1 Cycle-Free Models for Reed-Solomon Codes 4.1.1 Vardy and Be’ery’s Decomposition LetC RS be an [n = 2 m − 1,k,d = n− k + 1] RS code defined over F 2 m with roots α,α 2 ,...,α d−1 , whereαisprimitiveinF 2 m. AssociatedwithC RS isthe[n,k 0 ≤k,d 0 ≥ d] binary Bose-Chaudhuri-Hocquenghem (BCH) codeC BCH with roots α,α 2 ,...,α d−1 and their cyclotomic conjugates over F 2 . Let φ : F 2 m→ F m 2 be an F 2 -linear map with basis{γ 1 ,γ 2 ,...,γ m }. Any element c i ∈F 2 m can be written c i = m X j=1 γ j c (j) i where c (j) i ∈F 2 (4.1) and φ thus defines the binary image ofC RS . Vardy and Be’ery proved in [146] that a generator matrix of the binary image ofC RS can be found with the structure shown in Figure 4.1 where B is k 0 ×n and generates C BCH . The firstmk 0 rows of this structure are block diagonal. The lastm(k−k 0 ) rows of this structure are denoted glue vectors in [146]. The code generated by the glue vectors is denoted the glue code. 74 1 Efficient Soft-In Soft-Out Decoding of Reed-Solomon Codes Keith M. Chugg, Alex Grant, Thomas R. Halford and Vishakan Ponnampalam I. OPTIMALSISODECODING OFRSCODES A. The Vardy-Be’ery Decomposition Let C RS be a (n = 2 m − 1,k,d = n− k + 1) RS code defined over GF (2 m ) with generator polynomial: g RS (X) = d−1 ! i=1 " X +α i # (1) where α is primitive in GF (2 m ). Associated with C RS is the (n,k " ≤ k,d " ≥ d) binary BCH code C BCH generated by g BCH (X)withroots $ α,α 2 ,...,α d−1 % andtheircyclotomic conjugates over GF (2). Let φ : GF (2 m )→ (GF (2)) m be alinearmappingthatdefinesthebinaryimageof C RS .Vardy and Be’ery proved in [VaBe91] that a generator matrix of the binary image of C RS can be found with the block diagonal structure pictured in Figure 1 where B generates C BCH . Fig.1. Structureofthegeneratormatrixofthebinaryimageof C RS where B generates C BCH . Based on this structure, define codes B and L as: B = {b(X)|} (2) ThediagonalblocksofFigure ??correspondto B;codewords belonging to B are formed by interleaving BCH codewords. The glue vectors of Figure ?? correspond to L; codewords belonging to L are formed by interleaving some combination of coset leaders of C BCH . B. An Alternate Definition of the Glue Code II. SUBOPTIMALSISODECODING OFRSCODES glue vectors . . . (3) 0 1 Efficient Soft-In Soft-Out Decoding of Reed-Solomon Codes Keith M. Chugg, Alex Grant, Thomas R. Halford and Vishakan Ponnampalam I. OPTIMALSISODECODING OFRSCODES A. The Vardy-Be’ery Decomposition Let C RS be a (n = 2 m − 1,k,d = n− k + 1) RS code defined over GF (2 m ) with generator polynomial: g RS (X) = d−1 ! i=1 " X +α i # (1) where α is primitive in GF (2 m ). Associated with C RS is the (n,k " ≤ k,d " ≥ d) binary BCH code C BCH generated by g BCH (X)withroots $ α,α 2 ,...,α d−1 % andtheircyclotomic conjugates over GF (2). Let φ : GF (2 m )→ (GF (2)) m be alinearmappingthatdefinesthebinaryimageof C RS .Vardy and Be’ery proved in [VaBe91] that a generator matrix of the binary image of C RS can be found with the block diagonal structure pictured in Figure 1 where B generates C BCH . Fig.1. Structureofthegeneratormatrixofthebinaryimageof C RS where B generates C BCH . Based on this structure, define codes B and L as: B = {b(X)|} (2) ThediagonalblocksofFigure ??correspondto B;codewords belonging to B are formed by interleaving BCH codewords. The glue vectors of Figure ?? correspond to L; codewords belonging to L are formed by interleaving some combination of coset leaders of C BCH . B. An Alternate Definition of the Glue Code II. SUBOPTIMALSISODECODING OFRSCODES glue vectors . . . (3) 0 1 Efficient Soft-In Soft-Out Decoding of Reed-Solomon Codes Keith M. Chugg, Alex Grant, Thomas R. Halford and Vishakan Ponnampalam I. OPTIMALSISODECODING OFRSCODES A. The Vardy-Be’ery Decomposition Let C RS be a (n = 2 m − 1,k,d = n− k + 1) RS code defined over GF (2 m ) with generator polynomial: g RS (X) = d−1 ! i=1 " X +α i # (1) where α is primitive in GF (2 m ). Associated with C RS is the (n,k " ≤ k,d " ≥ d) binary BCH code C BCH generated by g BCH (X)withroots $ α,α 2 ,...,α d−1 % andtheircyclotomic conjugates over GF (2). Let φ : GF (2 m )→ (GF (2)) m be alinearmappingthatdefinesthebinaryimageof C RS .Vardy and Be’ery proved in [VaBe91] that a generator matrix of the binary image of C RS can be found with the block diagonal structure pictured in Figure 1 where B generates C BCH . Fig.1. Structureofthegeneratormatrixofthebinaryimageof C RS where B generates C BCH . Based on this structure, define codes B and L as: B = {b(X)|} (2) ThediagonalblocksofFigure ??correspondto B;codewords belonging to B are formed by interleaving BCH codewords. The glue vectors of Figure ?? correspond to L; codewords belonging to L are formed by interleaving some combination of coset leaders of C BCH . B. An Alternate Definition of the Glue Code II. SUBOPTIMALSISODECODING OFRSCODES glue vectors . . . (3) 0 mn mk mk ! m(k−k ! ) B B B glue vectors Figure 4.1: Structure of the generator matrix of the binary image ofC RS . The structure of Figure 4.1 implies that a codeword in the binary image ofC RS is formed by interleavingm codewords drawn fromC BCH and adding a glue code codeword. Formally, define codesB andL overF 2 m as 2 B = ( b(X) b(X)= m X j=1 γ j c (j) (X),c (j) (X)∈C BCH ) (4.2) L= ( l(X) l(X)= m X j=1 γ j l (j) (X), l (j) (X)∈E and l(β)=0 for β∈{α,...,α d−1 } ) (4.3) whereE is the set of coset leaders ofC BCH . The diagonal blocks of Figure 4.1 correspond to the binary image ofB; codewords belonging toB are formed by interleaving BCH codewords. ThegluevectorsofFigure4.1correspondtothebinaryimageofL;codewords belonging toL are formed by interleaving some combination of coset representatives of C BCH . The RS codeC RS is a binary linear combination ofB andL. 2 Throughout this section, codewords are described interchangeably as n-tuples: c c c = (c1,c2,...,cn), and as polynomials in an indeterminant X: c(X) = P n i=1 ciX i−1 . 75 Example: [7,5,3] RS Code LetC [7,5,3] be the [7,5,3] RS code defined overF 8 with generator polynomial g [7,5,3] (X)=(X +α) X +α 2 (4.4) andletφ:F 8 →F 3 2 havebasis 1,α,α 2 . Specifically, withthisbasis: 1→100,α→010, and α 2 → 001. The associated BCH codeC [7,4,3] has roots α, α 2 , and α 4 , and thus has dimension 4 and minimum distance 3. A generator matrix for the binary image ofC [7,5,3] with the structure shown in Figure 4.1 is G [7,5,3] =                         1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0                         . (4.5) If the coset leaders ofC [7,4,3] are labelled 1 =(0,0,0,0,0,0,0), 5 =(0,0,1,0,0,0,0), 2 =(1,0,0,0,0,0,0), 6 =(0,0,0,0,0,0,1), 3 =(0,1,0,0,0,0,0), 7 =(0,0,0,0,1,0,0), 4 =(0,0,0,1,0,0,0), 8 =(0,0,0,0,0,1,0), (4.6) 76 then the coset configurations that satisfy (4.3) are ( 1 , 1 , 1 ), ( 2 , 7 , 5 ), ( 3 , 8 , 4 ), ( 4 , 2 , 8 ), ( 5 , 6 , 7 ), ( 6 , 4 , 3 ), ( 7 , 3 , 6 ), ( 8 , 5 , 2 ). (4.7) To illustrate that the glue vectors of (4.5) generate the coset configurations of (4.7), consider the binary sum of the three glue vectors (1,1,1,0,0,0,0,0,0,0,0,1,1,1,0,0,1,1,1,0,0) (4.8) and note that (1,1,1,0,0,0,0)∈C [7,4,3] + 8 (0,0,0,0,1,1,1)∈C [7,4,3] + 5 (0,0,1,1,1,0,0)∈C [7,4,3] + 2 . (4.9) The coset configuration corresponding to the sum of the glue vectors is thus ( 8 , 5 , 2 ). 4.1.2 An Alternate Definition of the Glue Code Let 1 , 2 ,..., |E| =E be the set of coset leaders ofC BCH and let s s s 1 ,s s s 2 ,...,s s s |E| =S be the corresponding set of syndromes where|E|=2 n−k 0 . The map between coset leaders and syndromes is defined by s s s i = i H > BCH (4.10) where H BCH is an n−k 0 ×n parity-check matrix forC BCH . 77 LetT be the code obtained fromL by replacing each coset leader l (j) by the corre- sponding syndrome t (j) =l (j) H > BCH : T = ( t(X) t(X)= m X j=1 γ j t (j) (X),t (j) (X)∈S and t(β)=0 for β∈ n α,...,α d−1 o ) (4.11) where{γ 1 ,γ 2 ,...,γ m } defines the mapping φ :F 2 m→F m 2 and is a subset of a basis for the mapping φ 0 : F 2 n−k 0 → F n−k 0 2 . Codewords inT are thus formed by interleaving m binary syndrome vectors to form an m-tuple of symbols drawn fromF 2 n−k 0. The code defined by (4.11) has length m and is defined over F 2 n−k 0. The codes L andT are clearly closely related; the term glue code is used to denoteL andT interchangeably throughout this chapter. Codewords l(X) ∈ L and t(X) ∈ T must both satisfy l(β) = t(β) = 0 for β∈ α,...,α d−1 . When considering the generation or encoding of a codeword, theL representation of the glue code is more useful as was demonstrated in the example of Section 4.1.1. As will be seen in Sections 4.1.3 and 4.3, however, when considering the decoding of a codeword, theT representation of the glue code is more useful. 78 Example: [7,5,3] RS Code The glue codeT of the [7,5,3] RS code is defined over F 2 n−k 0 =F 8 , has roots α,α 2 , and is thus the [7,5,3] RS code shortened to length 3 with generator matrix G T = α 3 α 4 1 (4.12) overF 8 . The columns of the binary image of G T can be rearranged so that the bits from each syndrome are grouped together yielding a generator matrix for the resulting binary codeT 0 that is systematic in t (1) (X): t (1) 1 t (1) 2 t (1) 3 t (2) 1 t (2) 2 t (2) 3 t (3) 1 t (3) 2 t (3) 3 G T 0 =   1 0 0 0 1 1 0 0 1 0 1 0 1 1 1 1 1 0 0 0 1 1 0 1 0 1 1   . (4.13) Eight syndrome 3-tuples, or syndrome configurations, are generated by G T 0: (s s s 1 ,s s s 1 ,s s s 1 ), (s s s 2 ,s s s 7 ,s s s 5 ), (s s s 3 ,s s s 8 ,s s s 4 ), (s s s 4 ,s s s 2 ,s s s 8 ), (s s s 5 ,s s s 6 ,s s s 7 ), (s s s 6 ,s s s 4 ,s s s 3 ), (s s s 7 ,s s s 3 ,s s s 6 ), (s s s 8 ,s s s 5 ,s s s 2 ). (4.14) The set of syndrome configurations in (4.14) are in one-to-one correspondence with the set of coset configurations in (4.7) via (4.10). Note that not every syndrome 3-tuple corresponds to a codeword inT. For example, (s s s 1 ,s s s 1 ,s s s 1 ) / ∈T. Syndrome configurations inT are denoted valid syndrome configurations. The corresponding coset configurations inL are denoted valid coset configurations. 79 4.1.3 From Decomposition to Cycle-Free Graphical Model a (m) 1 c (m) 1 c (m) k c (m) k+1 c (m) n c (2) n c (2) k+1 c (2) k c (2) 1 c (1) 1 c (1) k c (1) k+1 c (1) n a (1) 1 a (1) k a (2) k a (2) 1 a (m) k Wolf parity-check trellis Uncoded bits for systematic trellis Glue code constraint node t t t (1) t t t (2) t t t (m) Figure 4.2: Cycle-free graphical model for Reed-Solomon codes based on Vardy and Be’ery’s decomposition. The generator matrix structure seen in Figure 4.1 implies the cycle-free graphical model for RS codes illustrated in Figure 4.2. This model consists of m n-stage binary trellises and an additional glue code constraint node. The binary trellises correspond to C BCH andareconstructedusingWolf’smethod[155]. Thefinaltrellisstageisa2 n−k 0 -ary 80 hiddenvariablecorrespondingtothecosets,orequivalentlythesyndromes,ofC BCH . The node connecting the final trellis stages corresponds to the glue codeT. Coded bits are labeled n c (j) i o for i = 1,...,n and j = 1,...,m in Figure 4.2 while uncodedbitsaresimilarlylabeled n a (j) i o fori=1,...,k andj =1,...,m. Ifthereisnoa priori soft information on uncoded bits then the corresponding graphical model sections are ignored. If there is a priori soft information on uncoded bits and if a systematic encoderisusedthentheequalityconstraintsinthecorrespondingsectionsofthegraphical model simply enforce a (j) i =c (j) i for i=1,...,k and j =1,...,m. The graphical model illustrated in Figure 4.2 is cycle-free and thus implies optimal SISO decoding algorithms. The following (non-unique) message passing schedule ensures optimal SISO decoding. An inward recursion is performed on each trellis in parallel cor- responding to the forward recursion of the standard forward-backward algorithm (FBA). The forward state messages for the the final state then act as soft input to an optimal SISO decoder for the glue code. The backward state metrics of the trellises are initialized withthesoftoutputofthegluecodeSISOdecoder. Theoutwardrecursiononeachtrellis is then performed in parallel corresponding to the backward recursion of the standard FBA. After the forward and backward metrics are computed at each trellis state, soft-out information on coded (and possibly uncoded) bits is obtained as per the FBA. Optimal SISO decoding ofC RS requires optimal SISO decoding of the glue code. One such optimal SISO decoder uses a trellis. When|T| is small, a trellis need not be used and the SISO decoding of the glue code proceeds as follows. The metric associated with each valid syndrome configuration t t t ∈ T is computed by combining soft input informationontheindividualsyndromes. Thesoftoutputoneachsyndromeisthenfound 81 by marginalizing over all configurations consistent with that syndrome. This process is denoted exhaustive combination and marginalization. 4.1.4 Cycle-Free Model Complexity The complexity of optimal soft-in soft-out decoding of Reed-Solomon codes using a Wolf trellis grows as the number of trellis states [104]: O 2 m(n−k) . (4.15) In the language of Chapter 3, such trellises constitute 2 m(n−k) -ary graphical models. For most RS codes, the complexity of the proposed optimal SISO decoder is domi- nated by the complexity of the glue code SISO decoder. If the glue code is decoded via exhaustive combination and marginalization then the complexity of the glue code SISO grows as the number of glue codewords: 2 m(k−k 0 ) . If the glue code is decoded with a trellis then the complexity of the glue code SISO grows as the number of states in the gluecodetrellis: 2 m(n−k) . InthelanguageofChapter3, theproposedcycle-freegraphical models are also 2 m(n−k) -ary. The complexity of the proposed optimal SISO decoder thus grows as O min 2 m(n−k) ,2 m(k−k 0 ) . (4.16) 82 Wolf Trellis Proposed SISO m n k k 0 Complexity Complexity 3 7 3 1 19.0 13.1 5 4 13.0 10.0 4 15 9 5 32.5 19.8 11 7 24.5 19.2 13 11 16.5 13.0 5 31 25 16 39.9 38.8 27 21 29.9 28.2 29 26 19.9 17.5 Table 4.1: Complexity comparison of proposed optimal SISO and Wolf trellis decoders for various Reed-Solomon codes. For RS codes where k−k 0 ≤n−k, the proposed optimal SISO decoder is much less complex than trellis decoding. This set includes the [7,3,5], [7,5,3], [15,9,7], [15,11,5], and [15,13,3] codes. For larger RS codes, the complexity of the Wolf trellis and proposed SISO decoders both grow as O 2 m(n−k) (indeed, both decoders are implied by 2 m(n−k) - ary models), and neither decoder is practically realizable in accordance with Wiberg’s Cut-SetBound(CSB)[151]. Moreover,theCSBprecludestheexistenceofanypractically realizable optimal SISO decoding algorithms for large RS codes. A more specific complexity comparison can be made by first estimating the number of real operations required by the FBA on an N-stage, S-state Wolf trellis. The forward and backward recursions each require 1 add-compare-select operation, or 3 real opera- tions, per-state, per-stage [125]. The completion step requires 2log 2 S real operations per stage. Exhaustive combination and marginalization of a length m code requires 2m real operations per codeword. Table 4.1 compares the complexity of the proposed optimal SISO and Wolf trellis decoders for a number of RS codes. Complexity is given as the base-2 logarithm of real operations per codeword. 83 4.2 Extensions of the Reed-Solomon Code Models In [146], Vardy and Be’ery noted that generator matrix with the structure shown in Figure 4.1 can be found for any code containing a subfield subcode. Accordingly, there exist cycle-free graphical models similar to that shown in Figure 4.2 for codes containing subfield subcodes. In this section, such models are described for shortened and extended RS codes followed by a brief description of a code construction technique developed by Ponnampalam [123] that uses a generalization of Vardy and Be’ery’s decomposition. 4.2.1 Shortened Reed-Solomon Codes Consider an [n=2 m −1,k,d] RS codeC RS that is systematically encoded. Without loss of generality, let the first k symbols of a codeword c c c∈C RS be message symbols and the lastn−k symbolsbeparitysymbols. An[n−s,k−s,d]shortenedRScodeC S RS isdefined by deleting the final s message symbols. Shortened RS codes can be similarly defined by shortening over any size s subset of the message symbols. The graphical model of Figure 4.2 is updated to accommodateC S RS by removing transitionscorrespondingtoc (j) i =1fori=k−s+1,...,kandj =1,...,m. Equivalently, these trellis stages can be removed altogether by virtue of the Wolf trellis construction. The glue codes associated withC RS andC S RS are identical. 4.2.2 Extended Reed-Solomon Codes The Vardy-Be’ery decomposition of RS codes extends naturally to extended RS codes. LetC RS be an [n=2 m −1,k,d] RS code andC BCH the associated binary BCH code. The 84 singly extended RS codeC E RS is defined by adding an overall parity constraint to each codeword: C E RS = ( n X i=1 c i X i−1 +X n n X i=1 c i X i−1 ∈C RS ) . (4.17) The extended BCH codeC E BCH is similarly defined. ToillustratetheVardy-Be’erydecompositionofC E RS ,firstexpressacodewordc E (X)∈ C E RS as the sum of an RS codeword c RS (X) = P n i=1 c i X i−1 and an overall parity con- straint: c E (X)=c RS (X)+ n X i=1 c i X n . (4.18) Decompose c RS (X) into the sum of some b(X)∈B and l(X)∈L (whereB andL are defined as per (4.2) and (4.3), respectively) and expand the parity term using c i =b i +l i : c E (X)=b(X)+l(X)+ n X i=1 (b i +l i )X n = m X j=1 γ j n X i=1 b (j) i X i−1 + n X i=1 b (j) i X n ! + m X j=1 γ j n X i=1 l (j) i X i−1 + n X i=1 l (j) i X n ! (4.19) where{γ 1 ,γ 2 ,...,γ m }definesthemappingφ:F 2 m→F m 2 . From(4.2)and(4.3)itfollows that: n X i=1 b (j) i X i−1 =c (j) BCH (X)∈C BCH (4.20) n X i=1 b (j) i X i−1 + n X i=1 b (j) i X n =c (j) BCH,E (X)∈C E BCH (4.21) n X i=1 l (j) i X i−1 =l (j) (X)∈E BCH (4.22) 85 n X i=1 l (j) i X i−1 + n X i=1 l (j) i X n =l (j) E (X)∈E E BCH (4.23) whereE BCH is the set of coset leaders ofC BCH andE E BCH is the set of coset leaders ofC E BCH . The glue codeT E is defined forC E RS over some subset ofS E BCH using the techniques of Section 4.1.2. A cycle-free graphical model for the binary image ofC E RS is thus constructed with m parallel Wolf trellises forC E BCH and a glue code node. Example: [8,5,4] Extended RS Code LetC E RS bethe[8,5,4]extendedRScodedefinedoverF 8 . Theassociatedextendedbinary BCH codeC E BCH is the [8,4,4] extended Hamming code with parity-check matrix H E BCH =    1 1 1 1 1 1 1 1 0 0 0 1 1 1 1 0 0 1 1 0 0 1 1 0 1 0 1 0 1 0 1 0   . (4.24) Let the syndromes of C E BCH be s s s 1 = (0,0,0,0),s s s 2 = (0,0,0,1),...,s s s 16 = (1,1,1,1). The valid syndrome configurations inT E are obtained from the valid coset configurations in (4.6): (s s s 1 ,s s s 1 ,s s s 1 ),(s s s 2 ,s s s 6 ,s s s 4 ),(s s s 3 ,s s s 7 ,s s s 5 ),(s s s 5 ,s s s 2 ,s s s 7 ), (s s s 4 ,s s s 8 ,s s s 6 ),(s s s 8 ,s s s 5 ,s s s 3 ),(s s s 6 ,s s s 3 ,s s s 8 ),(s s s 7 ,s s s 4 ,s s s 2 ). (4.25) Note that only half of the syndrome labels are involved in any valid configuration. 86 4.2.3 Code Constructions Based on Vardy and Be’ery’s Decomposition LetC 1 andC 2 be [n 1 ,k 1 ,d 1 ] and [n 1 ,k 2 ,d 2 ] binary codes, respectively, such thatC 2 is a subcode ofC 1 . Let H 2 be a parity-check matrix forC 2 . SinceC 2 ⊂C 1 , C 1 = [ ∈E 0 2 C 2 + (4.26) whereE 0 2 is a size 2 k 1 −k 2 subset of the coset leaders ofC 2 ,E 2 . Finally, letC 3 be an [n 3 ,k 3 ,d 3 ] code overF 2 k 1 −k 2 . Define a new codeC overF 2 n 3 as C = ( c(X) n 3 X j=1 γ j c (j) 2 (X)+g(X), c (j) 2 ∈C 2 and g(X)∈G ) (4.27) where{γ 1 ,γ 2 ,...,γ n 3 }definesthemappingφ:F 2 n 3→F n 3 2 andisasubsetofthemapping φ 0 :F 2 k 1 −k 2 →F k 1 −k 2 2 . The codeG is a generalization of the RS glue code defined by: G = ( g(X)= n 3 X j=1 γ j g (j) (X) n 3 X j=1 γ j g (j) (X)H > 2 ∈C 3 ) . (4.28) The binary image ofC is a generalized Vardy-Be’ery code with parameters n=n 1 n 3 , (4.29) k =n 3 k 2 +(k 1 −k 2 )k 3 , (4.30) d=min(d 2 ,d 1 d 3 ). (4.31) RS codes belong to this class of codes whereC 1 is the [n,n,1] universe code,C 2 is the appropriate [n,k 0 ,d 0 ] BCH code, andC 3 isT (as defined in (4.11)). Extended RS codes 87 also belong to this class of codes whereC 2 is an extended BCH code andC 1 is a union of cosets ofC 2 as seen in Section 4.2.2 Generalized Vardy-Be’ery codes have cycle-free graphical models similar to that seen in Figure 4.2. A graphical model for the binary image ofC is constructed withn 3 parallel Wolf trellises connected by a glue node corresponding toG. The Wolf trellises are pruned so that only paths correponding to the cosets inE 0 2 remain. Optimal SISO decoding of the binary image ofC can be achieved using the inward-outward activation schedule described in Section 4.1.3. ThevalueofsmallgeneralizedVardy-Be’erycodesliesintheirefficientSISOdecoders. The value of large generalized Vardy-Be’ery codes lies in the algebraic structure enforced by the glue code that may lead to efficient hard-out algebraic decoding algorithms. The construction of such codes is thus an interesting direction for future work. Example LetC 1 be a [16,11,4] extended BCH code and letC 2 be a [16,5,8] first order Reed-Muller code. It is known thatC 2 is a subcode ofC 1 [111]. LetC 3 be any [n 3 ,n 3 −1,2] code over F 64 . The resulting generalized Vardy-Be’ery codeC has length 16n 3 , dimension 11n 3 −6, and minimum distance 8. 88 4.3 Suboptimal SISO Decoding of Reed-Solomon Codes For high-rate RS codes, the complexity of optimal SISO decoder proposed in Section 4.1 is dominated by the complexity of the optimal glue code SISO decoder. Practically realizable suboptimal SISO decoding algorithms for high-rate RS codes can be developed byreplacingtheoptimalgluecodeSISOdecoderbyasuboptimalgluecodeSISOdecoder. This section describes such a decoder that uses a list-based glue code SISO decoder. List-basedSISOdecodersforlinearblockcodeshavebeenexaminedextensivelyinthe literature (see, for example, [113] and the references therein). List-based SISO decoders first produce a list of likely codewordsK ⊂ C and then perform the marginalization described in Section 4.1.3 overK rather than all codewordsC. The complexity of list- based SISO decoders depends on K =|K| and can thus be controlled. 4.3.1 Generic Glue Code List Generation The generation ofK is the most difficult design aspect of list-based SISO decoders [113]. The following presents a generic approach to list generation for the glue code. As described in Section 4.1.2, codewords ofT are BCH syndrome m-tuples (or con- figurations). Specifically, let w be the syndrome configuration w w w = s s s (1) w w w ,...,s s s (m) w w w ∈S m (4.32) whereS is the set of BCH syndromes. Associated with w w w is the syndrome configuration metric M[w w w]= m X i=1 SI[s s s (i) w w w ] (4.33) 89 where SI[s s s (i) w w w ] is the final state metric corresponding to the syndrome s s s (i) w w w in the i-th parallelBCHtrellis 3 . RecallfromSection4.1.2thatthesetofallsyndromeconfigurations is a superset of the glue codeT. Algorithm 4.3 generates a list of likely codewords by generating a list of likely syndrome configurations and throwing out those configurations not contained inT. The next shortest path algorithm described in [42] is used to obtain the likely syndrome configurations; this algorithm was used successfully for list detection in multiple access channels in [128] and is detailed in Section 4.6. Input: SI[s s s (i) ] for alls s s∈S and i=1,...,m. Output: List of glue codewords with K-shortest syndrome configuration metrics. K←∅; codewords found←0; repeat w w w← syndrome configuration with next smallest syndrome configuration metric; if w w w∈T then K←K∪{w w w}; codewords found←codewords found+1; end until codewords found=K; Figure 4.3: Generic glue code list generation algorithm. 4.3.2 Glue Code List Generation for the [n,n−2,3] RS Codes Algorithm4.3isinefficientbecauseinordertogeneratealistofK codewords, manymore than K syndrome configurations must be generated. The following presents a reduced- complexity list generation algorithm for the glue codes corresponding to the [n,n−2,3] RS codes that exploits the algebraic structure ofT. 3 Note that min-sum or min ? -sum processing is assumed and combination is achieved via addition of metrics rather than multiplication of probabilities. 90 LetC RS be an [n=2 m −1,n−2,3] RS code with roots α and α 2 where α is primitive in F 2 m. Since α 2 m = α in F 2 m, the union of the sets of cyclotomic conjugates of α and α 2 overF 2 is α,α 2 ,...,α 2 m −1 . The associated BCH codeC BCH thus has dimension k 0 =k−(m+2)=n−m. (4.34) For the [n,n−2,3] RS codes, the glue codeT is, therefore, a length m code defined over F 2 m with roots α,α 2 . Since|T| = 2 m(k−k 0 ) = 2 m(m−2) , the dimension of the glue code is m−2 andT is a shortened [n,n−2,3] RS code. As per the example of Section 4.1.2, let G T generateT overF 2 m and let G T 0 be the binary image ofG T with columns reordered so that bits from each syndrome are grouped together. For the specific [n,n−2,3] RS codes considered, binary generators matrices that are systematic in the bits corresponding tot t t (1) ,...,t t t (m−2) can be obtained. Define a syndrome sub-configuration w w w |m−2 as the first m−2 elements of a syndrome configurationw w w∈S m : w w w |m−2 = s s s (1) w w w ,...,s s s (m−2) w w w ∈S m−2 . (4.35) Associated withw w w m−2 is the syndrome sub-configuration metric M[w w w |m−2 ]= m−2 X i=1 SI[s s s (i) w w w ] (4.36) where SI[s s s (i) w w w ] is defined as per Section 4.3.1. Algorithm 4.4 generates a list of likely code- wordsbygeneratingalistoflikelysyndromesub-configurationsandexploitsthestructure 91 ofT to determine a glue code codeword corresponding to each sub-configuration. Note that only K sub-configurations need be generated since each sub-configuration generates a single codeword. Also note that the lists of codewords produced by Algorithms 4.3 and 4.4 may not be identical since Algorithm 4.4 uses metrics from only m−2 trellises in its shortest path search. Input: SI[s s s (i) ] for alls s s∈S and i=1,...,m−2. Output: List of glue codewords with K-shortest syndrome sub-configuration metrics. K←∅; for k =1,...,K do w w w |m−2 ← syndrome sub-configuration with next smallest syndrome sub-configuration metric; w w w 0 |m−2 ← binary image ofw w w |m−2 ; t t t 0 ←w w w 0 |m−2 G T 0; t t t← glue code codeword corresponding tot t t 0 ; K←K∪{t t t}; end Figure 4.4: List generation for [n,n−2,3] RS codes algorithm. Since RS codes are maximum distance separable overF 2 m, Algorithm 4.4 can readily be adapted to generate a list of likely RS codewords by considering configurations ofF 2 m symbols and the corresponding symbol-level soft information. The resulting list-based SISO decoder is denoted the standard list-based SISO in Section 4.3.5 since it does not exploit the Vardy-Be’ery decomposition. 92 4.3.3 Glue Code List Generation for the [n,n−4,5] RS Codes The structure of the [n,n− 4,5] RS codes can also be exploited in order to obtain a reduced-complexity list generation algorithm similar to that for the [n,n−2,3] codes. Specifically, letC RS be an [n = 2 m−1 ,n−4,5] RS code with roots α,α 2 ,α 3 ,α 4 . It is readily verified that the associated BCH codeC BCH has roots n α 2 i o m−1 i=0 , n α 3·2 i o m−2 i=0 ,α 3·2 m−1 −n . (4.37) The dimension ofC BCH is thus k 0 = n−2m and the glue codeT is defined over F 2 2m. The binary imageT 0 of the glue code thus has length 2m 2 and dimension 2m 2 −4m. Let G T 0 generateT 0 . Because codewords inT are formed using a size m subset of the mapping φ 0 :F 2 2m→F 2m 2 (as per (4.11)), it is not possible to make G T 0 systematic in its first 2m 2 −4m positions. It is, however, possible to find a generator matrix with the structure shown in Figure 4.5. I P 1 P 2 0 m m 2m 2 − 5m 2m 2 −4m 2m 2 2m 2 − 5m 4m Figure 4.5: Structure of the generator matrix of the binary image ofT for the [n,n−4,5] Reed-Solomon codes where bits from each syndrome are grouped together. The structure of G T 0 shown in Figure 4.5 motivates the following reduced-complexity glue code list generation algorithm for the [n,n− 4,5] RS codes. Define a syndrome 93 sub-configuration w w w |m−2 and its metric M[w w w |m−2 ] as per (4.35) and (4.36), respectively. Letw w w 0 |m−2 be the binary image ofw w w |m−2 : w w w 0 |m−2 = s s s (1) w w w,1 ,...,s s s (1) w w w,2m ,...,s s s (m−2) w w w,1 ,...,s s s (m−2) w w w,2m . (4.38) A syndrome sub-configuration w w w |m−2 comprises the first m−2 symbols of a codeword t t t∈T only if w w w 0 |m−2 H > 1 =0 (4.39) where H 1 is the (2m 2 −4m)×m binary matrix H 1 = [P > 1 | I] and P 1 is the submatrix of G T 0 seen in Figure 4.5. From the structure shown in Figure 4.5, it follows that if the binary (2m 2 − 4m)-tuple w w w |m−2 satisfies (4.39) then so do each of the 2 m binary (2m 2 −4m)-tuples of the form w w w 0 |m−2 = s s s (1) w w w,1 ,...,s s s (1) w w w,2m ,...,s s s (m−2) w w w,1 ,...,s s s (m−2) w w w,m ,b 1 ,...,b m (4.40) where b i ∈F 2 for i = 1,...,m. Thus, for each (m−2)-tuple of syndromes w w w |m−2 with binary image satisfying (4.39), 2 m glue codewords can be generated. Algorithm 4.6 generates a list of likely syndrome configurationsK of length K =|K| by first generating a list of most likely sub-configurations. Those sub-configurations that satisfy (4.39) are then used to generate codewords t t t∈T. As with Algorithm 4.4, Algorithm 4.6 is of lower complexity than Algorithm 4.3 because less total configurations are required to generate the same number of valid configurations. 94 Input: SI[s s s (i) ] for alls s s∈S and i=1,...,m−2. Output: List of glue codewords with K-shortest syndrome sub-configuration metrics. K←∅; repeat w w w |m−2 ← syndrome sub-configuration with next smallest syndrome sub-configuration metric; w w w 0 |m−2 ← binary image ofw w w |m−2 ; if w w w 0 |m−2 H > 1 =0 then foreach (b 1 ,...,b m )∈F m 2 do u u u← s s s (1) w w w,1 ,...,s s s (1) w w w,2m ,...,s s s (m−2) w w w,1 ,...,s s s (m−2) w w w,m ,b 1 ,...,b m ; t t t 0 ←u u uG T 0; t t t← glue code codeword corresponding tot t t 0 ; K←K∪{t t t}; end end until|K|≥K; Figure 4.6: List generation for [n,n−4,5] RS codes algorithm. 4.3.4 Complexity of the Suboptimal SISO It is shown in [42] that the K syndrome configurations with smallest metric can be generated in time O 2 n−k 0 +(m+1)log 2 (m+1)+K . (4.41) The complexity of the proposed suboptimal glue SISO decoder can thus be chosen to match that of the parallel BCH Wolf trellis decoders yielding an overall complexity that grows as O 2 n−k 0 . (4.42) For high rate codes (e.g. the [n,n−2,3] and [n,n−4,5] codes), n−k 0 is small enough for the proposed suboptimal SISO decoder to be practically realizable. 95 4.3.5 Simulation Results and Discussion In this section the performance of the proposed suboptimal SISO decoder is compared to that of both the proposed optimal SISO decoder and the standard list-based SISO decoder that was described at the end of Section 4.3.2. Note that the reduced-complexity list generation algorithms described in Sections 4.3.2 and 4.3.3 are used rather than Algorithm 4.3. Binary antipodal signaling over additive white Gaussian noise channels is assumed throughout. Figure 4.7 compares the performance of the three algorithms when used as stand- alone decoders for the [15,13,3] and [15,11,5] codes. The glue codes of the [15,13,3] and [15,11,5] contain 256 and 65,536 codewords, respectively. A negligible performance loss is incurred by the proposed suboptimal decoders when respective glue code list sizes of 32 and 1024 are used. Observe that the standard list-based SISO decoder with list size 1024 incurs a 0.5 dB loss with respect to the proposed algorithms. Generating 1024 codewords of the [15,11,5] glue code, which is a length 4 code overF 256 , is substantially less complex than generating 1024 codewords of the full [15,11,5] RS code overF 16 . The proposed suboptimal SISO was also compared to the proposed optimal SISO for the [31,29,3] and [63,61,3] codes. It was found that respective glue code list sizes of 64 and 128 were required in order to approximate optimal performance for these codes. In order to investigate the quality of soft-out information produced by the proposed suboptimal SISO decoder, its performance was compared to that of the proposed optimal and standard list-based SISO decoders in an iteratively decoded system. Specifically, a rate 13 17 [15,13,3]×[15,13,3] Reed-Solomon turbo product code was considered with 96 10 -4 10 -3 10 -2 3.5 4 4.5 5 5.5 6 6.5 7 (15,13) Optimal (15,13) Suboptimal (32) (15,13) Std. List (64) (15,11) Optimal (15,11) Suboptimal (1024) (15,11) Std. List (1024) Codeword Error Rate Eb/No (dB) Figure 4.7: Codeword error rate performance comparison of the proposed optimal SISO, proposed suboptimal SISO, and standard list-based SISO for the [15,13,3] and [15,11,5] RS codes. List sizes appear in parentheses. input block length 2704 bits. A high-spread pseudo-random bit-level interleaver was constructed using the real-relaxation optimization method described in [35]. Figure 4.8 illustrates the performance of the iterative turbo product decoder after 10 iterationsusingfivedifferentRSSISOdecoders: theproposedoptimalSISO,theproposed suboptimal SISO with glue code list sizes 32 and 64, and the standard list-based SISO with list sizes 256 and 1024. With a list size of 64, the decoder employing the suboptimal SISO incurs a negligible loss with respect to the decoder employing the optimal SISO. With a list size of 256, the decoder employing the standard list-based SISO performs approximately 0.3 dB worse than the decoder employing the optimal SISO at a bit error rate of 10 −4 ; increasing the list size to 1024 narrows, but does not close, this performance gap. Generating64codewordsofthe[15,13,3]gluescode,whichisalength4codedefined 97 over F 16 , is much less complex than generating 1024 codewords of the full [15,13,3] RS code overF 16 . 10 -5 10 -4 10 -3 10 -2 2.4 2.6 2.8 3 3.2 3.4 Optimal Suboptimal (32) Suboptimal (64) Std. List (256) Std. List (1024) Bit Error Rate Eb/No (dB) Figure 4.8: Bit error rate performance comparison of the proposed optimal SISO, pro- posedsuboptimalSISO,andstandardlist-basedSISOfortherate 13 17 [15,13,3]×[15,13,3] RS turbo product code. Ten decoding iterations were performed. List sizes appear in parentheses. 98 4.4 Cycle-Free Models for First-Order Reed-Muller Codes Consider again the extended Hamming codeC H that was studied in Chapter 2. As was described in that chapter, there exists a degree-1 generalized extension forC H with parity-check matrix: e H H =      1 1 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 0 0 0 1 0 0 0 1 1      . (4.43) The generalized Tanner graph corresponding to (4.43) is illustrated in Figure 4.9. p 1 V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure 4.9: Generalized Tanner graph forC H implied by (4.43). It is well-known that cycle-free models for codes can be obtained via hidden variable conditioning in models with cycles (cf. [74]). Hidden variable conditioning is particularly well-understood in the context of tail-biting trellises (cf. [1, 5, 150]). A cycle-free model forC H can be obtained by conditioning on p 1 in Figure 4.9. The resulting conditionally 99 cycle-free generalized Tanner graph is illustrated in Figure 4.10. Optimal SISO decoding ofC H can be accomplished by decoding twice on the conditionally cycle-free model (once with p 1 =0 and once with p 1 =1) and then appropriately marginalizing over the results of the two decoding rounds [74]. 0/1 0/1 0/1 0/1 V 1 V 5 V 2 V 6 V 3 V 7 V 4 V 8 Figure 4.10: Conditionally cycle-free generalized Tanner graph forC H implied by (4.43). The model illustrated in Figure 4.10 contains one conditioned binary variable and is denoted a degree-1 conditionally cycle-free generalized Tanner graph (GTG). More generally, a degree-f conditionally cycle-free GTG contains f variables which can be fixed to remove all cycles. Optimal SISO decoding can be achieved by decoding 2 f times on a degree-f conditionallycycle-freeGTG(andappropriatelymarginalizingovertheresults). It is well-known thatC H is also a first-order Reed-Muller code. In this Section, conditionally cycle-free GTGs are derived for the family of first-order RM codes and their duals - the extended Hamming codes. Before this derivation in Section 4.4.2, RM 100 codes and their properties are reviewed in Section 4.4.1. In Section 4.4.3, the complexity of the optimal SISO decoding algorithms implied by the derived models is examined. 4.4.1 Reed-Muller Codes LetC RM(r,m) denote the r th -order Reed-Muller code with length n=2 m . The dimension and minimum distance ofC RM(r,m) are k = r X j=0 m j and d=2 m−r , (4.44) respectively. The universe code of length 2 m is thus C RM(m,m) while the length 2 m repetition code isC RM(0,m) . The dual of an RM code is also an RM code [111]: C ⊥ RM(r,m) =C RM(m−r−1,m) . (4.45) It is well-known that the duals of the first-order RM codes are the [2 m ,2 m −m−1,4] extended Hamming codes. Reed-Muller codes can be constructed recursively via the squaring construction [111]. Specifically, any codewordc c c∈C RM(r,m) can be decomposed as c c c=(u u u,u u u+v v v) where u u u∈C RM(r,m−1) ,v v v∈C RM(r−1,m−1) . (4.46) As will be seen in Section 4.4.2, this construction takes a particularly useful form for the family of first-order RM codes. If c c c is a codeword in the length 2 m first-order RM 101 code, then it can be decomposed as per (4.46) where u u u is drawn from the length 2 m−1 first-order RM code andv v v from the length 2 m−1 repetition code. Asanexample, considerthe[8,4,4]extendedHammingcodeC H (whichisalsoafirst- order RM code by (4.45)), any codeword of which can be decomposed via the squaring construction as c c c=(c 1 ,c 2 ,c 3 ,c 4 ,c 5 ,c 6 ,c 7 ,c 8 ) =(u 1 ,u 2 ,u 3 ,u 4 ,u 1 +v 1 ,u 2 +v 2 ,u 3 +v 3 ,u 4 +v 4 ) =(u u u,u u u+v v v) (4.47) whereu u uisdrawnfromthe[4,3,2]singleparity-checkcodeandv v v isdrawnfromthe[4,1,4] repetition code. It is precisely this decompostion that motivates the generalized Tanner graph illustrated in Figure 4.9. Specifically, the hidden variable p 1 is c 4 +c 8 =v 4 . Figure 4.11 re-illustrates this model in order to emphasize the squaring construction. c 2 c 1 c 3 c 4 c 5 c 6 c 7 c 8 c 4 + c 8 u 1 u 2 u 3 u 4 v 4 v 3 v 2 v 1 Figure 4.11: Generalized Tanner graph forC H implied by a squaring construction. 102 4.4.2 Conditionally Cycle-Free Generalized Tanner Graphs from the Squaring Construction The existence of degree-(m−2) conditionally cycle-free Tanner graphs for the length 2 m first-order Reed-Muller code is proven constructively in the following. It is well-known that an optimal SISO decoding algorithm for a codeC implies an optimal SISO decoding algorithm for its dualC ⊥ [59, 116]. The conditionally cycle-free GTGs for first-order Reed-Muller codes described below thus imply optimal SISO decoding algorithms for the extended Hamming codes. Theorem 4.1. There exists a degree-(m−2) conditionally cycle-free generalized Tanner graph forC RM(1,m) , m≥3. Proof. By induction on m. The m = 3 case was shown by construction in Section 4.4.1. Suppose that there exists a degree-(m−3) conditionally cycle-free graphical model for C RM(1,m−1) . There exists a squaring construction forC RM(1,m) such that if c c c∈C RM(1,m) then c c c=(u u u,u u u+v v v) (4.48) where u u u∈C RM(1,m−1) and v v v∈C RM(0,m−1) [111]. A GTG for this code is illustrated in Figure 4.12. The code vertex labeledC RM(1,m−1) corresponds to a GTG for that code. By the induction hypothesis, a degree-(m−2) conditionally cycle-free GTG is formed by conditioning on the repetition constraintC RM(0,m−1) and on m−3 variables in the model corresponding toC RM(1,m−1) . 103 c 2 m−1 +1 c 2 m−1 +2 c 2 m −1 c 2 m c 2 m−1 c 2 m−1 −1 c 1 c 2 u 1 u 2 u 2 m−1 −1 u 2 m−1 −2 v 2 m−1 −2 v 2 m−1 −1 v 2 v 1 C RM(1,m−1) C RM(0,m−1) Figure4.12: GeneralizedTannergraphforthe[2 m ,m+1,2 m−1 ]Reed-Mullercodeimplied by a squaring construction. 4.4.3 Decoding Complexity Let A m and C m denote the number of addition and comparison 4 operations required by the optimal SISO decoding algorithm implied by the conditionally cycle-free GTG for C RM(1,m) described above. Counting operations in the binary graphical model illustrated in Figure 4.12 yields the following recursion for A m and C m : A m =3·2 m +2A m−1 , C m =2 m +2C m−1 . (4.49) Note that these formulae include the comparisons required for marginalizing over the resultsoftheindividualdecodingrounds. ItisclearthatthecomplexityofSISOdecoding 4 Note that min-sum or min ? -sum processing is assumed so that Cm counts the number of min or min ? operations. 104 usingtheproposedconditionallyacyclicbinarygraphicalmodelsthusgrowsasO(nlogn). Ashikhim and Litsyn proposed a MAP decoding algorithm for the family ofC RM(1,m) codes with similar asymptotic complexity based on fast Hadamard transforms [8]. Note that the complexity of bit-level trellis decoding of this code family grows asO n 2 . In order to specifically demonstrate that conditionally cycle-free generalized Tanner graphs (CCF-GTGs) can imply substantially less complex optimal SISO decoding al- gorithms than bit-level trellises, the number of addition and comparison operations for each SISO decoding algorithm is tabulated in Table 4.2 for a number of first-order Reed- Muller codes. The trellis complexity was evaluated by considering the total number of addition and comparison operations required to perform the full SISO BCJR algorithm [10] (rather that soft-in hard-out Viterbi decoding) on bit-level trellises with minimal state complexities as given by Lu and Huang in [77]. Trellis CCF-GTG Add Compare Add Compare C RM(1,3) 176 50 72 32 C RM(1,4) 1008 426 192 80 C RM(1,5) 4720 2202 480 192 C RM(1,6) 20336 9850 1152 448 C RM(1,7) 84336 41530 2688 1024 C RM(1,8) 343408 170426 6144 2304 Table 4.2: Complexity of trellis decoding vs. the proposed optimal SISO decoding algo- rithm for first-order Reed-Muller codes. 105 4.5 Application: High-Rate Serially Concatenated Codes 4.5.1 Motivation Very high-rate codes are of great interest for a number of practical applications including datastorageandhigh-speedfiberlinks. Thedesignofmoderncodeswhichsimultaneously have very low error floors (e.g. < 10 −10 bit error rate) and very high rates (e.g. > 0.9) is a particularly challenging problem of practical interest. Due to the inherent difficulty of simulating the performance of codes in the very low floor region, the design of such codes relies on the principles of uniform interleaver analysis [14, 15, 16, 39]. To review, on the additive white Gaussian noise (AWGN) channel with binary antipodal signaling, the bit error rate (P b ) and codeword error rate (P cw ) of a large class of modern codes vary asymptotically with block size as P b ∼N αmax P cw ∼N αmax+1 (4.50) whereN istheinterleaversizeandα max isthemaximumexponentofN inanasymptotic union bound approximation. Note that the maximum exponent depends on both the specific code construction and constituent codes used. If the bit (codeword) error rate decays with N, then the code is said to exhibit interleaver gain in bit (codeword) error rate. Designs for codes with low floors require interleaver gain in both bit and codeword error rates and thus require α max ≤−2. (4.51) 106 Serially concatenated code constructions (i.e. codes composed of an inner code and outer code separated by a random-like interleaver) are well-suited to low-floor design because, provided the inner code is recursive, the maximum exponent of (4.50) is [14] α max =− d o +1 2 (4.52) where d o is the minimum distance of the outer code. Since the rate of a serially concate- nated code (SCC) is equal to the product of the rates of its constituent codes, the design of a high-rate, low-floor SCC requires a high-rate outer code satisfying d o ≥3. However, itisverychallengingtofindsuchoutercodesforwhichthereexistlow-complexityoptimal SISO decoding algorithms. To this end, Graell i Amat et al. introduced a class of high- rate convolutional codes with optimal SISO decoding algorithms of moderate complexity based on their respective dual codes [79, 80]. An alternative approach to the design of high-rate low-floor codes are the systematic with serially concatenated parity (S-SCP) codes proposed in [33], of which Jin et al.’s generalized repeat accumulate (GRA) codes [85] are an example. The S-SCP code struc- ture can be viewed as a systematic code with a parity generating concatenated system that resembles an SCC. It was demonstrated in [33] that S-SCP codes have the same maximum error exponent and design rules as SCCs: codes constructed with a parity generating system composed of a recursive inner parity generator and an outer code sat- isfying d o ≥ 3 achieve interleaver gain in both bit and codeword error rates. In contrast to SCCs, good S-SCPs can be constructed with inner parity generators that have rate greater than 1 so that the rate of outer code can be lower than the overall code rate thus 107 alleviating the aforementioned challenge of finding high-rate, d o ≥ 3, outer codes with low-complexity SISO decoding algorithms. The design of good high-rate, low-floor codes has thus been largely solved for the AWGN channel. However, the S-SCP design philosophy is not directly applicable to the large class of systems which have recursive channels. The term recursive channel is introduced to describe systems in which the aggregate of the modulation and (possible) precoding with the channel is recursive. Two examples of recursive channels are: 1. Continuous phase modulations over AWGN and fading channels. 2. Certain models for precoded magnetic recoding channels (e.g. EPR4 [103]). In light of the above discussion, Figure 4.13 illustrates two potential structures for high- rate, low-floor codes for use in systems with recursive channels. The first structure (Fig. Recursive Channel 1 Recursive Channel 1 Modern π (1) Code 1 Modern π (1) Code 1 Modern π (1) Code 1 (a) Recursive Channel 1 Recursive Channel 1 Modern π (1) Code 1 Modern π (1) Code 1 Algebraic π (1) Conv. / 1 Algebraic π (1) Conv. / 1 (b) Figure 4.13: Potential structures for high-rate, low-floor codes for use in systems with recursive channels. Random-like interleavers are labeled π. 4.13(a)) consists of a high-rate modern code (e.g. an SCC or S-SCP) in serial concate- nation with the recursive channel while the second structure (Fig. 4.13(b)) replaces the modern outer code with a classical code (e.g. convolutional or algebraic block code). While both structures have the potential to offer the desired bit and codeword error rate performance (i.e. low floors), the second structure is more attractive for application in practical systems. Specifically, the use of a classical outer code for which there exists a 108 simple non-iterative SISO decoding algorithm offers reductions in decoding complexity, decoding latency and required memory with respect to a modern, iteratively decoded, outer code. As with the design of SCCs for the AWGN channel, the design of such codes requires a high-rate, d≥ 3 outer classical code for which there exists a low-complexity optimal SISO decoding algorithm. The results of Section 4.4.3 indicate that the family of extended Hamming codes provide a set of such codes. In the remainder of this section, it is shown that extended Hamming codes offer an attractive alternative to the high-rate convolutional codes studied in [79, 80] for use as outer codes in serial concatenation with recursive inner channels. 4.5.2 Simulation Results and Discussion In [80], Graell i Amat et al. studied a serially concatenated coding scheme consisting of a high-rate convolutional outer code and a recursive rate-1 inner code corresponding to a simplified discrete-time model of a precoded EPR4 magnetic recording channel. Specifically, the recursive channel model comprises a 1/1⊕D 2 precoder followed by a digital recording channel subject to intersymbol interference (ISI) with partial response polynomial1+D−D 2 −D 3 followedfinallybyanAWGNchannel. Notethattheprecoder and ISI can be jointly decoded on an 8-state trellis [30]. In this section, the performance of this scheme is compared to one which replaces the high-rateconvolutionalcodeswithextendedHammingcodes. Specifically,4convolutional outer codes with input block size 4000 bits and respective rates 8/9, 9/10, 11/12 and 16/17 are compared to four algebraic outer codes composed of the following mixtures of extended Hamming codes: 109 • The mixture of 3C RM(3,5) and 69C RM(4,6) codewords resulting in a code with input block size 4011 bits and rate 4011/4512=0.8890≈8/9. • The mixture of 56C RM(4,6) and 7C RM(5,7) codewords resulting in a code with input block size 4032 bits and rate 4032/4480=9/10. • The mixture of 30C RM(4,6) and 19C RM(5,7) codewords resulting in a code with input block size 3990 bits and rate 3990/4352=0.9168≈11/12. • The mixture of 2C RM(4,6) , 26C RM(5,7) and 3C RM(6,8) codewords resulting in a code with input block size 3975 bits and rate 3975/4224=0.9411≈16/17. As reported in [80], the serially concatenated codes using convolutional codes utilize s- random interleavers [38]. The codes using mixed extended Hamming outer codes utilize highsum-spreadpseudo-randominterleaversthatweregeneratedusingthereal-relaxation optimization method described in [35]. Figure4.14comparestheperformanceoftherespectiverate8/9and11/12codeswhile Figure4.15comparestheperformanceoftherespectiverate10/11and16/17codes. Note that the performance of the serially concatenated codes with convolutional outer codes is reported for 10 decoding iterations while the performance of the codes with mixed extended Hamming outer codes is reported for 10 and 20 decoding iterations. All curves correspond to min ? -sum processing (or its dual-domain equivalent [116]). Observe in Figures 4.14 and 4.15 that the codes constructed with mixed extended Hamming outer codes consistently outperform those with convolutional outer codes of similar rates and input block sizes. The mixed extended Hamming codes compare favor- ablytotheirhigh-rateconvolutionalcodecounterpartsintermsofcomplexityalso. Table 110 10 -6 10 -5 10 -4 10 -3 10 -2 4.4 4.6 4.8 5 5.2 5.4 5.6 11/12 CC (10) 11/12 RM (10) 11/12 RM (20) 8/9 CC (10) 8/9 RM (10) 8/9 RM (20) Bit Error Rate Eb/No (dB) Figure 4.14: Bit error rate performance of the respective rate 8/9 and 11/12 serially concatenated codes. 10 -7 10 -6 10 -5 10 -4 10 -3 10 -2 4.5 5 5.5 6 16/17 CC (10) 16/17 RM (10) 16/17 RM (20) 9/10 CC (10) 9/10 RM (10) 9/10 RM (20) Bit Error Rate Eb/No (dB) Figure 4.15: Bit error rate performance of the respective rate 9/10 and 16/17 serially concatenated codes. 111 Rate Add / Bit Compare / Bit 8/9 20.2 7.9 9/10 20.7 8.0 11/12 21.5 8.2 16/17 22.8 8.7 Table4.3: Averagenumberofoperationsperinputbit,perdecodingiteration,foroptimal SISO decoding of the mixed extended Hamming codes. 4.3 tabulates the average number of add and comparison operations per input bit, per decodingiteration, requiredforoptimalSISOdecoding(usingtheconditionallycycle-free generalized Tanner graphs decoding algorithm described in Section 4.4.2) for each of the mixed extended Hamming codes. The rate k/k+1 convolutional outer-codes in [80] were decoded on the 16- (for k = 8,9,10) and 32- (for k = 16) state trellises corresponding to their respective rate 1/1+k duals. Optimal SISO decoding on a rate 1/1+k, 16- (32-) state trellis requires at least 96 (192) additions and 64 (128) comparisons per input bit, per decoding iteration. Thus, if one assumes that an addition and comparison operation have roughly the same complexity, 5 then the complexity of the proposed mixed extended Hamming code SISO decoding algorithms are approximately 5 to 10 times less than that of the respective high-rate convolutional code decoding algorithms proposed in [80]. 5 This assumption is clearly reasonable for min-sum processing. For min ? -star processing, this assump- tion is also reasonable provided a table-lookup is used for the correction term in the min ? operator. 112 4.6 Appendix: K-Shortest Paths Algorithm In this section, theK-shortest paths algorithm used in the suboptimal glue SISO decoder of Section 4.3 is defined. This algorithm is an adaptation of that used in [128] which in turn is due to [42]. The algorithm receives as input r sorted lists of syndrome metrics and finds syndrome r-tuples with shortest total metric. Denote the length of each list S. Syndrome r-tuples are of the form s (1) ,...,s (i) ,...,s (r) where i is the path index. Givenaspecificpathandpathindexi,thepaththatresultswhens s s (i) isreplacedwiththe syndromewiththenextsmallestmetricisdenotedtheincrementedpath andthedifference in the total metrics of the path and the incremented path is denoted the incremental metric. The algorithm operates on a tree where each node n (in addition to the child-parent relationships) has the following memebers: n.path, the syndrome r-tuple associated with this node; n.metric, the sum 6 of the syndrome metrics associated with n.path; and, n.state ∈ {ACTIVE,INACTIVE,POTENTIAL}. The algorithm also maintains a list of nodes in the POTENTIAL state: POTENTIAL LIST. The algorithm is initialized by cre- ating a root node Λ with a path corresponding to the smallest metric in each list and Λ.state← POTENTIAL. Algorithm 4.16 describes the procedure for finding the next shortest path. 6 If the sum-product or max-product semi-ring is to be used in the SISO decoder then sum is replaced by product. 113 Input: POTENTIAL LIST - list of nodes in the POTENTIAL state. Output: Path with next shortest metric. // Choose the best potential node. if POTENTIAL LIST=∅ then // There are no more potential paths. return else p← node p 0 ∈ POTENTIAL LIST with smallest p 0 .metric; end // Extend p and its parent nodes. foreach n∈{p,p.parent} do if n.state= ACTIVE then Let i be the path index of n.path with smallest incremental metric such that there exists no node corresponding to the incremented path; if No such i exists then // n can have no more children. n.state← INACTIVE; else // Append a child to n. Create a child node n 0 of n with members: n 0 .metric←n.metric+incremental metric; n 0 .path←incremented path; n 0 .state← POTENTIAL; Append n 0 to POTENTIAL LIST; end end end Remove p from POTENTIAL LIST; // Return the shortest path. return p.path Figure 4.16: Next shortest path algorithm. 114 Chapter 5 The Extraction of Graphical Models for Linear Codes II: Heuristics for Cyclic Model Extraction This chapter is the second of two focusing on the extraction of graphical models for linear codes. This chapter focuses on cyclic graphical models with the aim of extracting modelswhichimplytractablesuboptimalsoft-insoft-out(SISO)decodingalgorithmsthat outperform simple Tanner graph decoding. This chapter introduces a novel approach to cyclic model extraction: graphical model transformation. Whereasminimalconventionalandtail-bitingtrellismodelscanbecharacterizedalge- braically via trellis-oriented generator matrices [145], there is in general no known analog of such algebraic characterizations for arbitrary cycle-free graphical models [51], let alone cyclic models. In the absence of such an algebraic characterization, it is initially un- clear as to how cyclic graphical models can be extracted. In Section 5.1, a set of basic transformation operations on graphical models for codes is introduced and it is shown in Section 5.2 that any graphical model for a given code can be transformed into any other graphical model for that same code via the application of a finite number of these basic transformations. The transformations studied in Section 5.1 thus provide a mechanism 115 for searching the space of all all graphical models for a given code. In Section 5.3, the basictransformationsintroducedinSection5.1areusedtoextractnovelgraphicalmodels for linear block codes. Starting with an initial Tanner graph for a given code, heuristics for extracting other Tanner graphs, generalized Tanner graphs, and more complex cyclic graphical models are investigated. Section 5.4 investigates the performance of the decod- ing algorithms implied by the models extracted using these heuristics for two extended Bose-Chaudhuri-Hocquenghem (BCH) codes. Finally, the complexity of graphical model extraction is considered briefly in Section 5.5. 5.1 Graphical Model Transformation LetG C be a graphical model for the linear codeC over F q . This dissertation introduces eight basic graphical model operations, the application of which toG C results in a new graphical model forC: • The merging of two local constraintsC i 1 andC i 2 into the new local constraintC i which satisfies C i =C i 1 ∩C i 2 . (5.1) • The splitting of a local constraintC j into two new local constraintsC j 1 andC j 2 which satisfy C j 1 ∩C j 2 =C j . (5.2) • The insertion/removal of a degree-2 repetition constraint. • The insertion/removal of a trival length 0, dimension 0 local constraint. 116 • The insertion/removal of an isolated partial parity-check constraint. Note that some of these operations have been introduced implicitly in this dissertation and in the literature already. For example, the proof of the local constraint involvement property of q m -ary graphical models presented in Chapter 3 utilizes degree-2 repetition constraint insertion. Local constraint merging has been considered by a number of au- thors under the rubric of clustering (e.g. [50, 97]). This dissertation introduces the term merging specifically so that it can be contrasted with its inverse operation: splitting. In Section 5.2, it is shown that these basic operations span the entire space of graphical models forC thus demonstrating the efficacy of this choice of operations. Detailed defini- tionsofeachoftheeightbasicgraphicalmodeloperationsaregiveninSections5.1.1-5.1.5 where it is assumed that a q m -ary graphical modelG C with behaviorB for a linear code C overF q on an index set I is given. 5.1.1 Local Constraint Merging SupposethattwolocalconstraintsC i 1 andC i 2 aretobemerged. Withoutlossofgenerality, assume that there is no hidden variable incident on bothC i 1 andC i 2 (since if there is, a degree-2 repetition constraint can be inserted). The hidden variables incident onC i 1 may be partitioned into two sets I S (i 1 )=I (c) S (i 1 )∪I (nc) S (i 1 ) (5.3) whereeachS j ,j∈I (c) S (i 1 ),isalsoincidentonaconstraintC j whichisadjacenttoC i 2 . The hidden variables incident onC i 2 may be similarly partitioned. The set of local constraints 117 incident on hidden variables in both I (c) S (i 1 ) and I (c) S (i 2 ) are denoted common constraints and indexed by I (c) C (i 1 ,i 2 ). Figure 5.1 illustrates this notation. C i1 C i2 C c2 C c1 I (c) S (i 1 ) I (nc) S (i 1 ) I (nc) S (i 2 ) I (c) S (i 2 ) Figure 5.1: Local constraint merging notation. The local constraintsC c 1 andC c 2 are common. The merging of local constraintsC i 1 andC i 2 proceeds as follows. For each common local constraintC j , j∈ I (c) C (i 1 ,i 2 ), denote by S j 1 (S j 2 ) the hidden variable incident on C j andC i 1 (C i 2 ). Denote byC j|{j 1 ,j 2 } the projection ofC j onto the two variable index set {j 1 ,j 2 } and define a new q k(C j|{j 1 ,j 2 } ) -ary hidden variable S j 1 ,j 2 which encapsulates the possible simultaneous values of S j 1 and S j 2 (as constrained byC j|{j 1 ,j 2 } ). After defining such hidden variables for eachC j , j∈ I (c) C (i 1 ,i 2 ), a set of new hidden variables results which is indexed by I (c) S (i 1 ,i 2 ). The local constraintsC i 1 andC i 2 are then merged by replacingC i 1 andC i 2 by a code defined over I V (i 1 )∪I V (i 2 )∪I (nc) S (i 1 )∪I (nc) S (i 2 )∪I (c) S (i 1 ,i 2 ) (5.4) which is equivalent toC i 1 ∩C i 2 and redefining each local constraintC j , j∈I C (i 1 ,i 2 ), over the appropriate hidden variables in I (c) S (i 1 ,i 2 ). 118 As an example, consider again the cycle-free 4-ary graphical model for the extended Hamming codeC H derived in the appendix to Chapter 3, which is re-illustrated in Figure 5.2. The hidden variables S 1 through S 8 are binary while S 9 through S 12 are 4-ary. The C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 S 8 V 8 V 7 V 6 V 5 V 1 V 4 V 2 V 3 S 3 S 2 S 4 S 1 S 5 S 6 S 7 C 9 C 10 C 11 C 12 C 13 S 11 S 12 S 9 S 10 Figure 5.2: Cycle-free 4-ary graphical model forC H . local constraintsC 1 throughC 8 simply enforce equality whileC 9 throughC 13 are defined by the following generator matrices: S 1 S 4 S 9 S 2 S 3 S 10 G 9 = 1 0 11 0 1 10 , G 10 = 1 0 11 0 1 10 , (5.5) 119 S 8 S 7 S 11 S 6 S 5 S 12 G 11 = 1 0 11 0 1 10 , G 12 = 1 0 10 0 1 11 , (5.6) S 9 S 10 S 11 S 12 G 13 =    10 10 10 10 01 00 00 01 00 01 00 01 00 00 01 01   . (5.7) Suppose thatC 9 andC 10 are to be merged as illustrated in Figure 5.3. The hidden S 10 S 9 S 11 S 12 S 1 S 4 S 2 S 3 S 2 S 3 S 11 S 12 S 1 S 4 C 9 C 10 C 13 C 13 C 14 S 14 Figure 5.3: The merging of constraintsC 9 andC 10 in a 4-ary graphical model forC H . The resulting graphical model is 8-ary. variables incident onC 9 are partitioned into the sets I (c) S (9) ={9} and I (nc) S (9) ={1,4}. 120 Similarly, I (c) S (10) ={10} and I (nc) S (10) ={2,3}. The sole common constraint is thus C 13 . The projection ofC 13 onto S 9 and S 10 has dimension 3 and the new 8-ary hidden variable S 14 is defined by the generator matrix S 14 S 9 S 10 G 9,10 =   100 10 10 010 01 00 001 00 01   . (5.8) The local constraintsC 9 andC 10 when defined over S 14 rather than S 9 and S 10 , respec- tively, are generated by S 1 S 4 S 14 S 2 S 3 S 14 G 0 9 =   1 0 110 1 0 111 0 1 100   , G 0 10 =   1 0 101 1 0 111 0 1 100   . (5.9) Finally,C 13 is redefined over S 14 and generated by S 14 S 11 S 12 G 13 =    100 10 10 010 00 01 001 00 01 000 01 01    (5.10) whileC 9 andC 10 are replaced byC 14 which is equivalent toC 9 ∩C 10 and is generated by S 14 S 1 S 4 S 2 S 3 G 14 =   100 0 1 0 1 010 1 1 0 0 001 0 0 1 1   . (5.11) 121 Note that the graphical model which results from the merging ofC 9 andC 10 is 8-ary. Specifically, S 14 is an 8-ary hidden variable while n(C 13 )−k(C 13 )=3 and k(C 14 )=3. 5.1.2 Local Constraint Splitting Local constraint splitting is simply the inverse operation of local constraint merging. Consider a local constraintC j defined on the visible and hidden variables indexed by I V (j) and I S (j), respectively. Suppose thatC j is to be split into two local constraintsC j 1 andC j 2 defined on the index sets I V (j 1 )∪I S (j 1 ) and I V (j 2 )∪I S (j 2 ), respectively, such thatI V (j 1 ) andI V (j 2 ) partitionI V (j) whileI S (j 1 )∪I S (j 2 )=I S (j) butI S (j 1 ) andI S (j 2 ) need not be disjoint. Denote by I S (j 1 ,j 2 ) the intersection of I S (j 1 ) and I S (j 2 ). Local constraint splitting proceeds as follows. For each S i , i∈I S (j 1 ,j 2 ), make a copy S i 0 of S i and redefine the local constraint incident on S i (which is notC j ) over both S i and S i 0. Denote by I 0 S (j 1 ,j 2 ) an index set for the copied hidden variables. The local constraint C j is then replaced byC j 1 andC j 2 such thatC j 1 is defined over I V (j 1 )∪I S (j 1 ) andC j 2 is defined over I V (j 2 )∪I S (j 2 )\I S (j 1 ,j 2 )∪I 0 S (j 1 ,j 2 ). (5.12) Following this split procedure, some of the hidden variables in I S (j 1 ,j 2 ) and I 0 S (j 1 ,j 2 ) may have larger alphabets than necessary. Specifically, if the dimension of the projection ofC j 1 (C j 2 ) onto a variable S i , i∈I S (j 1 ,j 2 ) (i∈I 0 S (j 1 ,j 2 )), is smaller than the alphabet index set size ofS i , thenS i can be redefined with an alphabet index set size equal to that dimension. 122 The merged code in the example of the previous sectionC 14 can be split into two codes:C 9 defined on S 1 , S 4 , and S 14 , andC 10 defined on S 2 , S 3 , and S 14 0. The projection of S 14 ontoC 9 has dimension 2 and S 14 can thus be replaced by the 4-ary hidden variable S 9 . Similarly, the projection of S 14 0 ontoC 10 has dimension 2 and S 14 0 can be replaced by the 4-ary hidden variable S 10 . 5.1.3 Insertion/Removal of Degree-2 Repetition Constraints Suppose that S i is a hidden variable involved in the local constraintsC i 1 andC i 2 . A degree-2 repetition constraint is inserted by defining a new hidden variable S j as a copy of S i , redefiningC i 2 over S j and defining the repetition constraintC j which enforces S i = S j . Degree-2 repetition constraint insertion can be similarly defined for visible variables. Conversely, suppose thatC j is a degree-2 repetition constraint incident on the hidden variables S i and S j . SinceC j simply enforces S i = S j , it can be removed and S j relabeled S i . Degree-2 repetition constraint removal can be similarly defined for visible variables. The insertion and removal of degree-2 repetition constraints is illustrated in Figures 5.4(a) and 5.4(b) for hidden and visible variables, respectively. C i1 C i2 C i1 C i2 C j S i S i S j (a) V i V i C i C i C j S j (b) Figure 5.4: Insertion and removal of degree-2 repetition constraints. 123 5.1.4 Insertion/Removal of Trivial Constraints Trivialconstraintsarethoseincidentonnohiddenorvisiblevariablessothattheirrespec- tive block lengths and dimensions are zero. Trivial constraints can obviously be inserted or removed from graphical models. 5.1.5 Insertion/Removal of Isolated Partial Parity-Check Constraints Suppose thatC i 1 ,...,C i j are j q-ary repetition constraints (that is each repetition con- straint enforces equality on q-ary variables) and let β i 1 ,...,β i j ∈ F q be non-zero. The insertion of an isolated partial parity-check constraint is defined as follows. Define j +1 new q-ary hidden variables S i 1 ,...,S i j and S k , and two new local constraintsC p andC k such thatC p enforces the q-ary single parity-check equation j X l=1 β i l S i l =S k (5.13) andC k is a degree-1 constraint incident only on S k with dimension 1. The new local constraintC p defines the partial parity variable S k and is denoted isolated since it is incident on a hidden variable whichis involved in a degree-1, dimension 1 local constraint (i.e.C k does not constrain the value of S k ). SinceC p is isolated, the graphical model that resultsfromitsinsertionisindeedavalidmodelforC. Similarly, anysuchisolatedpartial parity-check constraint can be removed from a graphical model resulting in a valid model forC. 124 As an example, Figure 5.5 illustrates the insertion and removal of an isolated partial- paritycheckonthebinarysumofV 7 andV 8 inaTannergraphfortheextendedHamming codeC H . V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 V 1 V 2 V 3 V 4 V 5 V 6 V 7 V 8 Figure5.5: Theinsertion/removalofanisolatedpartialparity-checkconstraintonV 7 and V 8 in a Tanner graph forC H . 125 5.2 Extraction via Transformation As discussed in Chapter 1, a primarily aim of this dissertation is the formalization of graphical model extraction. In order to describe extraction is the language of optimiza- tion, the following issues must be addressed: • The space of all graphical models for a given code must be defined. • A mechanism for searching this space must be found. • Provided this space can be searched, meaningful constraint and cost functions for such a search must be defined. • Given that combinatorial optimization is intractable in general, efficient search heuristics must be developed. ThespaceofallgraphicalmodelsforagivencodewasdefinedinChapter2. Itisshownin the following that the eight basic graphical model operations introduced in the previous section span the space of graphical models for a given code. Theorem5.1. LetG C and e G C be two graphical models for the linear codeC overF q . Then G C can be transformed into e G C via the application of a finite number of basic graphical model operations. Proof. Define the following four sub-transformations which can be used to transformG C into a Tanner graphG T C : 1. The transformation ofG C into a q-ary modelG q C . 2. The transformation ofG q C into a (possibly) redundant generalized Tanner graphG r C . 126 3. The transformation ofG r C into a non-redundant generalized Tanner graphG g C . 4. The transformation ofG g C into a Tanner graphG T C . Since each basic graphical model operation has an inverse,G T C can be transformed into G C by inverting each of the four sub-transformations. In order to prove thatG C can be transformed into e G C via the application of a finite number of basic graphical model operations, it suffices to show that each of the four sub-transformations requires a finite number of operations and that the transformation of the Tanner graphG T C into a Tanner graph e G T C correspondingto e G C requiresafinitenumberofoperations. Thisproofsummary is illustrated in Figure 5.6. G C G q C G r C G g C G T C ! G T C ! G C Figure 5.6: The transformation ofG C into e G C via five sub-transformations. That each of the five sub-transformations fromG C to e G T C illustrated in Figure 5.6 requires only a finite number of basic graphical model operations is proved below. 1) G C →G q C : The graphical modelG C is transformed into the q-ary modelG q C as follows. Each local constraintC i inG C is split into the n(C i )−k(C i ) q-ary single parity- check constraints which define it. A degree-2 repetition constraint is then inserted into every hidden variable with alphabet index set sizem>1 and these repetition constraints are then each split into m q-ary repetition constraints as illustrated in Figure 5.7. Each local constraintC j in the resulting graphical model satisfies n(C j )−k(C j )=1. Similarly, each hidden variable S j in the resulting graphical model satisfies|T j |=1. 127 m m m m S j S j S j ! Figure 5.7: Transformation of the q m -ary hidden variable S j into q-ary hidden variables. 2)G q C →G r C : A(possiblyredundant)generalizedTannergraphissimplyabipartiteq- arygraphicalmodelwithonevertexclasscorrespondingtorepetitionconstraintsandone tosingleparity-checkconstraintsinwhichvisiblevariablesareincidentonlyonrepetition constraints. By appropriately inserting degree-2 repetition constraints, the q-ary model G q C can be transformed intoG r C . 3)G r C →G g C : Let the generalized Tanner graphG r C correspond to an r H ×n(C)+g redundant parity-check matrix H (r,g) C for a degree-g generalized extension ofC with rank rank(H (r,g) C )=n(C)−k(C)+g. (5.14) AfinitenumberofrowoperationscanbeappliedtoH (r,g) C resultinginanewparity-check matrix the last r H −rank(H (r,g) C ) rows of which are all zero. Similarly, a finite number of basic operations can be applied toG r C resulting in a generalized Tanner graph containing r H −rank(H (r,g) C ) trivial constraints which can then be removed to yieldG g C . Specifically, consider the row operation on H (r,g) C which replaces a row h h h i by e h h h i =h h h i +β j h h h j (5.15) 128 where β j ∈F q . The graphical model transformation corresponding to this row operation first merges the q-ary single parity-check constraintsC i andC j (which correspond to rows h h h i and h h h j , respectively) and then splits the resulting check into the constraints e C i and C j (which correspond to rows e h h h i and h h h j , respectively). Note that this procedure is valid since C i ∩C j = e C i ∩C j . (5.16) 4)G g C →G T C : Let the degree-g generalized Tanner graphG g C correspond to an n(C)− k(C) + g× n(C) + g parity-check matrix H (g) C . A degree-(g− 1) generalized Tanner graphG g−1 C is obtained fromG g C as follows. Denote by b H (g) C the parity-check matrix for the degree-g generalized extension defined by H (g) C which is systematic in the position corresponding to the g-th partial parity symbol. Since a finite number of row operations can be applied to H (g) C to yield b H (g) C , a finite number of local constraint merge and split operations can be be applied toG g C to yield the corresponding generalized Tanner graph b G g C . Removing the now isolated partial-parity check constraint corresponding to the g-th partial parity symbol in b G g C yields the desired degree-(g−1) generalized Tanner graph G g−1 C . By repeatedly applying this procedure, all partial parity symbols can be removed fromG g C resulting inG T C . 5) G T C → e G T C : Let the Tanner graphsG T C and e G T C correspond to the parity-check matrices H C and e H C , respectively. Since H C can be transformed into e H C via a finite number of row operations,G T C can be similarly transformed into e G T C via the application of a finite number of local constraint merge and split operations. 129 Theorem 5.1 provides a mechanism for searching the space of all graphical models for agivencodeanditthusremainstoinvestigateoptimizationcostandconstraintfunctions, as well as extraction heuristics. Constraint functions for graphical model extraction are in general readily defined. For example, one such function is an indicator on inclusion in the sub-class of q m -ary models for some specific value of m. It is often difficult, how- ever, to define a cost measure on the space of graphical models which is simultaneously meaningful in some real sense (e.g. highly correlated with decoding performance) and computationally tractable. The cost measures considered in this chapter are functions of the short cycle structure of graphical models. The use of such cost measures is motivated first by empirical evidence concerning the detrimental effect of short cycles on decoding performance (cf. [7, 50, 57, 66, 112, 140, 143, 151]) and second by the existence of an effi- cientalgorithmforcountingshortcyclesinbipartitegraphs[66](whichisdescribedinthe appendix). The following sections investigate heuristics for graphical model extraction. 5.3 Greedy Heuristics for Cyclic Model Extraction 5.3.1 A Greedy Heuristic for Tanner Graph Extraction Following the results of Chapter 3, the Tanner graphs corresponding to many linear block codes of practical interest necessarily contain many short cycles. Suppose that any Tanner graph for a given codeC must have girth at least g min (C); an interesting problem is the extraction of a Tanner graph forC containing the smallest number of g min (C)-cycles. The extraction of such Tanner graphs is especially useful in the context of ad-hoc decoding algorithms which utilize Tanner graphs such as Jiang and Narayanan’s 130 stochasticshiftingbasediterativedecodingalgorithmforcycliccodes[83]andtherandom redundant iterative decoding algorithm that will be described in Chapter 6. Algorithm 5.8 performs a greedy search for a Tanner graph forC with girth g min (C) and the smallest number of g min (C)-cycles starting with an initial Tanner graph TG(H C ) which corresponds to some binary parity-check matrixH C . Define an (i,j)-row operation as the replacement of row h h h j in H C by the binary sum of rows h h h i and h h h j . As detailed in the proof of Theorem 5.1, ifC i andC j are the single parity-check constraints in TG(H C ) corresponding toh h h i andh h h j , respectively, then an (i,j)-row operation in H C is equivalent to mergingC i andC j to form a new constraintC i,j =C i ∩C j and then splittingC i,j intoC i and e C j (where e C j enforces the binary sum of rows h h h i and h h h j ). Algorithm 5.8 iteratively finds the rowsh h h i andh h h j in H C with corresponding (i,j)-row operation that results in the largest short cycle reduction in TG(H C ) at every step. This greedy search continues until there are no more row operations that improve the short cycle structure of TG(H C ). 131 Input: r H ×n(C) binary parity-check matrix H C . Output: r H ×n(C) binary parity-check matrix H 0 C . H 0 C ←H C ; i ? ←0; j ? ←0; g ? ← girth of TG(H 0 C ); N ? g ?← number of g ? -cycles in TG(H 0 C ); N ? g ? +2 ← number of g ? +2-cycles in TG(H 0 C ); repeat if i ? 6=j ? then Replace rowh h h j ? in H 0 C with binary sum of rows h h h i ? andh h h j ?; i ? ←0; j ? ←0; for i,j =1,...,r H , i6=j do Replace rowh h h j in H 0 C with binary sum of rows h h h i andh h h j ; g← girth of TG(H 0 C ); N g ← number of g-cycles in TG(H 0 C ); N g+2 ← number of g+2-cycles in TG(H 0 C ); if g >g ? then g ? ←g; i ? ←i; j ? ←j; N ? g ←N g ; N ? g+2 ←N g+2 ; end else if g =g ? AND N g <N ? g then i ? ←i; j ? ←j; N ? g ←N g ; N ? g+2 ←N g+2 ; else if g =g ? AND N g =N ? g then if N g+2 <N ? g+2 then i ? ←i; j ? ←j; N ? g+2 ←N g+2 ; end Undo row replacement; end until i ? =0 & j ? =0; return H 0 C Figure 5.8: Greedy heuristic for the reduction of short cycles in Tanner graphs for binary codes. 132 5.3.2 A Greedy Heuristic for Generalized Tanner Graph Extraction Anumberofauthors have studiedthe extractionofgeneralizedTannergraphs(GTGs)of codes for which g min (C)=4 with a particular focus on models which are 4-cycle-free and which correspond to generalized code extensions of minimal degree [98, 132]. Minimal degree extensions are sought because no informationis available to the decoder aboutthe partial parity symbols in a generalized Tanner graph and the introduction of too many suchsymbolshasbeenobservedempiricallytoadverselyaffectdecodingperformance[98]. Generalized Tanner graph extraction algorithms proceed via the insertion of partial parity symbols, an operation which is most readily described as a parity-check matrix manipulation. 1 Following the notation introduced in Chapter 2, suppose that a partial parity on the coordinates indexed by J⊆I∪{p 1 ,p 2 ,...,p g } (5.17) is to be introduced to a GTG forC corresponding to a degree-g generalized extension e C with parity-check matrix H e C . A rowh h h p is first appended to H e C with a 1 in the positions corresponding to coordinates indexed by J and a 0 in the other positions. A column is then appended to H e C with a 1 only in the position corresponding to h h h p . The resulting parity-check matrix H b C describes a degree-g + 1 generalized extension b C. Every row h h h i 6= h h h p in H b C which contains a 1 in all of the positions corresponding to coordinates indexed by J is then replaced by the binary sum of h h h i and h h h p . Suppose that there are 1 Note that partial parity insertion can also be viewed through the lens of graphical model transforma- tion. The insertion of partial parity symbol proceeds via the insertion of an isolated partial parity check followed by a series of local constraint merge and split operations. 133 r(J) such rows. It is readily verified that the tree-inducing cut size b X T of the GTG that results from this insertion is related to that of the initial GTG, e X T , by ΔX T = e X T − b X T =(|J|−1)(r(J)−1). (5.18) Algorithm 5.11 performs a greedy search for a 4-cycle-free generalized Tanner graph forC with the smallest number of inserted partial parity symbols starting with an initial Tanner graph TG(H C ) which corresponds to some binary parity-check matrix H C . Algo- rithm 5.11 iteratively finds the symbol subsets that result in the largest tree-inducing cut sizereductionandthenintroducesthepartialparitysymbolcorrespondingtooneofthose subsets. At each step, Algorithm 5.11 uses Algorithm 5.10 to generate a candidate list of partial parity symbols to insert and chooses from that list the symbol which reduces the most short cycles when inserted. This greedy procedure continues until the generalized Tanner graph contains no 4-cycles. Algorithm5.11iscloselyrelatedtotheGTGextractionheuristicsproposedbySankara- narayanan and Vasi´ c in [132] and Kumar and Milenkovic in [98] (henceforth referred to as the SV and KM heuristics, respectively). It is readily shown that Algorithm 5.11 is guaranteed to terminate using the proof technique of [132]. The SV heuristic considers only the insertion of partial parity symbols corresponding to coordinate index sets of size 2 (i.e.|J| = 2). The KM heuristic considers only the insertion of partial parity symbols corresponding to coordinate index sets satisfying r(J) = 2. Algorithm 5.10, however, considers all coordinate index sets satisfying|J|=2,3,4 and r(J)=2,3,4 and then uses (5.18) to evaluate which of these coordinate sets results in the largest tree-inducing cut 134 size reduction. Algorithm 5.11 is thus able to extract GTGs corresponding to general- ized extensions of smaller degree than the SV and KM heuristics. In order to illustrate this observation, the degrees of the generalized code extensions that result when the SV, KM, and proposed (HC) heuristics are applied to parity-check matrices for three codes are provided in Table 5.1. Figure 5.9 compares the performance of the three extracted GTG decoding algorithms for the [31,21,5] BCH code in order to illustrate the efficacy of extracting GTGs corresponding to extensions of smallest possible degree. Code SV KM HC [23,12,7] Golay 18 11 10 [31,21,5] BCH 47 19 12 [63,30,13] BCH 264 121 69 Table 5.1: Generalized code extension degrees corresponding to the 4-cycle-free GTGs obtained via the SV, KM, and HC heuristics. 10 -5 10 -4 10 -3 10 -2 3 3.5 4 4.5 5 5.5 6 6.5 7 SV KM HC Bit Error Rate Eb/No (dB) Figure5.9: BiterrorrateperformanceofthreeGTGdecodingalgorithmsforthe[31,21,5] BCH code. One-hundred iterations of a flooding schedule were performed. Binary an- tipodal signaling over an additive white Gaussian noise (AWGN) channel is assumed. 135 5.3.3 A Greedy Heuristic for 2 m -ary Model Extraction For most codes, the decoding algorithms implied by generalized Tanner graphs exhibit onlymodestgainswithrespecttothoseimpliedbyTannergraphs, ifany,thusmotivating the search for more complex graphical models. Algorithm 5.12 iteratively applies the constraint merging operation in order to obtain a 2 m ? -ary graphical model from an initial Tanner graph TG(H C ) for some prescribed maximum complexity m ? . At each step, Algorithm 5.12 determines the pair of local constraintsC i andC j which when merged reduces the most short cycles without violating the maximum complexity constraint m ? . In order to ensure that that the efficient cycle counting algorithm of [66] can be utilized, only pairs of checks which are both internal or both interface are merged at each step. Since the initial Tanner graph is bipartite with vertex classes corresponding to interface (repetition)andinternal(singleparity-check)constraints,thegraphicalmodelsthatresult from every such local constraint merge operations are similarly bipartite. 136 Input: Binary generalized parity-check matrix H e C . Output: ListS of best partial parity symbols sets. S←∅; ΔX ? T ←0; // Consider pairs of columns of H e C . J 2 ← coordinate pair that maximizes r(J 2 ); Append J 2 toS; ΔX ? T ←r(J 2 )−1; // Consider 3-tuples of columns of H e C . J 3 ← coordinate 3-tuple that maximizes r(J 3 ); if 2(r(J 3 )−1)>ΔX ? T then S←∅; Append J 3 toS; ΔX ? T ←2(r(J 3 )−1); end else if 2(r(J 3 )−1)=ΔX ? T then Append J 3 toS; // Consider 4-tuples of columns of H e C . J 4 ← coordinate 4-tuple that maximizes r(J 4 ); if 3(r(J 4 )−1)>ΔX ? T then S←∅; Append J 4 toS; ΔX ? T ←3(r(J 4 )−1); end else if 3(r(J 4 )−1)=ΔX ? T then Append J 4 toS; // Consider pairs of rows of H e C . J i ← largest coordinate subset such that r(J i )=2; if|J i |−1>ΔX ? T then S←∅; Append J i toS; ΔX ? T ←|J i |−1; end else if|J i |−1=ΔX ? T then Append J i toS; // Consider 3-tuples of rows of H e C . J j ← largest coordinate subset such that r(J j )=3; if 2(|J j |−1)>ΔX ? T then S←∅; Append J j toS; ΔX ? T ←2(|J j |−1); end else if 2(|J j |−1)=ΔX ? T then Append J j toS; // Consider 4-tuples of rows of H e C . J k ← largest coordinate subset such that r(J k )=4; if 3(|J k |−1)>ΔX ? T then S←∅; Append J k toS; ΔX ? T ←3(|J k |−1); end else if 3(|J k |−1)=ΔX ? T then Append J k toS; returnS Figure 5.10: Heuristic for generating candidate partial parity symbols. 137 Input: Binary parity-check matrix H C . Output: Binary generalized parity-check matrix H e C . H e C ←H C ; while GTG(H e C ) contains 4-cycles do S← set of candidate partial parity symbol subsets from Algorithm 5.10; J ? ← subset inS the insertion of which reduces the most 4–cycles in GTG(H e C ); Insert symbol corresponding to J ? in H e C ; end return H e C Figure 5.11: Greedy heuristic for the removal of 4-cycles in binary generalized Tanner graphs. Input: Tanner graph TG(H C ). Max. complexity m ? . Output: 2 m ? -ary graphical model GM forC. GM←TG(H C ); repeat (C i ,C j )← pair of incident or internal constraints the removal of which removes the most 4-cycles from GM while not violating the 2 m ? -ary complexity constraint; Merge local constraintsC i andC j in GM; until No allowed 4-cycle reducing merge operations remain; return GM Figure 5.12: Greedy heuristic for the extraction of 2 m -ary graphical models. 138 5.4 Simulation Results and Discussion TheproposedextractionheuristicswereappliedtotwoextendedBCHcodeswithparam- eters[32,21,6]and[64,51,6],respectively. InbothFigures5.13and5.14theperformance of a number of suboptimal SISO decoding algorithms for these codes is compared to al- gebraic hard-in hard-out (HIHO) decoding and optimal trellis SISO decoding. Binary antipodal signaling over AWGN channels is assumed throughout. Initial parity-check matricesH were formed by extending cyclic parity-check matrices for the respective [31,21,5] and [63,51,5] BCH codes [111]. These initial parity-check matrices were used as inputs to Algorithm 5.8, yielding the parity-check matrices H 0 , which in turn were used as inputs to Algorithm 5.11, yielding 4-cycle-free generalized Tanner graphs. The suboptimal decoding algorithms implied by these graphical models are labeled TG(H), TG(H 0 ), and GTG(H 0 ), respectively. The generalized Tanner graphs extracted for the [32,21,6] and [64,51,6] codes correspond to degree-17 and degree-40 generalized extensions, respectively. Finally, the parity-check matrices H 0 were used as inputs to Algorithm 5.12 with various values of m ? . The number 4-, 6-, and 8-cycles (N 4 , N 6 , N 8 ) contained in the extracted graphical models for the [32,21,6] and [64,51,6] codes are given in Tables 5.2 and 5.3, respectively. The utility of Algorithm 5.8 is illustrated in both Figures 5.13 and 5.14: the TG(H 0 ) algorithms outperform the TG(H) algorithms by approximately 0.1 dB and 0.5 dB at a bit error rate (BER) of 10 −4 for the [32,21,6] and [64,51,6] codes, respectively. For both codes, the 4-cycle-free generalized Tanner graph decoding algorithms outperform Tanner graph decoding by approximately 0.2 dB at a BER of 10 −4 . Further performance 139 improvements are achieved for both codes by going beyond binary models. Specifically, at a BER of 10 −5 , the suboptimal SISO decoding algorithm implied by the extracted 16-ary graphical model for the [32,21,6] code outperforms algebraic HIHO decoding by approximately1.5dB.Theminimaltrellisforthiscodeisknowntocontainstatevariables with alphabet size at least 1024 [99], yet the 16-ary suboptimal SISO decoder performs only 0.7 dB worse at a BER of 10 −5 . At a BER of 10 −4 , the suboptimal SISO decod- ing algorithm implied by the extracted 32-ary graphical model for the [64,51,6] code outperforms algebraic HIHO decoding by approximately 1.2 dB. The minimal trellis for this code is known to contain state variables with alphabet size at least 4096 [99]; that a 32-ary suboptimal SISO decoder loses only 0.7 dB with respect to the optimal SISO decoder at a BER of 10 −4 is notable. 140 10 -5 10 -4 10 -3 10 -2 3 4 5 6 7 HIHO TG(H) TG(H') GTG(H') 4-ary GM 16-ary GM Trellis Bit Error Rate Eb/No (dB) Figure 5.13: Bit error rate performance of different decoding algorithms for the [32,21,6] extended BCH code. Fifty iterations of a flooding schedule were performed for all of the suboptimal SISO decoding algorithms. N 4 N 6 N 8 TG(H) 1128 37404 1126372 TG(H 0 ) 453 11152 260170 GTG(H 0 ) 0 62 298 4-ary GM 244 3852 50207 16-ary GM 70 340 724 Table 5.2: Short cycle structure of the initial and extracted graphical models for the [32,21,6] extended BCH code. 141 10 -5 10 -4 10 -3 10 -2 3 4 5 6 7 HIHO TG(H) TG(H') GTG(H') 8-ary GM 32-ary GM Trellis Bit Error Rate Eb/No (dB) Figure 5.14: Bit error rate performance of different decoding algorithms for the [64,51,6] extended BCH code. Fifty iterations of a flooding schedule were performed for all of the suboptimal SISO decoding algorithms. N 4 N 6 N 8 TG(H) 9827 1057248 111375740 TG(H 0 ) 3797 270554 19374579 GTG(H 0 ) 0 163 1229 8-ary GM 847 19590 304416 32-ary GM 201 1384 0 Table 5.3: Short cycle structure of the initial and extracted graphical models for the [64,51,6] extended BCH code. 142 5.5 The Complexity of Graphical Model Extraction ItwasdemonstratedinSection5.2thatgivenanappropriatecostmeasureonthespaceof graphical models, extraction can be formulated as a combinatorial optimization problem. In general, combinatorial optimization problems are intractable. While it is extremely challenging to evaluate the computational complexity of extraction (and thus address the conjecture of [61] that extraction is NP-hard), there are a number of known complexity resultsinthecodingliteraturethatindicatethatextractionisindeedverydifficult. These results are discussed briefly below. 5.5.1 Minimal Trellis Extraction For a given graphical modelG C , define a constraint function which indicates inclusion in the class of conventional trellis realizations and a cost function which measures the total state complexity ofG C . Minimal solutions to this optimization problem are minimal trellises. Horn and Kschischang proved that minimal trellis extraction is NP-hard in [76]. 5.5.2 Minimum Dual Distance Extraction For a given graphical modelG C , define a constraint function which indicates inclusion in the class of (non-redundant) Tanner graphs and a cost function which measures the minimum repetition constraint degree inG C . Minimal solutions to this optimization problemareTannergraphscontainingatleastonerepetitionconstraintvertexwithdegree d ⊥ (C)+1. Solutions to this problem thus yield the minimum dual distance of a code, the determination of which is known to be NP-complete [144]. 143 5.5.3 Tanner Graph Extraction for the Binary Erasure Channel For a given graphical modelG C , define a constraint function which indicates inclusion in the class of (possibly redundant) Tanner graphs with stopping distance equal to the minimum distance [134] and a cost function which measures the number of redundant checks inG C . Minimal solutions to this optimization problem are redundant Tanner graphs that imply decoding algorithms suitable for use on the binary erasure channel [134]. Evaluation of the constraint function for this problem requires computation of the stopping distance of a redundant Tanner graph which is known to be NP-hard [96]. 144 Chapter 6 Random Redundant Iterative Decoding In the context of error-correcting codes, the ultimate goal of graphical model extraction is the development of iterative soft-in soft-out (SISO) decoding algorithms with desirable performance and complexity characteristics. Chapter 5 sought to address this goal via an exploration of the space of graphical models for a given fixed code. In this chapter, a different approach is taken to the development of iterative SISO decoding algorithms. This chapter presents an approach based on (massively) redundant Tanner graphs that is applicable to arbitrary linear codes. Thestudyofredundantgraphicalmodels is notnew tothisdissertation. Anumberof authors have considered SISO decoding algorithms for linear codes that utilize redundant Tanner graphs. Jiang and Narayanan developed a SISO decoding algorithm for cyclic codes that effectively decodes on the graph implied by a different parity-check matrix at eachdecodingiterationbyrandomlycycliclyshiftingsoftinformation[83]. Kothiyalet al. developedaSISOdecodingalgorithmforarbitrarylinearcodesthatdecodesonthegraph implied by a different parity-check matrix at every iteration which is chosen adaptively in order to minimize the propagation of unreliable soft information [94]. Related algorithms 145 were presented in [84, 93]. Furthermore, a number of authors have considered redundant graphicalmodelsinothercontextssuchastheiterativedecodingofone-stepmajoritylogic decodable codes [106], low-density parity-check (LDPC) decoding [6, 95, 109], pseudo- noise sequence acquisition [158], and iterative decoding on the binary erasure channel [134]. This chapter presents a novel redundant Tanner graph SISO decoding algorithm - random redundant iterative decoding (RRD) - that is both practically realizable and ap- plicable to arbitrary linear codes. Section 6.1 briefly reviews redundant Tanner graphs and describes the implementation issues associated with the decoding algorithms implied by massively redudant Tanner graphs. Section 6.2 proposes a novel redundant Tanner graph SISO decoding algorithm - random redundant iterative decoding (RRD) - that addressesthese issues andis thuspracticallyrealizable. Theperformanceoftheproposed algorithm is examined empirically in Section 6.3 and analytically in Section 6.4. With the exception of Section 6.4, the results described in this chapter were presented first at the 2006 International Symposium on Information Theory in Seattle, WA [69] and were submitted to the IEEE Transactions on Communications in March of 2006 (revised November 2006) [64]. Note that in the first three sections of this chapter, the convention of indexing vectors and matrices from 1 that has been adopted thus far is replaced by indexing from 0 so that certain permutations can be more clearly described. Section 6.4 sees a return to the familiar convention of indexing from 1. 146 6.1 Review of Redundant Tanner Graph Decoding Recall from Chapter 2 that a redundant Tanner graph is simply the Tanner graph corre- sponding to a redundant parity-check matrix. Let H C be an r≥ n−k×n parity-check matrix for the codeC over F 2 . The codes considered in this chapter are binary; how- ever, the proposed algorithm can be applied to codes over arbitrary fields. This chapter focuses specifically on redundant parity-check matrices where r is a multiple of n−k, r = m(n−k); the Tanner graph corresponding to such a matrix is denoted a degree-m redundant Tanner graph. As an example, consider again the [8,4,4] extended Hamming codeC H which can be represented equivalently by the two parity-check matrices: H 1 =    1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 1 1 1 1 0 1 1 0 0 1 1 0   , H 2 =    0 1 0 1 0 1 0 1 1 1 0 0 0 0 1 1 1 0 1 0 1 0 1 0 0 1 0 1 1 0 1 0   . (6.1) A redundant parity-check matrix forC H can be formed by concatenating the rows of H 1 and H 2 to form H R . The corresponding degree-2 redundant Tanner graph TG(H R ) is illustrated in Figure 6.1. Degree-m redundant Tanner graphs imply the following standard iterative decoding algorithm (note that the message passing schedule described below is reasonable but not unique). For each set of checks H i , i = 1,...,m, denote by M M M i the vector of messages passed from checks to variables and denote by M M M i the vector of messages passed from variables to checks. Note that for all i = 1,...,m, M M M i is initialized to zero. 1 For 1 Throughout this chapter, decoding is assumed to be performed in the −log(·) domain, i.e. either min-sum or min ? -sum processing is assumed. 147 H 2 H 1 V 0 V 1 V 2 V 3 V 4 V 5 V 6 V 7 C 1,0 C 1,1 C 1,2 C 1,3 C 2,3 C 2,2 C 2,1 C 2,0 Figure 6.1: Degree-2 redundant Tanner graph for the [8,4,4] extended Hamming code corresponding to the concatenation of the rows of H 1 and H 2 . i = 1,...,m, the M M M i messages are first updated at the repetition constraints (RCs) using the channel observations and theM M M j messages (for j6=i). TheM M M i messages are then updated at the single parity-check constraints (SPCs) using M M M i . This updating procedure repeats for a prescribed number of iterations, I. Note that this schedule can be modified so that a number of decoding iterations are performed on each set of checks before proceeding to the next. 6.1.1 Practical Implementation via Massive Redundancy and Permutation Groups From the viewpoint of practical low-complexity implementation, the decoding algorithm described above suffers from two drawbacks: 1. Forthestandardmessagepassingrules,theintermediatemessagesvectorsM M M i must be stored for i = 1,...,m resulting in an m-fold increase of the memory required with respect to standard Tanner graph decoding. 148 2. Since each parity-check matrix H i defines a different set of checks, there is either an m-fold increase in the number of SPC trellises which must be implemented or the SPC trellises must be implemented in a reconfigurable fashion. The first drawback may be addressed by using a massively redundant Tanner graph. If a large degree of redundancy is used then I (the number of decoding iterations on the aggregate model) can be set to one and the intermediate message vectors M M M i need not be stored. Before addressing the second drawback, permutation groups of codes must first be reviewed. LetC be a length n code. The permutation group ofC, Per(C), is the set of permuta- tionsofcoordinateplaceswhichsendC ontoitself[111]. Thepermutationgroupsofmany classical linear codes are well-understood (cf. [105, 111]) and for codes with short block lengths, permutation groups can be obtained using the methods described in [102, 135]. Returning to the [8,4,4] extended Hamming code, it can be shown that [111]: 2 σ =(5,3,1,7,0,6,4,2)∈Per(C H ) (6.2) It is readily verified that applying σ to the columns of H 1 (as defined in (6.1)) yields H 2 so that H 2 = σH 1 . In light of this example, it is clear that redundant parity-checks for a given codeC can be generated by applying permutations drawn from Per(C) to the columns of some initial parity-check matrix H. Observe that decoding with soft-input vector SI SI SI on TG(βH) (where β ∈ Per(C)) is equivalent to decoding with soft-input 2 Throughout this chapter, permutations of n coordinate places (indexed from 0) are described by n-tuples. For example, the application of the permutation (1,4,0,2,3,6,5) to a 7 bit codeword (c0,c1,c2,c3,c4,c5,c6) yields the permuted codeword: (c2,c0,c3,c4,c1,c6,c5). The identity permutation is denoted and the inverse of a permutation β is denoted β −1 . 149 vector β −1 SI SI SI on TG(H). It is this observation that allows for the efficient implemen- tation of redundant Tanner graph decoding: provided that the redundant parity-checks are column permuted versions of some base matrix H, redundant Tanner graph decoding can be implemented by permuting soft information vectors and decoding with a constant set of constraints. 6.1.2 Redundant Tanner Graphs and Previous Work Fromtheabovediscussion,itisapparentthatKothiyalet al.’sadaptivebeliefpropagation (ABP) algorithm [94] and Jiang and Narayanan’s stochastic shifting based iterative de- coding (SSID) algorithm [83] are redundant Tanner graph decoding algorithms. Kothiyal et al.’s scheme adaptively chooses new parity-check sets based on soft information reli- ability. Although the ABP algorithm can be applied to arbitrary linear codes, it does not imply a practical low-complexity implementation because the check sets change with every iteration. Furthermore, the ABP algorithm requires the computationally expensive Gaussian elimination of potentially large parity-check matrices at every iteration. Jiang and Narayanan’s scheme is an example of a practical, low-complexity redundant Tanner graph decoding algorithm for cyclic codes which uses the permutation group approach described above. The random redundant decoding algorithm proposed in this chapter is, in fact, an extension of Jiang and Narayanan’s algorithm to arbitrary linear codes with a known permutation group. 150 6.2 Practical Redundant Tanner Graph Decoding Algorithm 6.2 describes the proposed decoding algorithm. The inner for-loop of Algo- rithm 6.2 describes an efficient redundant Tanner graph decoding algorithm with the addition of a damping coefficient α. The outer for-loop of Algorithm 6.2 iterates over different values of α. By varying the damping coefficient α, the algorithm avoids local minima in the solution space. Many authors have considered the introduction of such damping coefficients in iterative soft decoding algorithms to achieve this end (cf. [24]). Forpracticalimplementationswherealargenumberofouteriterationsisundesirablefrom a time complexity standpoint, a single damping coefficient (or a small set of coefficients) could be used depending on the operating noise power. Algorithm 6.2 takes as input a received soft information vector, SI SI SI, a parity-check matrix for the code, H, and four parameters: α 0 : The initial damping coefficient. I 1 : The number of Tanner graph decoding iterations to perform per inner iteration. I 2 : Themaximumnumberofinneriterationstoperformperouteriteration. Eachinner iteration considers a different random permutation of the codeword elements. I 3 : The maximum number of outer iterations to perform. Each outer iteration uses a different damping coefficient. Let s s s be the sum of the input soft information SI SI SI and the output soft information produced by all previous inner iterations. During the i 2 -th inner iteration, I 1 Tanner graph decoding iterations are performed on TG(H) with damping coefficient α and soft 151 input s s s producing the soft output vector s s s 0 and hard decision c c c 0 . The cumulative soft information vector s s s is then updated to include s s s 0 . The inner iteration concludes by applyingarandompermutationθ fromthepermutationgroupofthecodetos s s. Decoding concludes when either a valid codeword is returned by the Tanner graph decoding step or when a maximum number of iterations is reached. Before returning the final soft output and hard decision vectors, the random permutations are undone by applying the inverse of the product of the permutations that were applied tos s s. Input: Length n soft-input vectorSI SI SI. n−k×n binary parity-check matrix H. Parameters I 1 , I 2 , I 3 , α 0 . Output: Length n soft-output vectorSO SO SO. Length n hard-decision vectorHD HD HD. α←α 0 ; for 1≤i 3 ≤I 3 do Θ←; s s s←SI SI SI; for 1≤i 2 ≤I 2 do Perform I 1 decoding iterations ofs s s on TG(H) with damping coefficient α and place soft output ins s s 0 and resulting hard decision inc c c 0 ; s s s←s s s+s s s 0 ; if Hc c c 0 =0 then Apply Θ −1 tos s s andc c c 0 ; SO SO SO←s s s−SI SI SI; HD HD HD←c c c 0 ; returnSO SO SO and HD HD HD end θ←random element of Per(C); Apply θ tos s s; Θ←θΘ; end α←α 0 +(1−α 0 ) i 3 I 3 −1 ; end Figure 6.2: Random redundant iterative decoding. 152 6.2.1 Initial Parity-Check Matrix Selection It was observed empirically that the performance of the proposed decoding algorithm depends heavily on the choice of parity-check matrix (and thus Tanner graph) used to representthecode. Itiswidelyacceptedthattheperformanceofthedecodingalgorithms implied by Tanner graphs are adversely affected by short cycles (cf. [66, 112]). Recall thatAlgorithm5.8searchesforasuitableparity-checkmatrixbygreedilyperformingrow operations on an input binary parity-check matrix H in order to reduce the number of short cycles contained in the Tanner graph defined byH. Algorithm 5.8 can thus be used to generate suitable input parity-check matrices for Algorithm 6.2. 6.2.2 Generation of Random Permutation Group Elements Algorithm 6.2 requires the efficient generation of random elements of the permutation group of a code. Cellar et al. presented an algorithm for generating random elements of an arbitrary finite group in [23]. Briefly, suppose that G is a finite group with size k generating set 3 X ={x 0 ,x 1 ,...,x k−1 }. (6.3) Cellar et al.’s product-replacement algorithm constructs a vector S S S = (s 0 ,s 1 ,...,s N−1 ) of length N > k containing all of the elements of X with repeats. The basic operation of this algorithm is to randomly choose two elements of S S S, s i and s j , and to replace s j by the product of s i and s j . Generation of random group elements is initialized by performing this basic operation K times. After initialization, successive basic operations 3 That is, every element g ∈ G can be expressed as a finite product g = x n 1 i 1 x n 2 i 2 ···x n t i t where xi j ∈ X and nj ∈N (the set of natural numbers) for all j. 153 yieldrandomelementsofGbyreturningtheupdatedvalueofs j . Notethattheexecution of this basic operation requires the generation of two random integers and only one group multiplicationandis thus efficient(permutation multiplicationis particularly easy). Also note that after every execution, the elements contained in S S S generate G. Cellar et al. found that setting N = max(2k + 1,10) and K = 60 provides near-uniform random generation of group elements in practice. 6.3 Empirical Performance Study 6.3.1 The Extended Golay Code Consider the extended Golay codeC G defined by the parity-check matrix H G =                   1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 1 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 1 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 1 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 1 0 0 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 1 0 0 0 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1                   . (6.4) The permutation group ofC G is generated by four permutations [111]: σ (1) G =(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,0,23), (6.5) σ (2) G =(0,2,4,6,8,10,12,14,16,18,20,22,1,3,5,7,9,11,13,15,17,19,21,23), (6.6) σ (3) G =(23,22,11,15,17,9,19,13,20,5,16,2,21,7,18,3,10,4,14,6,8,12,1,0), (6.7) 154 σ (4) G =(23,17,22,15,19,8,14,12,7,20,9,1,10,21,2,5,6,11,18,13,16,4,0). (6.8) TheTannergraphcorrespondingto(6.4)containsmanyshortcycles. Theapplication of Algorithm 5.8 to this matrix yields H 0 G =                   1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 1 0 1 0 0 1 0 0 0 1 1 0 0 0 1 0 0 0 0 1 0 1 0 0 1 1 1 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 1 0 0 1 0 0 0 0 1 1 0 1 0 1 0 0 0 0 1 1 0 1 0 0 0 0 1 0 1 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 1 1 1 0 1 0 1 0 1 0 1 0 1 0 0 1 1 0 0 1 1 0 1 0 1 1 1 1 0 0 0 0 0 1 0 0 1 1 0 1 0 1 1 0 0 0 1 0 0 1 1 1 1 1 0 1 0 0 0 0 0 1 1 1 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 0 0 1 1 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 0 1 0 0 0 1 1 0 0 0 0 1 1 0 1 0 1 1 0 0 1 1 0 0 1 0 1 0 0 0 0 1 1 1 0                   . (6.9) Whereas the Tanner graph corresponding to (6.4) contains 1,551 4-cycles and 65,632 6- cycles, the Tanner graph corresponding to (6.9) contains 295 4-cycles and 6,204 6-cycles. Recall from Chapter 3 that any Tanner graph representingC G necessarily contains 4- cycles. Figure6.3comparestheperformanceoffivedecodingalgorithmsforC G : optimalSISO decodingviaatrellis; standarditerativedecodingusingtheTannergraphsimpliedbyH G and H 0 G (labeled TG(H) and TG(H 0 ), respectively); and, the proposed RRD algorithm usingH G andH 0 G asinputparity-checkmatrices(labeledRRD(H)andRRD(H 0 ),respec- tively). One hundred decoding iterations were performed for the Tanner graph decoders while the RRD algorithms both use input parameter sets: α 0 =0.08, I 1 =2, I 2 =30 and I 3 = 20. Flooding message passing schedules were used for all Tanner graph decoders and binary antipodal signaling over additive white Gaussian noise (AWGN) channels is assumed throughout this chapter. 155 10 -5 10 -4 10 -3 10 -2 3 3.5 4 4.5 5 5.5 6 6.5 7 TG (H) TG (H') RRD (H) RRD (H') Trellis Bit Error Rate Eb/No (dB) Figure 6.3: Bit error rate performance comparison of different decoding algorithms for the [24,12,8] extended Golay code. Note first in Figure 6.3 that the performance of the RRD algorithm is highly sensitive tothechoiceofparity-checkmatrix: thedecoderusingH 0 G outperformsthedecoderusing H G by approximately 1.75 dB at a bit error rate (BER) of 10 −4 . It is known that any optimal SISO decoder for the extended Golay code must contain hidden variables with alphabet size at least 256 [117]. Furthermore, there exists a well-known tail-biting trellis for this code which contains 16-ary hidden variables and has near-optimal performance [22]. The RRD algorithm using H 0 G , which contains only binary variables, performs approximately 0.3 dB worse than optimal SISO decoding at a bit error rate of 10 −5 . 156 6.3.2 The [63,39,9] BCH Code The performance of the proposed algorithm is investigated empirically in this section for the [63,39,9] BCH codeC 63 . The performance of Algorithm 6.2 was studied using three choices of input parity-check matrix (labeled HC, H2, and HR), two choices of permutation(sub)group(labeledGFandGC),andtwochoicesofparametersets(labeled PA and PB) all of which are defined below. Figure 6.4 compares the BER performance of four combinations of these choices to algebraic hard-in hard-out (HIHO) decoding. HC: The standard 24×63 cyclic parity-check matrix forC 63 (cf. [111]). The correspond- ing Tanner graph contains 32,625 4-cycles and 6,981,190 6-cycles. H2: The output of Algorithm 5.8 when HC is used as input. The corresponding Tanner graph contains 3,162 4-cycles and 212,301 6-cycles. HR: A 1200×63 redundant parity-check matrix generated by concatenating 50 column- permuted versions of H2 (using permutations drawn randomly from Per(C 63 )). GF: The full permutation group Per(C 63 ) ofC 63 which is generated by the 6 permuta- tions: σ (j) = 1,2 j +1,2·2 j +1,...,62·2 j +1 (6.10) for 0≤j≤5 (where each permutation element is taken modulo 2 m −1) [105]. GC: The cyclic subgroup of Per(C 63 ) generated by σ (0) alone. PA: α 0 =0.08, I 1 =2, I 2 =50, and I 3 =20. PB: α 0 =0.08, I 1 =2, I 2 =1, and I 3 =20. 157 OptimalSISOdecodingofC 63 isintractableinpractice. Specifically,theleastcomplex known optimal SISO decoding algorithm for this code is obtained by decoding on a sixteen stage trellis for the dual of an extension ofC 63 containing 2 15 states [87]. Random redundant decoding with{H2,GF,PA}, which is suboptimal yet tractable in practice, is shown to outperform algebraic HIHO decoding by approximately 1.75 dB at a BER of 10 −6 in Figure 6.4. Figure 6.4 demonstrates the sensitivity of the perfomance of random redundantdecodingtothechoiceofinitialparity-checkmatrix: the{H2,GF,PA}decoder outperforms the{HC,GF,PA} decoder by approximately 2 dB at a BER of 10 −5 . The {H2,GF,PA} decoder outperforms the{H2,GC,PA} decoder by approximately 0.25 dB at a BER of 10 −6 . The former considers all possible permutations in Per(C 63 ), rather than only those corresponding to cyclic shifts, and is in some sense more random than the latter, which is equivalent to Jiang and Narayanan’s algorithm [83]. Note finally that althoughthe{H2,GF,PA}and{HR,GF,PB}decoderscanbeviewedasdifferentmessage- passing schedules on the same graphical model, the former outperforms the latter by approximately 0.9 dB at a BER of 10 −5 . This performance difference may be attributed tothefactthatthefloodingscheduleusedinthe{HR,GF,PB}decoderdoesnotallowfor the full exploitation of the gains that can be realized by damping thus indicating that the performance of the proposed random redundant decoding algorithm may be attributed to both the use of redundancy and the use of damping to slow convergence. 158 10 -6 10 -5 10 -4 10 -3 3 4 5 6 7 {HC,GF,PA} HIHO {HR,GF,PB} {H2,GC,PA} {H2,GF,PA} Bit Error Rate Eb/No (dB) Figure 6.4: Bit error rate performance comparison of different decoding algorithms for the [63,39,9] BCH code. 6.4 Analytical Performance Study As discussed in Chapter 2, the development of tools to analytically describe the perfor- manceofiterativemessagepassingalgorithms, particularlyfortheAWGNchannel, isnot yet fully mature. While the performance of iterative decoding techniques for the binary erasure channel (BEC) can be completely characterized by stopping sets [36], stopping sets do not adequately describe decoding on the AWGN channel [149]. A number of authors have considered the girth or short cycle structure of a graphical model as a proxy for performance on the AWGN channel based on the empirical observation that short cycles adversely affect the performance of iterative message passing algorithms (cf. [66]). However, short cycle structures fail to explain the performance of random redundant 159 decoding. Specifically, returning to the extended Golay code example of Section 6.3.1, the Tanner graph decoder, TG(H 0 ), contains 295 4-cycles, 6,204 6-cycles, and 114,783 8-cycles. By using a simple extension of the short cycle counting algorithm described in the appendix of this dissertation to random redundant models, it can be shown that, on average, a random redundant Tanner graph for this code contains 431 4-cycles, 11,187 6-cycles, and 311,578 8-cycles. Linearprogramming(LP)decodingwasintroducedinFeldman’sthesis[44]. Themost successful approach thus far developed to describe the performance of iterative message passing algorithms is to characterize the performance of LP decoding and then extrap- olate the results to message passing decoding. It is this approach that is taken in this section. Because it is relatively new, care is taken to briefly review LP decoding below. The remainder of this section is organized as follows. Mathematical notation specific to this analysis is defined in Section 6.4.1. LP decoding and its associated nomencla- ture (fundamental polytope, pseudo-codewords, etc.) are described in Section 6.4.2. The presentation of Section 6.4.2 draws heavily on that of Vontobel and Koetter [149]. The effect of redundancy on the fundamental polytope and pseudo-weight spectrum of a Tan- ner graph are described in 6.4.3. Section 6.4.4 discusses the bridge from LP decoding with redundancy, which is characterized exactly by the fundamental polytope, to mes- sage passing decoding on redundant Tanner graphs, for which the fundamental polytope is an inexact, yet empirically successful, characterization of performance. 160 6.4.1 Mathematical Preliminaries and Notation The l 1 -norm of a vector x x x is denoted||x x x|| 1 and the l 2 -norm is denoted||x x x|| 2 . The inner product of two vectors x x x and y y y is denotedhx x x,y y yi. The support set of a vector (i.e. the set of non-zero coordinates) is denoted supp(x x x). The all zeros and all ones vectors are denoted0 and1, respectively. Letx x x (1) ,...,x x x (k) bek points inR n . A convex combination of x x x (1) ,...,x x x (k) is any point of the form θ 1 x x x (1) +···+θ k x x x (k) such that θ 1 +···+θ k = 1 and θ i ≥0∀i∈[1,k]. A set of pointsS inR n is called convex if any convex combination of points inS is also inS. The convex hull of a setS conv(S) is the smallest convex set containingS. A polytope inR n is defined as the convex hull of a finite set of points. A polyhedron inR n is defined as the solution of a finite number of linear equalities and inequalities. A bounded polyhedron is a polytope by the Weyl-Minkowski Theorem [12]. 6.4.2 LP Decoding and the Fundamental Polytope Suppose that a codeword c c c∈C (whereC is a [n,k,d] binary linear code with parity- check matrix H C ) is transmitted over a memoryless binary channel with channel law Pr Y|X (y|x) (where the transmission of each codeword is equiprobable). The receiver observes the channel output vector y y y∈ R n and forms the channel log-likelihood ratio vector λ=(λ 1 ,...,λ n ) where λ i =log Pr Y|X (y i |x i =0) Pr Y|X (y i |x i =1) . (6.11) 161 The maximum a-posteriori (MAP) decoder can then be formulated as ˆ x x x MAP (y y y)=argmin c c c∈C hx x x,λi. (6.12) In order to define LP decoding, it is useful to first reformulate the MAP decoding function of (6.12). The convex hull ofC conv(C) has as its vertices the codewords ofC. Becausethecostfunctionhx x x,λiislinear, thereexistminimaofhx x x,λioverconv(C)which correspond to codewords inC. Thus, the MAP decoding function can be rewritten as ˆ x x x MAP (y y y)=arg min c c c∈conv(C) hx x x,λi. (6.13) Equation (6.13) defines MAP decoding as a linear program. In its most general form, LP decoding is defined as a minimization over some relaxation of the polytope conv(C): ˆ x x x LP (y y y)=arg min c c c∈relax(conv(C)) hx x x,λi. (6.14) Feldman et al. introduced the following relaxation [45]. Let H C be composed of n−k rows, r r r i for i∈ [1,n−k], each of which defines a length n single parity-check codeC i which depends on the choice of parity-check matrix H C . Observe first that C = \ i∈[1,n−k] C i (H C ). (6.15) 162 A reasonable relaxation is thus P(H C )= \ i∈[1,n−k] conv(C i (H C ).) (6.16) Equation (6.17) defines this relaxation polytope, denoted the fundamental polytope, ex- plicitly. P(H C )=                        x x x∈R n ∀j∈[1,n]: 0≤x j ≤1 ∀i∈[1,n−k], P j∈J i x j + P j∈supp(r r r i )\J i (1−x j )≤ ∀J i ⊆supp(r r r i ), |supp(r r r i )|−1 |J i |odd:                        (6.17) Theverticesofthefundamentalpolytopecorrespondtofeasiblesolutionsofthelinear program defined by (6.14) and are denoted pseudo-codewords. Note that every codeword is a pseudo-codeword but the converse is not necessarily true. Given a pseudo-codeword x x x, its pseudo-weight on the AWGN channel 4 is defined by w p (x x x)= ||x x x|| 1 ||x x x|| 2 2 . (6.18) Note that if x x x is a codeword then its pseudo-weight is precisely its Hamming weight. Whereas the performance of the MAP decoder is dictated by the Hamming weight spec- trumofthecode,theperformanceoftheLPdecoderforaspecificparity-checkmatrixH C 4 Although the definition of fundamental polytope and thus the pseudo-codeword set is channel inde- pendent, the definition of pseudo-weight is channel dependent. 163 isdictatedbythepseudo-weightspectrumofthe fundamentalpolytopeP(H C ). The per- formanceofLPdecodingthusdependsonthechoiceofH C . Theminimumpseudo-weight of a code specified by H C is denoted w H C p,min . Note that w H C p,min ≤d (6.19) since all codewords are pseudo-codewords. 6.4.3 Redundancy and the Fundamental Polytope In order to motivate how the addition of redundant parity-checks can improve LP decod- ing, consider as an example the [7,2,3] binary cycle code,C C , studied in [91, 149] with parity-check matrix H C =        1 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 1 1 0 0 0 0 0 0 1 1 0 1 0 0 0 0 1 1 0 0 0 0 0 0 1 1        . (6.20) After simplification, the fundamental polytope corresponding to H C is [149] P(H C )=                        x x x∈R 7 0≤x i ≤1∀i∈[1,7] x 1 =x 2 =x 3 x 5 =x 6 =x 7 x 4 ≤2min(x 2 ,1−x 2 ,x 5 ,1−x 5 )                        (6.21) 164 and it can be shown thatP(H C ) has five vertices: x x x (1) C =(0,0,0,0,0,0,0), x x x (2) C =(1,1,1,0,0,0,0), x x x (3) C =(0,0,0,0,1,1,1), x x x (4) C =(1,1,1,0,1,1,1), x x x (5) C = 1 2 , 1 2 , 1 2 ,1, 1 2 , 1 2 , 1 2 . (6.22) Notethatx x x (1) C ,x x x (2) C ,x x x (3) C ,andx x x (4) C arethecodewordsofC C whilex x x (5) C isapseudo-codeword with pseudo-weight 6.4. Consider next the redundant parity-check matrix formed by adding the binary sum of the first 4 rows of H C to H C : H 0 C = H C 0 0 0 1 0 0 0 . (6.23) After simplification, fundamental polytope corresponding to H 0 C is P H 0 C =                        x x x∈R 7 0≤x i ≤1∀i∈[1,7] x 1 =x 2 =x 3 x 5 =x 6 =x 7 x 4 =0                        (6.24) andP(H 0 C ) has only four vertices: x x x (1) C , x x x (2) C , x x x (3) C and x x x (4) C . In fact, the fundamental polytopeP(H 0 C ) is identical to the convex hull ofC C and LP decoding overP(H 0 C ) is equivalent to maximum likelihood decoding. 165 In light of the above example, it is natural to ask whether for all codes, there exists a redundant parity matrix H 0 C so thatP(H 0 C )=conv(C). The extended Hamming code, C H provides a counterexample, however. Let H 0 H be the parity-check matrix formed from the 15 non-zero codewords ofC H (note thatC H is self-dual). Using a software tool such as PORTA [29], it can be shown thatP(H 0 H ) has 144 vertices of which 128 do not correspond to codewords andP(H 0 H )6= conv(C H ). However, it can also be shown that ω H 0 H p,min =d=4 whereas ω H 1 p,min =ω H 2 p,min =3 (where H 1 and H 2 were defined by (6.1)). The following illustrates the positive effect that the addition of redundant parity- checks can have on the minimum pseudo-weight. Claim 6.1. Suppose H C is a (possibly redundant) parity-check matrix for a codeC (with dualC ⊥ ) and letH 0 C be a redundant parity-check matrix forC formed by adding a codeword ofC ⊥ to H C . Then ω H 0 C p,min ≤ω H C p,min . (6.25) Proof. Suppose that y y y is a vertex ofP(H 0 C ) such that ω p (y y y) = ω H 0 C p,min . Since y y y meets all of the constraints which defineP(H C ), it must either be a vertex ofP(H C ) or lie on a face ofP(H C ) and is thus a convex combination of vertices ofP(H C ): y y y =θ 1 x x x (1) +···+θ L x x x (L) (6.26) 166 where θ 1 +···+θ L =1 andx x x (1) ,...,x x x (L) are vertices ofP(H C ). By Lemma 41 of [149]: ω H 0 C p,min =ω p (y y y)=ω p L X l=1 θ l x x x (l) ! ≤min ω p x x x (1) ,...,ω p x x x (L) ≤ω H C p,min . (6.27) Given Claim 6.1, it is tempting to conjecture that for any linear codeC, there exists a redundant parity-check matrix H 0 C such that ω H 0 C p,min = d(C). While is possible to make such a claim for the binary erasure channel [134], the of hierarchy of minimum pseudo- weights for different channels [149] prevents the immediate extension of this result to the AWGN channel. Nevertheless, Claim 6.1 demonstrates that the addition of redundant checks can only improve the performance LP decoding. 6.4.4 From LP to Iterative Message Passing Decoding While the fundamental polytope characterizes the performance of LP decoding (LPD) exactly, the same cannot be said for message passing decoding. Vontobel and Koetter introducedgraphcoverdecoding(GCD)whichisessentiallyequivalenttoLPDandstated that [149]: Sometimes [GCD] is an exact model but usually it is just a very good approx- imation [of message passing decoding]. Vontobel and Koetter went on further to say in reference to redundancy that: 167 ...when adding redundant rows to a parity-check matrix it is clear that the decoding performance of GCD and LPD can only become better. A question remains as how far GCD is still a good model of [message passing decoding] when the parity-check matrix contains many more rows than columns. ThefollowingexploresthisquestionanddemonstratesempiricallythatLPdecodingwith theadditionofredundantrowsmaybetterdescriberandomredundantiterativedecoding than message passing with a flooding schedule on a redundant Tanner graph. Consider again the [8,4,4] extended Hamming codeC H which can be represented by the parity-check matrices H 1 and H 2 (as defined by (6.1)). SinceC H is self-dual, a redundantparity-checkmatrixH 3 forthiscodecanbedefinedwith14rowscorresponding to the codewords inC H \{0,1} (all of which have Hamming weight 4). Note that this definition of H 3 ensures that a random redundant Tanner graph using H 2 as input can use as checks all of the single parity-checks defined by the rows of H 3 . Figure 6.5 illustrates the pseudo-weight spectra of the Tanner graphs corresponding to H 1 , H 2 , and H 3 . Note first that since the 1 codeword is not included in H 2 , there are manylesspseudo-codewordsintheTannergraphimpliedbyH 2 comparedtothatimplied by H 1 although the minimum pseduo-weights for these Tanner graphs are identically 3. Note also that ω p,min (H 3 )=d=4. Figure 6.6 compares the performance of 5 decoding algorithms forC H : optimal SISO decoding via a trellis; standard iterative decoding (with 50 iterations) using the Tanner graphs implied by H 1 and H 2 (labeled TG(H 1 ) and TG(H 2 ), respectively); standard iterative decoding (with 50 iterations) - i.e. a flooding message passing schedule - on the redundant Tanner graph implied by H 3 (labeled RTG(H 3 )); and, random redundant 168 iterative decoding (with α 0 = 0.08, I 1 = 1, I 2 = 30 and I 3 = 20) using H 2 as the input parity-check matrix (labeled RRD(H 2 )). The performance difference between TG(H 1 ) and TG(H 2 ) is consistent withP(H 2 ) containing less low weight pseudo-codewords than P(H 1 ). For this toy code, the performance of random redundant decoding is near op- timal while flooding on the redundant Tanner graph loses 0.5 dB compared to random redundant decoding at a BER of 10 −5 . The performance difference between random redundant decoding and standard flood- ing decoding on a redundant Tanner graph is more pronounced in the following example. Consider again the [24,12,8] extended Golay codeC G which was studied in Section 6.3.1. Via computer search, Schwartz and Vardy found a redundant parity check matrix forC G , H R , with 34 rows and 5 w BEC p,min =d = 8 [134]. Figure 6.7 repeats the results of Figure 6.3 with the removal of those algorithms based on H G and the addition of standard iterative decoding (with 50 iterations) on the redundant Tanner graph implied by H R (labeled TG(H R )). Note that although flooding on TG(H R ) outperforms flooding on TG(H 0 G ), it loses 1.2 dB compared to random redundant iterative decoding at a BER of 10 −5 . Claim 6.1 demonstrates that the addition of redundant checks provably improves LP decoding. The two examples above demonstrate that random redundant iterative decodingmayoutperformdecodingwithafloodingscheduleonaredundantTannergraph. Thus, LP decoding with redundant checks may be a better model for random redundant iterative decoding than for flooding on a redundant Tanner graph, i.e. the performance gains that result from redundancy are better realized with the RRD schedule than with flooding. 5 The minimum pseudo-weight on the BEC, w BEC p,min , is equal to the minimum stopping set size. 169 0 20 40 60 80 100 120 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 Number of Pseudo-codewords Pseudo-weight (a) Tanner graph corresponding to H1. 0 20 40 60 80 100 120 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 Number of Pseudo-codewords Pseudo-weight (b) Tanner graph corresponding to H2. 0 20 40 60 80 100 120 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 Number of Pseudo-codewords Pseudo-weight (c) Redundant Tanner graph corresponding to H3 which includes all 14 weight 4 dual code- word checks. Figure 6.5: Pseudo-weight spectra of three graphical models for the [8,4,4] extended Hamming code. 170 10 -5 10 -4 10 -3 10 -2 4 5 6 7 8 9 TG(H1) TG(H2) RTG(H3) RRD(H2) Trellis Bit Error Rate Eb/No (dB) Figure 6.6: Bit error rate performance comparison of different decoding algorithms for the [8,4,4] extended Hamming code. 10 -6 10 -5 10 -4 10 -3 10 -2 3 3.5 4 4.5 5 5.5 6 6.5 7 TG (H') TG (HR) RRD (H') Trellis Bit Error Rate Eb/No (dB) Figure 6.7: Bit error rate performance comparison of different decoding algorithms for the [24,12,8] extended Golay code. 171 Chapter 7 Conclusions Our interest in the extraction of graphical models for linear codes was sparked first by a comment made by Forney in his first Codes on Graphs paper ([50], Sec. VI.D): Future work might explore whether there exists some intermediate clustering technique that reduces but does not eliminate cycles, and that obtains near- [optimal] decoding performance. While a number of other authors have similarly hinted at the importance of extraction (e.g. [2, 97]), this dissertation is, to the best of our knowledge, the first work to identify extraction explicitly and treat it formally. While this dissertation has developed some of the foundations of the theory of extrac- tive graphical modeling, this theory is far from complete. The results of Chapter 6, for example, indicate that quality measures for graphical models must go beyond short cycle structure. It is anticipated that as the coding community’s understanding of the perfor- mance iterative message passing algorithms evolves, so too will the theory and practice of model extraction. 172 The remainder of this chapter describes directions for future work that are motivated by the results of this dissertation. Section 7.1 describes directions for future work con- cerning extraction in the context of coding on a chapter-by-chapter basis. Important directions for the development of a theory of cyclic graphical model extraction for more general systems are described in Section 7.2. 7.1 Future Directions: Coding 7.1.1 Future Directions: Chapter 3 There are a number of interesting directions for future work motivated by the statement of the tree-inducing cut-set bound (TI-CSB) in Chapter 3. While the minimal trellis complexity s(C) of linear codes is well-understood, less is known about minimal tree complexity t(C) and characterizing those codes for which t(C)<s(C) is an open problem. A particularly interesting open problem is the use of the Cut-Set Bound to establish an upper bound on the difference between s(C) and t(C); such a bound would allow for a re-expression of the TI-CSB in terms of the more familiar minimal trellis complexity. A study of those codes which meet or approach the TI-CSB is also an interesting direction for future work which may provide insight into construction techniques for good codes with short block lengths (e.g. 10s to 100s of bits) defined on graphs with a few cycles (e.g. 3, 6, or 10). The development of statements similar to the TI-CSB for alternative measures of graphical model complexity and for graphical models of more general classes of codes (e.g. group, nonlinear) is also interesting. 173 7.1.2 Future Directions: Chapter 4 One interesting direction for future work concerning the cycle-free models for Reed- Solomon (RS) codes presented in Chapter 4 is to investigate the use of soft-in soft-out (SISO) ordered statistics decoding [54]of the glue code andto compare the resultingsub- optimal SISO with a SISO employing ordered statistics decoding of the full RS codes. A further study of generalized Vardy-Be’ery codes may also yield interesting constructions. A number of interesting directions for future work are motivated by cycle-free models presentedforfirst-orderReed-Muller(RM)codesinChapter4. Itwouldbeinterestingto searchformorecodesforwhichconditionallycycle-freegeneralizedTannergraphs(GTGs) offer complexity savings over trellis decoding. It would also be interesting to study the performance of the suboptimal iterative SISO decoding algorithms implied by GTGs in which only a subset of hidden variables required to break all cycles are conditioned. Specifically, a study of codes with known algebraic decompositions (cf. [4, 122, 146, 147]) may lead to novel low-complexity SISO decoding algorithms. There are also a number of interesting avenues of study concerning the application of the optimal SISO decoder for extended Hamming codes. Specifically, the algorithm presented in this work has a naturaltreestructurewhichcanleadtoparticularlyefficientimplementationsinhardware [13, 30]. 7.1.3 Future Directions: Chapter 5 There are a number of interesting directions for future work motivated by the study of graphical model transformation in Chapter 5. While the extracted graphical models pre- sented in Chapter 5 are notable, ad-hoc techniques utilizing massively redundant models 174 and judicious message filtering (i.e. random redundant iterative decoding) outperform the presented models. Such massively redundant models contain many more short cycles than the models presented in Chapter 5 indicating that short cycle structure alone is not a sufficiently meaningful cost measure for graphical model extraction. It is known that redundancy can be used to remove pseudo-codewords (cf. [149]) thus motivating the study of cost measures which consider both short cycle structure and pseudo-codeword spectrum. Finally, it would be interesting to study extraction heuristics beyond simple greedy searches, as well as those which use all of the basic graphical model operations (rather than just constraint merging). 7.1.4 Future Directions: Chapter 6 An interesting direction for future work concerning random redundant iterative decoding (RRD) is the application of the RRD algorithm to codes that are not necessarily block codes. For example, it would be interesting to investigate the application of random redundant iterative decoding to the lattice Tanner graphs introduced by Banihashemi and Kschischang [11] with the goal of developing low-complexity soft-in soft-out lattice decoders with reasonable complexity. It would also be interesting to investigate the performanceofveryshortlengthLDPCs(10sto100sofbits)withprescribedpermutation groups and to see whether such codes, with random redundant iterative decoding, have performance characteristics that are competitive with classical block codes. 175 7.2 Future Directions: Beyond Coding Graphical model extraction, while interesting in the context of codes, is more important forfixedsystemsthatcannotbedesigneda priori withiterativemessagepassinginference algorithms in mind. There are five major challenges which must be addressed in order to develop a general theory of graphical model extraction akin to that introduced for linear codes in this dissertation: 1. The space of valid graphical models for general systems must be specified. 2. Global and local complexity measures for systems must be developed that are both tractableandhighlycorrelatedtothecomplexityoftheinferencealgorithmsimplied by graphical models. 3. Quality measures on the space of models must be developed that are both tractable and highly correlated to the performance of the inference algorithms implied by graphical models. 4. A mechanism for searching the space of graphical models must be established. 5. Heuristics for searching the space of graphical models in order to extract models which imply inference algorithms with desirable performance and complexity char- acteristics must be developed. While the solution to these problems may be distant, it is anticipated that the formal treatment of extraction will ultimately lead to better inference algorithms than ad hoc approaches alone. 176 References [1] S.M.Aji,G.B.Horn,andR.J.McEliece,“Iterativedecodingongraphswithasin- gle cycle,” in Proc. IEEE International Symp. on Information Theory, Cambridge, MA, August 1998, p. 276. [2] S. M. Aji and R. J. McEliece, “The generalized distributive law,” IEEE Trans. In- formation Theory, vol. 46, no. 2, pp. 325–343, March 2000. [3] N. Alon, R. Yuster, and U. Zwick, “Finding and counting given length cycles,” Algorithmica, vol. 17, no. 3, pp. 209–223, 1997. [4] O. Amrani and Y. Be’ery, “Reed-Muller codes: Projections onto GF(4) and multi- levelconstructions,”IEEETrans.InformationTheory,vol.47,no.6,pp.2560–2565, September 2001. [5] J.B.AndersonandS.M.Hladik,“TailbitingMAPdecoders,”IEEEJ.Select.Areas Commun., vol. 16, no. 2, pp. 297–302, February 1998. [6] K. Andrews, S. Dolinar, and F. Pollara, “LDPC decoding using multiple repre- sentations,” in Proc. IEEE International Symp. on Information Theory, Lausanne, Switzerland, July 2002, p. 456. [7] D. M. Arnold, E. Eleftheriou, and X. Y. Hu, “Progressive edge-growth Tanner graphs,” in Proc. Globecom Conf., vol. 2, San Antonion, TX, November 2001, pp. 995–1001. [8] A.AshikhminandS.Litsyn, “SimpleMAPdecodingoffirst-orderReed-Mullerand Hamming codes,” IEEE Trans. Information Theory, vol. 50, no. 8, pp. 1812–1818, August 2004. [9] D. Augot and F. L. vit Vehel, “Bounds on the minimum distance of the duals of BCH codes,” IEEE Trans. Information Theory, vol. 42, no. 4, pp. 1257–1260, July 1996. [10] L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” IEEE Trans. Information Theory, vol. IT-20, pp. 284–287, March 1974. [11] A. H. Banihashemi and F. R. Kschischang, “Tanner graphs for group block codes and lattices: Construction and complexity,” IEEE Trans. Information Theory, vol. 47, no. 2, pp. 822–834, February 2001. 177 [12] A. Barvinok, A Course in Convexity. Providence, RI: American Mathematical Society, 2002. [13] P. A. Beerel and K. M. Chugg, “A low latency SISO with application to broadband turbo decoding,” IEEE J. Select. Areas Commun., vol. 19, no. 5, pp. 860–870, May 2001. [14] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “Serial concatenation of interleaved codes: Performance analysis, design, and iterative decoding,” IEEE Trans. Information Theory, vol. 44, no. 3, pp. 909–926, May 1998. [15] S. Benedetto and G. Montorsi, “Design of parallel concatenated convolutional codes,” IEEE Trans. Communications, vol. 44, pp. 591–600, May 1996. [16] ——, “Unveiling turbo codes: Some results on parallel concatenated coding schemes,” IEEE Trans. Information Theory, vol. 42, no. 2, pp. 408–428, March 1996. [17] Y. Berger and Y. Be’ery, “Bounds on the trellis size of linear block codes,” IEEE Trans. Information Theory, vol. 39, no. 1, pp. 203–209, January 1993. [18] C. Berrou, “The ten-year-old turbo codes are entering into service,” IEEE Com- munications Mag., vol. 41, no. 8, pp. 110–116, August 2003. [19] T. Beth, D. Jungnickel, and H. Lenz, Design Theory. Cambridge University Press, 1999. [20] R.E.Blahut,Algebraic Codes for Data Transmission. CambridgeUniversityPress, 1993. [21] B. Bollobas, Extremal Graph Theory. New York, NY: Academic Press, 1978. [22] A. R. Calderbank, G. D. Forney, Jr., and A. Vardy, “Minimal tail-biting trellises: The Golay code and more,” IEEE Trans. Information Theory, vol. 45, no. 5, pp. 1435–1455, July 1999. [23] F. Celler, C. R. Leedham-Green, S. H. Murray, A. C. Niemeyer, and E. A. O’Brien, “Generating random elements of a finite group,” Communications in Algebra, vol. 23, pp. 4931–4948, 1995. [24] J. Chen and M. P. C. Fossorier, “Near optimum universal belief propagation based decodingoflow-densityparitycheckcodes,” IEEE Trans. Communications,vol.50, no. 3, pp. 406–414, March 2002. [25] X.ChenandK.M.Chugg,“Iterativesoft-in/soft-outalgorithmsforarbitrarysparse channels,” IEEE Trans. Communications, 2006, accepted for publication. [26] X. Chen, K. M. Chugg, and M. A. Neifeld, “Near-optimal parallel distributed data detection for page-oriented optical memories,” IEEE J. Select. Topics Quantum Electron., vol. 4, no. 5, pp. 866–879, Sept./Oct. 1998, (special issue on Advanced Optical Storage Technologies). 178 [27] V. Chernyak, M. Chertkov, M. Stepanov, and B. Vasic, “Instaton method of post- error-correction analytical evaluation,” in Proc. IEEE Inform. Theory Workshop, San Antonio, TX, October 2004, pp. 220–224. [28] S. K. Christensen and H. Giese, “Genetic analysis of the obligate barley powdery mildew fungus based on RLFP and virulence loci,” Theor. Appl. Genetics, vol. 79, pp. 705–712, 1991. [29] T. Christof and A. L¨ obel, “PORTA - POlyhedron Rep- resentation Transformation Algorithm,” 2004. [Online]. Available: http://www.zib.de/Optimization/Software/Porta [30] K. M. Chugg, A. Anastasopoulos, and X. Chen, Iterative Detection: Adaptivity, Complexity Reduction, and Applications. Kluwer Academic Publishers, 2001. [31] K.M.Chugg, X.Chen, andM.A.Neifeld, “Two-dimensionalequalizationincoher- ent and incoherent page oriented optical memory,” J. Opt. Soc. Amer. A, vol. 16, no. 3, pp. 549–562, March 1999. [32] K. M. Chugg, X. Chen, A. Ortega, and C.-W. Chang, “An iterative algorithm for two-dimensional digital least metric problems with applications to digital image compression,” in Proc. Intl. Conf. Image Processing, Chicago, IL, 1998, pp. 722– 726. [33] K. M. Chugg, P. Thiennviboon, G. D. Dimou, P. Gray, and J. Melzer, “A new class of turbo-like codes with universally good performance and high-speed decoding,” in Proc. IEEE Military Comm. Conf., Atlantic City, NJ, October 2005. [34] S. Y. Chung, G. D. Forney, Jr., T. J. Richardson, and R. Urbanke, “On the design of low-density parity-check codes within 0.0045 dB of the Shannon limit,” IEEE Communications Letters, vol. 5, no. 2, pp. 58–60, February 2001. [35] S. Crozier, “New high-spread high-distance interleavers for turbo-codes,” in Proc. 20th Biennial Symp. Comm., Kingston, ON, Canada, 2000, pp. 3–7. [36] C. Di, D. Proietti, I. E. Teletar, T. J. Richardson, and R. L. Urbanke, “Finite- length analysis of low-density parity-check codes on the binary erasure channel,” IEEE Trans. Information Theory, vol. 48, no. 6, pp. 1570–1579, June 2002. [37] R. Diestel, Graph Theory, 2nd ed. New York, NY: Springer-Verlag, 2000. [38] D. Divsalar and F. Pollara, “Turbo codes for PCS applications,” in Proc. Interna- tional Conf. Communications, Seattle, WA, 1995, p. 5459. [39] ——, “Hybrid concatenated codes and iterative decoding,” JPL-TDA, Tech. Rep., August 1997, 42–130. [40] S. Dolinar, D. Divsalar, and F. Pollara, “Code performance as a function of block size,” Jet Propulsion Labs., Pasadena, CA, Tech. Rep. TDA Progress Report 42- 133, May 1998. 179 [41] D.Edwards,IntroductiontoGraphicalModelling,2nded. NewYork,NY:Springer- Verlag, 2000. [42] D.Eppstein,“Findingthekshortestpaths,”Dept.Info.andComp.Sci.,University of California, Irvine, CA, Tech. Rep., March 1997. [43] T. Etzion, A. Trachtenberg, and A. Vardy, “Which codes have cycle-free Tanner graphs?” IEEE Trans. Information Theory, vol. 45, no. 6, pp. 2173–2181, Septem- ber 1999. [44] J. Feldman, “Decoding error-correcting codes via linear programming,” Ph.D. dis- sertation, MIT Press, Cambridge, MA, 2003. [45] J. Feldman, M. J. Wainwright, and D. R. Karger, “Using linear programming to decode binary linear codes,” IEEE Trans. Information Theory, vol. 51, no. 3, pp. 954–972, March 2005. [46] J. Flum and M. Grohe, “The parameterized complexity of counting problems,” in Proc. IEEE Symp. Foundations of Comp. Sci., November 2002, pp. 538–547. [47] G.D.Forney,Jr.,“CosetcodesI:Introductionandgeometricalclassification,”IEEE Trans. Information Theory, vol. 34, pp. 1123–1151, 1988. [48] ——, “Coset codes II: Binary lattices andrelated codes,” IEEE Trans. Information Theory, vol. 34, pp. 1152–1187, 1988. [49] ——, “Density/length profiles and trellis complexity of lattices,” IEEE Trans. In- formation Theory, vol. 40, no. 6, pp. 1753–1771, November 1994. [50] ——, “Codes on graphs: Normal realizations,” IEEE Trans. Information Theory, pp. 520–548, February 2001. [51] ——, “Codes on graphs: Constraint complexity of cycle-free realizations of linear codes,” IEEE Trans. Information Theory, vol. 49, no. 7, pp. 1597–1610, July 2003. [52] G. D. Forney, Jr., F. R. Kschischang, B. Marcus, and S. Tuncel, Codes, Systems, and Graphical Models (Minneapolis, MN, 1999). New York, NY: Springer Verlag, 2001, ch. Iterative decoding of tail-biting trellises and connections with symbolic dynamics, pp. 239–264, eds.: B. Marcus and J. Rosenthal. [53] G. D. Forney, Jr. and M. D. Trott, “The dynamics of group codes: State spaces, trellisdiagrams,andcanonicalencoders,”IEEE Trans. Information Theory,vol.39, no. 9, pp. 1491–1513, September 1993. [54] M.P.C.FossorierandS.Lin,“Soft-inputsoft-outputdecodingoflinearblockcodes basedonorderedstatistics,”inProc. Globecom Conf.,Sydney,Australia,November 1998, pp. 2828–2833. [55] B. J. Frey, Graphical models for maschine learning and digital communications. Cambridge, MA: MIT Press, 1998. 180 [56] T. Fujiwara, T. Kasami, R. Morelos-Zaragoza, and S. Lin, “The state complexity of trellis diagrams for a class of generalized concatenated codes,” in Proc. IEEE International Symp. on Information Theory, Trondheim, Norway, June 1994, p. 471. [57] X.-P.Ge, D.Eppstein, andP.Smyth, “Thedistributionoflooplengthsingraphical models for turbo decoding,” IEEE Trans. Information Theory, vol. 47, no. 6, pp. 2549–2553, September 2001. [58] J. W. Gibbs, Elementary Principles of Statistical Mechanics. New Haven, CT: Yale University Press, 1902. [59] J. Hagenauer, E. Offer, and L. Papke, “Iterative decoding of binary block and convolutional codes,” IEEE Trans. Information Theory, vol. 42, no. 2, pp. 429–445, March 1996. [60] T. R. Halford, “Optimal soft-in soft-out decoding of Reed-Solomon codes,” Com- munication Sciences Institute, USC, Los Angeles, CA, Tech. Rep. CSI-04-05-03, April 2003. [61] ——, “Systematic extraction of good cyclic graphical models for inference,” Com- munication Sciences Institute, USC, Los Angeles, CA, Tech. Rep. CSI-05-01-02, November 2004, (Ph. D. Disertation proposal). [62] ——, “Conditionally cycle-free generalized Tanner graphs: Theory and applica- tions,” Communication Sciences Institute, USC, Los Angeles, CA, Tech. Rep. CSI- 06-09-01, September 2006. [63] T. R. Halford and K. M. Chugg, “Conditionally cycle-free generalized Tanner graphs: Theoryandapplicationtohigh-rateseriallyconcatenatedcodes,”submitted to ISIT 2007. [64] ——, “Random redundant iterative soft-in soft-out decoding,” submitted to IEEE Trans. Communications. [65] ——, “Counting and enumerating cycles in bipartite graphs,” in IEEE Communi- cation Theory Workshop (no proceedings), Capri Island, May 2004, (recent results session). [66] ——,“Analgorithmforcountingshortcyclesinbipartitegraphs,” IEEE Trans. In- formation Theory, vol. 52, no. 1, pp. 287–292, January 2006. [67] ——, “Conditionally cycle-free generalized Tanner graphs: Theory and application to high-rate serially concatenated codes,” Communication Sciences Institute, USC, Los Angeles, CA, Tech. Rep. CSI-06-09-01, September 2006. [68] ——, “The extraction and complexity limits of graphical models for linear codes,” submitted to the IEEE Trans. Information Theory, November 2006. 181 [69] ——, “Random redundant soft-in soft-out decoding of linear block codes,” July 2006, presented at ISIT 2006. [70] ——, “The tradeoff between cyclic topology and complexity in graphical models of linear codes,” in Proc. Allerton Conf. Commun., Control, Comp., September 2006. [71] T. R. Halford, A. J. Grant, and K. M. Chugg, “Which codes have 4-cycle-free Tanner graphs?” IEEE Trans. Information Theory, vol. 52, no. 9, pp. 4219–4223, September 2006. [72] ——, “Which codes have 4-cycle-free tanner graphs?” in Proc. IEEE International Symp. on Information Theory, July 2006. [73] T. R. Halford, V. Ponnampalam, A. J. Grant, and K. M. Chugg, “Soft-in soft- out decoding of Reed-Solomon codes based on Vardy and Be’ery’s decomposition,” IEEE Trans. Information Theory, vol. 51, no. 12, pp. 4363–4386, December 2005. [74] J.HeoandK.M.Chugg,“Constrainediterativedecoding: Performanceandconver- gence analysis,” in Proc. Asilomar Conf. Signals, Systems, Comp., Pacific Grove, CA, November 2001, pp. 275–279. [75] S.Hoory,“Thesizeofbipartitegraphswithagivengirth,” J. Comb. Theory, Series B, vol. 86, no. 2, pp. 215–220, 2002. [76] F.B.HornandF.R.Kschischang, “Ontheintractabilityofpermutingablockcode to minimize trellis complexity,” IEEE Trans. Information Theory, vol. 42, no. 6, pp. 2042–2048, November 1996. [77] C.-C. Hu and S.-H. Huang, “On bit-level trellis complexity of Reed-Muller codes,” IEEE Trans. Information Theory, vol. 41, no. 6, pp. 2061–2064, November 1995. [78] X.-Y. Hu, E. Eleftheriou, and D. M. Arnold, “Regular and irregular progressive edge-growth Tanner graphs,” IEEE Trans. Information Theory, vol. 51, no. 1, pp. 386–398, January 2005. [79] A. G. i Amat, S. Benedetto, and G. Montorsi, “Optimal high-rate convolutional codes for partial response channels,” in Proc. Globecom Conf., vol. 2, Taipei, Tai- wan, November 2002, pp. 1031–1036. [80] A. G. i Amat, G. Montorsi, and S. Benedetto, “New high-rate convolutional codes for concatenated schemes,” in Proc. International Conf. Communications, vol. 3, New York, NY, May 2002, pp. 1661–1666. [81] V. Isham, “An introduction to spatial point processes and Markov random fields,” Int. Stat. Rev., vol. 49, pp. 21–43, 1981. [82] F. V. Jensen, An Introduction to Bayesian Networks. Springer-Verlag, 1996. [83] J. Jiang and K. R. Narayanan, “Iterative soft decision decoding of Reed-Solomon codes,” IEEE Communications Letters, vol. 8, no. 4, pp. 244–246, April 2004. 182 [84] ——, “Iterative soft decision decoding of Reed-Solomon codes based on adaptive parity check matrices,” in Proc. IEEE International Symp. on Information Theory, Chicago, Il., June 2004, p. 261. [85] H. Jin, A. Khandekar, and R. McEliece, “Irregular repeat-accumulate codes,” in Turbo Code Conf., Brest, France, 2000. [86] S.J.JohnsonandS.R.Weller,“Resolvable2-designsforregularlow-densityparity- checkcodes,” IEEE Trans. Communications, vol.51, no.9, pp.1413–1419, Septem- ber 2003. [87] T. Kasami, T. Takata, T. Fujiwara, and S. Lin, “On complexity of trellis structure of linear block codes,” IEEE Trans. Information Theory, vol. 39, no. 3, pp. 1057– 1064, May 1993. [88] ——, “On structural complexity of thel-section minimal trellis diagrams for binary linear block codes,” IEICE Trans., vol. E76-A, pp. 1411–1421, January 1993. [89] ——, “On the optimum bit orders with respect to the state complexity of trellis diagrams for binary linear codes,” IEEE Trans. Information Theory, vol. 39, no. 1, pp. 242–245, January 1993. [90] R. Kindermann and J. L. Snell, Markov Random Fields and their Applications. American Mathematical Society, 1980. [91] R. Koetter, W.-C. W. Li, P. O. Vontobel, and J. L. Walker, “Pseudo-codewords of cycle codes via zeta functions,” in Proc. IEEE Inform. Theory Workshop, San Antonio, TX, October 2005, pp. 7–12. [92] R. Koetter and A. Vardy, “The structure of tail-biting trellises: Minimality and ba- sicprinciples,”IEEE Trans. Information Theory,vol.49,pp.2081–2105,September 2003. [93] A.KothiyalandO.Y.Takeshita,“Acomparisonofadaptivebeliefpropagationand the best graph algorithm for the decoding of linear block codes,” in Proc. IEEE International Symp. on Information Theory, Adelaide, Australia, September 2005, pp. 724–728. [94] A. Kothiyal, O. Y. Takeshita, W. Jin, and M. Fossorier, “Iterative reliability-based decoding of linear block codes with adaptive belief propagation,” IEEE Communi- cations Letters, vol. 9, no. 12, pp. 1067–1069, December 2005. [95] Y. Kou, J. Xu, H. Tan, S. Lin, and K. Abdel-Ghaffar, “On circulant low density parity check codes,” in Proc. IEEE International Symp. on Information Theory, Lausanne, Switzerland, June 2002, p. 200. [96] K. M. Krishnan and P. Shahkar, “On the complexity of finding stopping distance,” December 2005, arXiv:cs.IT/0512101. 183 [97] F. Kschischang, B. Frey, and H.-A. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Trans. Information Theory, vol. 47, pp. 498–519, Feb. 2001. [98] V. Kumar and O. Milenkovic, “On graphical representations for algebraic codes suitable for iterative decoding,” IEEE Communications Letters, vol. 9, no. 8, pp. 729–731, August 2005. [99] A. Lafourcade and A. Vardy, “Lower bounds on trellis complexity of block codes,” IEEE Trans. Information Theory, vol. 41, no. 6, pp. 1938–1954, November 1995. [100] ——, “Optimal sectionalization of a trellis,” IEEE Trans. Information Theory, vol. 42, no. 3, pp. 689–703, May 1996. [101] S. L. Lauritzen and D. J. Spiegelhalter, “Local computation with prob- abilities on graphical structures and their application to expert systems,” J. Roy. Statist. Soc. B, pp. 157–224, 1988. [102] J. Leon, “Computing automorphism groups of error-correcting codes,” IEEE Trans. Information Theory, vol. 28, no. 3, pp. 496–511, May 1982. [103] J. Li, K. R. Narayanan, E. Kurtas, and C. N. Georghiades, “On the performance of high-rate TPC/SPC codes and LDPC codes over partial response channels,” IEEE Trans. Communications, vol. 50, no. 5, pp. 723–734, May 2002. [104] S.Lin,T.Kasami,T.Fujiwara,andM.Fossorier,Trellises and Trellis-Based Decod- ing Algorithms for Linear Block Codes. Norwell,MS:KluwerAcademicPublishers, 1998. [105] C.-C. Lu and L. R. Welch, “On automorphism groups of binary primitive BCH codes,” in Proc. IEEE International Symp. on Information Theory, Trondheim, Norway, June 1994, p. 51. [106] R. Lucas, M. P. C. Fossorier, Y. Kou, and S. Lin, “Iterative decoding of one-step majority logic decodable codes based on belief propagation,” IEEE Trans. Com- munications, vol. 48, no. 6, pp. 931–937, June 2000. [107] D. J. C. MacKay. Encyclopedia of Sparse Graph Codes. [Online]. Available: http://www.inference.phy.cam.ac.uk/mackay/codes/data.html [108] ——, “Relationships between sparse graph codes,” in Proc. Information-Based In- duction Sciences, Izu, Japan, July 2000. [109] D.J.C.MacKayandR.M.Neal, Codes, Systems and Graphical Models. Springer- Verlag, 2000, ch. Evaluation of Gallager codes for short block lengths and high rate applications, pp. 113–130, eds.: B. Marcus and J. Rosenthal. [110] D. J. C. MacKay and M. S. Postol, “Weaknesses of Margulis and Ramanujan- Margulis low-density parity-check code,” Electronic Notes in Theoretical Computer Science, vol. 74, 2003. 184 [111] F. J. MacWilliams and N. J. A. Sloane, The Theory of Error-Correcting Codes. North-Holland, 1978. [112] Y. Mao and A. H. Banihashemi, “A heuristic search for good low-density parity- check codes at short block lengths,” in Proc. International Conf. Communications, vol. 1, Helsinki, Finland, June 2001, pp. 41–44. [113] P. A. Martin, D. P. Taylor, and M. P. C. Fossorier, “Soft-input soft-output list- based decoding algorithm,” IEEE Trans. Information Theory, vol. 52, no. 2, pp. 252–262, February 2004. [114] R. J. McEliece, “On the BCJR trellis for linear block codes,” IEEE Trans. Infor- mation Theory, vol. 42, no. 4, pp. 1072–1092, July 1996. [115] J. McGowan and R. C. Williamson, “Loop removal from LDPC codes,” 2003 IEEE Inform. Theory Workshop, 2003. [116] G. Montorsi and S. Benedetto, “An additive version of the SISO algorithm for the dual code,” in Proc. IEEE International Symp. on Information Theory, Washing- ton, DC, June 2001. [117] D. J. Muder, “Minimal trellises for block codes,” IEEE Trans. Information Theory, vol. 34, no. 5, pp. 1049–1053, September 1988. [118] S. Neuwirth, “The size of bipartite graphs with girth eight,” February 2001, arXiv:math/0102210. [119] P. Oswald and A. Shokrollahi, “Capacity-achieving sequences for the erasure chan- nel,” IEEE Trans. Information Theory, vol. 48, no. 12, pp. 3017–3028, December 2002. [120] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible In- ference. Morgan Kaufmann, 1988. [121] H. D. Pfister, I. Sason, and R. Urbanke, “Capacity-achieving ensembles for the bi- nary erasure channel with bounded complexity,” IEEE Trans. Information Theory, vol. 51, no. 7, pp. 2352–2379, July 2005. [122] V. Pless, “Decoding the Golay codes,” IEEE Trans. Information Theory, vol. 32, no. 4, pp. 561–567, July 1986. [123] V. Ponnampalam, September 2004, private correspondence. [124] V. Ponnampalam and A. Grant, “An efficient SISO for Reed-Solomon codes,” in Proc. IEEE International Symp. on Information Theory, Yokohama, Japan, 2003. [125] V. Ponnampalam and B. Vucetic, “Soft decision decoding of Reed-Solomon codes,” IEEE Trans. Communications, vol. 50, pp. 1758–1768, November 2002. [126] C.J.Preston, Gibbs States on Countable Sets. Cambridge,U.K.: CambridgeUniv. Press, 1974. 185 [127] A. Ramamoorthy and R. D. Wesel, “Analysis of an algorithm for irregular LDPC code construction,” in Proc. IEEE International Symp. on Information Theory, Chicago, IL, June 2004, p. 69. [128] A. B. Reid, A. J. Grant, and P. D. Alexander, “List detection for multi-access channels,” in Proc. Globecom Conf., vol. 2, November 2002, pp. 1083–1087. [129] I. Reiman, “ ¨ Uber ein problem von K. Zarankiewicz,” Acta. Math. Acad. Sci. Hun- gary, vol. 9, pp. 269–273, 1958. [130] T. Richardson, “Error floors of LDPC codes,” in Proc. Allerton Conf. Commun., Control, Comp., Monticello, IL, October 2003. [131] T. Richardson, M. Shokrollahi, and R. Urbanke, “Design of capacity-approaching irregularlow-densityparity-checkcodes,”IEEE Trans. Information Theory,vol.47, no. 2, pp. 619–673, February 2001. [132] S. Sankaranarayanan and B. Vasi´ c, “Iterative decoding of linear block codes: A parity-check orthogonalization approach,” IEEE Trans. Information Theory, vol. 51, no. 9, pp. 3347–3353, September 2005. [133] T. Schaub, “A linear complexity approach to cyclic codes,” Ph.D. dissertation, Swiss Federal Institute of Technology, Zurich, CH, 1998. [134] M.SchwartzandA.Vardy,“Onthestoppingdistanceandthestoppingredundancy of codes,” in Proc. IEEE International Symp. on Information Theory, Adelaide, Australia, September 2005, pp. 975–979. [135] N. Sendrier and G. Skersys, “On the computation of the automorphism group of a linear code,” in Proc. IEEE International Symp. on Information Theory, Washing- ton, DC, June 2001, p. 13. [136] Y. Shany and Y. Be’ery, “Linear tail-biting trellises, the square-root bound, and applications for Reed-Muller codes,” IEEE Trans. Information Theory, vol. 46, no. 4, pp. 1514–1523, July 2000. [137] A. Shokrollahi, “LDPC codes: An introduction,” Digital Fountain Inc., Tech. Rep., December 2002. [138] J. Snyders and Y. Be’ery, “Maximum likelihoodsoft decoding of binary block codes and decoders for the golay codes,” IEEE Trans. Information Theory, vol. 35, no. 5, pp. 963–975, September 1989. [139] M. Stepanov, V. Chernyak, M. Chertkov, and B. Vasic, “Diagnosis of weaknesses in modern error correction codes: A physics approach,” June 2005, available online under http://www.arxiv.org/cond-mat/0506037. [140] R. M. Tanner, “A recursive approach to low complexity codes,” IEEE Trans. In- formation Theory, vol. IT-27, pp. 533–547, September 1981. 186 [141] S. ten Brink, “Convergence behavior of iteratively decoded parallel concatenated codes,” IEEE Trans. Communications, vol. 49, no. 10, pp. 1727–1737, October 2001. [142] A. G. Thomason, “Hamiltonian cycles and uniquely edge-colourable graphs,” An- nals of Discrete Mathematics, vol. 3, pp. 259–268, 1978. [143] T. Tian, C. R. Jones, J. D. Villasenor, and R. D. Wesel, “Selective avoidance of cycles in irregular LDPC code construction,” IEEE Trans. Information Theory, vol. 52, no. 8, pp. 1242–1247, August 2004. [144] A.Vardy, “Theintractabilityofcomputingtheminimumdistanceofacode,” IEEE Trans. Information Theory, vol. 43, no. 6, pp. 1757–1766, November 1997. [145] ——, Handbook of Coding Theory. Amsterdam, The Netherlands: Elsevier, 1999, ch. Trellis structure of codes, eds.: V. S. Pless and W. C. Huffman. [146] A. Vardy and Y. Be’ery, “Bit level soft-decision decoding of Reed-Solomon codes,” IEEE Trans. Communications, vol. 39, pp. 440–444, March 1991. [147] ——, “More efficient soft decoding of Golay codes,” IEEE Trans. Information The- ory, vol. 37, no. 3, pp. 667–672, May 1991. [148] ——, “Maximum-likelihoodsoftdecisiondecodingofBCHcodes,” IEEE Trans. In- formation Theory, vol. 40, no. 2, pp. 546–554, March 1994. [149] P. O. Vontobel and R. Koetter, “Graph-cover decoding and finite-length analysis of message-passing iterative decoding of LDPC codes,” December 2005, submitted to IEEE Trans. Information Theory. [150] Y. Wang, R. Ramesh, A. Hassan, and H. Koorapaty, “On MAP decoding for tail- biting convolutional codes,” in Proc. IEEE International Symp. on Information Theory, Ulm, Germany, June 1997, p. 225. [151] N.Wiberg, “Codesanddecodingongeneralgraphs,” Ph.D.dissertation, Link¨ oping University (Sweden), 1996. [152] N.Wiberg,H.-A.Loeliger,andR.K¨ otter,“Codesanditerativedecodingongeneral graphs,” European Trans. Telecommun., vol. 6, no. 5, pp. 513–525, Sept-Oct. 1995. [153] J. C. Willems, Dynamics Reported, Volume 2. New York, NY: Wiley, 1989, ch. Models for Dynamics, pp. 171–269, eds.: U. Kirchgraber and H. O. Walther. [154] ——, “Paradigms and puzzles in the theory of dynamical systems,” IEEE Trans. Automatic Control, vol. 42, pp. 259–294, March 1991. [155] J. K. Wolf, “Efficient maximum-likelihood decoding of linear block codes using a trellis,” IEEE Trans. Information Theory, vol. 24, pp. 76–80, 1978. 187 [156] J. S. Yedidia, J. Chen, and M. C. Fossorier, “Generating code representations suit- able for belief propagation decoding,” in Proc. Allerton Conf. Commun., Control, Comp., Monticello, IL, October 2002. [157] O. Yeung and K. M. Chugg, “An iterative algorithm and low complexity hard- ware architecture for fast acquisition of long PN codes in UWB systems,” in Proc. Allerton Conf. Commun., Control, Comp., Allerton House, Illinois, October 2005. [158] ——, “An iterative algorithm and low complexity hardware architecture for fast acquisition of PN codes in UWB systems,” Springer J. VLSI Signal Processing (special issue on Ultrawideband Systems), vol. 43, no. 1, April 2006. [159] H. Zhang and J. M. F. Moura, “The design of structured regular LDPC codes with large girth,” in Proc. Globecom Conf., Taipei, Taiwan, December 2003, pp. 4022–4027. 188 Appendix Counting Short Cycles in Bipartite Graphs Exactly counting cycles and paths in arbitrary graphs is known to be intractable [46]. Alon, Yuster, and Zwick presented methods for counting cycles of length less than eight in [3]; however, their methods are prohibitively complex for longer cycles. This appendix addressestherestrictedproblemofcountingcyclesoflengthg,g+2,andg+4inbipartite graphs with girth g. The algorithm described in this appendix was first presented at the 2004 Communication Theory Workshop in Capri, Italy [65] and later appeared in the IEEE Transactions on Information Theory [66]. The remainder of this appendix is organized as follows. Section A.1 presents a brief review of the required graph theory and introduces the notation used throughout this appendix. SectionA.2presentsarecursionusedtocountpathsandthuscyclesinbipartite graphs. Section A.3 presents the proof technique used to solve said recursion. Section A.4 presents the specific expressions required to count cycles of length g, g+2, and g+4 in bipartite graphs with girth g. The proofs of the expressions presented in Section A.4 are detailed in Section A.5. Section A.6 presents a number of examples of the counting algorithm. Finally, the complexity of the proposed algorithm is discussed in Section A.7. 189 A.1 Review of Graph Theory and Notation Throughout this appendix, the shorthand [a,b] is used to abbreviate the set {a,a+1,...,b−1,b}⊂Z (A.1) wherea<b∈Z(thesetofintegers). Similary,[c,d] e and[e,f] o areusedtoabbreviatesets of consecutive even and odd integers, respectively, where c<d∈2Z and e<f∈2Z+1. A.1.1 Graph Theory A graphG = (V,E,H) consists of a finite non-empty set of verticesV, a set of edgesE which is any subset of the pairs{{u,v}:u,v∈V,u6=v}, and a set of half-edgesH which is any subset ofV. While it is non-standard to define graphs with half-edges, as seen in Chapter 2, half-edges are useful in the context of graphical models for codes. When H =∅, as is assumed in the remainder of this appendix, a graph can be simply defined by the pairG =(V,E). A walk of length n inG = (V,E) is a sequence of vertices α 1 ,α 2 ,··· ,α n ,α n+1 such that α i ∈V and{α i ,α i+1 }∈E for all i∈ [1,n]. A walk of length n is termed closed if α 1 = α n+1 ; that is, if α 1 and α n+1 correspond to the same vertex v∈V. A walk is a trail if the edges that form the walk are distinct; a trail is a path if the vertices are distinct. A closed walk of length n is a cycle if α 1 ,α 2 ,··· ,α n are distinct; that is, if α i 6= α j for all i6= j∈ [1,n]. The lengths of the shortest and longest cycles of a graph are respectively denoted its girth, g, and circumference, c [37]. A graph with no cycles is a tree and has g =∞ and c = 0. Two vertices u,v∈V are adjacent if a single edge 190 {u,v}∈E connects u to v. A graph is connected if any two of its vertices are linked by a walk. A cut in a connected graphG is some subset of edgesX⊆E the removal of which yields a disconnected graph. Cuts thus partition the vertex setV. The term (m,n−m)-lollipop 1 walk was introduced in [65, 66] to describe length n walks where α 1 ,α 2 ,··· ,α n are distinct and α n+1 = α m+1 for some m∈ [1,n]. Cycles of length n are thus (0,n)-lollipop walks. Figures A.1 and A.2 illustrate a (4,2)-lollipop walk and a (2,4)-lollipop walk, respectively. α 1 α 2 α 3 α 4 α 5 α 6 α 7 v 1 v 3 v 4 v 6 v 2 v 5 Figure A.1: A (4,2)-lollipop walk where α 7 =α 5 while α 1 ,α 2 ,...,α 6 are distinct. α 1 α 2 α 3 α 4 α 5 α 6 α 7 v 1 v 2 v 3 v 4 v 5 v 6 Figure A.2: A (2,4)-lollipop walk where α 7 =α 3 while α 1 ,α 2 ,...,α 6 are distinct. A graphG = (V,E) is bipartite with vertex classesU ={u i } |U| i=1 andW ={w i } |W| i=1 if V =U∪W,U∩W =∅, and each edge joins a vertex inU to one inW. All cycles in bipartite graphs contain an even number of edges. Consequently, all (m,n−m)-lollipop walks in a bipartite graph satisfy n−m≡0 mod 2. Associated with the bipartite graph G = (U∪W,E) is the|U|×|W| edge matrix E = [e ij ] where e ij = 1 if (u i ,w j )∈E and 0 otherwise. Finally, the number of cycles of length 2m in a bipartite graph is denoted N 2m . 1 This terminology is nonstandard and should not be confused with the lollipop graph introduced by Thomason in [142]. 191 A.1.2 Matrix Notation Thefollowingmatrixnotationisusedthroughoutthisappendix. LetAandB bematrices. The matrix product is denoted AB while A T is the transpose of A. If A = [a ij ] and B =[b ij ] have the same dimensions then the direct (element-wise) product is defined by C =A◦B =[c ij ] where c ij =a ij b ij . (A.2) If A is a square matrix then Tr(A) is the trace of A and theZ[·] operator is defined by: Z[A]=A−A◦I (A.3) where I is the identity. ThusZ[A] is A with the diagonal elements replaces by zeros. The following notation is also employed for arbitrary matrices A=[a ij ] where k∈Z: B = A k =[b ij ] where b ij = a ij k (A.4) = a ij ! k!(a ij −k)! B =A+k =[b ij ] where b ij =a ij +k (A.5) B =kA=[b ij ] where b ij =ka ij (A.6) B =max(A,k)=[b ij ] where b ij =max(a ij ,k). (A.7) 192 A.2 The Lollipop Recursion This section presents a recursion that transforms the problem of counting paths and thus cycles in a bipartite graph into the problem of counting lollipop walks. Eight matrices that count the number of paths and lollipop walks in a bipartite graphG = (U∪W,E) withedgematrixE mustfirstbedefined. LetP U 2k bea|U|×|U|matrixwherethe(i,j)-th element is the number of paths of length 2k from u i ∈U to u j ∈U; similarly define P W 2k . The diagonal elements of P U 2k and P W 2k are necessarily zero. Let P U 2k+1 be a|U|×|W| matrix where the (i,j)-th element is the number of paths of length 2k+1 from u i ∈U to w j ∈W; similarly define P W 2k+1 . Let L U (2k 0 ,2k−2k 0 ) be a|U|×|U| matrix where the (i,j)-th element is the number of (2k 0 ,2k−2k 0 )-lollipop walks from u i ∈U to u j ∈U; similarly define L W (2k 0 ,2k−2k 0 ) . Let L U (2k 0 +1,2k−2k 0 ) be a|U|×|W| matrix where the (i,j)-th element is the number of (2k 0 +1,2k−2k 0 )-lollipop walks from u i ∈U to w j ∈W; similarly define L W (2k 0 +1,2k−2k 0 ) . Based on the above definitions, there is a simple recursion for the path matrices with the initial conditions P U 0 =I, P W 0 =I and L U (0,0) =L W (0,0) =0: P U 2k+1 =P U 2k E− k−1 X i=0 L U (2i+1,2k−2i) (A.8) P U 2k =P U 2k−1 E T − k−1 X i=0 L U (2i,2k−2i) (A.9) P W 2k+1 =P W 2k E T − k−1 X i=0 L W (2i+1,2k−2i) (A.10) P W 2k =P W 2k−1 E− k−1 X i=0 L W (2i,2k−2i) . (A.11) 193 To prove (A.8), note that a path of length 2k + 1, α 1 ,α 2 ,··· ,α 2k+2 , is formed by augmenting a single edge to a path of length 2k, α 1 ,α 2 ,··· ,α 2k+1 , such that α 2k+2 6=α i ∀i∈ [1,2k +1]. All walks α 1 ,α 2 ,··· ,α 2k+1 ,α 2k+2 where the first 2k +1 vertices form a path are counted by P U 2k E. The product P U 2k E thus counts both length 2k +1 paths and all length 2k+1 lollipop walks, hence (A.8). Equations (A.9)-(A.11) are proved in a similar manner. The recursion defined by equations (A.8)-(A.11) transforms the problem of counting paths and cycles to that of counting lollipop walks and is thus denoted the lollipop recursion. Recall that cycles of length 2k are (0,2k)-lollipop walks. Since (0,2k)-lollipop walks are paths of length 2k−1 with an added edge such that α 2k+1 = α 1 , there is a simple expression for counting the number of cycles of length 2k passing through each vertex: L U (0,2k) = P U 2k−1 E T ◦I (A.12) L W (0,2k) = P W 2k−1 E ◦I. (A.13) The total number of cycles of length 2k is thus N 2k = 1 2k Tr L U (0,2k) = 1 2k Tr L W (0,2k) . (A.14) 194 It is clear from equations (A.12) and (A.13) that in order to count cycles of length g, g+2, andg+4 in a bipartite graph of girthg, the lollipop recursion must be solved up to andincluding2k+1=g+3. Inabipartitegraphofgirthg,thereareno(m,n−m)-lollipop walks for which n−m<g. Expanding equations (A.8)-(A.11) yields, for 0<k < g 2 : P U 2k =P U 2k−1 E T −L U (2k−2,2) (A.15) P W 2k =P W 2k−1 E−L W (2k−2,2) (A.16) P U 2k+1 =P U 2k E−L U (2k−1,2) (A.17) P W 2k+1 =P W 2k E T −L W (2k−1,2) (A.18) and for k≥ g 2 : P U g =P U g−1 E T −L U (0,g) −L U (g−2,2) (A.19) P W g =P W g−1 E−L W (0,g) −L W (g−2,2) (A.20) P U g+1 =P U g E−L U (1,g) −L U (g−1,2) (A.21) P W g+1 =P W g E T −L W (1,g) −L W (g−1,2) (A.22) P U g+2 =P U g+1 E T −L U (0,g+2) −L U (2,g) −L U (g,2) (A.23) P W g+2 =P W g+1 E−L W (0,g+2) −L W (2,g) −L W (g,2) (A.24) P U g+3 =P U g+2 E−L U (1,g+2) −L U (3,g) −L U (g+1,2) (A.25) P W g+3 =P W g+2 E T −L W (1,g+2) −L W (3,g) −L W (g+1,2) . (A.26) 195 Expressions and detailed proofs for the (m,n−m)-lollipop walk matrices in (A.15)- (A.26) for which m> 0 are given in Sections A.4 and A.5, respectively. In the following section, a general proof structure for the lollipop matrix equations is detailed. A.3 Lollipop Matrix Proof Structure Figure A.3 illustrates an (m,n−m)-lollipop walk where m is even. 2 Throughout the remainder of this appendix, lollipop walks are illustrated with vertices from classU rep- resented by white circles and vertices from classW represented by black circles. The lollipop matrix L U (m,n−m) thus counts walks of the form shown in Figure A.3. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α m+1 α m+2 α n+1 α n Figure A.3: An (m,n−m)-lollipop walk from α 1 to α n+1 =α m+1 where m is even. TheproductEL W (m−1,n−m) counts(m,n−m)-lollipopwalksbutalsowalkswhereα 1 = α i for i∈[3,n−1] o . In order to count (m,n−m)-lollipop walks, the inclusion/exclusion principle that was used to counts paths in equations (A.8)-(A.11) is applied: expressions for walks where α 1 = α i for i∈ [3,n−1] o are found and subtracted (or excluded) from the product EL W (m−1,n−m) . As an example, consider the case where m = 2 and n = 8 illustrated in Figure A.4. The product EL W (1,6) counts both (2,6)-lollipop walks and walks where α 1 =α 3 , α 1 =α 5 , 2 The case where m is even is presented without loss of generality; the proposed proof technique also applies to the case where m is odd. 196 α 1 α 2 α 3 α 4 α 5 α 6 α 7 α 8 α 9 Figure A.4: A (2,6)-lollipop walk from α 1 to α 9 =α 3 . andα 1 =α 7 . Inordertocount(2,6)-lollipopwalks, expressionsforwalkswhereα 1 =α 3 , α 1 = α 5 , and α 1 = α 7 are found and then subtracted (or excluded) from the product EL W (1,4) . As m and n get large, the number of terms that must be excluded from the product EL W (m−1,n−m) also gets large. However, many of these terms are zero as they violate the girth constraint. Returning to the m = 2, n = 8 example illustrated in Figure A.4, if g =6thenneitherwalkswhereα 1 =α 5 norα 1 =α 7 arepossiblesincethesecasesrequire the existence of a length 4 cycle. A.4 Summary of the Lollipop Matrix Equations In the following, only expressions for the L U (m,n−m) lollipop matrices are given explicitly. InordertoobtainanexpressionforL W (m,n−m) fromthecorrespondingL U (m,n−m) expression, exchange allU’s andW’s and replace E with E T . For example, L U (1,2) =Emax L W (0,2) −1,0 (A.27) becomes L W (1,2) =E T max L U (0,2) −1,0 (A.28) 197 and, L U (2,2) =Z h EL W (1,2) i (A.29) becomes L W (2,2) =Z h E T L U (1,2) i . (A.30) Equations (A.31) and (A.32) hold for 1<k < g 2 : L U (2k−1,2) =EL W (2k−2,2) −max L U (0,2) −1,0 L U (2k−3,2) (A.31) L U (2k,2) =EL W (2k−1,2) −max L U (0,2) −1,0 L U (2k−2,2) . (A.32) In equations (A.33)-(A.39), Kronecker delta notation, where δ i,j = 1 if i = j and 0 otherwise, is employed in order to present the g = 4, g = 6, and g > 6 cases in a unified fashion: L U (g−1,2) =EL W (g−2,2) −max L U (0,2) −1,0 L U (g−3,2) −P U g−1 ◦E (A.33) L U (g,2) =Z h EL W (g−1,2) i −max L U (0,2) −1,0 L U (g−2,2) +P U g−2 ◦P U 2 −δ g,4 P U 2 (A.34) L U (g+1,2) =EL W (g,2) −max L U (0,2) −1,0 L U (g−1,2) +L U (g−1,2) ◦E −L U (0,g) L U (1,2) +2 h P U g−1 max L W (0,2) −2,0 i ◦E +2P U g−1 ◦E +2δ g,4 P U 2 2 E−P U 3 ◦E−P U g+1 ◦E (A.35) L U (1,g) =EL W (0,g) −2P U g−1 ◦E (A.36) L U (2,g) =Z h EL W (1,g) i −6δ g,4 P U 2 3 (A.37) 198 L U (3,g) =EL W (2,g) −max L U (0,2) −1,0 L U (1,g) −6δ g,6 P U 3 3 −4δ g,4 P U 3 2 ◦E +6δ g,4 E P W 2 2 −P U 3 ◦E +4δ g,4 P U 2 2 E−P U 3 ◦E (A.38) L U (1,g+2) =EL W (0,g+2) −2P U g−1 ◦E−2δ g,4 P U 3 2 ◦E +2δ g,4 E P W 2 2 −P U 3 ◦E +2δ g,4 P U 2 2 E−P U 3 ◦E. (A.39) A.5 Detailed Proofs for the Lollipop Matrix Equations Proofs are provided only for the L U (m,n−m) equations in this section; the L W (m,n−m) equa- tions are proved analogously. A.5.1 Counting (1,2)-lollipop walks L U (1,2) =Emax L W (0,2) −1,0 (A.40) L W (1,2) =E T max L U (0,2) −1,0 (A.41) Walks of the form shown in Figure A.5 from α 1 to α 4 = α 2 are counted by L U (1,2) . The product EL W (0,2) counts (1,2)-lollipop walks and walks where α 1 = α 3 . Walks where α 1 =α 3 are excluded by instead considering the product Emax L W (0,2) −1,0 . α 1 α 2 α 3 α 4 Figure A.5: A (1,2)-lollipop walk from α 1 to α 4 =α 2 . 199 A.5.2 Counting (2,2)-lollipop walks L U (2,2) =Z h EL W (1,2) i (A.42) L W (2,2) =Z h E T L U (1,2) i (A.43) Walks of the form shown in Figure A.6 from α 1 to α 5 = α 3 are counted by L U (2,2) . The product EL W (1,2) counts (2,2)-lollipop walks and walks where α 1 = α 3 . Eliminating the diagonal elements of EL W (1,2) with theZ[·] operator excludes the α 1 =α 3 walks. α 3 α 5 α 4 α 2 α 1 Figure A.6: A (2,2)-lollipop walk from α 1 to α 5 =α 3 . A.5.3 Counting (2k−1,2)-lollipop walks for 1 < k < g 2 L U (2k−1,2) =EL W (2k−2,2) −max L U (0,2) −1,0 L U (2k−3,2) (A.44) L W (2k−1,2) =E T L U (2k−2,2) −max L W (0,2) −1,0 L W (2k−3,2) (A.45) Walks of the form shown in Figure A.7 from α 1 to α 2k+2 =α 2k are counted by L U (2k−1,2) . The productEL W (2k−2,2) counts (2k−1,2)-lollipop walks along with walks whereα 1 =α 3 . Note that walks where α 1 =α i for i∈[5,2k+1] o violate the girth constraint. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α 2k α 2k+1 α 2k+2 α 2k−1 Figure A.7: A (2k−1,2)-lollipop walk from α 1 to α 2k+2 =α 2k . TheproductL U (0,2) L U (2k−3,2) countswalkswhereα 1 =α 3 andwalkswherebothα 1 =α 3 and α 2 = α 4 . Note that walks where α 1 = α 3 and α 2 = α j for j∈ [6,2k] e violate the 200 girth constraint. By instead considering the product max L U (0,2) −1,0 L U (2k−3,2) , walks where α 1 =α 3 and α 2 =α 4 are excluded. A.5.4 Counting (2k,2)-lollipop walks for 1 < k < g 2 L U (2k,2) =EL W (2k−1,2) −max L U (0,2) −1,0 L U (2k−2,2) (A.46) L W (2k,2) =E T L U (2k−1,2) −max L W (0,2) −1,0 L W (2k−2,2) (A.47) Walks of the form shown in Figure A.8 from α 1 to α 2k+3 =α 2k+1 are counted by L U (2k,2) . The product EL W (2k−1,2) counts (2k,2)-lollipop walks along with walks where α 1 = α 3 . Note that walks where α 1 =α i for i∈[5,2k+1] o violate the girth constraint. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α 2k α 2k+1 α 2k+2 α 2k+3 Figure A.8: A (2k,2)-lollipop walk from α 1 to α 2k+3 =α 2k+1 . TheproductL U (0,2) L U (2k−2,2) countswalkswhereα 1 =α 3 andwalkswherebothα 1 =α 3 and α 2 =α 4 . Note that walks where α 1 =α 3 and α 2 =α j for j∈[6,2k+2] e violate the girth constraint. By instead considering the product max L U (0,2) −1,0 L U (2k−2,2) , walks where α 1 =α 3 and α 2 =α 4 are excluded. 201 A.5.5 Counting (g−1,2)-lollipop walks L U (g−1,2) =EL W (g−2,2) −max L U (0,2) −1,0 L U (g−3,2) −P U g−1 ◦E (A.48) L W (g−1,2) =E T L U (g−2,2) −max L W (0,2) −1,0 L W (g−3,2) −P W g−1 ◦E T (A.49) Walks of the form shown in Figure A.9 from α 1 to α g+2 = α g are counted by L U (g−1,2) . The product EL W (g−2,2) counts (g−1,2)-lollipop walks, walks where α 1 = α 3 and walks where α 1 = α g+1 . Note that walks where α 1 = α i for i∈ [5,g−1] o violate the girth constraint. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α g+1 α g+2 α g α g−1 Figure A.9: A (g−1,2)-lollipop walk from α 1 to α g+2 =α g . A.5.5.1 Walks where α 1 =α 3 The product L U (0,2) L U (g−3,2) counts walks where α 1 = α 3 and walks where both α 1 = α 3 and α 2 =α 4 . Note that walks where α 1 =α 3 and α 2 =α j for j∈[6,g] e violate the girth constraint. By instead considering the product max L U (0,2) −1,0 L U (g−3,2) , walks where α 1 =α 3 and α 2 =α 4 are excluded. A.5.5.2 Walks where α 1 =α g+1 A walk where α 1 =α g+1 requires that α 1 and α g be connected by both a path of length g−1 and a single edge. The direct product P U g−1 ◦E counts such walks. 202 A.5.6 Counting (g,2)-lollipop walks L U (g,2) =Z h EL W (g−1,2) i −max L U (0,2) −1,0 L U (g−2,2) +P U g−2 ◦P U 2 −δ g,4 P U 2 (A.50) L W (g,2) =Z h E T L U (g−1,2) i −max L W (0,2) −1,0 L W (g−2,2) +P W g−2 ◦P W 2 −δ g,4 P W 2 (A.51) Walks of the form shown in Figure A.10 from α 1 to α g+3 = α g+1 are counted by L U (g,2) . TheproductEL W (g−1,2) counts(g,2)-lollipopwalks, walkswhereα 1 =α 3 andwalkswhere α 1 = α g+1 . Walks where α 1 = α i for i∈ [5,g−1] o require the existence of a cycle of length less than g and thus violate the girth constraint. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α g α g+1 α g+2 α g+3 Figure A.10: A (g,2)-lollipop walk from α 1 to α g+3 =α g+1 . A.5.6.1 Walks where α 1 =α 3 The product L U (0,2) L U (g−2,2) counts walks where α 1 = α 3 along with walks where α 1 = α 3 and α 2 = α 4 and walks where α 1 = α 3 and α 2 = α g+2 . Note that walks where α 1 = α 3 andα 2 =α j forj∈[6,g] e violatethegirthconstraint. Walkswhereα 2 =α 4 areexcluded by instead considering the product max L U (0,2) −1,0 L U (g−2,2) . A walk where α 2 =α g+2 requires that α 1 and α g+1 be connected by both a path of length g−2 and a path of length 2. The direct product P U g−2 ◦P U 2 counts such walks. A.5.6.2 Walks where α 1 =α g+1 Eliminating the diagonal elements of EL W (g−1,2) with theZ[·] operator removes the α 1 = α g+1 walks. 203 A.5.6.3 The special case of g =4 When g = 4 the direct product P U 2 ◦ P U 2 includes pairs of identical length 2 paths. This double counting is remedied by replacing P U 2 ◦ P U 2 with 2 P U 2 2 or, equivalently, P U 2 ◦P U 2 −P U 2 . A.5.7 Counting (g+1,2)-lollipop walks L U (g+1,2) =EL W (g,2) −max L U (0,2) −1,0 L U (g−1,2) +L U (g−1,2) ◦E −L U (0,g) L U (1,2) +2 h P U g−1 max L W (0,2) −2,0 i ◦E +2P U g−1 ◦E +2δ g,4 P U 2 2 E−P U 3 ◦E−P U g+1 ◦E (A.52) L W (g+1,2) =E T L U (g,2) −max L W (0,2) −1,0 L W (g−1,2) +L W (g−1,2) ◦E T −L W (0,g) L W (1,2) +2 h P W g−1 max L U (0,2) −2,0 i ◦E T +2P W g−1 ◦E T +2δ g,4 P W 2 2 E T −P W 3 ◦E T −P W g+1 ◦E T (A.53) Walks of the form shown in Figure A.11 from α 1 to α g+4 =α g+2 are counted by L U (g+1,2) . The product EL W (g,2) counts (g + 1,2)-lollipop walks as well as walks where α 1 = α 3 , α 1 = α g+1 and α 1 = α g+3 . Note that walks where α 1 = α i for i∈ [5,g−1] o violate the girth constraint. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α g+1 α g+2 α g+3 α g+4 Figure A.11: A (g+1,2)-lollipop walk from α 1 to α g+4 =α g+2 . 204 A.5.7.1 Walks where α 1 =α 3 The product L U (0,2) L U (g−1,2) counts walks where α 1 = α 3 , walks where both α 1 = α 3 and α 2 = α 4 and walks where both α 1 = α 3 and α 2 = α g+2 . Note that walks where α 1 = α 3 and α 2 = α j for j ∈ [6,g] e violate the girth constraint. By instead considering the product max L U (0,2) −1,0 L U (g−1,2) , walks where α 1 = α 3 and α 2 = α 4 are excluded. A walk where α 1 = α 3 and α 2 = α g+2 requires that α 1 and α g+1 be connected by both a (g−1,2)-lollipop walk and a single edge. Such walks are counted by the direct product L U (g−1,2) ◦E. A.5.7.2 Walks where α 1 =α g+1 The product L U (0,g) L U (1,2) counts walks where α 1 =α g+1 along with the following individ- ually treated cases 1) Walks where α 2 = α g+2 : As shown in Figure A.12, walks where α 1 = α g+1 and α 2 =α g+2 require that α 1 be connected to α 2 by both a path of length g−1 and a single edge. Additionally, there must exist a vertex α g+3 to which α 2 is connected that is distinct from both α 1 and α 3 . The expression h P U g−1 max L W (0,2) −2,0 i ◦E counts such walks. 2) Walks where α g =α g+2 : By symmetry, walks where α 1 =α g+1 and α g =α g+2 are counted with the same expression as case 1 above. 3) Walks where α 2 = α g+2 and α 3 = α g+3 : A walk where α 1 = α g+1 , α 2 = α g+2 and α 3 = α g+3 requires that α 1 be connected to α g+2 by both a path of length g−1 and a single edge. Such walks are counted by the direct product P U g−1 ◦E. 205 ... (1) w 1 = w g 2 +1 (2) u 2 (3) u 1 = u g 2 +1 (4) w k+1 (5) u g 2 (6) u g 2 +1 (7) w g 2 (8) w g 2 +1 (9) u g 2 +2 (10) 1 ... (1) w 1 = w g 2 +1 (2) u 2 (3) u 1 = u g 2 +1 (4) w k+1 (5) u g 2 (6) u g 2 +1 (7) w g 2 (8) w g 2 +1 (9) u g 2 +2 (10) 1 α 3 α g+3 α g−1 α g α 2 =α g+2 α 1 =α g+1 Figure A.12: The case where α 1 =α g+1 and α 2 =α g+2 . 4) Walks where α g = α g+2 and α g+1 = α g+3 : By symmetry, walks where α 1 = α g+1 , α g =α g+2 and α g+1 =α g+3 are counted with the same expression as case 3 above. A.5.7.3 Walks where α 1 =α g+1 for g =4 In the special case of g = 4, there is a fifth case, in addition to cases 1-4 above, which must be excluded from the productL U (0,4) L (1,2) . Wheng =4, the productL U (0,4) L (1,2) also counts walks where α 1 = α 5 , α 3 = α 7 but α 4 6= α 6 . Figure A.13 illustrates this special case. α 4 α 6 α 2 α 3 =α 7 α 1 =α 5 Figure A.13: The case where α 1 =α 5 and α 3 =α 7 for g =4. By examining Figure A.13, it is observed that walks where α 1 = α 5 and α 3 = α 7 require that α 1 and α 6 be connected by both a single edge and two length 3 paths that 206 share the common vertex α 3 . The direct product h 2 P U 2 2 E i ◦E counts such walks along with walks where either α 2 = α 6 or α 2 = α 4 . Walks where α 2 = α 6 occur when α 1 and α 6 are connected by both a single edge and a path of length 3 and are thus counted by the direct product P U 3 ◦E. By symmetry, walks where α 2 = α 4 are also counted by the direct product P U 3 ◦E. A.5.7.4 Walks where α 1 =α g+3 A walk where α 1 =α g+3 requires that α 1 be connected to α g+2 by both a path of length g+1 and a single edge. Such walks are counted by the direct product P U g+1 ◦E. A.5.8 Counting (1,g)-lollipop walks L U (1,g) =EL W (0,g) −2P U g−1 ◦E (A.54) L W (1,g) =E T L U (0,g) −2P W g−1 ◦E T (A.55) Walks of the form shown in Figure A.14 fromα 1 toα g+2 =α 2 are counted byL U (1,g) . The u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α g+1 α g+2 Figure A.14: A (1,g)-lollipop walk from α 1 to α g+2 =α 2 . productEL W (0,g) counts both (1,g)-lollipop walks as well as walks where eitherα 1 =α 3 or α 1 =α g+1 . Note that walks where α 1 =α i for i∈[5,g−1] o violate the girth constraint. A walk where α 1 = α 3 requires that α 1 and α 2 be connected by both a path of length 207 g−1 and a single edge. The direct product P U g−1 ◦E thus counts walks where α 1 = α 3 and, by symmetry, walks where α 1 =α g+1 . A.5.9 Counting (2,g)-lollipop walks L U (2,g) =Z h EL W (1,g) i −6δ g,4 P U 2 3 (A.56) L W (2,g) =Z h E T L U (1,g) i −6δ g,4 P W 2 3 (A.57) Walks of the form shown in Figure A.15 from α 1 to α g+3 = α 3 are counted by L U (2,g) . When g > 4, the product EL W (1,g) counts (2,g)-lollipop walks and walks where α 1 = α 3 . u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α 4 α g+2 α g+3 Figure A.15: A (2,g)-lollipop walk from α 1 to α g+3 =α 3 . Note that walks where α 1 = α i for i ∈ [5,g+1] o violate the girth constraint when g >4. Eliminating the diagonal elements ofEL W (g−1,2) with theZ[·] operator removes the α 1 =α 3 walks. When g = 4, walks where α 1 = α 5 do not violate the girth constraint and must also be excluded from the product EL W (1,g) . A walk where α 1 =α 5 requires that α 1 and α 3 be connected by 3 distinct length 2 paths. The expression 6 P U 2 3 counts such walks. 208 A.5.10 Counting (3,g)-lollipop walks L U (3,g) =EL W (2,g) −max L U (0,2) −1,0 L U (1,g) −6δ g,6 P U 3 3 −4δ g,4 P U 3 2 ◦E +6δ g,4 E P W 2 2 −P U 3 ◦E +4δ g,4 P U 2 2 E−P U 3 ◦E (A.58) L W (3,g) =E T L U (2,g) −max L W (0,2) −1,0 L W (1,g) −6δ g,6 P W 3 3 −4δ g,4 P W 3 2 ◦E T +6δ g,4 E T P U 2 2 −P W 3 ◦E T +4δ g,4 P W 2 2 E T −P W 3 ◦E T (A.59) Walks of the form shown in Figure A.16 from α 1 to α g+4 = α 4 are counted by L U (3,g) . When g > 6, the product EL W (2,g) counts (3,g)-lollipop walks along with walks where α 1 = α 3 . Note that walks where α 1 = α i for i∈ [5,g+3] o violate the girth constraint when g > 6. When g = 6, walks where α 1 = α 7 must also be considered. When g = 4, walks where α 1 =α 5 and α 1 =α 7 must also be considered. u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α 1 α 2 α 3 α 4 α 5 α g+3 α g+4 Figure A.16: A (3,g)-lollipop walk from α 1 to α g+4 =α 4 . A.5.10.1 Walks where α 1 =α 3 TheproductL U (0,2) L U (1,g) countswalkswhereα 1 =α 3 alongwithwalkswhereα 2 =α 4 and, for the special case of g = 4, walks where α 1 = α 3 and α 2 = α 6 . Walks where α 1 = α 3 and α 2 = α 4 are excluded by instead considering the product max L U (0,2) −1,0 L U (1,g) . Figure A.17 illustrates a walk where α 1 = α 3 and α 2 = α 6 for the g = 4 case. The 209 walk shown in Figure A.17 is similar to that shown in Figure A.13. Using a similar argument to case 2.3 of the proof for L U (g+1,2) , it is readily shown that the expression 2 h E P W 2 2 −P U 3 i ◦E counts such walks. α 5 α 4 α 7 α 1 =α 3 α 2 =α 6 Figure A.17: The case where α 1 =α 3 and α 2 =α 6 for g =4. A.5.10.2 Walks where g =6 and α 1 =α 7 For the special case of g = 6, walks where α 1 = α 7 do not violate the girth constraint. A walk where α 1 = α 7 requires that there exist 3 distinct length 3 paths from α 1 to α 4 . The expression 6 P U 3 3 counts such walks. A.5.10.3 Walks where g =4 and α 1 =α 5 For the special case of g =4, walks where α 1 =α 5 do not violate the girth constraint. A walk where α 1 =α 5 requires that α 1 be connected to α 4 by both a single edge and by 2 distinct paths of length 3. The expression 2 P U 3 2 ◦E counts such walks as well as walks whereα 1 =α 5 andα 2 =α 6 andwalkswhereα 1 =α 5 andα 3 =α 7 . Walkswhereα 1 =α 5 and α 3 = α 7 are similar to those illustrated in Figure A.13 and are thus counted by the expression 2 h P U 2 2 E−P U 3 i ◦E. Walks where α 1 = α 5 and α 2 = α 6 are similar to those illustrated in Figure A.17 and are thus counted by the expression 2 h E P W 2 2 −P U 3 i ◦E. 210 A.5.10.4 Walks where g =4 and α 1 =α 7 By symmetry, walks where g = 4 and α 1 = α 7 are counted by the same expression as those where g =4 and α 1 =α 5 . A.5.11 Counting (1,g+2)-lollipop walks L U (1,g+2) =EL W (0,g+2) −2P U g−1 ◦E−2δ g,4 P U 3 2 ◦E +2δ g,4 E P W 2 2 −P U 3 ◦E +2δ g,4 P U 2 2 E−P U 3 ◦E (A.60) L W (1,g+2) =EL U (0,g+2) −2P W g+1 ◦E T −2δ g,4 P W 3 2 ◦E T +2δ g,4 E T P U 2 2 −P W 3 ◦E T +2δ g,4 P W 2 2 E T −P W 3 ◦E T (A.61) Walks of the form shown in Figure A.18 from α 1 to α g+4 = α 2 are counted by L U (1,g+2) . When g > 4 the product EL W (0,g) counts (1,g +2)-lollipop walks as well as walks where α 1 = α 3 and walks where α 1 = α g+3 . Note that walks where α 1 = α i for i∈ [5,g+1] o violate the girth constraint when g > 4. When g = 4, the product EL W (0,4) also counts walks where α 1 =α 5 . u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 u 1 (1) w 1 (2) u 2 (3) ... (4) uk 2 +1 (5) wk 2 +1 (6) w n 2 (7) 1 α g+3 α g+4 α 1 α 2 α 3 Figure A.18: A (1,g+2)-lollipop walk from α 1 to α g+4 =α 2 . 211 A.5.11.1 Walks where α 1 =α 3 A walk where α 1 =α 3 requires that α 1 and α 2 be connected by both a single edge and a path of length g+1. The direct product P U g+1 ◦E counts such walks. A.5.11.2 Walks where α 1 =α g+3 By symmetry, walks where α 1 =α g+3 are counted by the same expression as the α 1 =α 3 case. A.5.11.3 Walks where g =4 and α 1 =α 5 A walk where α 1 = α 5 requires that α 1 and α 2 be connected by a single edge and two non-overlapping length 3 paths. Such walks were counted in case 3 of the proof of L U (3,g) . 212 A.6 Short Cycle Counting Examples A.6.1 A Golay Code Tanner Graph Consider the [23,12,7] Golay code with systematic parity-check matrix H G =                 1 0 0 1 1 1 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 1 0 1 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 1 0 1 1 0 1 1 0 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 1 1 0 1 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 1 1 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 1 1 0 1 1 0 0 1 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 1 1 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 1 0 1 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 1 1 1 1 0 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1                 . (A.62) The cycle counting algorithm computes N 4 =452, N 6 =10,319 and N 8 =199,994 in the correpsonding Tanner graph (which has incidence matrix E =H G ). A.6.2 A Small LDPC Tanner Graph Consider the small LDPC code due to [159] with parity-check matrix H L =                       1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0                       . (A.63) 213 This code was designed to have a Tanner graph with girth 12. Accordingly, the cycle counting algorithm computes N 4 = N 6 = N 8 = N 10 = N 14 = 0, N 12 = 28 and N 16 = 21 in the corresponding Tanner graph. A.6.3 LDPC Tanner Graphs In this example, the relationship between cycle structure and performance for two sets of LDPC codes is examined. First consider two rate-1/2 Gallager codes with block length 816. Definitions for both codes are due to MacKay [107]. Code A has column weight 3 and corresponds to MacKay’s code 816.3.174 while Code B has column weight 5 and correspondstoMacKay’scode816.55.178. TheshortcyclestructureoftheTannergraphs corresponding to these codes is tabulated in Table A.1. Codes A and B have the same girth g = 6 but Code B has many more short cycles. The effect of these cycles on code performance is dramatic: Code A outperforms Code B by approximately 0.75 dB at a bit-error rate of 10 −5 [107] (on the AWGN channel with binary antipodal signaling). Code A Code B N 6 132 7921 N 8 1494 210740 N 10 9278 6054731 Table A.1: Short cycle structure of the Code A and B Tanner graphs. Next consider two rate-1/2, block length 504 LDPC codes constructed using the Progressive-Egde Growth method due to Hu et. al [78]. Codes C and D correspond to MacKay’s codes PEGReg252x504 and PEGirReg252x504, respectively. The short 214 Code C Code D N 6 - 13244 N 8 802 420609 N 10 11279 13567791 N 12 86791 - Table A.2: Short cycle structure of the Code C and D Tanner graphs. cycle structure of the Tanner graphs corresponding to these codes is tabulated in Ta- ble A.2. Given that the Tanner graph corresponding to Code C has larger girth, one may initially expect it to outperform Code D. However, Code D outperforms Code C by approximately 0.5 dB at a bit-error rate of 10 −5 [78]. This seeming contradiction is explained by examining the regularity of the respective Tanner graph cycle structures. Table A.3 tabulates the mean, μ k , and standard deviation, σ k , of the number of cycles of length k incident on each code bit vertex in the Tanner graphs corresponding to Codes C and D. The small σ k values for Code C indicate an extremely regular cycle distribution whereas the σ k values are very large for Code D indicating a much more irregular, or random-like, cycle distribution. In this example, the negative impact of cycle regularity on the performance of Code C is greater than the positive impact of large girth. It is clear from this example that girth alone is not enough to judge the quality of a LDPC code design. The number and statistics of the short cycles are both important metrics for quality. 215 A.7 Complexity of Counting Short Cycles Thetimecomplexityoftheproposedalgorithmisdominatedbymatrixmultiplicationand may be estimated by counting the number of matrix multiplications required to compute the number of cycles of length g, g+2, and g+4 in a bipartite graph with girth g > 6. Carefully counting the total number of matrix multiplications yields an expression that is linear in g: 6g+37 (A.64) ThetimecomplexityofeachmatrixmultiplicationgrowsasO n 3 wheren=max(|U|,|W|). The time complexity of counting short cycles thus grows asO gn 3 . Thespacecomplexityoftheproposedalgorithmisdominatedbymatrixstoragewhich grows asO n 2 . More specifically, computing the number of cycles of length g, g +2, andg+4inabipartitegraphwithgirthg andwithvertexclasssizes|U|and|W|requires at most 11|U| 2 +11|W| 2 +21|U||W| (A.65) storage locations. Note that high bit-width storage locations (e.g. 64-bit integer or double-precision floating point) must be used in order to accommodate large dynamic Code C Code D μ k σ k μ k σ k k =6 - - 26.3 55.3 k =8 1.6 0.7 834.5 1767.2 k =10 22.4 1.8 26920.2 56369.6 k =12 172.2 8.3 - - Table A.3: Short cycle statistics of the Code C and D Tanner graphs. 216 ranges in the matrix entries. Also note that the space complexity of the proposed algo- rithm does not grow with girth. 217

Abstract (if available)

Abstract A decade ago, the introduction of turbo codes and iterative message passing algorithms revolutionized the theory and practice of coding. In the ensuing years, the coding theory community has become adept at designing codes from good graphical models -- that is, models which imply low-complexity, near-optimal iterative message passing algorithms. Specifically, modern codes are constructed by connecting a large number of simple local codes together via a rich, random-like, cyclic interconnection network. A key observation from this work is that the introduction of cycles to graphical models can enable massive complexity reductions in model, and thus decoding, complexity.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

H.264/AVC decoder complexity modeling and its applications

PDF

Implementation architectures for robust iterative receivers

PDF

Exploration of parallelism for probabilistic graphical models

PDF

Scalable exact inference in probabilistic graphical models on multi-core platforms

PDF

Linear free energy relationships for transition metal chemistry: case study of CH activation

PDF

Algorithms and frameworks for generating neural network models addressing energy-efficiency, robustness, and privacy

Asset Metadata

Creator Halford, Thomas Rhys (author)

Core Title The extraction and complexity limits of graphical models for linear codes

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Electrical Engineering

Defense Date 02/07/2007

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag codes on graphs,coding theory,graphical models for inference,iterative decoding algorithms,OAI-PMH Harvest

Language English

Advisor Chugg, Keith M. (committee chair), Arratia, Richard A. (committee member), Sholtz, Robert A. (committee member)

Creator Email halford@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-m310

Unique identifier UC1118806

Identifier etd-Halford-20070307 (filename),usctheses-m40 (legacy collection record id),usctheses-c127-398132 (legacy record id),usctheses-m310 (legacy record id)

Legacy Identifier etd-Halford-20070307.pdf

Dmrecord 398132

Document Type Dissertation

Rights Halford, Thomas Rhys

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Repository Name Libraries, University of Southern California

Repository Location Los Angeles, California

Repository Email cisadmin@lib.usc.edu