Computer Science Technical Report Archive

USC Computer Science Technical Reports, no. 839 (2004)

(USC DC Other)

USC Computer Science Technical Reports, no. 839 (2004)

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content EnergyEfcient Data Organization and Query Processing in Sensor Networks Ramakrishna Gummadi Xin Li Ramesh Govindan Cyrus Shahabi Wei Hong ABSTRACT Sensor networks are an emerging class of highly distributed systems that generate, process, and store data that must be queried in an energy-efcient manner. Recent sensor net- works research has produced a class of data storage and query processing techniques called Data-Centric Storage (DCS) [29] that leverages locality-preserving distributed indexes like DIM [24], DIFS [15], and GHT [28] to efciently store sensor tuples, and answer multi-dimensional range and range-aggregate queries. These distributed indexes offer a rich design space of a) log- ical decompositions of sensor relation schema into indexes, as well as b) physical mappings of these indexes onto sen- sors. In this paper, we explore this space for energy-efcient data organizations (logical and physical mappings of tuples and attributes to sensor nodes) and devise purely local query optimization techniques for processing queries that span such decomposed relations. We propose four design techniques: (a) fully decomposing the base sensor relation into distinct sub-relations, (b) spatially partitioning these sub-relations across the sensornet, (c) localized query planning and opti- mization to nd fully decentralized optimal join orders, and (d) locally caching join results. Together, these optimiza- tions reduce the overall network energy consumption by 4 times or more when compared against the standard single multi-dimensional distributed index approach on a wide va- riety of synthetic query workloads simulated over both syn- thetic and real-world datasets. We further validate the feasi- bility of our approach by implementing a functional proto- type of our data organizer and query processor on mica2 [11] motes and observing comparable savings in message cost. 1. INTRODUCTION Wireless sensor networks are an emerging class of highly distributed systems with widespread applicability. In such networks, nodes generate, process and store sensor readings within the network. This architecture is necessitated by the relatively high energy cost of wireless communicationthis Computer Science Department, University of Southern Cali- fornia, Los Angeles, CA 90089, USA. Email: fgummadi, xinli, ramesh, cshahabig@usc.edu Intel Research at Berkeley, 2150 Shattuck Ave., Suite 1300, Berkeley, CA 94704, USA.Email: wei.hong@intel.com cost makes it infeasible to consider centrally collecting and processing voluminous sensor data. An important compo- nent of these networks, then, is an energy-efcient system that enables users to query the stored data. Existing approaches to organizing data and processing queries fall under one of the two broad categories namely, Data- Centric Routing (DCR) and Data-Centric Storage (DCS). In DCR, the data generated by the sensors is stored at the nodes that generate them, and queries are ooded through- out the network. Data from the sensors in the sensornet is then aggregated along the query tree that is built during the query ooding phase on a per-query basis. This ap- proach, pioneered by early systems such as TinyDB [26] and Cougar [5], is efcient for continuous (long-running) queries, where the high energy cost incurred during the query ooding and per-query data aggregation phases is amortized over time. Compared to DCR, DCS is a relatively new class of data storage and query methodologies proposed in [29]. In DCS, data generated by a sensor is rst stored intelligently at re- mote nodes as soon as it is generated with an eye toward ex- ploiting data locality during querying because related sensor data gets stored together regardless of where in the sensornet the data originates. Consequently, queries can be directed to the precise locations of the network during the query propa- gation phase, and, data can be aggregated locally and more efciently without propagating it up a query tree during the data aggregation phase. Thus, the overall (insertion+query) cost for DCS is lower for many ad-hoc (short-lived) work- loads. DCS can use any locality-preserving geographically dis- tributed index structure such as DIM [24], GHT[28], DIFS [15], and DIMENSIONS [13]. Our focus in this paper is to im- prove the overall energy performance of vanilla DCS by a) exploiting the exibility offered by these underlying data structures during the data insertion phase and b) optimizing query plans during query execution phase. Thus, while each DCS system currently defaults to using a xed data orga- nization (by this we mean mappings of tuples and attributes to network nodes), we seek to understand the design space of possible data organizations, and identify more energy ef- cient (in terms of total insertion+query costs) candidates. Related to this effort, we also propose techniques for opti- mizing query planning and execution in DCS. In this paper, we use a distributed index called DIM [24], which serves as our basic storage, indexing, and querying layer, and is interesting because of its locality-preserving property. However, we emphasize that the choice of this distributed index is orthogonal to the data organization and query processing ideas described in this paper, and it is pos- sible to use other indexes like GHT[28], DIFS [15], and DI- MENSIONS [13]. DIM is overviewed in Section 3.1, and can be thought of as a search tree that is spatially overlaid on a sensor network. In this sense, it resembles classical database indexes. However, DIM's are also intended to store the primary copy of the data. Consider a sensor network with an m-relation schemah uuid;a 1 ;a 2 ;:::;a m i. Tuples in this schema can be stored in one DIM. Alterna- tively, we can fully decompose them into m DIM's each of which stores a single relation of the form huuid;a i i, and we can then join on uuid on demand to evaluate queries. A spectrum of partial decompositions of the base relation into sub-relations of the formhuuid;a i ;:::;a j i is, of course, also conceivable. Clearly, we can expect these different data organizations to yield different performance under different workloads. Our measure of performance is the total energy cost incurred for a given workload, including data inserts and query retrievals. This is because sensor networks tend to be energy-constrained, and communication is the single biggest energy component of a sensornet. We approximate the energy cost of a single message as a product of the size of the message (in bits) and the number of hops the message traverses. We found that, in many cases, fully decomposing the base relation performs better than zero or any partial decompo- sition, even if the decomposition is carried out with an eye toward a given query workload (we identify important pre- conditions and exceptions to this broad statement in Sec- tion 5.4). We then study three related mechanisms that can improve the efciency of query processing when a base re- lation is fully decomposed into multiple DIM's: Spatially Partitioning Sub-Relations Each fully decomposed sub-relation is stored in a DIM, and all DIM's are as- signed spatially disjoint sections of the sensor eld. Note that this partitioning preserves the load-balancing nature of DIM'sas we shall see in Section 4.1, it only determines which attributes are mapped to each node and does not alter the number of tuples stored at each node. In other words, this device preserves static load- balancing. Assuming queries are also balanced across all attributes (i.e., they place equal statistical load on all attributes) 1 , such a spatial partitioning also guaran- tees dynamic load-balancing. We strive to respect the load-balancing constraint because it is a simple way to restrict hotspots and maximize worst-case network 1 In Section 5.7, we explore simulation scenarios where this as- sumption is relaxed. lifetime. Efcient Query Planning via Decentralized Join-Ordering We import the familiar notion of equijoins (Section 3.2) into sensornets for range and range-aggregate query processing within our system. Our system processes the SQL query at the query issuer and constructs an ef- cient query plan that includes an optimal join order using only locally available information in the form of a histogram. We show that it is important to choose a good join order during query optimization and demon- strate how to do so (Section 4.2) using only summa- rized global information in the form of a low over- head coarse-grained multi-dimensional histogram that approximates the distribution of data stored within the network. The histogram is lazily constructed and prop- agated a priori to each node in the network. Efcient Query Execution via Optimistic Join-Caching We advance a simple and robust mechanism to cache the results of partial joins across sub-relations locally at each sensor node (Section 4.3). This caching strategy enhances query performance by eliminating redundant tuple movement during query execution. We show using extensive simulations that for a variety of data distributions (both synthetic and real-world) and syn- thetic query workloads, these schemes together provide more than a four-fold reduction in energy expenditure over storing the base relation in one DIM even for a small number (4) of sensor attributes; we argue analytically in Section 4.1 that we can expect this factor of improvement to improve with increasing number of sensor attributes. The rest of the paper is structured as follows. After a re- view of related work (Section 2), we outline the functional- ity provided by DIM and analyze, using a simple model, the performance of various data organizations (Section 3). In Section 4, we discuss in detail the four primary techniques that we employ for minimizing the total energy consump- tion, and evaluate the impact of their contribution using sim- ulations over both real-world and synthetic datasets in Sec- tion 5. We describe the detailed implementation of our sys- tem in Section 6. We conclude in Section 7 with a brief discussion of future directions. 2. RELATED WORK The problems of distributed query processing in general and join optimization in particular have been studied in the context of both sensor networks and distributed/federated/Internet/peer- to-peer databases. One line of research [7, 2, 8] has studied non-blocking joins on data streams. Another [3, 30, 18] has focused on dynamic adaptation of query plans in federations of databases. Our work follows a different line of research, one that is closer in spirit to the conventional querying of stored, distributed relations. However, the key twist in our approach is the cost metricthe cost of join-ordering de- pends not just on how much data is moved but also the net- work distance that the data is moved. This presents an en- tirely new dimension to join-ordering and query optimiza- tion that, to our knowledge, has not been studied before. In databases, using selectivity estimation for query opti- mization has been described in [27]. The use of histograms for query processing has been described in [20, 6, 19, 9]. The main distinction of our work from semi-joins[10] in distributed databases is that we have a different cost model, more nodes, and a different architecture. Query optimiza- tion in distributed databases in general is discussed in sev- eral places such as [4, 1, 23]. However, the primary goal there is to optimize for query response time than for energy consumption. 3. BACKGROUND AND MOTIV ATION In this Section, we describe the mechanics of insertion and querying in DIM, and motivate the performance advantages of decomposing base relations using qualitative arguments. 3.1 DIM Overview Sensor networks are typically tasked to individually or collaboratively sense an environment and produce higher- level events or features after local signal processing and l- tering. Examples of such events might be local micro-climate temperature gradients or bird sightings. It is these events that we are primarily interested in querying in an energy-efcient manner. These queries can be issued from anywhere within the network (sometimes by other distant sensor nodes them- selves), and are declarative in nature. An event can be thought of as a tuple consisting of a small (typically 4-6) number of attributes. Each attribute corre- sponds to a sensor type and can be treated as a column in a single relation table consisting of all possible sensor types. For example, a typical habitat-monitoring sensornet may gen- erate tuples consisting of 4 attributes: < l;t;x;y > corre- sponding to light and temperature readings, and the (x;y) co-ordinates of the sensor that sensed this reading. These events are generated either periodically, or due to some ex- ternal stimulus or condition, or even perhaps in response to other events. The events are then cast into a tuple and times- tamped. Each tuple is assigned a universally-unique iden- tier (uuid). The uuid can be constructed as a simple con- catenation of node number and a locally unique sequence number (which could be the timestamp itself). The relational schema for a general sensor network can thus be viewed as a single table of the form(uuid;a 1 ;a 2 ;:::;a k ) where k is the number of sensor attributes. Throughout the rest of this paper, we call this single logical table sensors. Whenever a sensor generates a tuple, it inserts it (or some decomposed version of it) into one or more DIM indexes, as described later. A DIM index is best described by visualizing a collection of sensor nodes distributed on a two-dimensional surface. In DIM, this geographic region occupied by the sensor nodes is spatially partitioned such that each node owns the part of the region around it (we call these spatial sub-divisions zones). This spatial partitioning can be logically thought of as recursive equal-sized subdivisions of the 2-D space al- ternately along the x and y axes. Each spatial region result- ing from a series of subdivisions can be assigned a unique bit code; for example in Figure 1, the zone assigned a code 1001 indicates that the zone is on the right side of the rst subdi- vision (along the x-axis) as indicated by the 1 in the rst bit, along the bottom half of the second subdivision (along the y- axis) as indicated by the 0 in the second bit, and so on. This spatial partitioning can be accomplished by a distributed al- gorithm that is described in [24]. Then, hyper-rectangles in the attribute space are mapped to zones. Given a tuple, nodes can compute which zone the tuple belongs to entirely locallythe only global infor- mation they need is an approximate boundary of the sen- sor eld. They do this by essentially subdividing the at- tribute space in the same way that the sensor eld is geo- graphically partitioned (this algorithm generalizes to more than 2 dimensions on the attribute space). Thus each tu- ple can be assigned a code and will be stored at the node whose code matches that of the tuple; for example, tuple (0:3;0:5;0:8;0:1) would be assigned to node 4 because its zone-code, 011, matches the prex of the tuple-code 01100010(etc, up to desired resolution). Tuples are then routed to the ap- propriate zone using a geographic routing algorithm [21]. Multi-dimensional range queries on a DIM can be described by hyper-rectangles in the attribute space. Given these hyper- rectangles, nodes can map them to DIM zones using the same mapping algorithm as was used for tuple insertions. The semantics of a DIM query are that, without global time synchronization [12], we only provide relaxed consistency guarantees similar to those in PIER [18]the sensornet pro- vides a best-effort service to execute a query and return those tuples that correspond to a local snapshot dened when the query hits the nodes, and not at the time the query was is- sued. When nodes can be time-synchronized, we can de- ne stricter semantics, because both queries and data are timestamped. Finally, the semantics are best effort in an- other sense that node failures may make some of the data unavailable. In such cases, we return as much data as possi- ble, but we can ag this condition reliably using end-to-end reliability on top of DIM's, as explained in Section 6. 3.2 Motivating Alternative Data Organizations We are now ready to consider the subject of the paper; the tradeoffs involved in storing a relation in a single DIM ver- sus decomposing it across multiple DIM's. A crucial aspect of this tradeoff is our cost metric: the energy cost of trans- porting a message from one node to another is proportional to the product of the message size and the number of hops traversed (in sensor nodes, each transmission costs signi- cant energy). In DIM, the cost of a query response is inuenced by the 3 010 001 000 011 4 9 8 2 1 110 111 10 1011 1010 7 5 1000 1001 6 Figure 1: Example DIM Organization node distribution (which denes the zone structure) and by how many zones need to be consulted as part of the query (i.e. which nodes contain data for parts of the query hyper- rectangle). For example, in Figure 1, if the DIM is organized as a 2-D index structure over 2 attributes (l;t), to answer a query Q1 of the form: select avg(t) from sensors where 0.25<=t<0.5 the nodes with zone prexes f0001;0011;1001;1011g (in this example, these would be nodes 1, 2, 6, and 8) would have to be consulted because the rst attribute, l, can take on either 0 or 1. Consider now an alternative organization in which we con- struct two separate DIM's, one each for l and t (we call these 1-DIM's). The 1-DIM's are fully-decomposed tables of the form (uuid;a i ) where a i is the value of the i'th sensor type in the tuple. The 1-DIM itself is constructed on the sensor attribute a i , and the uuid's are used for joining a i with other sensor attributes during query execution. In this organiza- tion, we need to only consult the two nodes 3 and 4 corre- sponding to zone prexes f010;011g to answer query Q1. Also, more importantly, these two zones are closer together in terms of geometric distance by a factor of 2 compared to the four zones that would need to be searched with 2-DIM. On the other hand, for a query Q2 of the form: select avg(l) from sensors where 0.25<=l<0.75 and 0.5<=t<0.75 we would have to consult nodes owning zones with prexes between 0110 and 1100 in case of a 2-DIM (assuming it is constructed on (l;t)). These nodes would be 4, 5, 6, 7, 8, and 9. With 1-DIM's, the prexes would be 0100 to 1011 for l, and 1000 to 1011 for t (thus, the nodes would be 3, 4, 5, 6, 7, 8), comparable in number and distribution to the 2-DIM case. Thus, we see that the size of the query hyper-rectangle can have a critical inuence on how efciently a query gets executed. Note that to answer queries of type Q1 using a 2-DIM, we needed to scan all zones for which the second attribute of the DIM, t, can be treated as a wildcard (this is true as soon as at least one attribute in a multi-dimensional query is absent from the range part). On the other hand, if we were to use two separate 1-DIM's, one each for l and t, we would not have this inefciency, as noted above. In general, we typi- cally end up having to visit fewer and more closely separated nodes with fully-decomposed DIM's. However, in order to answer queries of the form Q2, using 1-DIM's, we need to (a) select both attributes (uuid;l) that make up the table data for the 1-DIM on t, (b) transport them to the nodes contain- ing the zone 0:25< l < 0:75, (c) locally match the l tuples with t tuples on uuid's and lter out those l's that don't have a matching t within the query range, and (d) nally aggre- gate the remaining values of l and return the result to the query issuer. Thus, we see that this form of query execution on partially- or fully-decomposed relations is generalizable to a larger number of sensor attributes and naturally leads to the familiar database notion of joins in sensor networks. 3.3 The Focus of the Paper This paper focuses on efciently supporting multi-dimensional range and range-aggregate queries using DIM's. Before we describe the questions that the previous section motivates, we describe some important assumptions. We assume that queries can be issued from any node in the network and data can be inserted from any node in the network at any time. We wish to accommodate a wide variety of aggrega- tion operators because we allow for aggregation to be per- formed both locally at a node after collecting relevant tu- ples, and as a form of in-network processing. Thus, we aim to support a exible, wide-ranging set of range, aggre- gation and order-statistics operators (like medians[25], etc., that are not fully amenable to any form of hierarchical aggre- gation) without sacricing efciency when computing sim- pler aggregates like sums and averages. In all cases, our sys- tem supports SQL-like relational queries involving standard clauses like select, where, group by, etc. The previous subsection pointed out that decompositions of the base relation into multiple DIMs can have different query performance characteristics. The important tradeoff here is that when sub-relations are stored in DIMs, scans are more efciently supported than when the base relation is stored in one DIM. On the ip side, however, to answer queries in general, the sub-relations need to be joined on uuid's. Joins entail additional costs in the form of data move- ment between the various DIM's. Clearly, then, the per- formance of decomposition will depend on the query work- loads. This discussion motivates several questions, which form the basis of the rest of the paper: What are the performance advantages of different data organizations? Do partial decompositions perform bet- ter than full decompositions? Given a decomposed base relation, how might a node decide on an efcient join-ordering? X L (1,0) (1,0) (0,0) (1,1) Y T Figure 2: Example spatially-partitioned fully- decomposed DIM's Given that query hyper-rectangles might overlap, is it benecial to cache the results of joins to reduce data movement costs? If so, what mechanism might we use to do this? We address these questions in the next section. 4. DATA ORGANIZATION FOR EFFICIENT QUERYING In the previous section, we discussed the possibility that decomposing a base relation and storing it in multiple DIM's can potentially improve query performance. In this section, we rst discuss issues related to this decomposition, then describe an entirely decentralized and distributed method for join-ordering, and nally discuss issues related to caching join results. 4.1 Full Decomposition and Spatial Partition ing We offered qualitative arguments in Section 3.2 that a full decomposition of a relation of k attributes into multiple 1- DIM's can achieve signicant energy efciency over storing the base relation in a single DIM. A related obvious opti- mization we can leverage is spatial partitioning: the DIM's that store a sub-relation can be assigned to spatially dis- joint partitions of the original sensor eld. For example, in Figure 2, the l (for light) attribute values of a tuple are all stored in a 1-DIM in the lower left (quadrant 0) corner of the sensor-eld, all the t (for temperature) values are stored in the lower right (quadrant 1), and the x and y values in quad- rants 2 and 3 respectively. This organization constrains the distance data must move within each DIM by clustering re- lated attribute values more densely, thereby further reducing the overall cost compared to a 4-DIM. To motivate the utility of this optimization, consider the execution of the following simple query on this data organi- zation: select avg(l) from sensors The average message cost for answering such a query in this data organization can be expected to be n 1=4 where n is the total number of tuples in the system because these tu- ples would have to move an average of 1=4 (assuming the whole square to be a unit square) to get aggregated some- where near the centroid of the lower left quadrant. Analyt- ically, the average distance an attribute would move would be R 1=2 0 R 1=2 0 jx1=4j+jy1=4jdxdy 1 4 (because the area of this quad- rant=1/4; assuming anL 1 norm). With k> 4 attributes, we can analytically show that we can get correspondingly higher multiple of message reduction, although this multiple grows as O( p k). For a non-partitioned data organization with separate over- lapping DIM's for individual attributes, intuitively, the av- erage execution cost would be n 1=2 because the tuples now have to move an average of 1=2 to get aggregated near the center of the unit square. Thus, the non-partitioned case moves tuples twice farther than the partitioned one, and, hence, incurs double the energy cost in this scenario. It also quadruples its hotspot concentration compared to parti- tioned. In what follows, then, we assume that when we decom- pose a base relation into n sub-relations, these are stored in spatially-partitioned DIM's. One important question that arises with respect to our data organization scheme is whether we can fundamentally im- prove our scheme knowing the nature of the query workload beforehand. For example, if we happen to know that most (but not all) range or range-aggregate queries are of the form: select avg(l,t) from sensors where l1<l<l2 and t1<t<t2 a natural question to ask is if we can construct a joint DIM on (l;t) to optimize this case. As we shall see in Section 5.4, there are instances when we may protably leverage alternate data organizations when sufcient preconditions exist. We treat them as important but special cases because of two main reasons: Even if most of the queries are on those two attributes, the tuples and not the queries need to exhibit a high de- gree of correlation between the joint attributes. Thus, it is not sufcient for the ranges (l 1 ;l 2 ) and (t 1 ;t 2 ) to be queried together; rather, the generated tuples must be correlated within the same range. For example, we may know a-priori that t tends to increase with l but if the query is for high l and low t, we still end up paying high query and aggregation costs. Even if a high fraction of the queries happen to be on (l;t), the remaining queries (which are essentially se- lect on one attribute and scan on the other) on indi- vidual l and t's cause higher aggregation penalty than for 1-DIM's (because our cost metric depends on the distance the data is moved), thereby offsetting some of the advantages accrued through a joint DIM construc- tion. Thus, generally speaking, a full decomposition will perform better than a partial decomposition, even with a good degree of knowledge of the query distribution. Section 5.4 identies scenarios where partial table decomposition outperforms full decomposition. A second interesting question is whether and how to opti- mize the data organization and query processing for queries following a hot/cold model on attributes. By this, we mean most of the queries involve joins or aggregations on a sub- set of the full set of attributes (the hot set of attributes). We show results in Section 5.3 that we do not need to do anything special to optimize for this case: we retain our data organization and query optimizer unchanged, and can automatically leverage the benets on the hot attribute set through caching (discussed in Section 4.3). However, if the queries happen to aggregate a lot on a hot attribute, the nodes in the 1-DIM built on the hot attribute are stressed even though we minimize overall energy consumption. Thus, while storage is load-balanced, querying would not be, in terms of individual energy consumed per node. Related to the hot/cold model is the concept from OLAP literature[17] of dimensions and measures on attributes. In many query workloads, some of the range attributes hap- pen to be dimensions like (x;y) that are never measured (or aggregated, as in our case), but are only used for query ranging. Once again, this places heavier load on measure attributes, thereby creating hotspots within the DIM's con- taining the measure attributes. In fact, in this case, we can decrease the total energy consumption while worsening the hotspot intensity by decreasing the size of the measure DIM's while correspondingly increasing the dimensions DIM's because we have a priori information about DIM's that would not participate in aggregation. We revisit such questions in Section 5.7. Finally, we note that there are a couple of common opti- mizations that are possible in real-world, but which we don't exploit in this paper. Batching and Deltas: since there is greater clustering (roughly 4 times for the same loading factor) with decomposed relations on 1-DIM's than with a 4-attribute base relation on a 4-DIM, we can batch the inserts of several successive tuples into one message, and use simple semantic compression/encoding schemes to re- duce the message size. Eliminating redundant updates: with a 4-DIM, we have to generate a new tuple each time there is a change in only one of the sensor readings. Inserting this tu- ple would mean even the attributes that are mostly un- changed (like (x;y), for example) or haven't changed for some time (for example, l reading might be un- changed even if t changes) would have to be included. With a 1-DIM, we can generate a full logical tuple by creating a new uuid, and inserting (uuid;t) into the t- DIM. At query time, we can interpolate for the miss- ing (uuid;x)'s and (uuid;y)'s because queries naturally support ranging, and interpolation is a simple local op- eration of replicating missing values. Such optimiza- tions can result in signicant insertion energy savings. 4.2 Optimizing Join Orders Having established that fully decomposing a base relation is at least a reasonable data organization, it remains to show how to efciently determine join orders. Clearly, different join orders can have vastly different costs, but we make the following key observation: knowing which DIM's exist in the system, and given an approximate joint data distribution, each node can independently compute an efcient join order for the query. This is possible for three reasons: First, DIM's are spatially distributed indexes, and the mapping between a data item and the location it is stored can be computed locally. Analogously, the geo- graphical location of a query hyper-rectangle is known. Second, we use a histogram to give us an approxi- mate indication of the number of tuples generated by selecting the appropriate range from each of the sub- relations. Third, knowing the selectivity and the locations of the query hyper-rectangles, each node can compute the ap- proximate cost of a query plan by estimating the mes- saging cost by either the Euclidean or theL 1 distance. In our design, each node in the network contains a query optimizer. When the query optimizer is presented with a range or range-aggregate query, it outputs a query plan con- sisting of a sequence of select and join operations that needs to be executed before the nal aggregation. In our current in- stantiation of the design, the optimizer considers all possible join orders (O(k!) (this is feasible if the number of attributes k is small; also, we don't consider join trees that are not sim- ple chains because chains are simple and robust to execute distributively and tend to be good enough) and determines the least cost one. As described above, to estimate the cost of a query plan, it needs to be able to estimate the number of tuples that would be produced by an initial select on a DIM (this num- ber would be inuenced by the range selectivity factor) as well as the fraction of these tuples that would be successfully joined with another attribute as part of the subsequent steps in the join phase of the query execution (this number would be inuenced by the join selectivity factors). To analytically understand the importance of good join or- dering, consider Figure 2 once again. If our query is of the form: select avg(l),avg(t) from sensors where 0<=l<0.5 The cost incurred for executing this query by moving t to l is approximately n=2 1=2 for join because roughly n=2 tuples would be selected by the range query 0<= l < 0:5, and moving these n=2 tuples incurs a cost of n=2 1=2 as can be intuitively seen and analytically shown. The query also has to pay an aggregation cost of 2n=21=4 because the 2 attributes (l,t) numbering n=2 need to be moved 1=4 distance after reaching the quadrant 1. The total cost is n=2. On the other hand, if we were to execute this query by rst joining l with t, we would have incurred a join cost of n 1=2 and an aggregation cost of 2 n=2 1=4 for a total cost of 3n=4. It is instructive to consider this query cost with that of a 4-DIM: there is only aggregation cost, and it is n=2 2 1=2= n=2 because the each node in the 4-DIM has to locally retrieve the l and t attributes of each tuple and ship them to the centroid of the unit square. While this is as expensive as with spatially-partitioned 1-DIM's that use optimal join ordering, we can readily see that the join component of the query cost in the 1-DIM case can be eliminated if we can cache the join results. This would double the efciency of the data organization scheme employing normalized DIM's, and one such caching scheme is described in Section 4.3. Once this consistent and high join cost component of a query is eliminated, we can intuitively see why it is possi- ble to get a small multiple (indeed, more than 2 even with 4 attributes) benet in total energy savings by using nor- malized DIM's compared to 4-DIM's: the data values re- turned by a query come from nodes all over the sensor eld (due to full-interleaving of attributes in 4-DIM) and end up getting aggregated near the center of the unit square after traveling long distances, while a normalized 1-DIM that can minimize join overhead through caching only has to pay a small variable localized aggregation cost (because of zero interleaving, the values desired by the attribute range of a query tend to come from close by nodes whose dispersion is dened only by the selectivity of a query). Additionally, spatial partitioning adds a small, on an average constant, but useful percentage to the proceedings. A secondary, and, in some scenarios, potentially critical benet of our approach is that it distributes the aggregation hotspots over the entire network in contrast to 4-DIM which tends to reinforce the single hotspot region around the network centroid with each query. Returning to join order processing, our query optimizer computes the total query energy cost as: E =j ¯ a 1 j D( ¯ a 1 ; ¯ a 2 )+jJ( ¯ a 1 ; ¯ a 2 )j D( ¯ a 2 ; ¯ a 3 )+ jJ( ¯ a 1 ; ¯ a 2 ; ¯ a 3 )j D( ¯ a 3 ; ¯ a 4 )j+:::+ jJ( ¯ a 1 ;:::; ¯ a k1 )j D( ¯ a k1 ; ¯ a k )+jJ( ¯ a 1 ;:::; ¯ a k )j å ¯ a k (1) where ¯ a i denotes the query range on attribute a i ,j ¯ a 1 j denotes the number of tuples that would be produced by the rst step of the range selection, D( ¯ a i ; ¯ a j ) denotes the average distance between the nodes in the DIM's containing ¯ a i and ¯ a j (for the fully decomposed case, this component can be approximated as the distance between the centroids of DIM's holding ¯ a i and ¯ a j ), andjJ( ¯ a 1 ; ¯ a 2 ;:::; ¯ a i )j denotes the number of tuples that would be produced after the ith join operation in the query. In particular, jJ( ¯ a 1 ; ¯ a 2 ;:::; ¯ a k1 )j denotes the num- ber of tuples that would be produced by k 1 joins before the nal join step; this latter yieldsjJ( ¯ a 1 ;:::; ¯ a k )j tuples that need to be aggregated within DIM DIM a k for an average ag- gregation cost per tuple of å ¯ a k (this can be estimated as the cost to move the nal aggregates to the centroid of ¯ a k , and has the advantage that it can both be computed directly from the histogram and the query, and used to reduce hotspots by fully distributing the centroid through all nodes). Note that both the join costs and the aggregation costs are clearly rep- resented in the above equation. The way the query optimizer estimates the termsj ¯ a 1 j and J( ¯ a 1 ;:::; ¯ a i ) is by using a histogram of joint density distribu- tions on all attributes. Each attribute range is divided into a small number b equal-sized bins (for simplicity, throughout this paper, we assume each attribute value a i 2 [0;1) with- out loss of generality), and a count of the number of tuples that fall into each k-dimensional slot: (b a 1 ;b a 2 ;:::;b a k ) 3 b a i 2f0;1;:::;b 1g is kept. To compute j ¯ a 1 j, we simply marginalize (sum over) all other dimensions for the range specied inj ¯ a 1 j and use linear interpolation where necessary (for partial ranges, missing histogram data, etc.) to arrive at a tuple count. Likewise, to compute J( ¯ a 1 ;:::; ¯ a i ) we count the number of tuples in the cube bounded by f ¯ a 1 ;:::; ¯ a i g after marginalizing the remaining attributes (i.e., compute å a i+1 å a i+2 ::: å a k b a 1 ;b a 2 ;:::;b a k ). The D(:) terms can be approximated as the centroid distances, and the aggregation cost (the last term) tends to be roughly the same regardless of join order. One can imagine several ways to compute the histograms (either using gossip-based mechanisms or ooding-based ap- proaches). There is some recent research in this area [22], and we just assume histograms as a given. Regardless, we observe that the overall data distribution in a sensor network is likely to change very slowly (on the timescale of hours, following diurnal patterns for example), so the histogram does not need to be recomputed very often. It is critical to note that the energy estimate used by the query planner needs to be just thatan estimate. It need not be fully accurate because we are only interested in producing join orders, not the actual energy values. However, as we shall in Section 5.3, it is important to select optimal or close- to-optimal join orders in order to be efcient. 4.3 Reducing Join Costs through Caching As we have seen in Section 4.2, the energy cost of a query has two components: the join cost, and the aggregation cost. The aggregation bit energy cost differs for the various data organizations, and is primarily a function of the size of the sensor eld. It needs to be paid for every query by all schemes. From the energy equation, we can see that the join cost can be a substantial portion of the total query cost. However, it need not be paid for by every query, and can be very effec- tively reduced through use of simple caching techniques. The caching technique we study here is a very simple lo- calized scheme. Consider a join order in which node A has to send tuples from its sub-relation to node B for joining. A remembers which tuples it has earlier transferred to B that fall within the current queried range. It then refrains from sending these tuples during this join step. B receives both the query and a partial list of tuples and knows that it has to add missing ranges of the joining tuples from its local cache. Thus, the caching protocol is straightforward and robust because the sender nodes don't need to know which receiver has which data ranges, only whether it has sent the rele- vant tuples previously or not. Furthermore, there is no dis- tributed cache maintenance overhead to guarantee correct- ness in the presence of caching node failures. When node failures cause some cached data in the receiver to become unavailable, we use a simple on-demand resend-based fail- ure recovery scheme to replenish the unavailable data. DIM provides a way for nodes in adjacent zones to takeover the failed nodes. These newly responsible nodes can invalidate the caches for data ranges in the adopted zones. Then, dur- ing the join step, they inform the sending nodes that they need to resend the entire set of tuples in the query range. An important property of our caching scheme is that the query optimizer itself does not keep track of caches; i.e., there is no cache discovery protocol. In fact, the query op- timizer is completely oblivious to the existence of caching. When it evaluates various join plans to pick a join order, it does not take caching into account although this may result in sub-optimal plans. Even so, we found that we get good performance benets over other data organization schemes, and we leave this question of exploiting caching during query optimization for future work. Distributed cache consistency is maintained through in- cremental caching. When a decomposed tuple is inserted into the joiner, the joiner records internally that this tuple needs to be sent to joined in ( joiner; joined) when a query optimizer asks to execute a join step from joiner to joined as part of a next query. The joined node then adds this tuple to the cache of previously received tuples. joined uses an LRU cache replacement policy and asks joiner to resend an evicted range as necessary. We study the effect of limited caches in Section 5.4. Finally, we note that while caching can be gainfully de- ployed in any (partially- or fully-) decomposed data orga- nization, a full DIM organization cannot leverage caching because a tuple already contains the full set of attributes. Thus, caching can be thought of as a simple yet effective technique to mitigate the join cost of a data organization that tries to minimize the aggregation cost. Full DIM's have zero join cost but high aggregation costs, and spatially par- titioned and fully decomposed 1-DIM's are at the other end of this spectrum with the lowest aggregation cost for a given number of tuples but with relatively high join costs, and can thereby gain from caching the most. This will become ap- parent when we present our evaluation results. 5. PERFORMANCE EV ALUATION In this Section, we evaluate the performance of our ap- proach using simulations over both real-world and synthetic datasets on a wide variety of query workloads. Our goal is to quantify the total performance benets of our data or- ganization and query processing over more straightforward approaches, and we use a full-dimensional DIM as the base case against which we compare energy efciency. We note that even though the full-DIM is chosen as the base, it is more energy efcient than other partial decompositions for many workloads, and is therefore intended as a competent representative for alternate data organizations. 5.1 Simulator Design We explore the performance of the various components of our scheme: the impact of DIM decomposition, the inu- ence of caching, and the importance of using good join al- gorithms. We show that, even with four sensor attributes, in- telligently querying data that is organized according to these three optimizations along with spatial partitioning delivers more than 4x benets over a 4-DIM. The simulator consists of several modules: a synthetic query workload generator that produces both online and batched workloads, a dataset generator, a histogram generator, a topol- ogy generator, an index generator to try various possible de- compositions, a global cache module, a query evaluator, an omniscient query executor (so called because it executes all possible query plans in addition to the join order selected by the query evaluator in order to determine the optimal, and worst-case plans; however, it cannot exploit global cache in- formation for online queries), and a statistics and reporting module. The topology generator also builds DIM routing tables between every pair of nodes in the network. It does this by constructing zones according to the DIM algorithm and routes between any two nodes in the simulator according to routes imported from a DIM implementation. The query optimizer computes the efciency of various join orders us- ing histograms, and a centralized executor then simulates the joins and aggregations of queries. The workload gen- erator generates both online queries to simulate an ad-hoc workload as well as a batched workload that can possibly be processed more efciently by the query evaluator because it now has the entire workload before it (see Section 5.7 for details). The relationship between the various modules is shown in Figure 3. 5.2 Methodology Our primary metric is the total bit energy cost incurred by the network as part of query execution. We evaluate this cost by keeping track of the size and number of messages transmitted by each query, and the number of hops under- taken by each message. Message sizes include header and payload sizes, and these are set to be the same as in the implementation described in Section 6 (7 and 36 bytes re- spectively). This payload and header size choice adds an overhead of 20% throughout, and the graphs shown below can be read after adjusting for this (this is only approximate, DIM Index Generator Query Executor (Omniscient) Dataset Generator Histogram Generator Statistics/ Reporting Query Evaluator Global Cache Query Distribution Generator Dataset Histogram All Possible Indices Cache Misses Query Workload (online/batch) Join Order read write Topology Generator Figure 3: Software Components of the Simulator however, because not every message is of full size, and we don't pad). However, the relative values on the plots would approximately be the same, so we do not dwell on this aspect any further. We vary the number of sensor nodes in the network from 50-200 to better understand our scaleup properties. The node locations are generated by using radio and node-connectivity models assuming a radio range of 250m and node connectiv- ity of 9. This generates the node locations within a square grid whose size is determined by the node number, radio range, and node connectivity. Without loss of generality, we then normalize the locations to fall within a unit square. We also vary the query workload from 100-400 queries or more in which each query computes aggregates over up to four attributes with each attribute being included or excluded for aggregation with equal probability; we do likewise for the range attributes. Thus, a query conforming to the query format in Section 3.3 is generated, and the terms in both the aggregate part and the range part can take on any attribute value with each having up to four attributes. We use a variety of data workloads including uniform dis- tributions, correlated Gaussian distributions (to better model natural sensor readings and to skew and stress the network), and data from a real-world set (measurements from the great duck island dataset) to better understand the behavior of our approach on varying workloads. We normalize all the data in the real-world dataset to fall within [0;1). The synthetic dataset generator also produces values in this range. For the Gaussian distribution, we gen- erate a four-attribute tuple in which the rst two attributes are positively correlated with the mean vector (0:5;0:5) and co-variance matrix 0:04 0:039 0:039 0:04 and the next two attributes are negatively correlated with the same mean vector and co- variance matrix 0:09 0:089 0:089 0:09 . The great duck island set called gdinet consists of 23548 tuples while the synthetic datasets had 4000 tuples on av- erage each. Thus, the query-data ratio varied from 10% to 0:5%. We normalize our energy cost by dividing the total cost by the number of nodes in the system as well as the number of tuples in the dataset. This gives us the aver- age cost per tuple per query. Note that this cost includes the tuple's original insertion cost as well. We also tested our approach on a wide variety of synthetic query workloads. We varied the number of dimensions and nature of both the aggregate and the range attributes in the query to simulate hot/cold behavior of attributes. We use a Poisson distribution with m = 1 to simulate queries that are popular on some attributes but can occasionally ask for the non-popular ones as well. We also tested our approach on both range and aggregate attributes drawn randomly and uniformly with an appearance probability of 1/2. We further independently varied the range size of the range attributes to see the results with both large (both exponential- and Pareto-distributions) and small (uniform) query range sizes. The start location of each of the ranges in the range attributes is picked uniformly within [0;1). This combination of datasets (synthetic with uniform/Gaussian or real-world gdinet) and query attributes (hot/cold or uni- form) and query box sizes (uniform, exponential, Pareto) gives rise to a simulation workload abbreviated as (u h e) for uniform dataset, hot/cold attributes, and exponential query size, (g u u) for Gaussian data, uniform attributes, and uni- form query size, etc. For each such variable workload, we re-ran our simula- tions with multiple instances of the workload until the stan- dard deviation of the set of averaged results fell below 20% of that of any one run. The yerrors for all points within a se- ries (like optimized in the graphs) are then calculated as the worst case yerrors and plotted. We compare the performance of our proposed organiza- tion technique with four other cases, giving us a total of ve scenarios for each workload and dataset: 4-DIM (abbreviated as 4-DIM in the graphs). This is the straightforward case with all four attributes being stored in a single DIM. 1 DIM o (abbreviated as optimized or optimal). This is the performance with optimized join-ordering and with caching enabled on 4 fully-decomposed 1-DIM's that are spatially partitioned and each assigned to one quadrant of a unit square. However, note that the op- timal join ordering does not use knowledge of the cur- rent state of the global cache, and only uses informa- tion that an oblivious query issuer described in Sec- tion 4.2 would have access to. This is computed by the omniscient query executor in Figure 3, and need not be the same as the one computed by the query evaluator in the same Figure. 1 DIM u (abbreviated as uncached). This is same as 1 DIM o above, but with caching turned off. 1 DIM r (abbreviated as random). This is the perfor- mance with random join ordering but otherwise same as 1 DIM o . 1 DIM m (abbreviated as worst). This is the perfor- mance of a worst-case join ordering, but otherwise same as 1 DIM o . 5.3 Main Results Main results from graphs in Figure 4 as well as from lots of different combinations of simulation runs not shown are: Caching eliminates a large fraction of joins, and gets us to the point of efciency of 4-DIM's after which we can use optimizations like optimal join orders, full decompositions, and spatial partitioning to deliver us signicant benets. The relative spreads of the various series (optimized, random, 4-DIM, worst, uncached) is roughly the same across all simulation runs, workloads, and number of queries. The absolute values depend on the simulation parameters. Random joining is nearly as bad as worst, and we also found (but not plotted) that average (of all possible join combinations) is also nearly as bad; this means optimal join ordering is crucial (as will become clear later on, most good enough join orders are OK too). Gaussians give much better performance over both uni- form and gdinet data because of high attribute den- sities within the bell; but this also creates hotspots, and performance degenerates if we employ additional load-balancing. However, they occur abundantly in nature, and gdinet approximates Gaussians more than uniform. While we don't report the hotspot values here, we ob- served that 4D has the highest hotspot value, and optimized has lowest (less intense than 4D by a factor of 4, as to be expected). 5.4 Queries with Correlated Attributes and Nodes with Limited Caches There are two specic scenarios when alternatives to full decomposition might shine: these are when queries have a high degree of correlation on some attributes (by this, we mean the ranges in a query tend be correlated or some range- aggregates tend to occur together, like asking for averages of high values for l and t, etc.; this notion of degree of correlation can be precisely dened). Also, in many cases, the sensors may be memory starved, and this may prevent them from fully exploiting the ideas we have described in a fundamental way: especially when queries have correlated attributes, and the query distribution is known a priori, it is better to use alternate decompositions like a 2-DIM that avoid costly redundant joins due to cache misses. In Figure 5, we examine these two scenarios where there are regimes when a 2-DIM delivers better performance than 1-DIM's. To the left, we plot the behavior of 2D (for 2- DIM) and O (for 1-DIM optimal) as the degree of corre- lation is varied. Clearly, 2D decreases sharper than O and this is because O still has to pay a near-constant aggrega- 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 0 0.2 0.4 0.6 0.8 1 1.2 Degree of Correlation Energy 2-DIM Optimized 0 2 4 6 8 10 12 0 20 40 60 80 100 % Original Cache Energy 2D(normal) 2D(correlated) Optimized(normal) Optimized(correlated) 4D(normal) 4D(correlated) Figure 5: Alternate choices with correlated queries (left) or nodes with limited caches (right) tion cost per query. To the right, we plot 2 sets of curves as we vary the amount of cache available at a node as a per- centage of the maximum-reached size: the rst set is with correlated queries, and the second is with normal queries. We consider optimal, 2-DIM, and 4-DIM. The workload is (gdinet queries on 2 attributes e). The degree of correlation here is xed at 0.7. We can see that, with correlated queries, 2D performs much better than others, but quickly loses out when there is no correlation. We also note that only a small amount of maximum cache size (20%) is necessary to make optimum better than 4D, and this can serve as a design guide- line. Also, we note that 4D-correlated is slightly better than 4D-non-correlated because 4D-correlated has two wild-card attributes. 5.5 Sensitivity Analysis and Impact of His togram Granularity In this Section, we look at the impact of histogram granu- larity on query performance. In Figure 6 we alter the num- ber of bins available per dimension between 10, 5, and 1. U stands for uniform data distribution, G stands for inde- pendent Gaussian on the attributes, and CG stands for cor- related Gaussian mentioned in the beginning of the Section. The query workload consists of uniformly chosen attributes with exponential query sizes. The three bar graphs on the left demonstrate the percentage of queries that were joined in a non-optimal way. Note that this is especially high for CG. This is small for U even with a single bin because we use interpolation. CG is worse than G because there is more correlation and predictability across attribute values, which is not captured very well by having only one bin. The three bar graphs on the right show the extra energy expended due to non-optimal joining, which turns out to be much smaller than the query percentage itself. This is because it is easy to get the query order wrong by mis-ordering even one join, and the impact of such non-optimal choices is cushioned by having caching because good orderings are adequate enough with this arrangement. 5.6 Query Saturation In this Section, we observe the average energy cost per query per tuple as the number of queries in the system is increased. We observe from Figure 7 that the query is not 0 5 10 15 20 25 0 50 100 150 200 250 Nodes Energy Optimized Random 4-DIM Uncached Worst 0 1 2 3 4 5 6 7 0 50 100 150 200 250 Nodes Energy Optimized Random Worst 4-DIM Uncached 0 5 10 15 20 25 0 50 100 150 200 250 Nodes Energy Optimized Random Uncached 4-DIM Worst Figure 4: Comparison of 4-DIM, Optimized, Random, Worst, Uncached for (u,u,u) with 100 queries (left), (g,u,e) with 200 queries (center), and (gdinet,h,u) with 400 queries (right) 0 10 20 30 40 50 60 70 80 90 U G CG U G CG Data Distributions 10 buckets 5 buckets 1 bucket % Non-optimal Queries % Energy Increase Figure 6: Histogram Sensitivity Analysis 0 1 2 3 4 5 6 7 0 500 1000 1500 2000 2500 3000 3500 Number of Queries Energy Optimized Random Worst 4-DIM Uncached Figure 7: Exploring Query Saturation saturated even at high query-data ratio ( 15%, for gdinet dataset) even on a workload with exponential query size (this means caching still has a non-zero role to play). Another nice feature is that the query tapers off monotonically with a long tail, and this means that we can quickly hit the useful knee point with only a few queries. Reaching this point is important because, then, benets of caching kick in vig- orously. Making sure queries did not saturate the dataset is important because otherwise caching could perhaps be deliv- ering only because it silently replicated the individual 1-DIM attributes. 0 1 2 3 4 5 6 0 50 100 150 200 250 Number of Nodes Energy O(online) O(batch) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 10 20 30 40 50 60 70 80 90 100 Additional Increase in Dimension % Energy Optimized Figure 8: Performance of batched queries (left), and ef- fect of increasing dimension attribute size (right) 5.7 Knowing the Query Mixture and Trade offs with Dimensions versus Measures In this last Section, we examine two miscellaneous effects: the impact of knowing the set of specic queries in a query mix a priori (that is, we are given a batched query set instead of being issued online queries, and are asked to see if we can provide some inter-query optimizations), and the tradeoffs we can make by knowing that some attributes are dimensions and not measures (see Section 5.7). In the left half of Figure 8, we show the impact of running optimized on a set of batch queries that were jointly opti- mized by trying various join orders even across queries by minimizing the total energy of all queries using Equation 1. We use a dynamic programming algorithm to achieve this minimum; the difference in performance would result from reduced joins due to increased caching. However, we see that most of the benets can be captured by using the more straightforward online approach, and this again reinforces the fact that we only need good-enough join orders; this, in turn, means that we can hope to make the query optimiza- tion process robust to various failures. In the right half of Figure 8, we show the graph of our efforts to try and extract higher efciency out of the system knowing that two attributes x and y are always present but are dimensional. The tradeoff is the increased total energy savings of the system versus increased hotspot intensityby decreasing the size of the spatial partitions allocated to the two measure attributes, we can gain in the aggregation com- Relational query Histogram Analyzer Join order selector Reliability Module Query Evaluator Histogram Generator (background, periodic) Multi-Dim Histogram Sequenced join/aggregate operators as DIM queries DIM GPSR Results More-specific queries for missing ranges, aggregates Query Engine Query Issuer Remote node overlapping first attribute range DIM GPSR To remote node-set overlapping next attribute range Multihop Mica2 radio network Attribute Value Selector Cache Manager Remaining query message 1 st attribute range Uid’s Tuple Insertion Manager DIM Insert Event Local CCP Remaining query + required uid’s Query Optimizer Query Processor Figure 9: Software Components of the Query Engine ponent (the dominant component with caching enabled) of the query execution cost. But this also results in increased hotspot levels in the nodes containing the measure attributes. We see from the Figure that we seem to gain roughly propor- tional to O( p n) where n is the additional sensornet space apportioned from a dimension attribute to its abutting mea- sure attribute; the extreme case is devoting only one node (zero area) each to the measure attributes (this extreme cost is non-zero because occasional joins still occur). Unfortu- nately, the concomitant rise in hotspot levels seems to grow linearly (not plotted), thereby making this approach gener- ally uninteresting except perhaps in special scenarios. 6. IMPLEMENTATION In this Section, we describe our design and implementa- tion of a prototype of the main data organization and query processing ideas described in the paper and report its per- formance. We have implemented this prototype on the pop- ular mica2 mote platform[11]. The software is written in NesC [14] and runs on top of an existing DIM implementa- tion done using the TinyOS [16] software platform so as to provide a realistic proof-of-concept demonstration of a prac- tical and workable system. It is designed to reuse as much of the existing software infrastructure on the mica2's as pos- sible, and achieve the desired functionality through cleanly introduced abstractions and API's. In particular, we use a full-edged implementation of DIM that doubles as the ge- ographic hash-based primary storage and range-index layer, and a complete implementation of GPSR [21] as the underly- ing routing and packet-delivery mechanism over a multihop mica2 radio network. This software stack organization and design is shown in Figure 9. In Figure 9, the query evaluator runs on the query issue node and consists of a query optimizer and an end-to-end reliability module (the reliability module is responsible for retrying missing parts of the received answers). There is also an instance of the histogram generator that runs in the background on each node that lazily collects and maintains a coarse-grained multi-dimensional histogram of the tuple values present in the DIM. The query evaluator and the his- togram generator taken together form the query processor. The query optimizer, in turn, has two sub-components: a 000 0(13) 010 011 110 111 101 1001 0011 0010 1000 3(13) 1(6) 2(6) 4(12) 9(12) 8(13) 7(12) 5(7) 6(6) Figure 10: Topology used for Implementation histogram analyzer that takes as input the current query and the k-dimensional histogram cube to compute the number of tuples that would be generated along each attribute dimen- sion, and the join-order selector that then selects the optimal join order of the attributes so as to minimize the total energy cost according to Equation 1. The output of the query optimizer is a sequenced join and aggregate operation chain that species the order in which the attributes must be moved for joining and aggregation. The resulting instruction chain is packaged as a DIM query that is then handed down to the DIM layer (using the stan- dard DIM query API) as a range query on the rst attribute in the join order, with the rest of the instruction sequence forming the query payload. Thus, the query processor con- sumes the standard DIM API in this phase, and exposes the SQL-like query API described in Section 3.3 to the user. The query engine running inside each node is responsible for controlling the cached data (timeouts, new data inserts), and is also responsible for propagating the remainder of the query and answers. Figure 10 shows an example 10 node topology of two 1- DIM's, one each on attributes l and t. The node id's are indicated in the circles. The number of tuples inserted into each of the 2 DIM's at each node is indicated in the paren- theses adjacent to the node id. The DIM zone code of a node is indicated in the upper left corner. The right half of Fig- ure 10 shows the physical arrangement of the nodes. They are congured to be within a few feet of each other, and the diameter of the network is four hops, as indicated by the logical link structure. We take advantage of the broadcast property during query execution (in the join caching phase). We ran several simple SQL queries on these two attributes, and observed a 2.7X performance improvement with 1 DIM o over standard 2-DIM. 7. FUTURE WORK AND CONCLUSION In this paper, we tried to understand the design space of data organization and query processing strategies built on top of DCS. We examined several general techniques that can be used to provide efcient and robust infrastructure support in sensornets. In particular, we identied a few key concepts like joins, decomposition, caching, and partition- ing. We wish to further our understanding of this eld along several directions, most notably caching because it seems to present further possibilities for optimization. We are also interested in examining DCS techniques in new contexts, es- pecially the idea of indexing queries instead of data as done currently, and using triggers and rendezvous to potentially achieve more energy efciency in answering queries than is possible today with data-centric indexing. 8. REFERENCES [1] P. M. G. Apers, A. R. Hevner, and S. B. Yao. Optimization algorithms for distributed queries. IEEE Transactions on Software Engineering, 9(1):5768, Jan. 1983. [2] A. Arasu, B. Babcock, C. Olston, G. Manku, J. Widom, J. Rosenstein, M. Datar, R. Motwani, R. Varma, and S. Babu. Query processing, resource management, and approximation. Nov. 14 2002. [3] R. Avnur and J. M. Hellerstein. Eddies: continuously adaptive query processing. In W. Chen, J. Naughton, and P. A. Bernstein, editors, SIGMOD 2000, volume 29(2) of SIGMOD Record, pages 261272, New York, NY 10036, USA, 2000. ACM Press. [4] P. A. Bernstein, N. Goodman, E. Wong, C. L. Reeve, and J. B. Rothnie. Query processing in a system for distributed databases (SDD-1). ACM Transactions on Database Systems, 6(4), Dec. 1981. [5] P. Bonnet, J. E. Gerhke, and P. Seshadri. Towards Sensor Database Systems. In Proceedings of the Second International Conference on Mobile Data Management, Hong Kong, January 2001. [6] N. Bruno, S. Chaudhuri, and L. Gravano. STHoles: A multi-dimensional workload aware histogram. In Proc. ACM SIGMOD, pages 211223, 2001. [7] D. Carney, U. Cetintemel, M. Cherniack, S. Lee, G. Seidman, M. Stonebraker, N. Tatbul, and S. Zdonik. Monitoring streams - A new class of DBMS applications. Technical Report CS-02-01, Department of Computer Science, Brown University, Feb. 2002. Fri, 8 Feb 102 15:36:33 GMT. [8] S. Chandrasekaran, O. Cooper, A. Deshpande, M. J. Franklin, J. M. Hellerstein, W. Hong, S. Krishnamurthy, S. R. Madden, F. Reiss, and M. A. Shah. TelegraphCQ: continuous dataow processing. In ACM, editor, Proc. ACM SIGMOD, pages 668668, New York, NY 10036, USA, 2003. ACM Press. [9] S. Chaudhuri, R. Motwani, and V . Narasayya. Random sampling for histogram construction: How much is enough? In Proc. ACM SIGMOD, pages 436447, 1998. [10] A. L. P. Chen and V . O. K. Li. Improvement algorithms for semijoin query processing programs in distributed database systems. IEEE Transactions on Computers. Vol.C33, Nov. 1984. [11] Crossbow Technology, Inc. MTS Data Sheet. http://www.xbow.com/Products/productsdetails.aspx?sid=72. [12] J. Elson, L. Girod, and D. Estrin. Fine-grained network time synchronization using reference broadcasts. SIGOPS Oper. Syst. Rev., 36(SI):147163, 2002. [13] D. Ganesan, D. Estrin, and J. Heidemann. DIMENSIONS: Why do we need a new Data Handling architecture for Sensor Networks? In Proc. HotNets-I, Princeton, NJ, October 2002. [14] D. Gay, P. Levis, R. von Behren, M. Welsh, E. Brewer, and D. Culler. The nesc language: A holistic approach to networked embedded systems. In Proc. PLDI, San Diego, CA, June 2003. [15] B. Greenstein, D. Estrin, R. Govindan, S. Ratnasamy, and S. Shenker. DIFS: A Distributed Index for Features in Sensor Networks. In Proc. IEEE WSNPA, Anchorage, AK, May 2003. [16] J. Hill, R. Szewczyk, A. Woo, S. Hollar, D. Culler, and K. Pister. System architecture directions for networked sensors. In Proceedings of ASPLOS 2000, Cambridge, MA, Novmeber 2000. [17] C.-T. Ho, R. Agrawal, N. Megiddo, and R. Srikant. Range Queries in OLAP Data Cubes. In Proc. SIGMOD, volume 26,2 of SIGMOD Record, New York, may 1997. ACM Press. [18] R. Huebsch, J. M. Hellerstein, N. Lanham, B. T. Loo, S. Shenker, and I. Stoica. Querying the Internet with PIER. In J. C. Freytag, P. C. Lockemann, S. Abiteboul, M. J. Carey, P. G. Selinger, and A. Heuer, editors, VLDB 2003, pages 321332, Los Altos, CA 94022, USA, 2003. Morgan Kaufmann Publishers. [19] H. Jagadish, J. Hui, B. C. Ooi, and K.-L. Tan. Global optimization of histograms. In Proc. ACM SIGMOD, pages 224234, 2001. [20] H. V . Jagadish, N. Koudas, S. Muthukrishnan, V . Poosala, K. C. Sevcik, and T. Suel. Optimal histograms with quality guarantees. In A. Gupta, O. Shmueli, and J. Widom, editors, VLDB'98, pages 275286. Morgan Kaufmann, 1998. [21] B. Karp and H. T. Kung. GPSR: Greedy Perimeter Stateless Routing for Wireless Networks. In Mobicom 2000, Boston, MA, August 2000. [22] Kempe, Dobra, and Gehrke. Gossip-based computation of aggregate information. In FOCS: IEEE Symposium on Foundations of Computer Science (FOCS), 2003. [23] D. Kossmann. The state of the art in distributed query processing. ACM Computing Surveys, 32(4):422469, 2000. [24] X. Li, Y . J. Kim, R. Govindan, and W. Hong. Multi-dimensional Range Queries in Sensor Networks. In Proc. Sensys, Los Angeles, CA, November 2003. [25] S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. TAG: a Tiny AGregation Service for Ad-Hoc Sensor Networks. In OSDI, Boston, MA, December 2002. [26] S. Madden, M. J. Franklin, J. M. Hellerstein, and W. Hong. The design of an acquisitional query processor for sensor networks. In Proc. ACM SIGMOD, pages 491502. ACM Press, 2003. [27] V . Poosala and Y . E. Ioannidis. Selectivity estimation without the attribute value independence assumption. In M. Jarke, M. J. Carey, K. R. Dittrich, F. H. Lochovsky, P. Loucopoulos, and M. A. Jeusfeld, editors, VLDB'97, pages 486495. Morgan Kaufmann, 1997. [28] S. Ratnasamy, B. Karp, L. Yin, F. Yu, D. Estrin, R. Govindan, and S. Shenker. GHT: A Geographic Hash Table for Data-Centric Storage. In Proc. WSNA, Atlanta, GA, September 2002. [29] S. Shenker, S. Ratnasamy, B. Karp, R. Govindan, and D. Estrin. Data-centric storage in sensornets. SIGCOMM Comput. Commun. Rev., 33(1):137142, 2003. [30] F. Tian and D. J. DeWitt. Tuple routing strategies for distributed eddies. In J. C. Freytag, P. C. Lockemann, S. Abiteboul, M. J. Carey, P. G. Selinger, and A. Heuer, editors, VLDB 2003, pages 333344, Los Altos, CA 94022, USA, 2003. Morgan Kaufmann Publishers.

Abstract (if available)

Linked assets

Computer Science Technical Report Archive

Conceptually similar

PDF

USC Computer Science Technical Reports, no. 852 (2005)

PDF

USC Computer Science Technical Reports, no. 873 (2005)

PDF

USC Computer Science Technical Reports, no. 872 (2005)

PDF

USC Computer Science Technical Reports, no. 826 (2004)

PDF

USC Computer Science Technical Reports, no. 771 (2002)

PDF

USC Computer Science Technical Reports, no. 848 (2005)

PDF

USC Computer Science Technical Reports, no. 841 (2005)

PDF

USC Computer Science Technical Reports, no. 692 (1999)

PDF

USC Computer Science Technical Reports, no. 959 (2015)

PDF

USC Computer Science Technical Reports, no. 813 (2004)

PDF

USC Computer Science Technical Reports, no. 939 (2013)

PDF

USC Computer Science Technical Reports, no. 835 (2004)

PDF

USC Computer Science Technical Reports, no. 840 (2005)

PDF

USC Computer Science Technical Reports, no. 739 (2001)

PDF

USC Computer Science Technical Reports, no. 828 (2004)

PDF

USC Computer Science Technical Reports, no. 750 (2001)

PDF

USC Computer Science Technical Reports, no. 888 (2007)

PDF

USC Computer Science Technical Reports, no. 745 (2001)

PDF

USC Computer Science Technical Reports, no. 744 (2001)

PDF

USC Computer Science Technical Reports, no. 748 (2001)

Description

Ramakrishna Gummadi, Xin Li, Ramesh Govindan, Cyrus Shahabi, Wei Hong. "Energy-efficient data organization and query processing in sensor networks." Computer Science Technical Reports (Los Angeles, California, USA: University of Southern California. Department of Computer Science) no. 839 (2004).

Asset Metadata

Creator Govindan, Ramesh (author), Gummadi, Ramakrishna (author), Hong, Wei (author), Li, Xin (author), Shahabi, Cyrus (author)

Core Title USC Computer Science Technical Reports, no. 839 (2004)

Alternative Title Energy-efficient data organization and query processing in sensor networks (title)

Publisher Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA (publisher)

Tag OAI-PMH Harvest

Format 14 pages (extent), technical reports (aat)

Language English

Unique identifier UC16270871

Identifier 04-839 Energy-Efficient Data Organization and Query Processing in Sensor Networks (filename)

Legacy Identifier usc-cstr-04-839

Format 14 pages (extent),technical reports (aat)

Rights Department of Computer Science (University of Southern California) and the author(s).

Internet Media Type application/pdf

Source 20180426-rozan-cstechreports-shoaf (batch), Computer Science Technical Report Archive (collection), University of Southern California. Department of Computer Science. Technical Reports (series)

Access Conditions The author(s) retain rights to their work according to U.S. copyright law. Electronic access is being provided by the USC Libraries, but does not grant the reader permission to use the work if the desired use is covered by copyright. It is the author, as rights holder, who must provide use permission if such use is covered by copyright.

Repository Name USC Viterbi School of Engineering Department of Computer Science

Repository Location Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089

Repository Email csdept@usc.edu

Inherited Values

Title Computer Science Technical Report Archive

Coverage Temporal 1991/2017

Repository Email csdept@usc.edu

Repository Name USC Viterbi School of Engineering Department of Computer Science

Repository Location Department of Computer Science. USC Viterbi School of Engineering. Los Angeles\, CA\, 90089

Publisher Department of Computer Science,USC Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, California, 90089, USA (publisher)