University of Southern California Dissertations and Theses

Augmenting knowledge reuse using collaborative filtering systems

(USC Thesis Other)

Augmenting knowledge reuse using collaborative filtering systems

PDF

Download

Open document

Flip pages

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content AUGMENTING KNOWLEDGE REUSE USING COLLABORATIVE FILTERING SYSTEMS by II Ini A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (INFORMATION SYSTEMS) May 2001 Copyright 2001 II Im R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. UMI Number: 3 0 2 7 7 3 0 Copyright 2001 by Im , I I All rights reserved. ® UMI UMI Microform 3027730 Copyright 2001 by Bell & Howell Information and Learning Com pany. All rights reserv ed . This microform edition is protected against unauthorized copying under Title 17, United S ta te s C ode. Bell & Howell Information and Learning C om pany 300 North Z eeb Road P.O. Box 1346 Ann Arbor, Ml 48106-1346 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. UNIVERSITY OF SOUTHERN CALIFORNIA The Graduate School University Park LOS ANGELES, CALIFORNIA 900894695 This dissertation, w ritten b y T L _ _ i M __________ Under th e direction o f hi?.... D issertation C om m ittee, and approved b y a ll its m em bers, has been p resen ted to an d accepted b y The Graduate School, in p a rtia l fulfillm ent o f requirem ents fo r th e degree o f DOCTOR OF PHILOSOPHY Dean o f G raduate Studies D ate May 11, 2001 DISSERTA n O N C O M M II j L C A s'-S '\ C hairperson / / / R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. TABLE OF CONTENTS CHAPTER I. INTRODUCTION 1 1.1. Motivation.................................................................................................................1 1.2. Goals of the study ....................................................... 4 1.3. Structure of this dissertation................................................................................... 5 CHAPTER II. COLLABORATIVE FILTERING TECHNOLOGY 6 2.1. Overview on collaborative filtering technology....................................................6 2.1.1. An algorithmic representation of CF process..................................................7 2.1.2. Recommendation generation algorithms..................................... ................. 8 2.1.3. Example of CF................................................................................................11 2.1.4. CF system architecture................... 12 2.2. Previous research on CF systems..........................................................................14 2.2.1. Research on CF algorithms............................................................................ 15 2.2.2. Research on applications of CF ............................................................18 2.3. Limitations of previous studies in C F .................................................................. 19 CHAPTER III. MANAGEMENT ISSUES IN COLLABORATIVE FILTERING 22 3.1. Benefits of C F............................. 22 3.1.1. Benefits from using CF for consumer products........................................... 23 3.1.2. Benefits from using CF for knowledge intensive products......................... 23 3.2. The status of current CF products.........................................................................25 3.2.1. Firefly............................................................................................................... 25 3.2.2. Net Perceptions (GroupLens)................................................ 26 3.2.3. Agentware......................................................................... 28 3.2.4. WiseWire ..............................................................................................30 3.2.5. LikeMinds........................................................................................................32 3.2.6. Alexa........................ 34 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER IV. RESEARCH ISSUES AND HYPOTHESES DEVELOPMENT 36 4.1. Impact of CF on information search and knowledge reuse................................ 36 4.2. Research issues in CF............................................................................................ 40 4.2.1. Critical mass of users and evaluations .............................................. 42 4.2.2. System characteristics.................................................................................... 43 4.2.3. Mode of search............................................................................................... 45 4.2.4. User characteristics ..................... 48 4.2.5. Preference cluster patterns ........ 53 4.2.6. Performance measures....................................... 64 4.3. Hypotheses development................. 6 8 4.3.1. Critical mass of users and evaluations.......................................................... 6 8 4.3.2. Reference group size ....... 70 4.3.3. Mode of search................................................................................................72 4.3.4. Characteristics of users.................................................................................. 74 4.3.5 Pattern of users’ preference.......................................................................... 74 CHAPTER V. EMPIRICAL STUDY 76 5.1. Research Design.............. 76 5.1.1 Selection of domains ................................................................................. 77 5.1.2. Evaluation methods (measures)..................................................................... 78 5.1.3. Similarity measure ................ 79 5.1.4. Experiment 1 (Pilot experiment)................................................................... 80 5.1.5. Experiment 2 (Main experiment).................. 81 5.2. Simulation.......................................................................................... 85 5.2.1. Simulation method.......................................................................................... 8 6 5.2.2. Simulation procedure.................................................................... 87 5.2.3. Settings for the simulation............................................................................. 89 5.3. Descriptive statistics..................................................................................... 94 5.3.1. Experiment 1 (Pilot experiment)................................................................. 94 ii R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.3.2. Experiment 2 (Main experiment)................................................................94 5.4. Hypothesis tests......................................................................................................96 5.4.1. Number of users and the performance of CF systems ...........................96 5.4.2. Reference group size and the performance of CF systems........................107 5.4.3 Mode of search and the performance of CF systems...................................113 5.4.4 Characteristics of users and the performance of CF systems......................115 5.4.5 Pattern of users’ preference and the performance of CF systems.............. 124 5.5. Summary of the result......................................................................................... 126 CHAPTER VI. CONCLUSIONS AND FUTURE RESEARCH 129 6.1. Conclusions...........................................................................................................129 6.2. Implications of the study..................................................................................... 130 6.2.1. Implications for researchers........................ 130 6.2.2. Implications for practitioners ..........................................................133 6.3. Limitations of the study.................................................................... 134 6.4. Future research directions ........................................................................... 135 REFERENCES 138 I. Books and articles.....................................................................................................138 II. World-wide Web sites............................................................................................146 APPENDIX147 A. Screen shots of the system for experiment 1 (Movies)....................................... 147 B. Screen shots of the system for experiment 2 (Research Papers)......................... 149 C. Questionnaire after the experiment (Research Papers).........................................152 D. The soliciting e-mail message sent to the subject pool........................................153 E. Screen Shot of the Simulation Program and an Example of Simulation Output 155 F. Source code of the collaborative filtering engine................................................. 156 iii R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. LIST OF FIGURES Figure 1. Algorithmic expression of CF process ........................................................ 8 Figure 2. Basic components of collaborative filtering system................. 12 Figure 3. Procedure of CF and Alternative Methods ............ 18 Figure 4. Firefly system architecture............................................................................... 26 Figure 5. GroupLens system architecture........................................................................27 Figure 6 . WiseWire system architecture..........................................................................31 Figure 7. LikeMinds system architecture.........................................................................33 Figure 8 . Factors affecting the accuracy of CF systems.................................................41 Figure 9. Different Patterns of Customer Preference Clusters with Two Items........... 55 Figure 10. Distribution of Two-person Proximity Measures......................................... 57 Figure 11. Hypothetical Relationships Between Total Number of Users and Performance of CF Systems.....................................................................................69 Figure 12. Reference group size and expected performance of CF system..................71 Figure 13. System Architecture for the Main Experiment.............................. 82 Figure 14. Holdout Set and Calibration Set in the Simulation.......................................87 Figure 15. Number of Users and CF Performance (Research Papers)..........................98 Figure 16. Actual Data and Fitted Data by Straight Line and S-Shape Curve........... 100 Figure 17. Number of Users and CF Performance (Movies)....................................... 103 Figure 18. Reference Group Size and Recommendation Accuracy (Research Papers) 108 Figure 19. Reference Group Size and Recommendation Accuracy (Movies)............109 Figure 20. Optimal Reference Group Size and Number of Users................................110 Figure 21. Search Mode and Recommendation Accuracy............................................114 Figure 22. Experience and Accuracy of C F ...................................................................119 Figure 23. Number of Evaluations and CF Performance (Research Papers).............. 122 Figure 24. Number of Evaluations and CF Performance (Movies)....................... 123 iv R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. LIST OF TABLES Table 1. Example of C F .................................................................................................... 11 Table 2. Summary of CF tools in market.........................................................................35 Table 3. Characteristics of scanning and problemistic search .............................47 Table 4. K-S Test Statistics for Different Cluster Patterns............................................ 60 Table 5. K-S D Statistic for Different Numbers of Items.............................................. 63 Table 6 . Example of Rank-Based Performance Measure.............................................. 67 Table 7. Number of Iterations and Standard Deviations of Accuracy...........................93 Table 8 . Test Results of Pattern of Performance Increase (Research Papers)............102 Table 9. Test Results of Pattern of Performance Increase (Movies)........................... 104 Table 10. Perceived Accuracy of Recommendations Across Time................. 106 Table 11. Test Results of Pattern of Performance Increase (Research Papers)...........111 Table 12. Descriptive Statistics of Subgroups............................................................... 116 Table 13. Simulation results for user groups with different experience......................118 Table 14. Number of Evaluations and Number of Years in IS Field .................. 121 Table 15. Number of Evaluations and CF Performance (Research Papers)............... 122 Table 16. Number of Evaluations and CF Performance (Movies).............................. 123 Table 17. K-S D Statistics of theThree Domains..........................................................125 Table 18. Accuracy of CF System for Movies and Research Papers.......................... 126 Table 19. Summary of Hypotheses Test.............................................. 127 v R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. II Im Chair: Dr. Alexander Hars Augmenting Knowledge Reuse Using Collaborative Filtering Systems In today’s networked business environment, with the endless increase in available information, relevant information is becoming more and more difficult to find. Collaborative filtering (CF) generates recommendations for users based on others’ evaluations. CF has great potential to improve information search and knowledge reuse. Previous studies have focused on the improvement of CF algorithms. Little research has been done on the effect of users and types of product domains on the performance of CF systems. In this study, four factors - product domain, user characteristics, user’s search mode, and number of users-that are expected to affect the accuracy of CF systems - were identified and investigated. Five main hypotheses were developed from previous CF research and related literature. HI. The performance of a CF system increases as the number of users increases. H2. Reference group size affects the performance of a CF system. H3. The performance of CF differs depending on the search mode of users. H4. The performance of a CF system is affected by the characteristics of users. H5. The performance of a CF system is affected by the patterns o f users ’ preference clusters. These hypotheses were tested using data collected from two experiments in two different product domains, movies and research papers. It was shown that reaching vi R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. critical mass takes longer in a consumer domain than in a knowledge-intensive domain. Users’ search mode and knowledge in a domain are important determinants of the accuracy of a CF system. Adapting the theories in the ‘mental model’ and ‘cognitive map’ areas, this study provides theoretical explanations on how CF helps users find relevant information and why the accuracy of CF systems differ in different search modes and with the user’s level of knowledge. The results show that managers need to invest in collecting enough initial data to reach critical mass quickly. System implementers have to design CF systems customized for different domains and user groups. A successful CF system needs to be designed to handle multiple modes of search, even within a domain and user group. Keywords: Collaborative filtering, knowledge management, e-commerce, information search, recommendation system R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER L INTRODUCTION 1.1. Motivation We are moving towards an economy where competitive advantage is determined by knowledge rather than by access to raw materials and cheap labor. In this environment, organizations are forced to create more knowledge in as short a time as possible. This requires that they improve the reuse of knowledge that already exists within their organization. A problem these companies encounter is in identifying reusable knowledge. Even with the support of classification and search tools, it takes a long time for a user to locate needed knowledge within a large repository. In a recent survey [Rozgus and Sicaras, 2000], respondents indicated that the major shortcoming of search engines was their inability to find needed information. If information search costs outweigh the benefits of the information, people will not try to search for information in the repository. Thus, faster and better tools to identify reusable knowledge are of critical importance for effective knowledge reuse. Several different approaches to this problem have been attempted in the information systems area. Natural language query for better information retrieval [Adam et. al., 1994], genetic algorithms for classifying documents [Chen and Kim, 1994/1995], and software agents [Wang, 1999] are examples of the research on more effective information search. 1 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. One of the solutions to this problem is “collaborative filtering (CF)” a technology that provides users with personalized recommendations based on their preferences [Schafer et al., 1999]. CF systems identify a group of people with similar preferences and use their evaluations to generate recommendations. CF is a relatively young, but not immature, technology [Kostan et al., 1997]. CF is used in an increasing number of online companies such as Amazon (www.amazon.com), CDNow (www.cdnow.com), Levi’s (www.levis.com). and Moviecritic (www.moviecritic.com) [Schafer et al., 1999]. As electronic commerce (e-commerce) becomes one of the major ways of doing business [Evans and Wurster, 1999] and the demand for customized services surges, CF systems become a readily applicable and crucial tool for e-commerce companies. Because it is easier to enter information on the Internet customers when they buy consumer goods in an Internet store, are more willing to enter their evaluations than they are in a real store. Also the customer knows that in the e-commerce environment evaluation information can be collected in real-time and fed back to the customer as useful recommendations. While CF has been successfully used by many companies to recommend CD’s, books and movies [Wingfield, 1998], only a few companies have applied it to their internal knowledge repositories such as documents or manuals. These knowledge repositories are less standardized and the need for knowledge is highly situation- dependent. CF has great potential in facilitating knowledge reuse. This is because CF helps users acquire new knowledge by reducing the time needed to search and 2 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. evaluate it. For example, a study of executive information systems (EIS) suggests that providing relevant information is one of the keys to successful EIS systems [Rainer and Watson, 1995], Therefore, CF may play an important role in knowledge-intensive domains, as well as in consumer product domains, by helping users find relevant information. Another venue of CF research, where little work has been done, is the behavioral aspect of CF systems. Since a CF system is also an information system, their success also depends on how they are used. Designers often focus on finding uses for tools, and deploying cutting-edge new features but overlook that their primary focus should be on providing value to the end users [Kramer et al., 2000]. The value of CF systems to end-users has not been adequately investigated in previous studies. For example, the characteristics of users have been only briefly explored in previous studies [Miller et al., 1996; Podberezniak, 1998]. Many questions about users’ behavioral issues still need to be answered; for example, how the effectiveness of CF varies with the characteristics of users, and also how other factors affect the performance of CF systems. To utilize CF systems more effectively, these questions need to be answered. This study is an effort to extend behavioral knowledge on CF systems. Because behavioral issues are as important as technical issues for the effective use of CF systems, more research is needed to guide the design and application of CF systems in various domains. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 1.2. Goals of the study This study investigates CF in ‘knowledge-intensive’ domains such as corporate repositories, which are different from consumer product information repositories. Investigation of CF in a knowledge-intensive domain will provide valuable knowledge for both researchers and practitioners. For researchers, a better understanding of CF will be obtained by comparing CF in a knowledge-intensive domain with CF in a consumer product domain. Contrasting the differences between CF for consumer products and CF for knowledge-intensive products will promote theoretical development of the phenomena underlying CF. For practitioners, the study will help provide guidelines for designing and implementing CF in knowledge-intensive domains. Another goal of this study is to answer the question, ‘what are the critical factors that determine the performance (accuracy) of a CF system in a knowledge-intensive domain?’ Before answering this question, important factors for CF performance will be identified. The effect of these factors on the performance of CF systems is also examined. Thus, a better understanding of the factors around CF systems other than algorithms will be obtained through this study. This will also provide practitioners with guidelines for designing and implementing CF systems in a knowledge-intensive domain. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 1.3. Structure of this dissertation As the first step to answering the questions raised above, an overview of CF and previous studies on CF technologies are reviewed in Chapter 2. Also in Chapter 2, previous studies on technical issues (algorithms to generate recommendations) are reviewed and discussed. Discussed at the end of Chapter 2 are the limitations of previous studies. Chapter 3 is about management issues surrounding CF. Theoretical explanations of how CF aids in information search and knowledge reuse in business repositories is discussed. Also reviewed are CF tools available in the market chapter 4 discusses critical success factors and research issues in CF systems. Hypotheses are developed based on the research issues. In Chapter 5, empirical studies conducted to test the hypotheses developed in Chapter 4 are described. The hypotheses are tested in using data collected from two experiments. Chapter 6 contains the conclusions and directions for future research to take limitations of the study and their implications for researchers and practitioners are also discussed. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER II. COLLABORATIVE FILTERING TECHNOLOGY This chapter discusses technology issues in CF. These issues include the underlying mechanism of CF, technical issues surrounding CF systems, and previous studies on CF technology. 2.1. Overview on collaborative filtering technology The concept of ‘collaborative filtering’ was enunciated by Goldberg et al. [1992], who were the first to publish an account of using collaborative filtering techniques in information retrieval. Collaborative filtering (hereafter referred to as CF) generates recommendations for users based on others’ evaluations. More precisely, CF systems make use of the reactions and opinions of people who have already seen a piece of information to make predictions about the value of that piece of information to people who have not yet seen it [Miller and Konstan, 1997]. CF reduces the time needed to search for information by systematically capturing and leveraging others’ evaluations. Without a CF system, evaluations on information items must be transferred by ‘word- of-mouth’, which is haphazard and inaccurate. By institutionalizing evaluations, CF makes it possible to transfer consistent and precise evaluations among larger numbers of people. A CF system needs a certain number of initial evaluations from a user in order to be able to generate recommendations for her/him. Each user is asked to evaluate a 6 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. certain number (usually 10-15) of items before he/she starts getting recommendations. The system stores and then compares these evaluations with those of other users to identify those who have similar evaluations (preferences). After a set of people with similar preferences - termed a ‘reference group’ in this study - is identified, the system looks up the items that have not been seen by the user who is to get recommendations. The system calculates the user’s expected evaluation score for each item based on the evaluations of the people in the reference group. This overall recommendation process is the same for most CF applications although the detailed algorithms for identifying reference groups and calculating expected evaluation scores differ with each application. 2.1.1. An algorithmic representation of CF process A good way of explaining the CF process is an algorithmic description. The algorithmic expression of the process to generate recommendations using CF is as follows. Stepl: Check if the user is a new user. If he/she is a new user, get initial evaluations. Step2: Based on the evaluations of the user, calculate similarity indices with other users or classify this user into a cluster. Step3: Calculate the users estimated evaluation for items that have not been seen by the user. Step 4: Recommend items that have the highest estimated evaluations to the user. If the user wants to see more recommendations, the next items are shown. 7 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. This algorithm is shown as a flow chart in Figure 1. C, rul • p Si r ■ rtly Step 2 AI Oti t Utr r j NeW'User?: C J* u 11- fir -ijte Fvr>udtrJ* » f e ) u Step 3 Step 1 R ecom m endations? K,hi, Step 4 Figure 1. Algorithmic expression of CF process 2.1.2. Recommendation generation algorithms Put simply, collaborative filtering systems are used to predict the evaluation of users from the users’ evaluations in the database. Many algorithms have been developed to conduct this task. Breese et al. [1998] propose two categories of CF algorithms - memory-based algorithms and model-based algorithms. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 2.1.2.1. Memory-based algorithms Memory-based algorithms predict the evaluations of a particular user based on some partial information regarding the active (current) user and a set of weights calculated from the user database. The predicted evaluation of the active user i for item j ( eif) is a weighted sum of the evaluations of the other users. Estimated evaluation: ei } = ef{st,,em j) where, w f(sit) : weight function ei}: Evaluation by ith user on j th item All evaluations by ith user (=en,ei2,...,eim ) e.j: All users’ evaluations on j th item (= elj,e2j,...,e2n) sab: Degree of similarity of user a and user b S a b ~ s f (ea * ’eb*) Two methods are widely used to calculate the similarities between two users - correlation and vector similarities. n Estimation function: efy (sit ,etj) H * 0 1=1 m Correlation: Vector similarity: sf (ea. ,eh J = |Eh - Ea where Ea = (eal,ea2,...,e<m)and Eb = (eM ehm ) 9 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 2.1.2.2. Model -based methods Model-based methods predict the expected evaluation of a user on each item based on mathematical models. Although there are many variations of the model-based algorithms, they can be categorized into two groups - cluster models and Bayesian network models. Cluster models classify users into a certain number of classes and calculate the probabilities that the members of a class give a rating score of k to an item j. The probability can be calculated using a standard Bayesian probability formula. n Bayesian probability: Pr(C = c,e,,...,en) = Pr(C = c)]JPr(ei IC = c) 1 = 1 where, et : evaluation for item I C : class membership The probabilities of class membership ( Pr(C = c) ) and the conditional probabilities of evaluations given class ( P r ^ .1C = c) ) are estimated from a training set of user evaluations. A Bayesian network model represents the items as nodes and the evaluations as the states of the nodes. The learning algorithm searches over various model structures in terms of dependencies for each item. In the resulting network, each item will have a set of parent items that are the best predictors of its evaluations. 10 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 2.1.3. Example of CF To explain how CF works, let us use movies as an example. Suppose there are five users, Ui to U5, and five movies, Mi to M5, to be evaluated in a CF system. Suppose the users’ ratings of movies are those shown in Table 1 and recommendations for Ui are being generated. Also, assume that this system uses correlation as the similarity function (sf) and selects 2 people most similar to a user and averages these two people’s evaluations to calculate estimated evaluations (etj). Table 1. Example of CF Movie U ser m2 m 3 m 4 M5 Correlation With U, (Slm ) Ui 2 5 3 - ua 4 4 3 5 1 0.19 U3 1 5 4 5 0.89 U4 3 5 3 2 5 0.94 U5 4 5 3 4 0.65 A verage of U3an d U4 (£,.) 2 5 3.5 2 5 (1 = bad, 5 = good) From the correlations, which are similarity indices (Su ) in this example, U3 and U4 are selected because they have the highest similarity indices with Ui (0.94 and 0.89). For the movies M4 and M5, which user 1 has not seen yet, user l ’s expected ratings (e,.) are calculated by averaging the ratings of U3 and U4. Average ratings for M 4 and M5 are 2 and 5, respectively. Thus, movie 5 is recommended to user 1, because it has the highest estimated evaluation (5). R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 2.1.4. CF system architecture CF is usually implemented on a database that works as a knowledge repository. There are three essential components of CF - a database that stores user profiles, a data entry module, and a recommendation module. Two additional modules are found in some advanced CF tools: a collection module and a monitoring module. A graphical representation of CF is shown in Figure 2. Notiftcatioi (Alerfesf Monitoring M odule Evaluations P referen ces Monitoring infornfation 2 fkiiL .jle .n■ dict-isir* > /r' \ R ecom m endation Module T!enr" Milder Cala I m o i i i . i I I i l l ■ n m . i ' ii m i Information Items Collection M odule Figure 2. Basic components of collaborative filtering system 2.I.4.I. Databases for user profiles and item master data For CF to work, the system must be able to store and use users’ preferences. The simplest form of profile is classifications by the users themselves. This type of profile can be collected by directly asking users for their preferences. More sophisticated CF 12 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. tools use special algorithms to indirectly discover similarities between users from their evaluations on the items in the database. 2.I.4.2. Data entry module Data entry accepts user preferences and new information items. The data entry module can be a web browser or a special application. In sophisticated CF tools, data entry modules record user behavior - the items the user browsed, the time spent in opening and reading each item, etc. Users’ behavior can be used as supplementary preference information because frequent retrieval and long stays imply higher levels of preference. 2.1.4 3 . Recommendation module The recommendation module is the key part of a CF tool. This module stores association information between the users and the information items - which user provided which evaluation score for which item. This association information is used to cluster users and to generate recommendations. A variety of algorithms are used to calculate the extent of similarity between users or between information items. 2.1.44. Collection module This module employs a ‘robot’, a program that goes out and visits WebPages as programmed, to collect information from specified websites or databases. The 13 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. collected information is filtered and pre-categorized according to the criteria specified by users. The categorized information items are stored in the master database. 2.1.4.5. Monitoring module This module continuously monitors the changes in the master database and checks if new information items are rated by a certain number of users. Once a new item receives a certain number of ratings, the monitoring module identifies the users who will, potentially, be interested in this item. It adds the new items to a customized newsletter or sends e-mail to announce the arrival of a new item. 2.2. Previous research on CF systems Previous research has studied CF on various issues. One of the research streams is about ‘algorithms’ used in CF systems. This stream of research has sought to improve the CF algorithms. Some of the studies compared different algorithms [Shardanand and Maes, 1995; Breese et al., 1998], sought variations of algorithms to improve performance [Herlocker et al., 1999], and investigated combinations of CF with other methods [Good et al., 1999; Ansari, 2000]. A second stream of research has focused on the application and use of CF. The studies in this stream investigated the applications of CF in e-mail messages [Goldberg et al., 1992], music [Shardanand and Maes, 1995], movies [Greening, 1996], Web pages [Rucker and Polanco, 1997], 14 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Usenet messages [Konstan et al., 1997; Miller et al., 1997], Internet resources [Terveen et al., 1997], and TV programs [Podberezniak, 1998]. 2.2.1. Research on CF algorithms Herlocker et al. [1999] tested several variations of CF algorithms. They tested similarity weighting methods, significance weighting, variance weighting. neighborhood selecting methods, and prediction producing methods. They found the following: • Similarity weighting methods: Two similarity indices (Pearson correlation coefficient and Spearman rank correlation) were compared, and it was found that the two methods performed similarly. • Significance weighting: Significance weighting refers to the weighting method that gives higher weights to the evaluations from those who have many items common with the current user. This method is expected to improve performance because evaluations of users who have only a few common evaluations are often terrible predictors for the current user. Significance weighting was found to improve the accuracy of CF systems. • Variance weighting methods: Variance weighting refers to the weighting method that gives higher weights to the evaluations from those who have been proved to be very accurate evaluators. This weighting did not have a significant effect on the accuracy of CF systems. 15 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. • Neighborhoods selecting methods: In order to generate recommendations, a CF system must select a ‘preference neighborhood’ (reference group) for the current user. This is a group of people who have the highest level of similarity with the current user. The neighborhoods can be selected using ‘correlation-thresholding ’ or ‘ best-n-neighbors’. The correlation-thresholding is the method that selects neighbors who have a correlation with the current user higher than a certain threshold (e.g. 0.5 or 0.7). The best n neighbors is the method that selects people who have the highest correlations with the current user. It was found that correlation-thresholding performed better than the best n neighbors, but with an unacceptable loss in coverage (percentage of people to whom the recommendations can be generated). • Prediction producing method: Once the neighborhoods have been selected, the ratings from those neighbors are combined to compute a prediction. One problem in combining the evaluations of the neighbors is that users may have different variances in their evaluations. The variances can be removed by adjusting the evaluations by users’ variances (bias-from-mean). Herlocker et al. found that the adjusted prediction method performed significantly better. They also showed that the performance of CF systems decreases when too small or too big a reference group size (number of people in reference group) is used for predictions. Breese et al. [1998] compared several different algorithms used for CF systems in 3 different application areas. They compared four CF algorithms - correlation, vector 16 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. similarity, Bayesian clustering, and Bayesian network - in Web page visits, TV programs, and movies. They found that correlation and Bayesian network performed almost equally in most cases. They also tested the effect of inverse user frequency on the performance of CF systems. Inverse user frequency is a method that assigns less weight to the items that most users commonly like. The idea is that universally liked items, for example ‘Titanic’ in movies, are not as useful in capturing similarity as less common items. It was found that the inverse user frequency consideration significantly improved performance. Shardanand and Maes [1995] compared two different algorithms for generating similarity indices - Pearson correlation coefficient and mean squared difference. The results show that the Pearson correlation coefficient performs best in terms of accuracy and coverage. They also compared the effect of increase in number of users. They found that, as expected, the increase in the number of users led to higher prediction accuracy. However, they have not investigated whether there emerges any patterns in the increase of accuracy as the number of users increases or the increase of accuracy commonly occurs for different conditions (different types of products and/or different user groups). Freund et al. [1998] proposed a new algorithm for calculating estimated evaluations based on ranks of the evaluations. They showed that their algorithm for calculating estimated evaluations, named RankBoost, performed better than other generally used algorithms such as nearest neighborhood. 17 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Ungar and Foster [1998] proposed a formal statistical approach to CF algorithms. They developed an algorithm which was based on a probabilistic model and applied it to a synthetic and to an empirical data set. They found that their system worked well for the dataset that had clusters of varying size. C F P ro ced u re Calculate Sintlairv idcx ^ -do-iffy r-ie'o cnee Oioup c .a ou.ato E.$ti-n a ted E valuation M eth o d s • Index type (Correlation, Vector similarity, • Best-n-neighbors • Threshold • Variance weighting • Mean adjusting Bayesian probability) • Significance weighting Figure 3. Procedure of CF and Alternative Methods Figure 3 shows a diagram that summarizes the previous research on CF algorithms. Previous studies have focused on methods to generate recommendations and investigated different alternatives, seeking better methods to generate recommendations. 2.2.2. Research on applications of CF An earlier study [Miller et al., 1997] showed a correlation between users’ evaluation patterns (how similar users’ evaluations are) and CF performance. In the study, a CF application was applied to several different Usenets - humor, food- recipes, and a Linux system. The results showed that different Usenets differ in average ratings - some Usenets have harsher average ratings than other Usenets - and in correlation between the evaluations of the users. Although rigorous investigations 18 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. were not conducted, the results show that the overall performance of the CF system was better for the Usenets with high correlations among users. However, in relative performance (differences between estimates by CF and estimates by everybody’s average), Usenets with low correlations among users showed much better performance. This implies that CF is more effective for the items for which users’ evaluations have large variations. There have been efforts to combine CF systems with other filtering methods such as personalized agents. Good et al. [1999] tested the performances of four different types of recommendation systems - a pure CF system, a single personalized agent (a machine learning or syntactic filter), a combination of many agents, and a combination of multiple agents and CF. They found that combining CF with agents resulted in better performance than CF or agents alone. They also found that although CF was more useful with a sparse dataset, it is not as accurate as expected with a small community of less than 50 people. 2.3. Limitations of previous studies in CF CF systems are also information systems. Studies on the success of information systems have shown that the characteristics of users is one of the most influential factors of system success. Numerous studies [Griffith and Northcraft, 1996; Newman and Shabherwal, 1996; Essex et. al., 1998] have shown that people were a critical success factor in information system development. When it comes to the information 19 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. systems that require active interactions between users and systems such as an executive information system (EIS), the user factor becomes more important for the success of the system. For example, Yandenbosch and Higgins [1995] examined the performance of EIS and found that the success of EIS may be contingent upon the ‘type of user’s learning. In CF systems, users play an even more important role because the information processed by the system is the record of the judgments received from users. Thus, the user is likely to be one of the most critical factors in the success of CF systems. Previous studies have focused on the improvement of CF algorithms and application of CF to new products. Little research has been done on the effect of users on the performance of CF systems. For example, the characteristics of users have been only briefly explored in previous studies [Miller et al., 1996; Podberezniak, 1998]. Another area that previous research in CF has not covered is the relationship between types of products (domains) and the performance of CF systems. Since CF systems generate recommendations based on other users’ evaluations, the accuracy of the recommendation is affected by the characteristics of the evaluations - mean, variance, how they are distributed, etc. The characteristics of the evaluations would differ from product to product. For example, the evaluations across users for movies may be quite different from evaluations across users for books. In previous research, the effect of differences in these evaluations - therefore, the effect of product type - on the performance of CF systems has not been explored sufficiently. Only a few studies, for example, Miller et al. [1997] indirectly investigated the effect of product 20 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. on the performance of the system. Understanding the effect of the product domain on the performance of the system is very important for both researchers and managers. By understanding the effect of “type of product” on the performance, theoretical explanations on the fundamental mechanisms of CF systems (how they work under different circumstances) can be developed, and more effective application of CF systems to different types of products made possible. 2 1 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER III. MANAGEMENT ISSUES IN COLLABORATIVE FILTERING In this chapter, managerial issues that arise in implementing CF systems are discussed. The issues are, what benefits are expected from a CF system if it is applied to business, and what CF tools are available in the market. 3.1. Benefits of CF CF provides many benefits to both organizations and to their customers. In a company, CF can be used for various tasks. Most current CF tools are being used for generating recommendations for consumer products such as movies, CDs, and clothes. The preferences of people for products in these domains are relatively consistent across time - if a person liked ‘Star Wars’, he/she will probably like it for long time. CF can also be applied to knowledge intensive products. An example of CF use for a knowledge-intensive product is in the selection and recommendation of documents appropriate for a user given his/her interests. Another example of CF use would be automatic e-mail alerts triggered by the addition of an information item to the repository. The emails are sent to people whose interests match with the new information. 22 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 3.1.1. Benefits from using CF for consumer products Companies benefit from using CF for consumer products. If CF technology is used for electronic storefronts, companies can collect detailed preference information about customers. Customers will voluntarily enter information since they understand that the more information they enter the more precise will be the recommendations they get. Customers can be classified into a range of groups - from a handful to hundreds - based on their preferences, instead of their self-reported demographic variables such as age and occupation. Moreover, customer information evolves and becomes more precise as more customers and more information from existing customers are added. The recommendations that customers receive are real time - if a customer adds more information, the recommendations immediately reflect this change. Customers will get better service because the information provided to them can be personalized based on their preference/interest profiles. 3.1.2. Benefits from using CF for knowledge intensive products If companies use CF for knowledge intensive precuts, an information search can be more efficient and effective, promoting more utilization of databases and better knowledge sharing and reuse. People seeking help in organizations often search for others who have similar tasks/interests or experience. When it comes to knowledge- intensive work, the need for finding people with common interests becomes more critical because a large part of knowledge work is stored in the human brain. Finding 23 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. the appropriate people is not an easy task; it is constrained by the searcher’s range of acquaintances and his/her accuracy of judgment. CF technology has great potential in identifying people with common interests. A CF system identifies people who evaluated items in a similar pattern. People working on similar projects will have similar knowledge needs and, thus, will show similar evaluation patterns. If users A and B, for example, have similar tasks, the knowledge/information they need to successfully complete the task will be similar. In other words, they have very similar criteria for evaluating knowledge items and their evaluations of knowledge items are likely to follow similar patterns. If a user wants to locate people who have similar interests or have had similar problems, he/she can use the CF system in the following way. The user evaluates a certain number of initial items and then asks the CF system to locate people who have entered similar evaluations. The system then compares these evaluations with other users’ evaluations and finds a set of people who have evaluation patterns similar to those of the user. The CF system responds with contact information, so the user can send emails or call them. Since CF systems keep all information - all users’ evaluations on all items - in the repository, locating people is not restricted by the user’s range of acquaintances. Also, any clustering by CF would be more accurate than human subjective judgment because it is rooted in the users’ own evaluations. 24 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 3.2. The status of current CF products The following are descriptions of the major CF applications available to businesses. 3.2.1. Firefly Firefly (www.firefly.com) is a CF product used in Yahoo! (my.yahoo.com), Barnes & Noble (www.barnesandnoble.com), and Filmfinder (www. filmfinder.com). Firefly is a system for recommending items - messages, WebPages, or products - to a user based on the similarities of users’ preferences. If a user registers for a Firefly account, called Firefly Passport, he/she can do a personalized search or get personalized book recommendations. Firefly stores rating data for items provided by users of the system. It uses a 1 to 7 rating system for recommendations. A user must rate at least six items before receiving any recommendations. Upon request of the first user, the system determines similarity values for the first user as compared with other users who have provided rating data for items that the first user has also rated. Based on the similarity values, a subgroup of users is selected that is then used to provide recommendations to the first user. Firefly's recommendation system is made up of several parts: the actual technology, called Advanced Collaborative Filtering (ACF), and various tools that help Webmasters implement ACF. To build Firefly's technology into their sites, 25 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Webmasters use either HTML extensions or Firefly's object-oriented C++ APIs (Application Program Interfaces). The Catalog Navigator tool is the main delivery mechanism for recommendations. The Passport Office tool can simplify the user account and profile management by automatically passing account information between sites, so users can log on to other sites that use Firefly's collaborative filtering without creating new accounts [Dragan, 1997]. FIREFLY PASSPORT OFFICE M s s r a m S E f K E f l SE SSIU N (Adapted from rwww.fireflv.coml) Figure 4. Firefly system architecture 3.2.2. Net Perceptions (GroupLens) GroupLens is a CF tool marketed by Net Perceptions and is being used by Amazon.com Inc. (www.amazon.com), Moviefmder (www.moviefinder.com) and the CDnow (www.cdnow.com). GroupLens’ filtering is based on the users’ evaluation of 26 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. pieces of information - usegroup messages, WebPages and documents - or products - books, movies, and CDs. Once a new user evaluates several samples of information, the system calculates correlations between his/her evaluations and those of other users to identify people with similar preferences. The system provides recommendations to the user based on the evaluations of the people with preferences similar to the user. Web Registration correlation program I ^ prediction daemon j / _ pool data management subsystem GroupLens Ratings Bureau daemon NNTP Server ] i ratings corre Collaborative DB lations DB Filtering Engine ^ ______ ^ (Adapted from [Miller et al., 1997]) Figure 5. GroupLens system architecture At CDnow Inc.'s Internet music store, for example, shoppers looking for new music that matches their tastes can type the names of three favorite bands into CDnow's Album Advisor, based on Net Perceptions' software. It then scours its records for the purchasing history of customers who liked the same three bands. Based on this data, Album Advisor predicts and lists the CDs the new shoppers are likely to 27 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. enjoy [Wingfield, 1998], GroupLens also lets the systems ascertain users’ preferences implicitly based on actions the users take on the system - the types of information sources users retrieve or the types of product they put into their shopping cart. As shown in Figure 5, the GroupLens’ filtering engine contains four primary modules - a prediction module, a ratings module, a correlation module, and a data management module [Miller et al., 1997], The ratings module collects evaluation information, which is similar to the data entry module in Figure 5. The prediction module calculates predicted evaluations of a user based on the user’s profile, which is the recommendation module in Figure 5. Pair-wise similarity between users is determined in the correlation module. The data management subsystem is responsible for maintaining both the ratings and the correlation databases. 3.2.3. Agentware Agentware is a software product provided by Autonomy. Using pattern-matching technology, Agentware analyzes the contents of documents from various sources such as websites, newsfeeds or email archives. Agentware identifies and encodes the unique “signature” of the key concepts within text documents. This signature of the concept then seeks out and uncovers the presence of similar concepts in volumes of content. The key component of the Agentware system is the Dynamic Reasoning Engine (DRE). The DRE applies pattern-matching technology to perform four main functions: 28 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. • Concept matching: The DRE accepts a source of text as input and returns references to documents in another source of text with the highest degree of relevance. • Agent creating: The DRE accepts text and returns an encoded representation of the most important ideas in the source (called concept agent), which includes each concept’s specific underlying patterns of terms. ’ • Agent retraining: The DRE accepts a Concept Agent and a text set and adapts the agent using the text. • Standard text search: The DRE accepts a Boolean term or natural language query and returns a list of documents containing the terms ordered by relevance to the query. Agentware automatically creates user profiles by observing users’ actions in the system. To determine an individual’s topical interests or expertise, Agentware applies the pattern-matching technology to extract key ideas from the articles a user reads online. As a user is served documents, Agentware automatically creates a set of Concept Agents. By weighting the frequency with which certain topics occur, the server can then encode a set of interests into Concept Agents. In addition to the above features, Agentware has more functionality for knowledge management. It can monitor specified Internet and Intranet sites, and seek out news and information that match the interest-profiles of each individual user. It, then, organizes the information and informs users about its availability. Agentware also automatically puts users with similar interests in touch with each other by matching 29 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. their individual-interest profiles, so that they have the opportunity to create communities of users involved in the same topic area. For example, in XOOM.com, an e-commerce-based web community, Agentware helps members identify and contact other members with similar interests. 3.2.4. WiseWire WiseWire is a CF tool used by LEXIS-NEXIS, an information vendor, and by Lycos, a major Internet search engine site. WiseWire has common CF features - discovering clusters of users based on their interests. In addition, WiseWire has an active information collection feature. It has a robot that automatically collects information from various sources such as newspaper websites. It categorizes the collected information according to specific groups of words, authors, and sources. The main components of WiseWire are its Document Subsystem, Filter Server Network, User Interface, and Learning Agents. The Document Subsystem collects documents, and the Filter Server Network is a set of systems that processes and classifies documents in real-time. The User Interface module handles all user interactions and the Learning Agents learn from feedback loops and adjust various parameters used for filtering in the Filter Server Network. The above diagram illustrates the architecture of the WiseWire system. A document initially enters WiseWire through the Documents subsystem. This subsystem gathers documents from premium news sources, news-groups, mailing lists, and the web. The Filter Server Network is a cluster of machines arranged into six 30 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. functional layers. The computers in each layer communicate through dedicated 100 megabit-per-second Ethernet backbones. Documents are actually moving, through the filter stages, in continuous streams, in real time rather than being physically clustered together in batches at any point. The filter extracts conceptual elements of each document and screens out any documents that do not fall into any of the broad interest areas. The documents are inspected in depth and rated based on how closely they fit into specific interest streams, or “Wires”. If a document makes it through these filtering stages, it has found its interested readers. The final leg of the filtering stage is collaborative filtering. The system compiles the document ratings of all users and The WiseWire System Documents L e a rn e d \ Knowledge Filter Server Network;"? . ^Ratings Users (Adapted from [www.wisewire-corp.coml) Figure 6. WiseWire system architecture 31 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. analyzes this data to recognize patterns among interests and people. Based on this pattern recognition, WiseWire is able to recommend information rated highly by one user to other users with similar interests. Collaborative filtering also enables WiseWire to predict the usefulness of completely new information, so users receive relevant documents in real time. The User Interface (UI) module handles all system interactions with human users. The UI assigns each document to users’ “in-boxes” or Wires, and ranks it among other documents depending on the aggregate score given to it by the filter network. WiseWire uses empirical models to infer required adjustments to the information retrieval, filter, and delivery processes based on multiple feedback loops among the various system components (Learning Agent). These loops adjust the profile concepts, model weights, and other learned parameters for the various levels of the Filter Server Network. The Advertising System, which accepts advertising input and creates targeted advertising programs, helps deliver personally targeted ads to each user of the WiseWire service. (http://www.wisewire-corp.com/Technology.html) 3.2.5. LikeMinds Another CF product is Like Minds from Macromedia (http://www.macromedia.com, formerly www.andromedia.com). LikeMinds is used in several websites such as Movie Critic (www.moviecritic.com) and Movie Matchmaker (www.cinemax.com). LikeMinds accepts users’ ratings on several initial items and then provides recommendations based on the ratings of others with similar 32 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. preferences. One unique feature of LikeMinds is that it provides more customized recommendations. It also provides confidence-level ratings for each user. Figure 7 illustrates the architecture of LikeMinds. LikeMinds is a software package that can be plugged into an existing Web server or database. LikeMinds Personalization Server (LPS) consists of several components. The Rules Engine makes product correlations based on similar attributes and enables LikeMinds to immediately recommend new products to customers. The Collaborative Filtering Engine is the core engine. Based on its algorithms, it uses individual ratings and preference information to generate recommendations. The Sifter continuously processes new user, product, and preference information to leverage the most current WEB SERVER PtfcSC.'UL.:,- t e n (Adapted from P www.macromedia.coml) Figure 7. LikeMinds system architecture 33 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. information possible. The Database Interface configures LikeMinds to work with any database format, as long as the database supports Open Database Connectivity (ODBC) standard. LikeMind’s API (Application Program Interface) supports JavaScript and VBScript for ASP-based applications or C++ and Java for CGI-based applications. 3.2.6. Alexa Alexa (http://www.alexa.com) is another variation of collaborative filtering. Alexa draws on the experience of all its users to recommend sites that Internet users might find interesting. To use Alexa, people visit its Web site and download client software. Once they have installed the program, a small toolbar appears below the window of the user’s Web browser. Each time a user calls up a Web page, the Alexa window suggests other sites he/she might want to visit [Weber, 1998], Alexa recommends web sites based on trails of the users’ past visits. If many users visit website A and website B together, then websites A and B are considered highly correlated. The next time a user visits website A, website B will be recommended to her/him. Alexa also provides additional information such as website traffic and freshness. The CF products above have different purposes and features that are summarized in Table 2. In summary, CF is expected to improve the search and reuse of knowledge- related items. CF systems store evaluation information and calculate similarities between users and estimated evaluations of items by each user. CF systems will help 34 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. users search across categories and keywords, identify people of similar interests, and browse an area with a guide from others. Table 2. Summary of CF tools in market Products Key Filtering Method Unique features Appropriat e target Clients Firefly • Statistical proximity of users • Personal profile available across multiple web sites • Uses statistical probability • Diverse Physical products • Messages www.vahoo.com www.filmfinder.com www.bamesandnoble.com www.virtualemDorium.com www.launch.com www.bianote.com GroupLens • Correlations between users’ evaluation on information items • Many additional features - Email, storing evaluations • Push feature • Physical products • Messages www.amazon.com www.moviefinder.com www.cdnow.com www.sierra.com www.eonline.com Agentware • Key concepts of documents • Similarities in document retrieval • Analysis of documents using pattern-matching technology • Alerting when new information is available or regular briefing • News • Documents • E-mails Reuters LA Times Reed Elsevier AP Wire Novell Xoom Shell Nomura Ericsson WiseWire • Groups of words Author, sources • User interests • Active collection - collects and analyzes documents • Push system - sends information of new documents to users with matching interests • Documents • News • Papers mv.lvcos.com LEXIS-NEXIS UMI SURETRADE, Inc. LikeMinds • Distance in evaluations • More customized filtering • Physical products www.moviecritic.com www.cinemax.com Alexa • Similarities of retrieving information (visiting trails) • Website information • Websites CF makes it possible to have a division of labor between human and machine. It combines the specialty of humans - judgment - and the strength of machines - precise and fast storage and calculation. 35 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER IV. RESEARCH ISSUES AND HYPOTHESES DEVELOPMENT In this chapter, the research issues derived from the previous chapters are discussed. Hypotheses are developed from these research issues and discussed in this chapter. When CF systems are applied to knowledge-intensive domains, new issues emerge in addition to the issues surrounding CF when they are applied to consumer products. 4.1. Impact of CF on information search and knowledge reuse Most of the current CF tools are used for consumer products such as CDs, movies, and books. Some CF tools, for example Agentware [http://www.autonomy.com] and WiseWire [http://www.wisewire-corp.com], are appropriate for knowledge-related items such as documents. Here, ‘knowledge-related items’ are defined as ‘codified knowledge objects that are results of or inputs to intellectual activities that yield economic or technological value for organizations. Documents, engineering drawings, and news are examples of knowledge-related items. Once knowledge-related items are codified into a database, the technical process of CF for them does not significantly differ from that of consumer products. However, the impact of CF in a knowledge intensive domain would be bigger than in consumer products. One of the most obvious impacts would take place in information search. 36 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Comparing search behavior with and without CF will highlight how CF facilitates knowledge reuse. A typical process of knowledge reuse - more precisely, codified knowledge reuse - without the support of CF can be described in the following way. A user finds reusable knowledge items by keyword search or by navigating different categories in the knowledge repository. The user scans the titles of the items and, one by one, opens items that are thought to be relevant to his/her task. If the system does not find the needed items or list too many items, the user sometimes alternates or narrows keywords. Users usually experience confusion, uncertainty and doubt during the search process; an inability to express precisely what information is needed makes communication, between the user and the system, awkward [Kuhlthau, 1999]. The problem of keyword search or category navigation is that it often takes too much time to reach the items the user needs. The user has to spend time opening and reading items to evaluate whether they are relevant to his/her work. Moreover, the user’s evaluations are sometimes inaccurate, and this inaccuracy of evaluation becomes more evident under time pressure [Ahituv et al., 1998]. Another problem is that the user’s search may miss some important items if the items use different keywords or categories. The operations of keywords using Boolean logic (OR, AND) are unforgiving and inflexible, and the retrieved results are often inadequate [Salton et al., 1994], In some cases, the information searcher asks others who have had similar problems to point out appropriate items to look at. The problem in asking others is that even if the person being asked has seen potentially useful information, human memory is not 37 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. perfect and often he/she is unable to precisely recall where he/she saw the information. In most cases, therefore, the user has to spend additional time searching for the information. There is also the problem of imprecise human judgment. This imprecision has two aspects relevant to the process of ‘finding by asking others’. First, people’s judgments on the usefulness of an item vary from person to person and from task to task. The information searcher needs to find a person whose information needs are identical to his/hers. People, however, cannot precisely judge others’ information needs. Identifying an appropriate person is also constrained by the range of the information searcher’s acquaintances. Second, even if the information searcher finds a person who has had similar information needs, the person’s judgments may be erroneous. Consequently, the information searcher needs to get evaluations from more people and aggregate these evaluations for a better judgment. This process of aggregation can be supported by information systems because the main strength of an information system is its fast and accurate storage and calculation. CF helps people reuse knowledge in several ways. First, CF keeps precise records on who retrieved which item and what evaluation scores they gave to each item. This alone can be a great help for search because a user, seeing what others think about each item, can save time and use it for browsing and evaluating the items. Second, CF identifies a set of people (reference group) who have interests similar to the user’s, and provides recommendations to the user based on those people’s evaluations. The people in a reference group have similar interests or preferences, so 38 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. their evaluations will match the interests of the user better than will average evaluations. Third, CF broadens the range of search by retrieving items across keywords and categories. Although keyword search is a dominant search method, it fails more often than not, because users do not understand how the system processes their keyword searches [Hildreth, 1997]. For ‘search by keyword’ to be effective, the knowledge items need to be neatly classified according to a few keywords. If the searcher does not have substantial knowledge of the keyword structure, the keyword search will miss important items that use different keywords. Because CF recommends items regardless of the category and keywords, users of CF will have more chances to look at items across categories and keywords. Finally, CF helps identify people with similar interests so that they can communicate with each other. The system can inform a user about the members of the reference group so that he/she can contact and communicate with them on- and off line for further help. This communication will help users form a community. Armstrong and Hagel [1996] proposed four types of community - transaction, interest, fantasy, and relationship. The community to be formed with the help of CF will be a ‘community of interest’. 39 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4.2. Research issues in CF One of the most important issues in CF research would be ‘How to improve the effectiveness (or performance) of the system?’ In case of CF, the effectiveness of the system is determined by the accuracy of the recommendation because what users expect from the system is accurate recommendations. The accuracy of recommendations is thus an important effectiveness factor for the system. The accuracy of CF systems is affected by many factors. As mentioned in Chapter 2, system characteristics - type of similarity index, reference group selection methods, and other methods for generating recommendations - is an obvious factor affecting accuracy of recommendations. Recommendation by CF is based on other people’s evaluations. Thus the evaluations - the amount and qualitative characteristics of evaluation - would affect the accuracy of the system. Information quality has been an important factor affecting system quality in other IS studies [Delone & McLean, 1992]. As the number of users increases, the information quality - accuracy in CF systems - increases. A study by Shardanand and Maes [1995] analyzed qualitative data - comments from users - and confirmed that accuracy increases as the number of users increases. Therefore, the amount of information (or number of users) in the system is a factor in CF systems performance. This factor (number of users) is a performance factor unique to CF systems and inherent in the nature of CF systems (recommendation generation). 40 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The type of product is one of the important qualitative dimensions of evaluation in CF systems. As Miller et al. [1997] showed, the characteristics of evaluations (e.g. correlations of across users) are dependent on the type of products and those characteristics of evaluation would affect the accuracy of CF systems. User Characteristics Product Characteristics Performance (Accuracy) of the System System Characteristics Number of Users Evaluation Density Figure 8. Factors affecting the accuracy of CF systems The discussions in the previous sections lead to a research framework shown in Figure 8. The four important factors that affect the accuracy of CF systems are user characteristics, product characteristics, system characteristics, and number of users. These factors are not the only determinants of CF performance, but they are the most significant ones. These factors are discussed further in the following sections. 41 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4.2.1. Critical mass of users and evaluations Another important issue is how the performance of CF systems evolves as the number of users increases. In order for a CF system to be effective - to have a certain level of recommendation accuracy - it should have a certain minimum number of users and a minimum number of evaluations per user. If a CF system does not reach this point quickly enough, it will fall into a vicious cycle - users will lack motivation to use the system due to the poor recommendations and, in turn, the system will not get enough inputs to produce accurate recommendations. To avoid this cycle it is vital to secure a certain minimum number of users. Increasing the number of users improves performance by improving the possibility of finding similar people. Research on human cognitive processes, as discussed in Section 4.2.4, implies that people have different mental models or cognitive maps [Montazemi and Conrath, 1986; Santhanam and Sein, 1994]; so, an increase in the number of users increases the probability of finding people with similar cognitive maps. For example, suppose a CF system selects 20 similar people as a reference group and consider two cases - one when the total number of users is 100 and the other when the total number of users is 1000. In the first case, the people in a reference group are 20 out of 100 while in the second case the people are 20 out of 1000. People in the latter case will show higher similarity than the former case, and thus, the results will be more accurate recommendations. Shardanand and Maes [1995] applied CF technology to music (songs) and analyzed the comments from users. They found that 42 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. users commented more and more positively as the number of users increased, which implies the accuracy of CF systems increases as the number of users increases. The number of users and number of evaluations per user that are required for the needed recommendation accuracy is termed 'critical mass of users and evaluations’. Although the required accuracy can be set at many different levels, the most meaningful required accuracy level is the point where the accuracy of CF exceeds the accuracy of ‘average of everybody.’ With a small number of users, the accuracy of CF will be similar to or lower than the accuracy of ‘average of everybody.’ However, As the number of users and evaluations increase, the accuracy of CF will exceed the accuracy of ‘average of everybody’. The number of users and evaluations at this crossover point is the critical mass of users and evaluations. Two important issues are involved in critical mass. It will be interesting to investigate whether any patterns emerge in the increase of performance as the number of users increases. Also, the impact of other factors, such as user characteristics and distribution of evaluations, on the critical mass need to be examined. 4.2.2. System characteristics Technical issues such as configuration of CF systems are also critical for more accurate recommendations. As discussed in Section 2.1 and 2.2, a CF system may use different recommendation algorithms. Two main methods to set reference group size are ‘correlation-thresholding’ and ‘best-n-neighbors’ [Herlocker et al., 1997]. They 43 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. found that ‘correlation-thresholding’ resulted in higher performance while sacrificing coverage. The extreme case of personalized recommendation is generating recommendations based on the ratings of the person who has the greatest commonality with the user. In this case, the reference group size is 1. As people who have the next greatest commonality with the user are added to the reference group, the recommendations are based on more people’s ratings. When the last person is added to the reference group, the reference group size becomes equal to the whole user group. In this research, the reference group with fewer people is termed the “narrow” reference group, and the reference group with more people is termed the “broad” reference group. As the reference group becomes narrower, the “level of personalization” increases. There are some technical variances in the size of reference groups for different CF tools. Instead of making a decision to ‘include in or exclude from’ a reference group, some CF applications generate personalized recommendations by summing up everybody’s ratings, but with different weights. LikeMinds, for example, generates estimated ratings by calculating the weighted average of ratings of all users [Greening, 1999]. For these applications changing the weights applied to individual users can alter the level of personalization. Assigning weights to users is a more general method than selecting a certain number of users because the latter is a special case of the former - assigning 0 weight to an unselected person. Therefore, the term ‘ reference group size’ is a conceptual representation of ‘how much weight is assigned to each user’. 44 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. When there are relatively few evaluations, a CF system will not do well because it uses only evaluations found at the intersection of the items both individuals have evaluated [Konstan et al., 1997; Breese et al., 1998; Sarwar et al., 1998]. Default evaluation is a value that is assigned to un-rated items. If default evaluation is assigned to the un-rated items, it will help avoid the ‘no common items’ problem. The ‘default evaluation’ method was used in previous studies [Breese et al., 1998; Sarwar et al., 1998]. The advantage of this method is that it increases the ‘coverage’ - percent of items/users for which estimated evaluations can be calculated. The disadvantage of the method is that it loses accuracy [Breese et al., 1998]. 4.2.3. Mode of search Even for the same CF system, performance will vary according to usage behavior and the current tasks of users. When users search for information, they do so in several different ways. Sometimes users search for information because they have an urgent problem that needs to be solved. In other cases, especially when they use a large networked system such as the Internet they just browse, looking for any information that may be directly or indirectly useful to them. Studies have categorized search behavior in various ways. Arguilar [1967] categorized search behavior into four modes. Undirected viewing, defined as general exposure occurring when the viewer has no specific purpose in mind; conditioned viewing, or directed exposure not involving active search; informal search, which is an unstructured effort to obtain specific information; md formal search, which refers to a 45 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. deliberate effort to find a piece of information [Aguilar, 1967]. Vandenbosch and Huff [1997] used two modes of search - scanning and focused search. If the underlying need driving the information search is, “what is the answer to this question,?” the person is performing a focused search. Scanning is the behavior of people when they are browsing through data in order to understand trends or sharpen their general understanding of the business [Vandenbosch and Huff, 1997]. El Sawy [1985] also categorized manager’s information retrieval behaviors into ‘scanning’ and ‘problemistic search.’ A problemistic search is a search stimulated by a problem and directed towards finding a solution to the problem while scanning is not directed towards any particular problem [El Sawy, 1985]. Although the categorization and labeling may differ depending on the study, a common criterion of categorization in information search and retrieval is whether or not users have specific problems when they search for information. The existence of a problem is an important criterion for characterizing user’s information search and retrieval. This is because the user’s behavior in the case of searches to solve a specific problem is substantially different from when the user searches without a specific problem in mind [El Sawy, 1985; Vandenbosch and Huff, 1997]. People sometimes go back and forth between these two modes. Most studies label the search without specific questions, ‘scanning.’ The search that tries to answer a specific question was labeled ‘focused search’ [Vandenbosch and Huff, 1997] or ‘problemistic search’ [El Sawy, 1985]. In this study, the labels ‘scanning’ and ‘problemistic search’ are used to highlight the difference between the 46 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. two searches - with and without specific problems. The characteristics of these two modes are illustrated in Table 3. In the problemistic search mode, the relevance or usefulness of an item of information is determined on the basis of a very narrow criterion - whether or not it is the answer the user is looking for. On the other hand, the relevance/usefulness of an item in scanning mode is determined by a loose criterion - whether it is potentially useful for the user’s task. Thus, a problemistic search has a higher context-dependency than scanning. With the ‘mental model’ framework [Daniels, 1995], the mental model of a user in scanning mode is broader and more stable while the mental model of the user in a problemistic search is more specific and changes from problem to problem. Table 3. Characteristics of scanning and problemistic search Scanning Problemistic Search Purpose of the search Discovering undeterm ined useful information Discovering specific information Frequency R egular basis W hen a problem occurs Search range Broad and shallow Specific an d d eep Value of information G eneral usefulness R elev an ce to current problem (task) End of search C ertain a re a is covered T he an sw er is found or declared im possible to find with given constraints The different search modes have different consequences. Vandenbosch and Huff [1997] found that the use of executive information systems (EIS) for focused search would improve the efficiency of the task, but the use of EIS for scanning was more associated with effectiveness of the task. The studies indicate that users have different mindsets when they are in different search modes [El Sawy, 1985; Vandenbosch and 47 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Huff, 1997], which implies that users may have different criteria for information evaluation. It is expected that the search mode, which a user adopts when he/she uses CF systems, will affect the evaluations of the user. This will, in turn, affect the accuracy of the CF system. 4.2.4. User characteristics The performance of CF systems would be affected by the characteristics of users. If CF is applied to a knowledge intensive domain, the knowledge of users will be a critical factor that affects the accuracy of the system. Theoretical explanations of why knowledge of users may affect the accuracy of CF systems can be found in the ‘human cognitive process’ and ‘experts and novices’ literature. 4.2.4.I. Human cognitive process view A number of studies have investigated the human mental structure and applied it to information system design. A cognitive map is a representation of the relationships that are perceived to exist among the elements of a given environment [Montazemi and Conrath, 1986]. Cognitive mapping techniques refer to methods used to elicit the structure and content of people’s mental models [Daniels et al., 1995]. A study that applied cognitive mapping techniques to information requirement analysis [Montazemi and Conrath, 1986] shows that the technique is useful for identifying irrelevant data, evaluating the factors that affect a given class of decisions, and 48 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. enhancing the overall understanding of a decision maker’s environment. Another study [Santhanam and Sein, 1994] investigated the relationship between people’s mental models and their learning in computer system education: It found that the mental models of the system formed by the users affected the success of learning. Another important theory on the human cognitive process is categorization theory [Dutton and Jackson, 1987]. A critical assertion of this theory is that people form cognitive categories based on their observations of the features or attributes of objects. Cognitive categories comprise objects with similar perceived attributes, and they reflect the structure of the objects in the environment. Cognitive categories influence how the user remembers the information he/she has encountered such that, over time, information congruent with established perceptions is more likely to be remembered. Another study showed that different categorizations (labeling) for a similar event result in different information processing [Sullivan and Nonaka, 1988]. “Schemas” is another concept related to categorization. Schemas are cognitive models that allow people to categorize events, assess consequences, and consider appropriate actions [Ireland et al., 1985]. Cognitive mapping and categorization theories imply that each person forms his/her own mental model of artifacts, and that this mental model affects his/her categorization and evaluation of new objects. Two complementary roles of CF in knowledge reuse are inferred from these theories. First, using people’s interests/preferences, CF identifies people who have similar mental models. People’s evaluations of information items partly reflect their 49 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. evaluations of features and attributes of the objects. Users’ evaluation of the ‘relevance’ of an item to a task depends on the perceived distance between the item and the task. Relevance, therefore, depends on how one categorizes or locates the item and task on his/her mental map. For example, suppose there are two users, A and B, working on two individual, but very similar projects. On a CF system, the evaluation by and perceived relevance to users A and B depend on where the item is categorized in each of their mental models. If they have similar mental structures, their evaluations will also be similar. Thus, people with similar evaluation structures are thought to have similar mental structures. Second, if the recommendation from the system is not the same as what the user thinks, CF may promote change in a user’s mental model. For example, suppose user A is searching for information items for her task. When she opens and reviews one of the recommended items, her own evaluations may differ from those suggested by the people in the reference group. Since she knows that the recommendations are based on the evaluations of people who share common interests with her, she will be more likely to try to discover the causes of the incongruence. Ausubel [1985] argues that when an individual cannot find a basis for reconciling apparently or genuinely contradictory ideas, he or she will (sometimes) attempt to resolve these differences so as to attempt synthesis and reorganization of his or her existing knowledge. 50 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4.2.4.2. Expert vs. novice Studies show that knowledgeable people are more selective in the information they acquire and more flexible in the manner in which they search for information [Johnson, 1988]. Spence and Brucks [1997] find that experts select fewer, but more diagnostic, information inputs and are more consistent when evaluating non-quantified inputs. As a result, they make more accurate and tightly clustered judgments than do novices. They are also are more confident in their decisions. Another study shows that more knowledgeable people agree more than novices on what information is important [Shanteau, 1988]. Experts appear to possess comparatively richer schemata for ascribing meaning to given information [Carter et al., 1988]. In systems analysis, the biggest difference between experts and novices was found in the knowledge structure - novice system analysts’ knowledge base is characterized best by its sparseness, lack of breadth, and generality [Schenk et al., 1998]. In summary, previous studies show that experts (knowledgeable people) are better at judging the value of information and have more consistent evaluation across items and higher cohesiveness across people. These characteristics of knowledgeable people will lead to a better performance of CF systems for them because of their precise and consistent evaluations. Motivations of users will also affect the performance of the system. Motivated users will evaluate the items more carefully, and this will lead to more accurate evaluations. More accurate evaluations will result in more precise identification of clusters of similar users, thereby improving performance. 51 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4.2.4 3 . Sense-making view Information science studies have attempted to bridge human cognition processes and information search. Two “spaces” in the information search have been proposed: document space and concept space [Baldonado, 1997]. Document space is the realm of possible information artifacts in the world, ranging from books to movies to bibliographic citations to user-created artifacts. Concept space is the cognitive realm of the context and the knowledge required to interpret documents. Humans are good at bundling documents - identifying similar documents - and detecting duplication - detecting equivalent documents. When a human searches for information from a repository such as a library information system, he/she repeats bundling and duplicate detection, gradually forming or modifying his/her concept space from the document space. If an information searcher is a novice in an area, he/she needs to spend more time for this ‘sense-making’ process when he/she searches for information in the area. The evaluation structure of users in a CF system may be a result of the users’ sense-making processes. Users knowledgeable in an area will evaluate those items that they think are effective in forming concept space in the area. If a person uses a CF system for information search, it will save his/her time for the ‘sense-making’ process because he/she can easily form concept space by browsing recommended items. The recommended items are the items that were positively evaluated by the people who have interests/tasks similar to the user. Thus, a user gives some items (initial 52 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. evaluations) that he/she thinks effective for forming a concept space in an area and receives recommendations from the people who already have formed concept space in the same area. One who browses these items with the help of CF will, with less effort, grasp ideas about the area. 4.2.5. Preference cluster patterns In e-commerce sites that leverage their customers’ preferences, the pattern of preference clusters - how many clusters exist and how tight the clusters are - is very important for the success of their system development. In terms of business strategy, interface design and functionality of e-commerce sites for customers with relatively homogeneous preferences or interests (e.g. Newsgroups or Specialty Stores) will be different from sites for a broader spectrum of customers (e.g. general retailing stores or portal sites). The pattern of customers’ preference clusters would affect the accuracy of CF recommendations. Since the recommendations by CF are based on evaluations by other people with similar preferences, differences in the pattern of preference clusters would affect the accuracy and consequently the effectiveness of CF systems. 4.2.5.I. Pattern of preference clusters One of the most important characteristics of a preference pattern would be how the customers are clustered. Two extreme cases in the pattern of customer preference clusters are: 53 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. • Each customer has his/her own taste. There is no cluster in the customers’ preference space. • Customers can be classified into one or more distinctive clusters. Customers in a particular cluster have very similar preferences, and these are clearly different from those of customers in other clusters In terms of both the number of clusters and the tightness (densities) of the clusters there are countless possible cluster patterns between these two extremes. Let us take an example domain, which consists of only two items. The domain can be a knowledge-intensive domain such as a document repository or a consumer preference domain such as movies or music. If, for example, movies is the domain, for example the two items will be two movies. Figure 9 shows four different cluster patterns of customer preferences. Each diagram in Figure 9 shows preferences of 200 fictitious people on a 0 (dislike) to 1 (like) scale. For example, in the Diagram (a), customer A likes both items and customer B dislikes both items. Customers D and C like one item and dislike the other, and customer E moderately likes both items. All data for Figure 9 were generated using random numbers. The cluster patterns were obtained by limiting the range of random numbers. The Diagram (a) uses random numbers ranging 0 to 1. The seven clusters in (b) and (c) were generated by restricting the range of random numbers. Random numbers for Diagram (b) were generated with a range of ±0.0625 from the cluster centers, and for the Diagram (c), with a range of 54 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. ±0.125 from the cluster centers. The Diagram (d) has 15 clusters with same range of random numbers as Diagram (b). 3 ro ♦♦ Dislike Like Item 1 (a) Case 1: No Clusters w r~ 7 T f ♦ ♦ ♦ * * * ♦ Like Dislike Item 1 (c) Case 3: Loose Clusters C D 3 ro Like Dislike Item 1 (b) Case 2: Tight Clusters ro 3 N ) C . C D t g c o ' ix f ♦% _ 4 --- 7 . ♦ 4^4 I Dislike Like Item 1 (d) Case 4: More Clusters Figure 9. Different Patterns of Customer Preference Clusters with Two Items Diagram (a) represents an example of the first extreme case mentioned above where the customers scatter almost equally over the preference space. In this case no clusters emerge in the preference space. Diagram (b) is the second extreme case where 55 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. customers can be grouped into seven clear clusters. Diagram (c) has the same number of clusters in the same locations as Diagram (b), but the clusters are looser than they are in the Diagram (b). The clusters in Diagram (d) have tightness similar to the clusters in Diagram (b), but there are more (15) clusters in this case. Similar cluster patterns were generated in a domain with five items using a method similar to the one used in Figure 9. In fact, Figure 9 was drawn using two items taken out of the five items in this domain. Although it is not possible to graphically show the preference patterns because 5 dimensions are involved, the four different cases similar to Figure 9 were generated using exactly the method used for Figure 9. Two-person proximity measures (proximity metrics between all possible combinations of two people) were calculated for the four cases. Correlation was used as the proximity measure in this analysis because, as mentioned in Section 2.2, Bayesian probability and correlation are the two best proximity measures and of the two, correlation is the simpler to calculate. Each customer has 199 (excluding self) correlations with other customers. Therefore, there are 39800 (= 199 x 200 people) two-person correlations in all, or 19900 (= 39800 / 2) unique correlations. Figure 10 shows the distributions of the 19900 unique correlations in the four cases. From the diagrams, several rules about the relationship between the pattern of preference clusters and distribution of the correlations are discovered. First, when there is no cluster in the preference space, the distribution of correlations has a symmetric hill shape. Second, if there are some clusters of preferences, the distribution 56 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. differs from the ‘no clusters’ case. In this example, it seems that the distribution becomes bi-modal when there are some clusters in the preference space. Finally, as a cluster pattern becomes closer to the ‘no clusters’ pattern - the number of clusters increases or the clusters become looser - the distribution of correlations becomes similar to the distribution of correlations of the ‘no clusters’. .15 ,26 .67 .05 .46 -.36 (a) No Clusters 3200 2800 2400 2000 1600 1200 800 400 0 k -.57 -.15 .26 .67 -.77 -.36 .05 .46 (b) Tight Clusters 3200 2800 2400 2000 1600 1200 800 400 0 -.57 -.15 .26 -.77 -.36 .05 .46 (c) Loose Clusters 3200 2800 2400 2000 1600 1200 800 400 L i . f f ■ i 1 1 — ■pspi 1 1 98 -.57 -.15 ,26 .67 .77 -.36 .05 .46 y (d) More Clusters Figure 10. Distribution of Two-person Proximity Measures 57 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4.2.5.2. Measures of the pattern of preference clusters The findings from Figure 10 show that the preference cluster pattern in a domain can be measured by capturing how much the distribution of proximity measures differs from the distribution in the ‘no clusters’ case. A statistical method to test differences of distributions - goodness of fit test - can be used to quantify the extent of the difference. The Kolmogorov-Smirnov test (hereafter referred to as K-S test) is the most widely used goodness of fit test for continuous data [Khamis, 2000], The K-S test can be used to test all types of differences in any two continuous distributions [McClave and Benson, 1988]. The K-S test statistic D is denoted as, D = Max\Fj where Ft = the value of theoretical cumulative distribution function (c.d.f.) at / ' ordered (ascending) observation Sj = the value of empirical c.d.f. at f h ordered (ascending) observation Larger D statistics in K-S tests imply more differences in the distribution. The critical value of the K-S test depends on the sample size [McClave and Benson, 1988]. Based on the rules developed in the previous section, several predictions about the K-S statistic D can be made. When the K-S tests are conducted for the ‘no clusters’ domain and for a ‘clustered’ domain, the D statistic will be large if the clusters are clear. Similarly, if a domain has a smaller number of clusters, the K-S D statistic that compares this domain with the ‘no clusters’ domain will be larger. Therefore, the following propositions are developed. 58 R eproduced with perm ission o f the copyright owner. Further reproduction prohibited without perm ission. Proposition 1: The K-S D statistic, that compares the distributions of proximity measures of a ‘clustered’ domain with the ‘no clusters’ domain, becomes larger when the clusters are tighter. Proposition 2: The K-S D statistic, that compares the distributions of proximity measures of a ‘clustered’ domain and the ‘no clusters’ domain, becomes larger when the number of clusters is smaller. Several simulations were carried out to examine the effects of ‘tightness’ and the ‘number’ of clusters. Table 4 shows the K-S test statistics for five different levels of tightness and four different numbers of clusters. Other conditions, number of items (=5) and number of people (=200), are the same as applied to Figure 10. In order to eliminate random number biases, 10 simulations were carried out for each condition and the results were averaged. For example, the number, 0.178, in the first row is the average of 10 K-S statistics of 10 randomly generated data sets, which have values within a ±0.025 range from the cluster centers. Table 4 shows that the K-S D statistic increases as the tightness of the clusters increases or the number of clusters decreases. All the average K-S D statistics in the table are significant at a=0.01 level1 (the critical value = 0.011). Propositions 1 and 2 are therefore supported. In Table 4, all the cases and the baseline of the K-S D statistics (‘no clusters’ domain) have 5 items. The distributions of the proximity measures (correlations), 1 The K-S D statistics in Table 4 are based on the populations - all the possible correlations were used. Strictly speaking, they are not the K-S tests because they are not testing samples. The D statistics in the table just show how different the distributions are from the ‘no clusters’ domain. 59 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. however, may change as the number of items changes. For example, the distribution of correlations of ‘no clusters’ with 5 items may differ from the distribution of correlations of ‘no clusters’ with 10 or 20 items. In real cases, customers evaluate only part of the items in a domain. For example, when customers evaluate movies, they evaluate only the movies that they have watched. In some cases, customers do not even evaluate all movies they have watched because they do not have enough time. In this case, the proximity measures are calculated using the limited number of items that a pair of customers commonly evaluated. Table 4. K-S Test S tatistics for Different Cluster Patterns Cluster Patterns K-S D Statistics Tightness of Clusters (N. of Clusters = 7) Range = ± 0.025* (Tightest) 0.178** Range = ± 0.05* 0.172** Range = ±0.075* 0.169** Range = ±0.10* 0.163** Range = ±0.15* (Loosest) 0.103** Number of Clusters (Range = ± 0.10*) N = 5 0.233** N= 10 0.113** N= 15 0.093** N = 20 0.076** * The range refers to the size of the clusters (e.g. ± 0.025 means the data were randomly generated within the distance of 0.025 from the cluster centers) ** Significant at a = 0.01 level Furthermore, in a real e-commerce environment, the proximity measures for all possible pairs of users are rarely available. Some users may have evaluated no items 60 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. and some pairs of users may have no common items. Therefore, in real data sets, it is usual that the available ‘two-person’ proximity measures are only part of all possible measures. However, if the available proximity measures are random, it is expected that the distribution of the sample proximity measures will be indifferent to the population distribution. For example, if there are 100 items in a domain and each pair of customers has, on average, 5 common items, the distribution of the proximity measures of this domain is likely to be similar to the distribution of proximity measures of a domain with 5 items, using all 5 items. If this is the case, the ‘average number of common items between people’ is a more important parameter for characterizing the distribution of proximity measures than the ‘total number of items in the domain. ’ If this holds in general, the baseline distribution in the example should be the distribution of the ‘no clusters’ domain with 5 items, not with 100 items. Suppose there is a domain with N items and, on an average, data on n out of the N items is available for calculating proximity measures for each pair of people. Furthermore, assume that proximity measures are not available for all possible pairs of people in the first domain. Also suppose there is another domain with n items and all n items are available. From the above it is expected that the distributions of the available proximity measures in the first domain may be similar to the distribution of the proximity measures in the second domain. This leads to two further propositions. 61 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Proposition 3: The distribution of proximity measures in the following two domains are not different (the K-S D statistic is insignificant) if the two domains have similar preference patterns. Domain A has N items and Domain B has n items (N > n). In Domain A, on an average, n items out of N items are available to calculate the proximity measures and also only part of the proximity measures are available. In Domain B, all items and proximity measures are available. Proposition 4: The pattern of preferences (tightness and number of clusters) of the Domain A in the Proposition 3 can be measured by comparing the distribution of the available proximity measures of Domain A against the distribution of proximity measures of ‘no clusters’ domain with n items, using all n items. Another round of simulations were carried out with different numbers of items in two cases - ‘no clusters’ and ‘7 clusters with a range of ± 0.05.’ Table 5 shows the K- S test results with different numbers of items in a domain. Each number in the table is an average of 10 simulation runs, which is same as in Table 4. Data sets for 200 fictitious people were generated with different numbers of items (10, 20, 50, and 100 items). In the first round of simulation, the correlations of each data set were calculated using randomly selected items. The number of selected items was also randomly selected ranging 3 to 7 so that the average becomes 5. In order to simulate the sparse data sets, 300 correlations were randomly selected out of 19,900 possible correlations in each case. The distribution of the correlations was compared with two cases. The first comparison was against the distribution of the correlations of a domain with 5 items, using all 5 items. The second was against the distribution of the correlations of a domain with the same number of items - i.e. 10, 20, 50, and 100 items, respectively. 62 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Table 5. K-S D Statistic for Different Numbers of Items Average Number of Items Used Per Pair of People Pattern of Clusters Total Number of Items in Domain Average K-S D Statistics Compared to a Domain with n (=5) Items Compared to a Domain with N (10, 20, 50, or 100) items 5 No Clusters 10 Items 0.064 0.132** 20 Items 0.057 0.116" 50 Items 0.054 0.175" 100 Items 0.052 0.165" 7 Clusters (Range = ±0.05*) 10 Items 0.074 0.205** 20 Items 0.076 0.284** 50 Items 0.077 0.485" 100 Items 0.088 0.623" Compared to a Domain with n (=10) Items Compared to a Domain with N (20, 50, or 100) items 10 No Clusters 20 Items 0.050 0.088 50 Items 0.052 0.093 100 Items 0.067 0.112" 7 Clusters (Range= +0.05*) 20 Items 0.086 0.239" 50 Items 0.069 0.509** 100 Items 0.090 0.657” * The data were randomly generated within the distance of ±0.05 from the cluster centers. * Significant at a = 0.01 level (Critical value = 0.094) For example, 0.052 in the fourth row is the K-S D statistic comparing 1) the ‘no clusters’ domain with 100 items that utilizes, on an average, 5 randomly selected items for each pair of people and 2) the ‘no clusters’ domain with 5 items that utilizes all the items. The second number in the same row, 0.165, represents the K-S D statistic comparing 1) the ‘no clusters’ domain with 100 items that utilizes, on average, 5 randomly selected items for each pair of people and 2) the ‘no clusters’ domain with 100 items that utilizes all the items. Another round of simulations was conducted for an average of 10 (range = 7 to 13) randomly selected items. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The average D statistics show that if only a portion of items (n out of T V ) are used to calculate the proximity measures, then the distribution of those measures is different from the distribution of proximity measures with all T V items. Rather, it is similar to the distribution of the proximity measures of a domain with n items where all n items are available. The results imply that the distribution of proximity measures is not different across the domains, even those with a different number of items, as long as the same number of items is used to calculate the proximity measures. Thus, when a distribution of proximity measures, with an average of n common items across people, needs to be compared against a ‘no clusters’ case, the reference ‘no clusters’ domain should be the one with n items. Therefore, propositions 3 and 4 are also supported. The difference between D statistics for T V items and n items seems to be more significant in the clustered domain than the ‘no clusters’ domain. The difference also seems to increase as the number of items increases. From results of the simulation above, a hypothetical relationship between the K-S D statistic and the accuracy of CF system is developed. CF systems will have higher accuracy in the domains with higher K-S D statistics. 4.2.6. Performance measures When the performance of CF systems are investigated, the most commonly used accuracy measures are mean absolute deviation (MAD) [Shardanand and Maes, 1995; Sarwar et al., 1998; Good et al., 1999; and Ansari et al., 2000], mean squared error 64 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. (MSE) [Miller et al., 1997], root mean squared error [Sarwar et al., 1998] and correlation between ratings and predictions [Hill et al., 1995]. MAD and root mean squared error have some limitations when used as a performance measures for CF. The limitations are because of ‘average adjustment’ and ‘scale conversion.’ If the average of actual evaluations of a user is 3 while the average of estimated evaluations is 2, then the estimation has an ‘average adjustment’ problem. If the estimated evaluations are 1,3, and 5 for the actual evaluations 2, 3, and 4, then the estimation has accurate average, but inaccurate ‘scale conversion’. If MAD or root of mean squared error is used as a measure in the above examples, the estimates are considered inaccurate even though they are in a correct order. From the user’s point of view, ‘correlation between actual evaluations and predicted evaluations’ also has a limitation as a measure. For example, suppose the actual evaluations are (4, 5, 6, and 7). If the estimated evaluations are (3, 4, 5, 6), the correlation is 1.0 and if estimated evaluations are (3, 4, 5, 10), the correlation is 0.91. Actual users would judge the accuracy of recommendation based on the order of recommended items rather than the estimated evaluation numbers. The two estimates are identical to users while the correlations are different. The use of a rank-based performance measure can eliminate the deviation owing to the inaccuracies in ‘average adjust’ and ‘scale conversion.’ The rank-based measure would be congruent with users’ perceived accuracy because it uses rank of the items, not estimated evaluations. Therefore, the rank-based measure is used as a primary 65 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. accuracy measure in this study. The mean squared error (MSE) is used as the secondary measure. 4.2.6.1.4 Average of everybody’ as the baseline performance of CF The ‘recommendations by average’ will be the baseline of performance comparison. If the system did not have the CF feature, the average ratings would be used to generate recommendations. In order for a CF system to work, the estimated evaluation by CF should be more accurate than by average ratings. Therefore, the ‘average of everybody’ becomes a sort of baseline when the accuracy of CF is measured. Performance (accuracy) = Estimation error by ‘average evaluations of everybody’ - Estimation error by CF Estimation error by CF and ‘average evaluations of everybody’ = n m ZZ|r K -)-r M » • = ! M _______________ nrn where r(ay) = rank of actual evaluation of user i on item j r(ejj) = rank of estimated evaluation (by CF or ‘average of everybody’) of user i on item j n = number of users in the holdout set m = number of items in the holdout set 66 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. In this study, two different measures - ‘rank-based’ measure and mean squared error (MSE) - are used to measure the performance of a CF system. These two measures are explained in detail in the following sections. 4.2.6.2. Rank-based performance measure The rank-based measure gauges accuracy based on the ranks of the items. The estimation errors are calculated using ranks of the items in the hold out sample. Table 6 shows an example of rank-based performance measure when the hold out sample size is 4. Table 6. Example of Rank-Based Performance Measure Items Actual Average CF Errors Evalu ation Rank3 Estim ation Rankb Estim ation Rankc Average (= la-bl) CF (= la-cl) 1 6 4 4.9 3 5.2 4 1 0 2 2 I 2.5 1 2.1 1 0 0 3 4 2 5.5 4 3.8 3 2 1 4 5 3 4.2 2 2.9 2 1 1 X 4 2 The final performance (accuracy) of CF in the example is the difference between error sum of ‘average of everybody’ ( = 4 ) and error sum of CF ( = 2 ). Thus, in the example, the final performance is 2 ( = 4 - 2 ). The bigger this number is, the better the CF system performance. In the simulation, the performance scores of multiple runs are averaged and used as the final performance measure. For example, if the final measure is 0.5, it means that CF generates recommendations that outperform the recommendations of ‘average of everybody’ by 0.5 in the order (rank) of the items. 67 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4.2.6.3. Perceived Accuracy An indirect measure of performance is the users’ perceived accuracy [DeLone and McLead, 1993]. The perceived accuracy of users can be an indirect measure for the accuracy of CF systems. In previous studies [Shardanand and Maes, 1995] comments from users were analyzed, but users’ perceived accuracy has not been used as a measure. 4.3. Hypotheses development When a CF system is applied to knowledge-related items, many factors are thought to influence the performance of the CF system. The performance of a CF system, in most cases, can be measured by the accuracy of recommendations by the system. The factors discussed in Section 4.2 - distributions of evaluations, user characteristics, critical mass of users/evaluations, and mode of search - are important factors that need to be investigated. 4.3.1. Critical mass of users and evaluations The performance of a CF system would largely depend on the number of users and the number of evaluations per user. When the user group is large, the accuracy level of the recommendations will be higher than when it is small. As discussed in Section 4.2.1, the minimum number of users and the number of evaluations per user are, as a whole, termed 'critical mass’. 68 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The relationship between the number of users and performance will differ depending on the given circumstances. Figure 11 shows three hypothetical patterns. For example, pattern C hypothesizes that the increase in accuracy is sluggish at first, but at a certain point the accuracy increases abruptly, and at another point the rate of increase diminishes. These three hypothetical patterns do not cover all possible patterns. In actual data, a pattern that is entirely different from these three may emerge. However, two hypotheses can be formulated regarding the pattern of accuracy increase: First, as discussed in Section 4.2.1., the accuracy will increase as the number of users increases. Second, there will be a pattern and the pattern will vary depending on the given circumstances including user characteristics, product characteristics, and search mode. Performance (Accuracy of Recommendations’ Required Accuracy Level Critical Mass Total Number of Users Figure 11, Hypothetical Relationships Between Total Number of Users and Performance of CF Systems 69 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. In summary, questions regarding critical mass relate to two issues. One is whether there is any pattern in the increase of performance as the number of users increases. The second is how the pattern, if there is a pattern, varies as the circumstances change. H1. The performance of a CF system increases as the number of users increases. H1a. The performance of a CF system increases as the total number of users increases. H1b. The increase in performance adopts one of the hypothetical patterns. Hie. The pattern of increase will differ depending on the type of product and the search mode of the users. 4.3.2. Reference group size Several questions on reference group size emerge. Is there an optimal reference group size for a given circumstance? If there is an optimal reference group size in a certain situation, does the optimal size differ for different situations? If the size of the reference group is too small, the CF system’s performance (recommendation accuracy) will not be high because it risks generating recommendations based on too few people’s opinions. On the other hand, if the size of the reference group is too large, the system loses discrimination power in identifying clusters of users with different preferences, which is necessary for better recommendations. As the reference group size increases, the recommendation approaches the average of all users’ evaluation. Therefore, it is expected that there is an optimal size of ‘reference group’ 70 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. that maximizes the CF system’s performance. This relationship between reference group size and performance is illustrated in Figure 12. Previous studies [Herlock et al., 1999] demonstrate that the performance of a CF system varies as the reference group size varies, and the performance becomes worse in the two extremes. However, it still needs to be answered whether the optimal reference group size varies as the circumstances vary and if yes, how different circumstances affect the optimal reference group size. Performance (Accuracy of Recommendations^ Optimal Size Reference Group Size Figure 12. Reference group size and expected performance of CF system For those CF systems that use fixed reference group size, an interesting issue is whether or not the relative optimal reference group size - the optimal reference group size compared to the whole user group - changes as the number of users increases. With a small number of users, the optimal reference group size will be relatively large, close to or the same as the whole group size, because there are not enough users to be clustered into different groups. The optimal reference group size, compared to the 71 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. whole group size, however, will decrease as the number of users increases. Other factors such as characteristics of users and search mode will also affect the optimal reference group size. H2. Reference group size affects the performance of a CF system H2a. There is an optimal reference group size under given conditions. H2b. The relative optimal reference group size decreases as the number of users increases. H2c. The optimal reference group size varies according to other conditions. 4.3.3. Mode of search The relevance of knowledge-related items depends on the mode o f search - whether the search is a ‘problemistic search’ or ‘scanning’. In problemistic search mode, users already have very specific ideas on what they are looking for while users in scanning mode do not. A problemistic search is similar to a Boolean search - an item is either a ‘right’ or ‘wrong’ item for the user. There is less room in a problemistic search for other people’s judgment to help the search. Therefore, CF should be more effective for ‘scanning’ than for a ‘problemistic search’. On the other hand, an earlier study [Miller et al., 1997] has shown that a CF system is more effective for items that have more highly correlated users. In problemistic search mode, users with common problems will have very high 72 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. correlations. This implies that CF systems would be more effective for a problemistic search mode, where there are highly correlated users. The conflicting predictions can be resolved by critical mass. In a problemistic search mode, the relevance of information items will be highly dependent on what the problem is, whereas the relevance in the scanning mode depends on the general usefulness of the items. In scanning mode, users evaluate items based on ‘potential usefulness.’ In problemistic search mode, users rate the items based on ‘usefulness for the current task’. Potential usefulness will have less variability among users than usefulness for current work. Less variability in evaluations means fewer required users (critical mass) for the same performance level. With a small number of users, therefore, a CF system will perform better when the users are in scanning mode. If there are a sufficient number of users, even the users in problemistic search mode can be classified into groups that are large enough but with small variations within groups. In this case, the performance of the CF system will be higher than in the case where there are not enough users to make large enough groups with small variations in the evaluations. The above discussion can be summarized as follows. With a small number of users the CF system will perform better for users in scanning mode. However, as the number of users increases, the performance of the problemistic search will increase rapidly. 73 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. H3. The performance of CF is different when users are in scanning mode from when users are in problemistic search mode. H3a. With a small number of users, the performance of a CF system is better for the users in scanning mode than in problemistic search mode. (Scanning mode needs less critical mass) H3b. The performance of a CF system for a problemistic search mode increases faster than for the scanning mode as the number of users increases. 4.3.4. Characteristics of users As discussed in Section 4.2.4, the characteristics of users are expected to affect the performance of CF systems. From the discussion, it is expected that knowledgeable users will evaluate more accurately than less knowledgeable users. Therefore, the performance of CF systems with knowledgeable users will be better. Similarly, motivated users will evaluate the items with more precision, and thus, improve the accuracy of CF systems. H4. The performance of a CF system is affected by the characteristics of users. H4a. The system with more knowledgeable users performs better. H4b. The system with more motivated users performs better. 4.3.5 Pattern of users’ preference As discussed in Section 4.2.5, the pattern of users’ preference clusters would affect the accuracy of CF systems. The accuracy will be higher when users have clearer 74 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. preference clusters - tighter clusters or a smaller number of clusters. The pattern of preference clusters can be captured using K-S D statistics as discussed in Section 4.2.5. H5. The performance of a CF system is affected by the patterns of users’ preference clusters. H5a. The K-S D statistics that measure the pattern of user preferences vary significantly across domains. H5b. The performance of CF systems is better in the domain with higher K-S D statistics. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER V. EMPIRICAL STUDY This chapter describes the empirical study conducted. The chapter begins with the descriptions of the experiments that were conducted to collect data for the study. Then, the results of the data analysis and hypothesis tests then follow. 5.1. Research Design In order to test the hypotheses developed in the previous chapter, several things need to be done. First, actual evaluation data on more than one product should be collected. The purpose of the study is to test the performance of CF in various conditions. Therefore, actual data from various conditions - including different types of products - needs to be collected. Second, once the data is collected, the performance of CF needs to be calculated. Performance of CF is the accuracy of recommendations - how close the recommendations are to the actual evaluations. The most widely used method for calculating accuracy is simulation [Breeze et al., 1998; Herlock et al, 1999; Ansari et al., 2000]. In simulation, data is divided into two sets - a holdout set and a calibration set - and the recommendations for the holdout set are generated using only the calibration set (detailed explanations are in Section 5.2). The difference between the recommendations (estimated evaluations) and actual evaluations becomes the performance of CF. 76 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Consequently, a ‘hybrid’ method is adopted in this study. - The experimental method is used to collect data and observe behavior, and the simulation method is used to calculate the performance or the changes in performance. The hybrid method is appropriate for this study because the empirical test requires actual evaluation data and other information from users, and simulation is an appropriate method for calculating the performance of CF systems. 5.1.1 Selection of domains Previous CF studies applied CF technology mostly to consumer preference goods - for example, movies and music [Balabanovic and Shoham, 1997]. One of the goals of this study is to investigate the differences of CF accuracies across different product domains - specifically, the difference between consumer products and knowledge-intensive products. Evaluations on two different types of products - consumer preference goods and knowledge intensive products - are needed to compare the similarities and differences in the applicability of CF to these products. Among consumer products movies form one of the best domains for this study because the evaluation of an individual toward a specific item is determined by his/her preferences. Movies are also widely used in CF research [Breese et al., 1998; Herlock et al., 1999; Ansari et al., 2000], The appropriate products for the other domain (knowledge intensive domain) are mostly document repositories. A technical support repository is an example of a knowledge intensive domain because the technical support repository usually contains 77 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. huge amounts of information that is required for a certain task (customer support) [El Sawy, 1997]. A research paper is also a knowledge intensive product. A research paper is used mainly for solving a problem or carrying out a certain task. An individual’s evaluation of a specific item depends on the task or job of the person evaluating the item. Products that have characteristics similar to research paper are project reports, white papers, and other knowledge-intensive items. In this study, therefore, two different types of products were investigated - movies in the first experiment and research papers in the second (main) experiment. By comparing these two different types of products, the issue about the “differences in CF performance for different types of products,” discussed in the previous chapter, can be addressed. 5.1.2. Evaluation methods (measures) The ratings by subjects are the primary data used in the analysis. Several methods can be used to avoid the ‘rating sparsity’ problem. Two possible solutions to the problem are - using other measures as the substitute for actual evaluation and using ‘default evaluations’ for the missing evaluations. The ‘time spent in reading the paper’ may be a good indicator of the user’s interest in the paper. Some studies showed that the time spent on each item is a good substitute for an actual rating [Konstan et al. 1997]. However, their study is different from this one because the subjects in this study evaluate items that they already have knowledge about while in Konstan et al.’s study the items were mostly new to the subjects. When 78 with perm ission of the copyright owner. Further reproduction prohibited without perm ission. users browse items they are not familiar with, they would spend more time on items that they are interested in. Thus, time would be a good surrogate of their interests. On the contrary, if users are evaluating items they are familiar with, as in this study, they would not spend more time on items they are interested in because they already have knowledge of the items. In this case, the time spent for reading items is not a good measure of their interest. Therefore, the time spent in reading the paper was not used as a parameter in this study. Since this study is not comparing different recommendation methods, an extensive comparison of different recommendation methods is not performed. The main method used in this study is direct evaluation of the subjects. 5.1.3. Similarity measure For CF systems, a similarity index is one of the most critical factors that determine the accuracy of the recommendation because a similarity index is the criterion for identifying people with similar preferences. Thus, it is very important to decide what type of similarity index is used in data analysis as well as in implementing actual CF systems. Previous studies [Shardanand and Maes, 1995; Breese et al., 1998; Herlocker et al., 1999] have shown that correlation is one of the best similarity indices for CF. Therefore; a correlation coefficient was used as the similarity index in this study. The correlation coefficient is calculated using the following formula. 79 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. [Griffiths et al., 1993] T S ^ t-y )2 t=i 5.1.4. Experiment 1 (Pilot experiment) The goal of the first experiment was twofold. First, through the first experiment, evaluations about movies, which are representative of consumer preference goods, are collected. Second, the stability and usability of the system is tested in the first experiment. 5.1.4.1. Experimental system The pilot system was developed using a Microsoft Access database and the Inprise (former Borland) Delphi development tool. The system contained about 150 movies from various genres. The screenshots of the system are shown in Appendix A. 5.1.4.2. Sample The subjects were recruited from undergraduate classes at the University of Southern California and the University of California, Los Angeles. As an incentive for the subjects, they were informed that two randomly selected subjects would each be awarded $50 in cash. After the experiment, the two randomly selected subjects were announced on the experimental web page and the checks were mailed to them. The 8 0 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. pilot experiment lasted for about a month. A total of 157 people participated in the experiment. 5.I.4.3. Experimental procedure In the initial screen, subjects enter the information on their movie watching behavior - frequency of watching movies and preferred genre (screen shots are in Appendix A). They are then asked to evaluate movies they have watched using a ‘1 to 13’ scale system. After they evaluate 10 or more movies, a list of movies recommended for them is displayed. After seeing the list of recommended movies, they are asked to assess the accuracy and the ease of use of the recommendations. 5.1.5. Experiment 2 (Main experiment) The product for the main experiment was a research paper, which is representative of knowledge-intensive products. The system was developed on the top of an existing web application, Cybrarium (http://cvbrarium.usc.edu), which is a prototype of an infrastructure to create, synthesize, and disseminate knowledge in IS research (for more information about Cybrarium, refer to [Hars, 1998; Hars et al., 2000]). 5.I.5.I. Experimental system The collaborative filtering engine for the main experiment was developed using the Oracle database and Inprise’s Delphi. The source code for the system is included in Appendix F. It was developed as a Dynamic Component Object Model (DCOM) 81 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. object because DCOM is an effective way to provide functionality to multiple applications in a multi-tier system environment [Eddon and Eddon, 1998; Harmon, 2000]. Figure 13 shows the architecture of the system used for the main experiment. Clients (Subjects) ■ B E * ' sllslliSSii Server Internet - m User/Item D a ta b a se Recommendation Requests Evaluations Recommendations Evaluation Database Cybrarium Server Collaborative Filtering Engine Internet Information Server Figure 13. System Architecture for the Main Experiment The system for the main experiment contained abstracts of about 2,000 academic articles from the recent issues of five leading IS journals - Communications of the ACM, Information Systems Research, Journal of MIS, MIS Quarterly, and Management Science. Soliciting e-mails were sent to the potential participants. Between 100 and 200 e-mails were sent everyday to observe users’ behavior across time and to avoid network traffic congestion. 82 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.I.5.2. Sample Ideal subjects for the second experiment should meet the following criteria. First, the subject should be knowledgeable in the subject area. If the experimental items are simple consumer products, the subjects do not have to have knowledge about the subject area because they just need to evaluate how much they liked (or would like) the items. However, since the subjects in the second experiment must evaluate knowledge-related items, they should have enough knowledge in the subject area. Second, the subjects should, to some extent, share, common interests with each other. This condition is required because of the underlying mechanism of CF. Since a CF system generates recommendations based on others’ opinion, there should be a substantial number of people who share common interests. Finally, ideal subjects for this experiment should have some specific problems for which they are willing to search for help. As discussed in the Section 4.2.3, only people searching for answers to specific problems in the subject area are defined as being in ‘problemistic search mode.’ People in academia are good subject candidates for this experiment. Academics have in-depth knowledge in a subject area, they often share common research interests, and most of them have specific research questions. In this study, academics in the IS field were selected as the subject pool because they satisfy the above criteria and also because of the their high accessibility to the author. Soliciting emails were 83 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. sent to the academics (faculties) in IS. The email addresses were obtained from the IS Faculty Directory of ISWorld (www.isworld.ore), which is public in fo rm a tio n . 5.I.5.3. Experimental procedure An ID and a password is given to each user and an URL is included in the soliciting email (Appendix D) so that subjects can automatically log on to the system by clicking on the URL. After the first briefing screen (Appendix B), subjects are asked to search papers using the integrated search engine and evaluate them using a ‘0 to 6’ scale system. Subjects are asked to evaluate two things per paper - overall usefulness/relevance o f the paper fo r IS research and usefulness/relevance o f the paper fo r the user’s specific research project. The former is the evaluation of ‘scanning mode’ and the latter is that of ‘problemistic search.’ After evaluating 10 papers, the subject is asked, as an option, if he/she wants to evaluate more papers for better performance. Once the subject decided to get the recommendations, five recommended papers for each search mode (scanning and problemistic search) are given to the subject (Appendix B). More papers are recommended if the subject requests. At the end of the session, each user is asked to fill out a questionnaire (Appendix C). The questionnaire was designed to elicit from the subjects background information and his / her perceptions of the accuracy and of the ease of use of the system. Two separate accuracy issues are probed - one is ‘accuracy of matching’ and the other is ‘providing new perspectives. ’ The ‘accuracy of matching’ refers to the extent to which 84 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. the recommended papers are similar to their research topics in terms of subject area. ‘Providing new perspectives’ refers to the extent to which the recommendations stimulate new ideas and show different approaches to their research topics. Good recommendations should satisfy two conditions. Recommended papers should be in the same area as the user’s research topic. At the same time the recommended papers should show new aspects and different approaches that the user would not have found otherwise. Subjects may also add written comments. These behaviors - additional evaluation, adding written comments - are the indicators of the users motivation. All the evaluations and behaviors (clicking and duration of reading each paper) are recorded. The experiment lasted for about 6 weeks and, on average, 10-15 people visited the experimental web page every day. Many people sent various types of feedback emails - expressing interest, error report (dead links), suggestions for interface, etc. -along with the written comments. 5.2. Simulation For accuracy calculations, a simulation method is used in this study. This section reviews the simulation method as a research method and describes detailed simulation methods and the simulation parameters used in this study. 85 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.2.1. Simulation method Simulation is “the process of designing a mathematical or logical model of a real system and then conducting computer-based experiments with the model to describe, explain, and predict the behavior of the real system” [Hoover and Perry, 1990]. In order for a simulation to be possible, there must exist an explicit mathematical or logical model. Also, it must be possible to express the model as an efficient and effective computer program. Analytical or mathematical models have definite advantages and disadvantages to be considered carefully when selecting a problem solving technique [Hoover and Perry, 1990]. These advantages are: conciseness in problem description, closed form solutions, ease of evaluating the impact of changes in inputs on output measures, and in some cases, the ability to produce an optimum solution. Disadvantages include: assumptions regarding the system description, which may be unrealistic and complex mathematical formulations, which defy solution. Simulation models have their own plus points and deficiencies some of which are quite unique [Hoover and Perry, 1990]. Simulation models can describe systems, which are very complex. They can be used to experiment with systems, which are not yet in existence, or to experiment with existing systems without actually altering the system. Analytical models can also be used in these ways only if the system is not too complex. On the other hand, there are no closed form solutions to simulation models; 86 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. each change of input variables requires a separate solution or set of runs, making it very difficult to establish the validity of the model. 5.2.2. Simulation procedure The simulation in this study simulates recommendation generations with different sets of users and items. The evaluation data are divided into two sets - holdout sets and calibration sets as shown in Figure 14. The estimated evaluations for the items in the holdout set are calculated using only the evaluations in the calibration set. Since we know the actual evaluations in the holdout set, the estimation error (recommendation accuracy) for the holdout set can be calculated by comparing the estimated evaluations and actual evaluations. It is a common method used in time series analysis [Griffiths et al., 1993] and other CF studies [Breeze et al., 1998; Herlock et al., 1999; Ansari et al., 2000] to calculate the accuracy of estimations. User 1 u s e r 3 S e t ------------------ r I ; .. User n ^ Figure 14. Holdout Set and Calibration Set in the Simulation 87 — I t f i a u - i t e m 4 ,.............— Caisb/afcFon Set I f p m m -1 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. For example, if the size of the user holdout set is 2 and the size of the item holdout set is 3, as shown in Figure 14, the estimated evaluations are calculated thus: Step 1: Select 2 users who will belong to the holdout set. Step 2: Select 3 items for each user as the holdout set. Step 3: Calculate estimated evaluations for the 3 items selected in the Step 2. When calculating the estimated evaluations, only the evaluations of the users in the calibration set and the part of the evaluations (the evaluations for the items that were not selected as the holdout set in the Step 2.) of the users in the holdout set are used. Step 4: Calculate accuracies by comparing the actual evaluations and the estimated evaluations generated in Step 3. The detailed measures are explained in Section 4.2.6. Step 5: Repeat Step 2 through Step 4 for all users in the holdout set. Step 6: Average the accuracies of all the items of all users in the holdout set. The average is the final accuracy measure. One issue here is that the result (accuracy of the recommendation) of simulation may depend on who is in the holdout set (and calibration set) and which item is in the holdout set (and the calibration set). The ideal case would be trying all possible combinations of users and items. However, the number of possible combinations is virtually infinite. Therefore, a certain number of iterations with randomly selected combinations of users/items are carried out in the simulation for this study. The 88 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. parameters of the simulation - the size of the holdout/calibration set and the number of iteration etc. - are discussed in the following section. 5.2.3. Settings for the simulation Several parameters need to be set for the simulations to calculate the accuracy of CF systems. This section discusses the issues involved in conducting the simulations. 5.2.3.I. Selecting users for simulation In the experiment the order in which participants participated may have, by chance, an effect on the accuracy of the recommendation. So, to eliminate bias stemming from the time at which a participant entered the study, one has to take the timeline away and consider also what would have happened if the participants had engaged in a different order. Therefore, the randomization of users is important when the effect of the increase of users is tested. When the effect of the number of users is investigated, recommendation accuracies for increasing number of users - for example, at first 100 users, then 110 users, and so forth up to 260 - are examined. If the users are selected in chronological order, the first 100 users are always included in the simulations and thus, may cause some bias in the performance measures. In this study, the users included in the simulations were randomly selected to eliminate the possible bias due to the participation order. 89 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.2 3 .2. Default evaluations The default value is a value that is assigned to the items that users have not evaluated [Breese et al., 1998; Sarwar et al., 1998]. The advantage of using default values is the increase of the coverage (the percentage of users or items for which recommendation can be generated) [Breese et al., 1998]. However, CF with default values may cause some biases when comparing different conditions - search modes, type of product, etc. For example, if condition A’s coverage is 80% while condition B’s is 60%, then the remaining 20% of condition A should be replaced with default values and 40% of condition B should be replaced with default values. Since the percent of default values is higher with condition B, the accuracy with condition B may degenerate (if CF is better than default values) more than condition A. This may prevent an accurate comparison of the two conditions. In this study, ‘CF without default values’ was used for two reasons. First, the ‘CF with default values’ has some problems as discussed earlier this section. Second, ‘CF without default value’ is faster in generating recommendations because it has less data to be processed. 5.2.3.3. Holdout Sample Size One of the settings that needs to be determined for the simulation is ‘holdout sample size’ - how many users and items are included in the holdout sample. The size of the item holdout sample should not be too big. For example, if the size of the 90 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. holdout item sample is 6 and a user has evaluated 10 items, it means that the estimated evaluations for 6 items are generated based on the evaluations for the remaining 4 items. If the size of the holdout item sample is too small, there is another problem. For example, if the size of the holdout item sample is 2, there is not much room for a recommendation system - the system can recommend the two items either in the order of 1, 2 or 2, 1 . There would, then, not be enough variance in the performance measures. Thus, the size of the holdout item sample in this study should be between 3 and 5. In order to examine whether the performance is affected by the item holdout size, simulations with three sizes (3, 4, and 5) were conducted for each condition. Similarly, the size of the holdout user sample should be neither too big nor too small. After several trials, the size of the holdout user sample was set to 10 because there was not a big variance in the performance for different sizes of holdout user samples. • Size of user holdout sample = 5 • Size of item holdout sample = 3, 4, and 5 The performance may vary depending on who/which items are in the holdout sample and who/which items are in the calibration sample. Therefore, the holdout samples (users and items) were randomly selected, and the results were averaged and used as the final performance measure. 91 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.2 3 .4. Iterations The ideal method, not considering time constraints, for this study is running simulations that explore all possible combinations of users, items, and conditions. However, the number of possible combinations is virtually infinite and it is necessary to limit the number of item iterations (how many random combinations of item are simulated) and the number of user iterations (how many random combinations of users are simulated).A balance between the output variance (reliability) and the simulation time (feasibility) has to be achieved. Several criteria for selecting the number of iterations were determined. • The number of iterations should be set at the point where the standard deviation of output measures (accuracies of CF) becomes steady enough. • The entire simulation should be possible to complete within a certain time period (about 2-3 months) with a personal computer of decent computing power (Pentium III 450MHz with 128MB of RAM). In order to determine the number of iterations, several exploratory simulations were conducted. In the exploratory simulations, the accuracies of CF for different numbers of iterations were examined. For each set of iterations (a certain number of item-iterations and a certain number of user-iterations), a total of 5 runs of simulations were carried out and the standard deviations of the 5 runs were averaged. The result of the exploratory simulations is shown in Figure 15. The figure indicates that the standard deviation of the accuracies is stabilized at a certain number of iterations: around 20 for item iterations and around 10 for user iterations. 92 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 0.16 0.15 0.14 0.13 0.12 0.11 0.1 Number of Iterations and Standard Deviation of Accuracy —♦—A verage S tandard Deviation 5 10 15 20 25 30 Number of Item- Iterations a. Number of Item- Iterations and Standard Deviation of Accuracy Number of Iterations and Standard Deviation of Accuracy Number of User- Iterations A verage S tan d ard Deviation b. Number of User-Iterations and Standard Deviation of Accuracy Table 7. Number of Iterations and Standard Deviations of Accuracy Therefore, the following simulations parameters were used in the simulations. • Number of different combinations for items = 20 • Number of user different combinations for users = 10 • Number of runs = 5 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. A program was developed for the simulation in this study. The program shares the recommendation generation modules with the collaborative filtering engine used in the main experiment. However, a substantial amount of modification was made to implement the simulation requirements - e.g. random selection of users and items, iterations, and performance calculation. Appendix E shows the screen capture of the simulation program developed for this study and an example of simulation output. 5.3. Descriptive statistics As described in the previous chapter, two experiments were conducted in this study - the pilot experiment with movies and the main experiment with research papers. 5.3.1. Experiment 1 (Pilot experiment) The total number of subjects participating in the first experiment was 150. The total number of evaluations was 1809. The average number of evaluations per subject was 12.06. 5.3.2. Experiment 2 (Main experiment) The total number of soliciting emails sent out for the main experiment was approximately 4,200. About 480 people visited the experimental site and 259 people 94 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. participated in the experiment. Thus, the response rate was about 11.4% ( 480/4,200). The total number of evaluations was 3,634 or 1,817 for each search mode (scanning and problemistic search). The average number of evaluations was 14.02 per user, or 7.01 per user/search mode. The number of research papers evaluated by at least one user was 1,063. As explained in Section 5.1.5.3, the subjects were asked to fill out a questionnaire after they got a list of recommended papers. The number of questionnaires completed was 68, significantly less than the total number of participants. There appear to be two reasons for the small number. First, the subjects saw the questionnaire only after they evaluated 10 or more papers. Many subjects (63.6%) quit before they finished evaluating 10 papers. Second, many early subjects did not see the questionnaire even after they finished evaluating 10 or more papers because due to the small number of subjects in the database recommendations for them could not be generated. Some other variables were used as auxiliary measures of the accuracy of the system. The subjects’ responses about the perceived accuracy of the recommendations were used as a measure of the performance of the CF system. The participants could also add written comments along with the questionnaire. Their written comments were also used as an auxiliary indicator of-the accuracy of the system. 95 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.4. Hypothesis tests In this section, hypotheses discussed in Chapter 4 are tested and the results are discussed. The empirical results of the study are simulation outputs whose statistical distributions are difficult to identify. If the statistical distribution is not identified, non- parametric methods should be used for statistical inferences [Conver, 1980]. Therefore, all the statistical tests were conducted using non-parametric methods. 5.4.1. Number of users and the performance of CF systems H1. The performance of a CF system increases as the number of users increases. H1a. The performance of a CF system increases as the total number of users increases. H1b. The increase in performance adopts one of the hypothetical patterns. H1c. The pattern of increase will differ depending on the type of product and search mode of users. In order to test the hypothesis, simulations for different number of users were conducted and the recommendation accuracies - both rank-based accuracy and MSE - were collected. For example, a number of iterations, as discussed in 4.3.2, were conducted with 60 users. The average performance of those iterations was calculated 96 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. and used as the performance of CF with 60 users. Next, the performance with 70 users was calculated in the same way, followed by calculations with 80 users and so on. 5.4.1.1. Hypothesis tests using simulation data Figure 15 shows the simulation results with different holdout sample sizes. It is obvious from the figure that the performance of a CF system, as hypothesized, increases as the number of users increases. More rigorous tests were conducted to investigate whether the increase is statistically significant. If the data satisfy the normality assumption of regression analysis, the hypothesis can be tested using regression analysis. However, ordinary regression analysis cannot be applied if the data do not satisfy the normality condition [Griffiths et al., 1993], It is not certain if the probability distribution of the data in this study, including the data in Figure 15, is normal. Thus, a non-parametric method should be used to test the pattern of performance increase. One way of testing whether the increase is statistically significant would be to check whether the data (accuracy measures) are significantly deviated from the non increasing line (horizontal line). The logic is similar to what is used in the t-test on the coefficients of independent variables in regression analysis. The only difference is that the non-parametric test in this study does not have any assumptions on the probability distribution of the errors. 97 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Average Performance (Ffoblerristic Search Mode) 0.16 0.14 0.12 5 0.08 8 0.06 < 0.04 0.02 Number of Users Average Performance (Scanning Mode) * * * * * j h h p s * 0,06 JN sT * 1 3 ----------- i - t- t* ■ ifn m 1 0.02 I ' 1 & & $ $ q f $ Number of Users a. Holdout Item Size = 3 Average Performance (Scanning Mode) 0.18 0.16 0.14 2 0.1 | 0.08 < 0.06 0.04 0.02 O ^ & & r f ? $ Number of Users — - ♦♦ Average Performance (Problemistic Search Mode) 0.3 0.25 >. 0.2 m 3 0.15 o < 0.1 0.05 0 Number of Users b. Holdout Item Size = 4 Average Performance (Problemistic Search Mode) 0.4 0.35 0.3 0.25 0.15 0.1 0.05 Number of Users Average Performance (Scanning Mode) 0.25- ............"TS****** °-2- o , i < c & & n'5’ d f Number of Users c. Holdout Item Size = 5 Figure 15. Number of Users and CF Performance (Research Papers) 98 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The next question raised by the hypotheses (Hlb) is about the pattern of performance increase. In Section 4.3.1, three possibilities were proposed regarding the pattern of performance increase. First, the performance increase is a straight line. Second, the performance increases follow an S-shape, which implies that there is a take-off point. Last, the performance increases follow an increasing line with a diminishing increase rate (square-root curve). It is apparent, however, that the increasing patterns in the charts are more approximate to a straight line or an S-shape rather than a square-root curve. In this study, therefore, only two possibilities - straight line and S-shape curve - were compared against the horizontal line. The test procedure and results are illustrated in the following sections. For the straight line, common linear regression analysis can be used to derive the best fitting line. It is not necessary to check whether the data satisfy the assumptions of regression analysis because the purpose of regression analysis here is to identify the best-fitting line, not to make statistical inferences. There are several different ways to identify an S-shape curve that best fits the actual data. Most commonly used S-shape line models are Bass’s [1969] and Mansfield’s [1961] diffusion models. One limitation of Mansfield’s model compared to Bass’s model is that it is symmetric. However, it is simpler than Bass’s model [Mahajan et al, 1990]. Mansfield’s model was used in this study because it is simpler and the test is about whether it is S-shaped or not, rather than whether it is symmetric or not. 99 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. y = - i \ + e~{c + q t) where, y = actual data t = time (independent variable) c, q = constants (5-1) Pattern of Performance Increase (Scanning Mode) 0.1 0.08 0.06 0.04 0.02 0 •Y -uy) Number of Users a. Fitted Line Using a Straight Line Pattern of Performance Increase (Scanning Mode) 0.08 - 0.06 - 0.04 -; 0.02 - ! E(Y) Number of Users b. Fitted Line with a S-Shape Line Figure 16. Actual Data and Fitted Data by Straight Line and S-Shape Curve In actual estimation, the following transformed equation is used. 100 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. - ln(— -1) = c + qt ..............................(5-2) y The y (actual data) is transformed to - ln(— -1) and c and q are estimated using y OLS (ordinary least square) regression analysis. After c and q is estimated, y is estimated using formula (5-1). Figure 16 shows an example of actual data and fitting data. The example is the data of scanning mode with holdout item size 3. The deviations of three different lines (straight line, S-shape curve, and horizontal line) were measured using mean absolute deviations (MAD) shown in formula (5-3). For each line, deviations for three different holdout item sizes (3, 4, and 5) were calculated and averaged. E h - : ? * ! MAD of line i = (5-3) N where, yk = actual performance for the kth set of users yik = estimated performance on hypothetical line i for the kt h set users N = number of user sets In order to test the statistical difference between two hypothetical lines - straight line and horizontal line, for example - a Wilcoxon Signed Rank Test was conducted for each pair of hypothetical lines. The Wilcoxon Signed Rank Test is a non- parametric method which is used to test the mean difference of two groups [Conver, 1980]. The test results are summarized in Table 8. 101 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Table 8. Test Results of Pattern of Performance Increase (Research Papers) Mean Absolute Deviation (MAD) Horizontal Line Straight Line S-Shape Curve Scanning Mode 0.0332 0.0061 0.0051 Problemistic Search Mode 0.0542 0.0055 0.0049 Z value (Wilcoxon Signed Rank Test) Horizontal Vs. Straight Horizontal Vs. S-Shape Straight Vs. S-Shape Scanning Mode -3.84*** -3.91*** -1.13 Problemistic Search Mode -3.88*** -3.95*** -0.88 *** Significant at a = 0.01 level The results imply that the increase pattern is significantly closer to a straight line or an S-shape line than to a horizontal line. However, the difference between a straight line and an S-Shape line is not statistically significant. Similar tests were conducted for the movie data. The patterns of performance increase are shown in Figure 17, and the test results are summarized in Table 9. From the tests, it can be concluded that the accuracy of CF for movies increases significantly as the number of users increases. The increase pattern of the accuracy is significantly closer to an S-shape curve than to a straight line. The difference between straight line and S-Shape curve is significant at a = 0.01 level. 102 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Average Performance (Movie) 0.05 i « -0.05 | -0.1 < -0.15 - 0.2 Number of Users a. Holdout Item Size = 3 Average Performance (Movie) 0.1 o 8 - 0.2 Number of Users b. Holdout Item Size = 4 e Performance (Movie) 0.2 1 o o o o 3 -0.2 -0.4 - - 0.6 Number of Users c. Holdout Item Size = 5 Figure 17. Number of Users and CF Performance (Movies) R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Table 9. Test Results of Pattern of Performance Increase Movies) Mean Absolute Deviation Horizontal Line Straight Line S-Shape Curve 0.1403 0.0344 0.0087 Z value (Wilcoxon Signed Rank Test) Horizontal Vs. Straight Horizontal Vs. S-Shape Straight Vs. S-Shape -3.236 -3.233 -2.919*** *** Significant at o c = 0.01 level It is interesting that the pattern of increase differs in research papers from that in. The accuracy of CF for research papers increases almost linearly while the accuracy of CF for movies increases following an S-shape curve. It implies that for consumer products such as movies, there is a ‘ take-off point’ at which accuracy increases dramatically). On the contrary, knowledge-intensive products such as research papers have a relatively steadily increasing accuracy. Movies, however, reach a state of saturation with a comparatively small number of users. In this study, the accuracy of movies seemed to reach the peak with the first 160 people. Research papers also have higher levels of accuracy in rank-based measures than do movies with the same number of users. This means that there is a greater opportunity for CF to provide recommendations for knowledge-intensive products than for consumer products. One possible explanation of the research paper’s higher accuracy levels would be the deviations of similarities across users. The appeal of consumer products, such as movies, would usually have less variance across users (if a movie is good, then most people would like it) than would be the case in knowledge- 104 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. intensive products like research papers (even if a research paper is good, it is not necessarily relevant to all users). Therefore, the correlations of evaluations on movies between a user and other users will vary less than those on research papers. Hence the deviation of correlations across users is lower in movies than in research papers. The actual data confirm it (standard deviation of correlations - movies: 0.347, research papers - scanning: 0.646 and problemistic search: 0.702). If users’ preferences or evaluations are relatively invariant, the recommendations by CF cannot significantly improve upon the recommendations by ‘average of everybody.’ This provides an explanation for the better performance of CF in the research papers. 5.4.I.2. Hypothesis tests using perceived accuracy In order to examine the changes in the accuracy of recommendations, indirect measures - satisfaction and perceived accuracy across time - were also collected and observed. Since the soliciting emails were sent out sequentially, the changes in users’ satisfaction and perceived accuracy across time can be used as a measure of the effect of the increase in the number of users. Two groups of questions were asked to the subjects regarding the accuracy of the recommendations. One was how much the recommendations matched their research area; the other was how much new perspective the recommendations provided to their research (refer to Appendix C). The responses were divided into two groups depending on the time of response. The first group (earlier response) consists of people who used the system between June 20 and July 9 and the second group (later response) consists of people who used the 105 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. system between July 10 and July 24. Table 10 shows the t-tests results on the subject responses across time. Table 10. Perceived Accuracy of Recommendations Across Time Accuracy of Matching Providing New Perspectives Average t-value Average t-value Scanning Mode Earlier Response (6/20 - 7/9) 4.73a (n = 26) 1.59 4.38a (n = 25) 1.02 Later Response (7/10-7/24) 4.10a (n = 10) 3.91a (n = 10) Problemistic Search Mode Earlier Response (6/20-7/9) 5.00a (n = 26) 0.82 4.80a (n = 25) ^ ^ A*'* 2.14 Later Response (7/10-7/24) 4.60a (n=ll) 3.80a (n = 10) a. I =Excellent, 7=Poor ** Significant at a = 0.05 level The table shows that the perceived accuracy improves (the average number decreases), as the number of users increases. However, only ‘providing new perspectives in problemistic search mode’ was statistically significant. This result is congruent with the simulation results. The results are statistically insignificant because of the small sample size. 106 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.4.2. Reference group size and the performance of CF systems H2. Reference group size affects the performance of a CF system H2a. There is an optimal reference group size under given conditions. H2b. The relative optimal reference group size decreases as the number of users increases. H2c. The optimal reference group size varies according to other conditions. The effect of reference group size was examined by observing the changes in accuracy as the reference group size changes. The reference group size was determined using the “best-n-neighborhood” method discussed in 4.3.2. The simulations were run for different sizes of holdout items - 3, 4, and 5. A total of 60 reference group sizes (1 to 60) were tested for each holdout item size, and the number of holdout users was 5. The results are summarized in Figure 18. From the figure, the research papers seem to have no clear optimal reference group size. The changes in the accuracy across different reference group sizes, except for small bumps, seem to be flat rather than convex as was hypothesized in Chapter 3. Similar simulations were carried out for the movie data. The results are summarized in Figure 19. 107 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Reference & oup Size and Accuracy (Scanning) 0.1 - >. °-oa m 5 0.06 - o « ^ 0.04 - 0.02 - . _ — ......... - ■ - i \ * > * * f> <5 < !> < $ > T? < $ > $ & 1? 4 Bsference Group Size Reference & oup Size and Accuracy (Problemistic Search) 0.16 0.14 0.12 S' 0.1 a 5 0.08 o ^ 0.06 0,04 0.02 0 N ^ < v N < p < § > 4 £ & # 4 Reference Group Size a. Holdout Item Size = 3 0.25 - 0.2 - S' 0.15 - J B 3 O ,3 0.1 . 0.05 - Reference Group Size and Accuracy (Scanning) V n .A a a A v ^ \ ® # 4 (? > tf> # 4> 4 Reference G-oup Size Reference Group Size and Accuracy (Problemistic Search) 0.3 0.25 0.2 a 0.15 0.1 0.05 N ^ £ & £ < $b $ Reference Gfroup Size b. Holdout Item Size = 4 Reference Group Size and Accuracy (Scanning) 0.3 0.25 8 0 .2 - 3 8 0.15 - < c 0.1 ■ 0.05 - i . » im lj* « A * j A m | r - u j c n c o h v i - i n a i c o h ' r - i n o j r t h - i - i - N W C M n w ^ t i f w i n Reference Group Size R eference Cfroup Size and A ccuracy (Problemistic Search) 0.45 0.4 0.35 0.3 m 0.25 0.1 0.05 U D O C O f'“ 05 n s t . io in Reference Group Size c. Holdout Item Size = 5 Figure 18. Reference Group Size and Recommendation Accuracy (Research Papers) 108 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Reference Group Size and Accuracy (Movie) 0.04 >. 0.02 -0.04 Reference Group Size a. Holdout Item Size = 3 Reference Group Size and Accuracy (Movie) 0.1 0.08 I.06 0.04 £ 0.02 0 I °-( "ex'*. T T T n -r r TT i - i ’T T T n - r n v i r i t i i r r n i i 11 111 1 1 i gv ^ ^ ^ - -< gr- ..gv-..^r . Reference Group Size b. Holdout Item Size = 4 Reference Group Size and Accuracy (Movie) 0.15 » 0.1 0.05 -0.0& Reference Group Size c. Holdout Item Size = 5 Figure 19. Reference Group Size and Recommendation Accuracy (Movies) R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Movies, unlike research papers, have a clear optimal reference group size. For holdout size 4 and 5, the optimal reference group size seems to be 73, and 95 seems to be optimal size for the holdout item size of 3. Optimal Reference Group Size 1 0 0 y 80 - - 60 -- 40 - 20 - - - - 0.6 I I Optim al Size Ratio - 0.2 Number of Users Figure 20. Optimal Reference Group Size and Number of Users The next hypothesis (H2b) is about the ‘relative size’ of the optimal reference group - whether the relative size (reference group size / total number of users) of the optimal reference group decreases as the number of users increases. Figure 20 shows the relationship between total number of users and optimal reference group sizes. It also shows the changes in the relative reference group size (ratio of the optimal reference group size to the number of users). The optimal reference group size increases along with the increase in the total number of users, but the relative size of the optimal reference group size decreases and 110 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. becomes stable at around 0.55 (55% of the total number of users). A statistical test similar to what was used to test hypothesis 1 was conducted to check whether the decrease in the relative reference group size is statistically significant. Deviations of the data from a horizontal line (average) and a declining line were tested as shown in Table 11. The difference is not statistically significant, although the straight line has smaller errors (closer to the actual data). Therefore, it cannot be concluded that the relative reference group size is decreasing. Table 11. Test Results of Pattern of Performance Increase (Research Papers) Horizontal Line Straight Line Mean Absolute Deviation (MAD) 0.0440 0.0308 Horizontal Line vs. Straight Line Wilcoxon Signed Rank Test (Z value and probability) -0.970 ( p = 0.332) It is interesting that research papers do not have a clear optimal reference group size, while movies do. One possible reason for this phenomenon would be the pattern of preference clusters. As shown in Section 5.4.5, the K-S D statistics of research papers is larger than of movies. This implies that the user group of movies may consist of users with overlapping preferences. In research papers, the large standard deviation of correlations implies that the users may consist of several clearly isolated clusters, but with small numbers of users in each cluster. In movies, therefore, quite accurate recommendations are possible with a small number of users who have high correlation with the current user. However, as the 111 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. reference group size increases beyond a certain point, the accuracy decreases because the added users have lower and lower correlations with users. In other words, if reference group size exceeds a certain point (optimal point), the loss by noise is larger than the gain by information. On the contrary, in the case of research papers, the addition of users to the reference group has little impact because the users consist of small isolated user groups. If you are a user in cluster A, for example, increasing reference group size would not affect the accuracy much because any users from clusters other than cluster A are almost equidistant from you. This explains why the research papers do not have a clear optimal reference group size. The result implies that the importance of reference group size varies depending on the characteristics of the items. When a CF system is used for knowledge-intensive items, the reference group size may be not as important as it is for consumer products such as movies. However, if CF systems are applied to consumer products, the reference group size needs to be frequently checked and tuned (optimized) because it affects the accuracy of the CF system. 112 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.4.3 Mode of search and the performance of CF systems H3. The performance of CF is different when users are in scanning mode from when users are in problemistic search mode. H3a. With a small number of users, the performance of a CF system is better for the users in scanning mode than for users in problemistic search mode. (Scanning mode has less critical mass) H3b. The performance of a CF system for a problemistic search mode increases faster than for the scanning mode as the number of users increases. In order to test the hypothesis, recommendation accuracies of the two search modes - scanning and problemistic search - were compared. Subjects were asked to evaluate two things for each paper - overall usefulness for IS research and relevance to the user’s current research project. The overall usefulness represents the user’s evaluation when he/she is in scanning mode, and the relevance to the current research project represents the evaluation when the user is in problemistic search mode. The differences in the performance for the two search modes were calculated and compared. The accuracies of scanning and problemistic search modes with a small number of users are similar. However, as the number of users increases, the accuracy of the problemistic search mode improves faster than in the scanning mode. This implies that CF systems would be more effective for those users who have a specific purpose (problemistic) for the search 113 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Average Performance 0.16 0.14 ^ 0 .1 2 § 0 .1 3 0.08 | 0 06 0.04 0.02 0 o o o o o o o o o o o COCOOCSJ'4> <£»COOCM'4'C0 r - r - r - r r C M O J W C M Number of Users Search Scanning a. Holdout Item Size = 3 Average Performance — • — Problemistic Search - Scanning o o o o o o o o o o o c o o o oo a' S - o oo o cM 'i - c o I-T-1-T- I-CJWWW Number of Users b. Holdout Item Size = 4 Average Performance 0.4 0.35 0.3 0.25 o 0.15 0.05 oooooooo COOOOC'J'^tCOCOO o o o OJ ^ CD W W W ■ Problemistic Search " Scanning Number of Users c. Holdout Item Size = 5 Figure 21. Search Mode and Recommendation Accuracy R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The result indicates that CF systems would work better for those users who need help with a specific problem. Currently, CF systems are used for consumer products whose evaluations are more homogeneous across users. However, the result shows that CF systems would be more effective in domains that have more heterogeneous evaluations across users. The knowledge-related items such as research papers and task-related items such as documents would be good domains to apply CF systems to because they are expected to have more heterogeneous evaluations than consumer products such as movies. 5.4.4 C haracteristics of users and the performance of CF systems H4. The performance of a CF system is affected by the characteristics of users. H4a. The system with more knowledgeable users performs better. H4b. The system with more motivated users performs better. As described in 5.1, a total of 68 people completed the post-experiment survey. The survey questionnaire is attached in Appendix C. The respondents’ had spent between 0 & 34 years, at an average of 13.97 (standard deviation = 9.08 ) years after a masters degree in IS or in a related field. The average number of years working on the specific project that they referred to when they were evaluating papers was 3.6 (standard deviation = 3.6) ranging from 1 month to 20 years. 115 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 5.4 A L The effect of knowledge and experience of users on accuracy of CF The effects of knowledge of users and their motivation on performance were examined using the subjects’ direct responses and other measures collected during the experiment. The years in the IS field was used as the measure for knowledge of users in a general IS field, and the years in a specific project was also used as a surrogate of the knowledge of users in the specific area. The subjects were divided into three sub groups depending on the number of years in the IS field and in a specific project. The respondents were sorted according to the years in the IS field and divided into three groups. The subjects were similarly divided into another three groups depending on the number of years in the project they referred to when evaluating papers. Table 12 shows descriptive statistics of the subgroups. Table 12. Descriptive Statistics of Subgroups Sub-groups (Group size) Average Number of Years (Range) Number of Evaluations Average Kruskal-Wallis x2 Years in IS Field Group 1 (17) Group 2(20) Group 3(23) 5.1 (1-8 years) 12.5 (9-15 years) 24.1 (16 years and over) 24.5 22.0 23.1 0.641 (p=0.726) Years in the Specific Project Group 1 (19) Group 2(20) Group 3(21) 1.4 (-1.5 year) 5.2 (1.5 - 3 years) 4.4 (4 years and over) 22.3 22.7 24.3 0.862 (p=0.650) The performance of the CF system for each group was simulated by iteratively calculating the accuracy of the system, as explained in the previous chapter, for the people in the group. The simulation was conducted with 50 different reference group sizes (1 to 50) for each group. The maximum accuracy across different reference 116 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. group sizes was taken as the accuracy of each group. This procedure is based on the assumption that the CF system is optimized with the given conditions. The accuracy of Group 1, for example, is the accuracy of the CF system with the optimal reference group size (reference group size that maximizes the performance) within the given group members. A difference in the number of evaluations in a sub-group is a concern because it may affect the accuracy of the CF system. Therefore, if the number of evaluations of groups are different, the “number of evaluations” effect may confound the “years in IS field (project)” effects. The Kruskall-Wallis Test, which is a non-parametric method to test mean differences of multiple independent groups [Conver, 1980], was conducted to examine whether there are any statistically significant differences in average number of evaluations across the groups. The p-values imply that there are no statistically significant differences in the number of evaluations across the sub-groups. The number of years in the IS field is a measure of the subjects’ knowledge in the IS area in general, while the number of years in a specific project is an indicator of the subjects’ knowledge in the specific topic. Therefore, the relationship between the number of years in IS field and the accuracy of scanning mode was examined first, and then the relationship between the number of years in a project and the accuracy of the problemistic search mode was investigated. Table 13 and Figure 22 show the summary of simulation results for the sub groups. In order to examine whether the differences are statistically significant, 117 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Kendall’s W-test can be used. Kendall’s W-test is a non-parametric method to test mean difference across multiple related groups [Conver, 1980]. The word, ‘related’ means that the groups have data from several different conditions. For example, the groups have simulation results with different holdout item sizes - 3, 4, and 5. In this case the result of a group with holdout item size 3 should be compared with another group’s simulations with the same holdout item size. The results show that the accuracies of CF in scanning mode for the three groups are statistically significant at a = 0.05 level, and the accuracies of problemistic search mode is significant at a = 0.01 level. Table 13. Simulation results for user groups with different experience Years in IS Field Accuracy3 Kendall’s W Group 1 (- 8 year) 0.158 0.418* Group2 (9-15 years) 0.145 Group3 (16 years and over) 0.149 Years in the Project Accuracy1 * Kendall’s W Group 1 (- 1.5 year) 0.213 0.510** Group2 (2-3 years) 0.229 Group3 (4 years and over) 0.225 a. Average accuracy of scanning mode (bigger number is better) b. Average accuracy of problemistic search mode (bigger number is better) * Significant at a = 0 .1 level ** Significant at oc=0.05 level Interestingly, the accuracies of the two search modes have reverse patterns. The accuracy is lowest in the group with moderate experience in the scanning mode. However, the accuracy is highest with moderate experience in the problemistic search mode. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Y ears in IS Field & Accuracy (Scanning Mode) 0.155 - 0.145 ■ 0.14 - Sroupl (-8 Group2 (9-15 Group3 (16 year) years) years and over) Y ears in IS Field a. Number of Years in IS Field and Accuracy of CF Y ears in Project & A ccuracy (Problem istic S earch Mode) 0.235 -r 0.23 - >• 0.225 - 0.215 - 0.21 - 0.205 - 0 . 2 - Groupl (Less than Group2 (1 .5 -3 Group3 (4 years 1.5 year) years) and over) Y ears in Project b. Number of Years in a Specific Project and Accuracy of CF Figure 22. Experience and Accuracy of CF The reversed patterns of the two modes may be due to the differences of the evaluations in the two modes. As a novice in an IS field or in a certain project, a user may have a relatively myopic view of the value of papers. Therefore, the evaluations from the less experienced users are fragmented. However, since the users are more fragmented in the problemistic search mode - the evaluation criteria are narrower (hit or miss) - there may not be a sufficient number of users in a cluster. As shown in the previous study [Carter et al., 1988], users may develop more dimensions (criteria) of evaluation as they gain more knowledge in the area. This addition of dimension may 119 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. affect the accuracies of the two modes differently. In the scanning mode, the extension of evaluation criteria means lower correlation within the clusters. In problemistic search mode, however, it may increase the number of people within a cluster. In other words, in problemistic search mode, as people add some more dimensions in evaluation, each cluster is more populated, which increases the performance. This explains the reversed pattern of the CF accuracy for the users with moderate experience. As users acquire more knowledge and experience, they may get a better consensus on evaluation criteria. Therefore, the opposite happens - the performance in scanning mode improves while the performance in problemistic search mode decreases. 5.4.42. The effect of motivation of users on the accuracy of CF One of the indicators showing how much a subject is motivated for participating in an experiment is how much time and effort he/she devotes to the experiment. In the experiment for the study, the subjects could get recommendations once they evaluate 10 papers (20 evaluations for two modes). However, they were given an option to evaluate more papers if they wished to get more accurate recommendations. If a subject had evaluated more than 10 papers, it meant that he/she voluntarily spends more time and effort to get more accurate evaluations. Therefore, the number of evaluations is an accurate measure of how much they were motivated by the experiment. 120 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The effect of motivation of users on the accuracy of CF was tested by comparing the accuracy of CF for three user groups that have different numbers of evaluations. The descriptive statistics of the three groups are summarized in Table 14. In order to check whether there is any correlation between the number of evaluations and the number of years in an IS field, a non-parametric test (Kruskal-Wallis Test) was conducted. As shown in the table, there were no statistical differences in the average number of years in the IS field across the three groups. Table 14. Number of Evaluations and Number of Years in IS Field Average Number of Evaluations (Range) Number of Years in IS Field Average Kruskal Wallis x2 Group 1 Group 2 Group 3 20.0 (20-20) 22.8(22-24) 29.5 (26-36) 13.7 a 15.6a 13.6a 0.49 (p=0.78) The accuracies of CF for the three groups were examined as summarized in Figure 23. The graphs show the differences in the accuracy of CF for the three groups. 8 S C 0) £ a ® 3 ■ & 2 < r o Number of Evaluations & Accuracy (Scanning) 40 30 2 0 -f 10 0 m 0.3 0.2 0.1 0 Group1Group2Group3 Number of Evaluations >. o 8 < i i Optim al R eference G roup Size — ■—A ccuracy 121 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Number of Evaluations & Accuracy (Problemistic Search) a 3 O o © o c a > -8 a > cc N 25 - 20 - 15 - 10 - 5 - 0 - 0.3 0.25 0.2 0.15 0.1 0.05 0 >. o « 3 O o < C m Optim al R eference G roup Size — H— A ccuracy Group1G roup23roup3 Figure 23. Number of Evaluations and CF Performance (Research Papers) Figure 23 shows that the accuracy of the CF is best for the groups with medium numbers of evaluations. It is the same for the both modes, although only one of them (problemistic search mode) is statistically significant. The statistical significances were tested using a non-parametric method, Kendall’s W Test. The test results are summarized in Table 15. Table 15. Number of Evaluations and CF Performance (Research Papers) Groups (Range of Number of Evaluations) Rank-based Accuracy Kendall’s W Scanning Mode Group 1 (20) 0.243 0.063 Group2 (22-24) 0.270 Group3 (26-36) 0.249 Problemistic Search Mode Group 1 (20) 0.122 0.813** Group2 (22-24) 0.269 GroupS (26-36) 0.183 ** Significant at a = 0.05 level 122 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. As shown in the table, the accuracies across the three groups are significantly different in the problemistic search mode. However, the differences for scanning mode are not statistically significant. Another Kendall’s W Test was conducted for movie data. The results are shown in Figure 24 and Table 16. Table 16. Number of Evaluations and CF Performance (Movies) Groups (Range of Number of Evaluations) Rank-based Accuracy Kendall’s W Movies Groupl (12 -14) 0.101 0.063 ( p = 0.779 ) Group2 (15 - 21) 0.108 Group3 (23 - 46) 0.102 Number of Evaluations & Accuracy (Movie) m < D 90 £ -S 85 | a 8 0 o 2 75 70 DC ( 5 npn —— L±_ -4 — -- 0.15 ^ 0.1 « 0.05 § < 0 G roupl G roup2G roup3 (12-14) (15-21) (23-46) Number of Evaluations d l l Optimal R eference G roup Size — B—A ccuracy Figure 24. Number of Evaluations and CF Performance (Movies) The accuracy of movies has a similar pattern - the accuracy is highest for the group with a medium number of evaluations - as with the research papers. The differences, however, are not statistically significant. 123 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The accuracy of CF is highest in the group with a moderate number of evaluations. The difference is more outstanding in the problemistic search mode than in the scanning mode and movies. One possible reason is that too many evaluations may result in a less acute identification of reference groups. If users evaluate too many research papers, the evaluations may span across too many areas, which may inhibit identifying the precise problem domain. That is probably why group 3 in problemistic search mode has a clearly lower accuracy than group 2. In problemistic search mode, users need recommendations in a very specific domain. If they evaluate too many papers, the domain for which they want to get recommendations may become less evident, resulting in relatively inaccurate recommendations. The result implies that for CF systems for specific recommendations more evaluations do not necessarily lead to a better result. For those systems, perhaps, selecting and presenting better items to users for evaluation may be more critical than soliciting more evaluations from users. 5.4.5 Pattern of users’ preference and the performance of CF systems H5. The performance of a CF system is affected by the patterns of users’ preference clusters. H5a. The K-S D statistics that measure the pattern of user preferences significantly vary across domains. H5b. The performance of CF systems is better in the domain with higher K-S D statistics. m R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. The K-S D statistics in the three domains - movies and two search modes in research papers - were calculated. Table 17 shows the K-S D statistics. The average number of evaluations for a pair of people was 5.48 for movies and 4.16 for research papers. Thus, the ‘no clusters’ distributions used for the K-S tests were, from proposition 3 and 4, for those with 5 and 4 items. Table 17. K-S D Statistics of theThree Domains Domain K-S D Statistic Movie 0.161* Research Paper Scanning 0.185** Problemistic Search 0.189** * Compared with the ‘no clusters’ with 5 items (with 16 intervals) ** Compared with the ‘no clusters’ with 4 items (with 16 intervals) The K-S D statistic for the movies is smaller than for the research papers. This implies that the research papers probably have tighter or fewer clusters than the movies. If this is true, the CF system for the research papers should outperform the CF system for the movies. The differences in K-S D statistics could not be tested statistically because only one observation is available for each domain. The accuracies of CF systems in three domains were calculated as shown in Table 18. The accuracies are the same as in Figure 15. Both rank-based measure and MAD show that the CF system is most accurate in research paper (problemistic search), and least accurate in movie. This order is same as the K-S D statistics of the three measures shown in Table 17. Therefore, it can be inferred that the pattern of 125 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. preference clusters, which is measured using K-S D statistics, affects the accuracy of CF systems. Table 18. Accuracy of CF System for Movies and Research Papers Accuracy Measure Holdout Sample size Movie Research Paper Scanning Problemistic Search Rank-based Measure* (Average Differences in Ranks) 3 0.016 0.076 0.133 4 0.046 0.149 0.239 5 0.079 0.264 0.367 MAD* (Average Differences in MAD) “ 3 0.050 0.457 0.603 4 0.042 0.452 0.605 5 0.038 0.445 0.589 * Significant at a = 0.05 level. ** (Error by average - Error by CF). Thus, a larger number indicates a better performance of CF The differences in rank-based measure and MAD were tested using the Kendall’s W test. The results show that the differences are statistically significant. This implies that the accuracy of CF systems varies across domains and probably this is due to the differences in the pattern of preference clusters discussed in the Section 4.2.5. 5.5. Summary of the result The accuracy, in both types of products (movies and research papers), of a CF system increases as the number of users increases. This implies that CF technology can be used in many different product domains. It is interesting that a CF system performs better for knowledge-intensive products such as research paper, implying that CF has more potential in knowledge-related products. 126 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. Table 19. Summary of Hypotheses Test Hypotheses Test Result HI. The performance of a CF system increases as the number of users increases. a. The performance of a CF system increases as the total number of users increases. ✓ b. The increase in performance adopts one of the hypothetical patterns. c. The pattern of increase will differ depending on the type of product and search mode of users. ✓ H2. Reference group size affects the performance of a CF system a. There is an optimal reference group size under given conditions. Research Papers X Movies b. The relative optimal reference group size decreases as the number of users increases. ( /) c. The optimal reference group size varies depending on the conditions. (O H3. The performance of CF is different when users are in scanning mode from when users are in problemistic search mode. a. With small number of users, the performance of a CF system is better for the users in scanning mode than problemistic search mode. (Scanning mode has less critical mass) (’0 b. The performance of a CF system for problemistic search mode increases faster than for scanning mode as the number of users increases. ✓ H4. The performance of a CF system is affected by the characteristics of users. a. The system with more knowledgeable users performs better. X b. The system with more motivated users performs better. X H5. The performance of a CF system is affected by the patterns of users’ preference clusters. a. The K-S D statistics that measure the pattern of user preferences significantly vary across domains. (O b. The performance of CF systems is better in the domain with higher K-S D statistics. ✓ S -Supported, X- Not Supported, ( / ) - Partial Support (supported, but statistically not significant) Movies have an optimal reference group size for a given set of circumstances. However, research papers do not have a clear optimal reference group size. Two 127 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. different search modes (scanning and problemistic search) show different levels of accuracies - CF was more accurate for users in problemistic search mode than for users in scanning mode. The hypotheses about the relationship between characteristics of users and CF performance were only partially supported. The accuracy of CF systems is better for users with moderate knowledge/experience than it is for novice users or, on the other hand, for users with extensive knowledge. The motivation of users is not highly related to the performance of a CF system. The hypothesis test results are summarized in Table 19. R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. CHAPTER VI. CONCLUSIONS AND FUTURE RESEARCH 6.1. Conclusions This study investigated behavioral aspects of CF systems, which are critical factors of the accuracy of CF systems but have been neglected in previous studies. CF systems for a consumer product domain (movie) and a knowledge-intensive domain (research paper) were investigated. Hypotheses were developed from previous studies on CF and related areas and two experiments were conducted to test the hypotheses. From the analyses of the data it was shown that the accuracy of CF is different for different domains. The accuracy of CF systems increases as the number of users increases. However, the patterns of the accuracy increase differ for different domains. The accuracy of CF for knowledge-intensive domain was much better than it was for consumer product domain. When CF is applied to a knowledge intensive domain, the accuracy of the CF system is also affected by the search mode of users. CF seems to work better when users are looking for specific information than when they are searching for information of general use. System configuration such as reference group size seems to be less important than user- and situation-related factors when the domain is knowledge intensive. Users’ experience and knowledge in the domain appears to be one of the most critical factors 129 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. influencing the accuracy of the system. In this study the accuracy was higher for users with moderate experience and knowledge than for both those with little or those with a lot of experience. 6.2. Implications of the study This study will help researchers and practitioners understand the issues surrounding the application of collaborative filtering systems. The study provides a better understanding on the design and implementation of CF systems. 6.2.1. Implications for researchers Previous studies on CF have focused on technical issues and little research, theoretical and empirical, about the effect of domain and user has been carried out. This study developed theoretical foundations that explain how the accuracy of CF systems varies across domains and how the characteristics of users and their behaviors affect the accuracy of CF systems. The hypotheses developed from the theoretical foundations were also empirically tested. The results show that the knowledge of users and their search mode affect the accuracy of CF systems. It has two important implications in CF research. First, the optimization or improvement in CF algorithms must be done with considerations of ‘user factors.’ Otherwise, it may end up as a sub-optimization. Second, users seem to adopt different evaluation criteria depending on their knowledge in the domain and 130 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. their search mode. People seem to use simpler evaluation criteria when they search for answers to a specific question (problemistic search) than when they browse information (scanning). Knowledgeable users seem to have more complicated evaluation dimensions compared to novice users. This shows an important research direction in CF. Theoretical and empirical research needs to be done on how different the evaluation criteria across different user groups and different search modes are. The research will provide valuable knowledge for designing CF systems and also other information systems such as search engines. The results also show that the type of product domain affects the accuracy of CF systems. This implies that users’ evaluations criteria vary across domains. Therefore, researchers in CF systems need to consider the effects of domain when conducting research on CF. For example, when a new CF algorithm is developed, it needs to be targeted at and validated in a specific domain. Adapting the theories in the ‘mental model’ and ‘cognitive map’ areas, this study provides theoretical explanations of how CF helps users find relevant information and acquire knowledge in a domain. This theoretical framework also provides explanations of why the accuracy of CF systems differs for different search modes and different levels of knowledge of users. These theories imply that the difference in the mental model and cognitive map may be the cause of the different accuracies of CF systems across different domains and user groups. This study showed the need for a critical mass of users of a CF system. The critical mass differs for different domains and different search modes. It was also shown in 131 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. this study that a pattern emerges in the increase of CF accuracy as the number of users increases. This study showed that the pattern differs depending on the product domain. This result implies that the effectiveness of a system may vary across different domains. It also implies that the number of users and the pattern of accuracy increase need to be considered in CF research when algorithms are compared. This study developed a new measure for the accuracy of CF systems. A ‘rank- based’ accuracy measure was developed. This measure has advantages over the traditional accuracy measures such as MAD or MSE because it is domain- independent, scale-independent, and average and variance independent. Since CF systems are applied to various domains, a robust measure that is applicable across different domains is crucial for CF research. This study showed that the pattern of preference clusters can be measured and the accuracy of a CF system is affected by the pattern of preference clusters. This study also calls for further development of measures for preference clusters. The measures will help understand the preference cluster patterns and predict the effectiveness of CF systems. A simulation tool for analyses of CF systems was developed in this study. It is flexible enough to be applied to various types of evaluation data, similarity indices, and simulation settings. This software reads preference data in various database formats and runs simulations, according to the given simulation settings, to calculate the accuracy of CF systems. The similarity index and recommendation generation method can be easily alternated by substituting the corresponding modules, which 132 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. have minimal connections to the main module. This tool will be helpful for further CF research. 6.2.2. Implications for practitioners CF has high potential for providing customized recommendations for e-business and knowledge discovery. However, there exist significant barriers to implementing successful CF systems. This study points out several areas that practitioners need to be aware of to implement successful CF systems. Implementers need to watch for the critical mass effect. For consumer products, the critical mass is larger than it is for knowledge-intensive products. In the initial stage, then, system managers need to invest in seeding information so as to reach the critical mass. The tools and methods used in this study can also be used for identifying critical mass. Since recommendation accuracy varies depending on the experience and knowledge of users, system managers need to collect information about the experience and knowledge of users and adjust recommendation generation methods to fit the level of experience and knowledge of the users. System managers need to design the CF system so that it collects data on users’ purpose of use - whether they are just scanning or they are searching for answers to a specific question - and change the recommendation method accordingly. Since users may have multiple search modes in a domain, system designers may have to consider 133 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. a new design for the database that allows users to keep multiple evaluations for same items. This study showed that the accuracy of CF systems varies substantially across domains. CF seems to be more promising in knowledge-intensive domains such as documents. Practitioners should analyze the characteristics of the domain the CF technology is applied to. The relationship between preference cluster measures and the accuracy of CF systems shows that the accuracy of CF systems can be estimated, before the system is actually implemented, by measuring the preference cluster patterns of a domain. Finally, this study shows that CF technology has the potential to be an effective tool for community development and training. CF can be used to identify people with similar preferences, which can lead to success in community development. CF works well for knowledge repositories as well as for consumer products. Since CF helps a user form his/her own ‘mental model’ or ‘concept space,’ it can be used for training. For example, CF systems can be used to suggest a customized list of ‘reading list’ for a user based on his/her interests. 6.3. Limitations of the study This study has some limitations due to the sample sizes and research methods. First, only two different types of products - movies and research papers - were examined in the study. Although the two products are good representatives of two 134 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. different categories of products, it is desirable for generalizable results to investigate more products. Second, the sample size is a limitation. Large-scale experiments or large data sets from actual CF systems would help understand the issues with large numbers of users. Third, the evaluation scale of this research was the Likert-scale (1 tol3 for movies and 0 to 6 for research papers). Although the Likert-scale evaluation is most common in CF systems, several alternative methods exist. For example, if users are not willing to spend additional efforts to enter evaluations, the system will have to capture indirect evaluations. Several indirect measures such as purchase, page visits, and duration of visit can be used instead of direct evaluations. This study used only direct evaluations and did not investigate the other types of measures. Finally, this study focused on empirical analyses and did not attempt to develop mathematical models of the determinants of the accuracy of CF systems. More analytical/mathematical research will complement the results of this study. 6.4. Future research directions Extending this research would contribute to better understanding the issues around CF systems. I suggest three possible directions to extend the studies on CF systems. First, research on various types of evaluations - binary data, indirect evaluations (time spent for each item), etc. - is needed. In the Internet environment, users’ surfing behavior is represented as ‘clickstream’ (series of mouse clicks) [Chatterjee et al., 135 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 1998]. Clickstream reflects users’ evaluations on the web sites. If the information on the web page is of value to the user, he/she will drill down further by clicking on the links on the web page. Otherwise, he/she will exit from the web page. The clickstream data are binary (visit/no visit). Binary data pose more challenges in recommendation generations because ordinary similarity indices such as correlation are not applicable. Another type of evaluation is the ‘time spent for each item.’ Time spent for each item is a good indirect measure for user preferences [Konstan et al. 1997], Clickstream and time have a benefit over direct evaluations because users do not have to enter their evaluations. More research about better ways to utilize these types of evaluation data is needed. Second, more research is needed to investigate the users’ usage behavior of CF systems. It is interesting to study how users accept CF systems and how recommendations made by a CF system affect the users’ behavior. The acceptance of CF may vary across users and product domains. Users may prefer making purchase decisions for themselves if the items are very important to them (for example, automobiles and houses) rather than getting recommendations from the system. Therefore, research on how users accept CF systems under different circumstances will extend our knowledge about CF systems and provide some guidelines for applications. Third, there is potential for research in developing mathematical models that explain the relationship between characteristics of products and CF accuracy. As alluded to in an earlier study [Miller et al., 1997], different products may have 136 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. different evaluation distributions, and this may lead to different evaluation accuracy levels. Developing mathematical models about the general relationships between the distribution of evaluations and CF accuracy is not easy, even with substantially simplified assumptions. However, mathematical models, even simplified models, would help us better understand CF systems and apply them in practice. As information systems become more powerful and the communication using IT becomes more pervasive, people will use computers more and more to interact with other people. People will use computers to collaborate, make decisions, and integrate their knowledge. Therefore, it is valuable to explore potential systems that may be a new way of interacting with others. CF is a technology that has great potential to help people use computers to systematically leverage others’ knowledge and experiences. Companies can use CF not only to generate recommendations for their customers but also to systematically leverage knowledge of their employees. It is crucial to investigate a better way of applying CF technology in various environments. How the performance of CF technology varies under different circumstances, especially user’s behavioral perspective was examined. This study provides many implications for designing and applying CF in practice. The study also explored a theoretical framework to understand how CF can help knowledge reuse and how performance of CF varies depending on various situations. This will be a basis for further theoretical development in user’s behavioral aspects of CF. 137 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. REFERENCES I. Books and articles 1. Adam, Nabil R.; Gangopadhyay, Aryya; Clifford, James, “A form-based approach to natural language query processing,” Journal of Management Information Systems. Fall 1994, 11:2. 2. Aguilar, F. J., Scanning the business environment. MacMillan, New York, 1967. 3. Ahituv, Niv; Igbaria, Magid; and Sella, Aviem, “The effects of time pressure and completeness of information on decision making,” Journal of Management Information Systems. Fall 1998, 15(2), pp. 153-172. 4. Ansari, Asim; Economides, Nicholas; Ghosh, Avijit, “Competitive positioning in markets with nonuniform preference.” Marketing Science. 13:3, Summer 1994. 5. Ansari, Asim; Essengaier, Skander; and Kohli, Rajeev, “Internet Recommender System,” Journal of Marketing Research, August 2000. 6. Arabie, Phipps; Corroll, J. Douglas; DeSarbo, Wayne; and Wind, Jerry, “Overlapping clustering: A new method for product positioning,” Journal of Marketing Research. 18:3, August 1981. 7. Armstrong, Arthur; and Hagel, John III, ‘The real value of on-line communities,” Harvard Business Review. May-June 1996, pp. 134-141. 8. Ausubel, D., “Learning as constructing meaning,” New Directions in Educational Psychology. N. Entwhistle (ed.), Falmer Press, London, 1985, pp. 71-82. 9. Avery, Christopher; and Zeckhauser, Richard, “Recommender systems for evaluating computer messages,” Communications of the ACM. March 1997, 14(3), pp. 88-89. 10. Balabanovic, Marko; and Shoham, Yoav, “Fab: Content-based, collaborative recommendation,” Communications of the ACM. 40:3, March 1997, pp. 66-72. 11. Baldonado, Michelle Q. Wang, An interactive structure-mediated approach to exploring information in a heterogeneous, distributed environment, Ph.D. Dissertation. Stanford University, 1997. 12. Bass, Frank M., “A new product growth model for consumer durables,” Management Science. January 1968, pp. 215-27. 13. Belkin, N.; and Croft, B. W., “Information filtering and information retrieval: Two sides of the same coin?” Communications of the ACM. 35:12, 1992, pp. 29- 38. 138 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 14. Benbasat, Izak; and Todd, Peter, “The effect of decision support and task contingencies on model formulation: A cognitive perspective,” Decision Support Systems. Vol. 17, 1996. 15. Breese, John S.; Heckerman, David; Kadie, Carl, “Empirical analysis of predictive algorithms for collaborative filtering,” Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, August, 1998. 16. Carter, Kathy; Cushing, Katherine; Sabers, Donna; Stein, Pamela; and Berliner, David, “Expert-novice differences in perceiving and processing visual classroom information,” Journal of Teacher Education. May-June, 1988. 17. Chatterjee, Patrali; Hoffman, Donna L.; Novak, Thomas P., “Modeling the clickstream: Implications for web-based advertising efforts,” Working paper series, Rutgers University, 1998. 18. Chaturvedi, Anil; Carroll, J. Douglas; Green, Paul E.; and Rotondo, John A., “ A feature-based approach to market segmentation via overlapping K-centroids clustering,” Journal of Marketing Research. August 1997, pp. 370-377. 19. Chen, Hsinchun; and Kim, Jinwoo, “GANNET: A machine learning approach to document retrieval,” Journal of Management Information Systems. Winter 1994/1995, 11:3. 20. Choudhury, Vivek; and Sampler, Jeffrey L., “Information specificity and environmental scanning: An economic perspective,” MIS Quarterly. March 1997. 21. Cohen, Wesley M. and Levithal, Daniel A., “Absorptive capacity: A new perspective on learning and innovation,” Administrative Science Quarterly. 1990, 35, 1 . 22. Conover, W. J., Practical Nonparametric Statistics. Wiley, New York, 1980. 23. Daniels, Kevin; de Chernatony, Leslie; Johnson, Gerry, “Validating a method for mapping managers' mental models of competitive industry structures,” Human Relations. September 1995, 48:9, pp. 975-991. 24. Davenport, Thomas H.; Jarvenpaa, Sirkka L.; Beers, Michael C , “Improving knowledge work processes,” Sloan Management Review. Summer 1996. 25. DeLone, W. H. and McLean, E. R., “Information systems success: The quest for the dependent variable,” Information Systems Research. 3:1, 1992. 26. Dragan, Richard V., “Firefly Network Inc.: Firefly Passport Office,” PC Magazine. September 9, 1997. 27. Dutton, Jane E.; and Jackson, Susan E., “Categorizing strategic issues: Links to organizational action.” Academy of Management Review. 1987, 12(1). 139 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 28. Eddon, G.; Eddon, H.: Inside Distributed COM. Redmond (Microsoft Press), 1998. 29. El Sawy, Omar A., “Personal information systems for strategic scanning in turbulent environments: Can the CEO go on-line?,” MIS Quarterly. March 1985. 30. El Sawy, Omar A.; and Bowles, Gene, “Redesigning the customer support process for the electronic economy: Insights from Storage Dimensions,” MIS Quarterly. December 1997, pp. 457-483. 31. Englis, Basil G.; and Solomon, Michael R., “To be and not to be: Lifestyle imagery, reference groups, and the clustering of America,” Journal of Advertising. 24:1, Spring 1995. 32. Essex, Patricia A.; Magal, Simha R.; and Masteller, Douglas E., “Determinants of information center success,” Journal of Management Information Systems. Fall 1998, pp.95-117. 33. Evans, Philip; and Wurster, Thomas S., “Getting real about virtual commerce,” Harvard Business Review. November-December 1999. 34. Freund, Yoav; Iyer, Raj; Schapire, Robert E.; and Singer, Yoram, “An efficient boosting algorithm for combining preferences,” Proceedings of International Conference on Machine Learning. 1998. 35. Goldberg, David; Nicolas, David; Oki, Brian M.; Terry, Douglas, “Using collaborative filtering to weave an information tapestry,” Communications of the ACM. December 1992, 35(12), pp. 61-70. 36. Good, Nathaniel; Schafer, Ben; Konstan, Joseph A.; Borchers, Al; Sarwar, Badrul; Herlocker, Jon; and Riedl, John, “Combining collaborative filtering with personal agents for better recommendations,’’ Proceedings of the 1999 Conference of the American Association of Artificial Intelligence. 1999. 37. Grant, Robert M., “Toward a knowledge-based theory of the firm,” Strategic Management Journal. Winter 1996, 17. 38. Green, Paul E.; Carmone, Frank J.; and Wachspress, David P., “Consumer segmentation via latent class analysis,” Journal of Consumer Research, 3:3, Dec. 1976. 39. Greening, Dan R., “Building consumer trust with accurate product recommendations - A white paper on LikeMinds Personalization Server,” http://www.andromeida.com/products/likeminds. Andromedia Inc. 40. Griffith, Terri L.; and Northcraft, Gregory B., “Cognitive elements in the implementation of new technology: Can less information provide more benefits?” MIS Quarterly. March 1996, pp. 99-110. 140 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 41. Griffiths, William E.; Hill, R. Carter; and Judge, George G., Learning and Practicing Econometrics. John Wiley & Sons, Inc., 1993. 42. Harmon, Eric, Delphi COM Programming. Macmillan Technical Publishing, 2000. 43. Hars, Alexander; El Sawy, Omar A.; Gosain, Sanjay; Hirt, Sabine; Im, II; Kang, David; Lee, Zoonky; Raven, Arjan, “Reengineering IS Research and Its Intellectual Infrastructure for the Electronic Economy”, Journal of Organizational Computing & Electronic Commerce. June 2000, 10:2, pp. 67-83. 44. Hars, Alexaner, “USC Cybrarium: An infrastructure for the creation and management of information systems knowledge,” In: Baets, W.R.J. (ed.): Proceedings 6th European conference on information systems. Aix-en-provence, 1998, pp. 75-85. 45. Heckerman, David; Chickering, David Maxwell; Meek, Christopher; Rounthwaite, Robert; and Kadie, Carl, “Dependency networks for inference, collaborative filtering, and data visualization,” Journal of Machine Learning Research. October 2000, pp. 49-75. 46. Herlocker, Jonathan L.; Konstan, Joseph A.; Borchers, Al; Riedl, John, “An Algorithmic Framework for Performing Collaborative Filtering,” Proceedings of the 1999 Conference on Research and Development in Information Retrieval. August, 1999. 47. Hibbard, Justin, Straight Line to Relevant Data,” Information Week. Nov. 17, 1997. 48. Hildreth, Charles R., ‘The use and understanding of keyword searching in a university online catalog,” Information Technology and Libraries. June 1997, pp. 52-62. 49. Hill, W.; Stead, L.; Rosenstein, M., and Furnas, G., “Recommending and evaluating choices in a virtual community of use,” Proceedings of ACM CHI’95, 1995, pp. 194-201. 50. Hoffman, Donna L.; Kalsbeek, Wiliam D.; and Novak, Thomas P., “Internet and Web use in the U.S.,” Communications of the ACM. 39:12, December 1996, pp. 36-46. 51. Hoover, Stewart V.; and Perry, Ronald F., Simulation - A Problem-Solving Approach. Addison-Wesley, 1990. 52. Huber, G. P., “Organizational learning: The literatures,” Organization Science. 2:1, 1991, PP- 88-115. 53. Iansiti, Marco, “Real-world R&D: Jumping the product generation gap,” Harvard Business Review. May-June 1993. 141 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 54. Ireland, R. Duane; Hitt, Michael; Bettis, Richard A.; and de Porras, Deborah Auld, “Strategy formulation processes: Differences in perceptions of strength and weaknesses indicators and environmental uncertainty by managerial level,” Strategic Management Journal. 1987, Vol. 8, pp.469-485. 55. Jobson, J. D., Applied Multivariate Data Analysis. Springer-Verlag, New York, 1992. 56. Johnson, Eric J., “Expertise and decision under uncertainty: Performance and process,” The Nature of Expertise, Michelene T.H., Chi, Robert Glaser, and M. J. Farr. Eds., London, 1988. 57. Kautz, Henry; Selman, Bart; and Shah, Mehul, “Referral web: Combining social networks and collaborative filtering - An interactive system for restructuring, visualizing, and searching social networks on the web,” Communications of the ACM, March 1997, Vol. 40 No. 3, pp.63-65. 58. Khamis, H. J., “The two-stage 8-corrected Kolmogorov-Smirnov test,” Journal of Applied Statistics, 27:4, 2000, pp. 439-450. 59. Konstan, Joseph A; Miller, Bradley N; Maltz, David; Herlocker, Jonathan L; et al, “GroupLens: Applying collaborative filtering to Usenet news,” Communications of the ACM. Mar 1997, 40(3), pp. 77-87. 60. Kramer, Joseph; Noronha, Sunil; and Vergo, John, “A user-centered design approach to personalization,” Communications of the ACM. 43:8, August 2000, pp. 44-48 61. Krieger, Abba M.; and Green Paul E., “Modifying cluster-based segments to enhance agreement with an exogenous response variable,” Journal of Marketing Research. August 1996, pp. 351-363. 62. Kuhlthau, Carol Collier, “Accommodating the user's information search process: Challenges for information retrieval system designers,” Bulletin of the American Society for Information Science. February/March 1999. 63. Leitheiser, Robert L.; and March, Salvatore T., “The influence of database structure representation on database system learning and use,” Journal of Management Information Systems. Spring 1996, 12(4). 64. Leonard-Barton, Dorothy, “Core capabilities and core rigidities: A paradox in managing new product development,” Strategic Management Journal. 1992, Vol. 13, pp. 111-125. 65. MacCrimmon, Kenneth R.; and Wagner, Christian, “Stimulating ideas through creativity software,” Management Science. November 1994, 40 (11). 142 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 66. Malone, Thomas W.; Grant, Kenneth R.; Turbak, Franklyn A.; Brobst, Stephen A.; and Cohen, Michael D., “Intelligent information-sharing systems,” Communications of the ACM. May 1987, 30(5), pp. 390-402. 67. Mansfield, Edwin, “Technical change and the rate of imitation,” Econometrica, October 1961, pp. 741-766. 68. McClave, James T.; and Benson, P. Geroge, Statistics for Business and Economics. Dellen, 1988. 69. Miller, B., Riedl, J., and Konstan, J., “Experiences with GroupLens: Making Usenet useful again.” Proceedings of the 1997 Usenix Winter Technical Conference. Jan 1997. 70. Montazemi, A.R.; and Conrath, D.W., ‘The use of cognitive mapping for information requirements analysis,” MIS Quarterly. March 1986. 71. Mookerjee, Vijay S.; and Mannino, Michael V., “Redesigning case retrieval to reduce information acquisition costs,” Information Systems Research. 8(1), March 1997, pp. 51-68. 72. Newman, Michael; and Sabherwal, Rajiv, “Determinants of commitment to information systems development: A longitudinal investigation,” MIS Quarterly. March 1996, pp. 23-54. 73. Nonaka, Ikujiro and Takeuchi, Hirotaka, The Knowledge Creating Company: How Japanese Companies Create the Dynamics of Innovation. 1995, Oxford University Press, New York. 74. Podberezniak, Elizabeth, Collaborative filtering in TV recommender, Master’s Thesis. New Jersey Institute of Technology, May 1998. 75. Prahalad, C. K.; Ramaswamy, Venkatram; Krishnan, M. S., “Consumer centricity,” Informationweek. April 10, 2000. 76. Pryor, Michael H., “The effect of singular value decomposition on collaborative filtering,” Computer Science Technical Report. Dartmouth College, June 1998. 77. Punj, Girish; and Stewart, David W., “Cluster analysis in marketing research: Review and suggestions for application,” Journal of Marketing Research. 20:2, May 1983. 78. Rainer Jr., R. Kelly; and Watson, Hugh J., “The keys to executive information system success,” Journal of Management Information Systems. Fall 1995, 12:2, pp. 83-98. 79. Rozgus, Amara; and Sicaras, Victoria K., “Searching for the perfect way to search,” Research & Development. June 2000, 42:6. 143 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 80. Rucker, James; and Polanco, Marcos J., “Siteseer: Personalized navigation for the Web,” Communications of the ACM. March 1997, 40:3, pp. 73-75. 81. Sachs, Patricia, “Transforming work: collaboration, learning, and design,” Communications of the ACM. September 1995, 38, 9. 82. Salton, Gerard; Allan, James; Buckley, Chris, “Automatic structuring and retrieval of large text files,” Communications of the ACM. New York, Feb 1994, 37(2), pp. 97-108. 83. Sampler, Jeffrey L. and Short, James E., “An examination of information technology’s impact on the value of information and expertise: Implications for organizational change,” Journal of Management Information Systems. Fall 1994, 11, 2. 84. Santhanam, Radhika; Sein, Maung K., “Improving end-user proficiency: Effects of conceptual training and nature of interaction,” Information Systems Research. December 1994. 85. Sarwar, B. M., Konstan, J. A.; Borchers, A.; Herlocker, J. L.; Miller, B. N.; and Riedl, J., “Using filtering agents to improve prediction quality in the Grouplens research collaborative filtering system,” Proceedings of CSCW ’98. 1998. 86. Saunders, J. A., “Cluster analysis for market segmentation,” European Journal of Marketing. 14:7, 1980. 87. Schafer, J. Ben; Konstan, Joseph; and Riedl, John, “Recommender systems in E- commerce,” Proceedings of the ACM Conference on Electronic Commerce, Nov. 1999. 88. Schenk, K. D.; Vitalari, Nicholas P.; and Davis, Shannon, “Differences between novice and expert systems analysts: What do we know and what do we do?” Journal of Management Information Systems. Summer 1998, 15:1, pp. 9-50. 89. Schonfeld, Erick, “The customized, digitized, have-it-your-way economy,” Fortune. September 28, 1998. 90. Shardanand, U.; and Maes, P., “Social information filtering: Algorithms for automating “word of mouth”,” Proceedings of ACM CHI’95. 1995. 91. Shanteau, James, “Psychological characteristics and strategies of expert decision makers,” Acta Psvchologica. 1988, pp. 203-215. 92. Shardanand, Upendra; and Maes, Pattie, “Social Information Filtering: Algorithms for Automating ‘Word of Mouth’,” Proceedings of CHI f Conference on Human Factors in Computing Systems). 1995. 93. Shibata, Yoshitaka; Fukuda, Manabu; and Tsumoto, Michiaki, “A hyper-media- based design image database system using a perceptional link method,” Journal of Management Information Systems, Winter 1996-1997, 13(3), pp.25-43. 144 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 94. Spence, Mark T.; and Bracks, Merrie, “The moderating effects of problem characteristics on experts’ and novices’ judgments, Journal of Marketing Research. May 1997, pp. 233-247. 95. Star, Susan Leigh; and Ruhleder, Karen, “Steps toward an ecology of infrastructure: Design and access for large information spaces,” Information Systems Research. 7(1), March 1996. 96. Storey, Veda C.; Straub, Detmar W.; Stewart, Kathy A.; and Welke, Richard J., “A conceptual investigation of the e-commerce industry,” Communications of the ACM. 43:7, July 2000, pp. 117-123. 97. Straub, Detmar W., “Validating instruments in MIS research,” MIS Quarterly. June 1989, pp. 147-169. 98. Sullivan, Jeremiah; and Nonaka, Ikujiro, “Culture and strategic issue categorization theory,” MIR. 1988, 28(3), pp.6-10. 99. Terveen, Loren; Hill, Will; Amento, Brian; McDonald, David; and Creter, Josh, “PHOAKS: A system for sharing recommendations,” Communications of the ACM. March 1997, 14(3), pp. 59-62. 100.Ungar, Lyle H.; and Foster, Dean P., “A formal statistical approach to collaborative filtering,” Proceedings of the Conference on Automated Learning and Discovery (CONALD). 1998. 101.Ungar, Lyle H.; and Foster, Dean P., “Clustering methods for collaborative filtering,” Proceedings of AAAI Workshop on Recommendation Systems. 1998. 102. Vandenbosch, Betty; and Higgins, Christopher A., “Executive support systems and learning: A model and empirical test,” Journal of Management Information Systems. Fall 1995, 12:2. 103. Vandenbosch, Betty; and Huff, Sid L., “Searching and scanning: How executives obtain information from executive information systems,” MIS Quarterly. March 1997. 104. Vessey, Iris; and Galletta, Dennis, “Cognitive fit: An empirical study of information acquisition,” Information Systems Research. March 1991, 2(1). 105. Wang, Shouhong, “Analyzing agents for electronic commerce,” Information Systems management. Winter 1999, 16:1, pp. 40-47. 106. Weber, Thomas E., “A Closer Look: A Guide to the Web,” Wall Street Journal, New York, May 28, 1998. 107. Whitworth, Brian; and Felton, Roy, “Measuring disagreement in groups facing limited choice problems,” Working Paper. New Jersey Institute of Technology, 2000. 145 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 108.Wierenga, Berend; and van Braggen, Gerrit H., “The dependent variable in research into the effects of creativity support systems: Quality and quantity of ideas,” MIS Quarterly. March 1998. 109. Wiley, Deborah Lynne, “Beyond information retrieval,” Database, August/September 1998, pp. 18-22. 110. Wingfield, Nick, “Unraveling the Mysteries Inside Web Shoppers' Minds,” Wall Street Journal, New York, Jun 18, 1998. II. World-wide Web sites 1. Agentware: www.autonomv.com 2. Alexa: www.alexa.com 3. Amazon bookstore: www.amazon.com 4. Barnes and Noble: www.barnesandnoble.com 5. Big Note: www.bignote.com 6. CDNow: www.cdnow.com 7. CDNow: www.cdnow.com 8. Cinemax: www.cinemax.com 9. E! Online: www.eonline.com 10. Film Finder: www.filmfinder.com 11. Firefly: www.firefly.com 12. IS World: www.isworld.org 13. Launch: www.launch.com 14. Levi’s: www.levis.com 15. Macromedia: www.macromedia.com 16. Movie Finer: www.moviefinder.com 17. Moviecritic: www.moviecritic.com 18. Sierra: www.sierra.com 19. Wise Wire: www.wisewire-corp.com 20. Yahoo!: www.yahoo.com 146 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. APPENDIX A. Screen shots of the system for experiment 1 (Movies) 1. User Registration Screen w Welcome to MofU’Rt-u ummemicr Microsoft Internet f.xploter DKKid&rf l\V MSN Ffe fca( View r'a' otitc? Too1 * Slop Reftesn Home S*a/ch^ t-aronie History* P<mt tcU D m js* Radfiuicfe m s n s e s i s i M i ‘ A ^Jresi l i j M t p //cylxarium-us&edu/'movie.htm ~ j3 . L m U Welcome to MovieRecommender Thank you for participating the survey. Please fifi out the following items. First N am e Last Name E-m ail address Logon I D Password C onfirm password H ow often da you watch movies optional ] ' optional atch movies ■r;r”' ~ (or videos)? IfSoteoiono) ((What is your favorite movie? j (Seleci one) ^ 0 S' (irj i $ IrlaintHl R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 2. Item Evaluation S I A, _j . 4 ' J . J ,> *d<i««E j a ht-p /Vcv.raiii™ use -du/Wc1 SA jiPeMi 1 1 ta i.B -2 3 i*i'er-1 I.sl6 tus--upda!I'l«ist.5 1 !^Miie.UFi:S2 0 ISX2 liEAUTirULS:bacU1 r'S n , f « ii ” .-! Please enter your evaluation for LIFE IS BEAUTIFUL lO H O Ifl • 9 ® <*~Y0UB a « f » ! T M IN G *i*lh uk» rr 9 9 9 UK# It 9 » a T o u tu rto i t a a a n tu i» n < a aiVtN'tatttit a ISSSS 3. Recommendations by the system wm tie £dit Tjy^rtes Tocte Help ,,. r f ^ i r a K s l i y h t t p V / c y b f a n u r m u s a e d u / c r M ^ ^ ^ ^ ^ ' 12l.Il y\ < " < " ' p '“ ” : T h e fo llo w in g m o v ie s a r e t h e m o iv e s y o u will lik e. T h e : r e c o m m e n d a t i o n s a r e b a s e d o n t h e r a t i n g s o f o t h e r p e o p l e f w h o h a y e t a s t e s sim ila r t o y o u r s . . . . > 1 a S H A L L W E D A N C E ? ( 1 9 9 7 1 r - V ;j a M A D H O U SE (1 9 9 0 1 » W E L C O M E T O S A R A J E V O • K IS S TH E G IR LS . \':."?f; * W E S T S ID E S T O R Y (1 9 6 3 .1 V j • BIRD ON a w i r e > 5 1 a HEARTBREAK RIDGE ; > | • f u l l m e t a l j a c k e t • .G O D S AND M O N S T E R S ... :> 5j a G O N E W IT H T H E W IM P 1 1 9 3 9 ) ^ I n t i s f r t i R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. B. Screen shots of the system for experiment 2 (Research Papers) 1. Initial Screen (Informed Consent Form) ^ ' X 3 *2 /A -J J ’ Jr J A A*o-* * jitj] (■|ty//cj,iidr ' Jrruxedii/c>ibdlL'*- (.tei-ll*'m; pj5Jvv~c»--0R<*Cf J jJ i> fi * . ... ‘ ? ...a p ro to ty p e in frastructure fo rtfie c re a tio n , sy n th e sis a n d a p p lic a tio n o f re s e a r c h k n o w led g e. S' Collaborative filtering experiment Thank you for participating ! « This experiment is apart of a research project about collaborative filtering system. If you would like to seethe abstract of the working paper, please click here. • The entire experiment should not take more than about 10 minutes of your time. • As a reward.for participation, you will get recommended papers for your research * At the end, you wiU.be asked to fill out a brief questionnaire. s Any information obtained m connection with this study and that can be identified with you will remain confidential as required by law, All analysis and reporting will be done anonymously and in aggregation. * Youmay stop, without anypenaky, at any point of the survey ifyouso desire. * If you would H ke to participate in this experiment, pleasechck on "START’ and proceed. Clicking on the "START'* Enk will constitute consent to participate in this research project. START • If you have any questions or comments about this study, please contact one of the following investigators, • . H 3m Alexander Hars ; ■ ■ ■ ■ IOM Department IOM Department j | * V R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 2. Integrated Search Engine & » j & t f y * ^ to a w r l < y * t ti - h x * Q | & .j ^ a jk ’ : i, j Utah ’i S tr- B e)t«h •Seateh (-BWfl'es' M<i Piirt E d' Ojm'-.to W ^//c>toaH^u«.ecW c^dlV?*i5wfofni=CF_Ba5cSeadi J'j f" C > > : Search fo r papers Please use the search, engine below to get a list of papers to evaluate. You can search by Title, Author, Keywords, or Journal. Search words: j " " .... in jTttle 3 . : ~ "^3{ © p f c j O I © } ^ "3 ( o p t i o n a l ) • The search is an ’ exact search’ - ft searches for papers that contain the exact phrases. (Case insensitive) • The search will find papers from recent issues of 5 IS journals. (CACM, ISR. JMIS. Management Science, and M 1SQ). M to + li'& fr-A 'J 4 1 > ' cm 1 re * * * * * " -* !gny*n»!.; | J S 4 B v K S B « p » " 3. Search Result m m m m M vtjO l Hf,j^CJhTar-muit.edu/Cv4iCV?ScdchNiiwrBaCafuJOivNol fiboarch2-lLi{LgotvNo2-’ IWounTo 3 S earch resu lts • Please click on the title of the paper that you would like to evaluate • You don't have to evaluate ail the papers in the list. Please evaluate papers that you know or would like to evaluate after reading the abstract. :i • Click on New Search if you would like to search another set of papers. Click this button to get recommendations » §. Risks of content filtering Peter G. Nmun^n - l-anrcn Wewstain. CACM. Corrnnunicationg of tha ACM. 11-1999. HCmnpT ,ens: Applying coUaborative filtering to Usenet news JossKh A K onsfta, Bradley N Miller, David Mate, Jonathan L Heilocker. CACM. Communications of tha ACM, 03-1997. ^ Referral Web: Combining social networks and collaborative filtering Henry AKauts, Bart Sefrnati, Mehtii Shah. CACM. Communications of tha ACM. 03-1997. H Information Filtering and Information Retrieval: Two Sides Nicholas J. Belkin. W. Bruce Croft. CACM Coirnramications of the ACM 12-1992. New Search a #Ve,ns* m i ^ ’ * '5 2 f " M R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 4. Item Evaluation ••i'’ J p liv i Apjilvif 'i i oil. t'lff.itiv •’ •iiK-nnit to I! i ft T i f f g f e * J t f o t * f c c f c » IdU*. jib] http/Vc/waium. irtc.edu/cj4idll/show?chi8ct«1030313 i Tart 0 3/1/97 GroupLens^ Applying collaborative filtering to Usenet news Joseph A K onstan. Bradley N Miller, David M altz. Jonathan L Herlocker The GroupLens project designed, implemented, and evaluated a collaborative filtering system for Usenet news, a high-volume, high- turnover discussion list service on the Internet. The GroupLens project was started in 1992, and a pilot study at 2 sites was conducted to determine the feasibility of using collaborative filtering for Usenet news. Usage data gathered during the 7-week public trial shows that predictions are meaningful and valuable to users. GroupLens shows promise as a viable service for all Usenet news users. A second public trial is now underway which will test the effect of providing predictions to new users more rapidly by providing overall averages until the user has rated enough articles to correlate, and it will make full use of time spent reading measures to capture implicit ratings unobtrusively. GroupLens: Applying collaborative filtering to Usenet news, Konstan, Joseph A : Miller, Bradl^r N : Maltz, David ; Herlocker, Jonathan L ; et al, Communications of the ACM, 40:3, Mar 1997 Relevance for your general 1 Relevance for your specific 1 research areas I research project 1 o o C 1 0 2 0 9 0 4 0 5 o6 |ro pi r2 C 3 C 4 05 O-neghgjblB l*sa»am 2 * a c c e d e 3-«v*m e 5 * eaceJtest " Mart: on the rating scores and click on the "submit" button to turn in your ratings. r m m sm 1 VMrwl 5. After Initial Evaluations t - f c .. , , _ r —....— Zl-. ^ ■ T hank you ! : V ; You have entered enough information to get recommendations. . 1. You can click "CONTINUE" to evaluate more papers if you'd like, : ■ 2. Click on 'RECOMMENDATIONS' to see the recommended papers. Remember that the recommendations will be more accurate if you evaluate more papers, ■ ' „ ..... w,. „ J f i . _zl \.y- i t , M H u , \[z ,h < m BW, c . 'i |b u t w i r *. } f tw j [ijifMcitrfril Oft.. j * S J'K t \ R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. 6. Recommendations from the System wmMn mmmg n mmm & E o * S fftK t * j?; f-JV K xfr* H ie fffp < ; J h M p ://c y b r« iu m .U ! Papers Recommended for you 1. F or general IS research • ^ P rivacy, info rm atio n technology, an d health c a re Thomas C Bm'dfi'eisch, CACM. Commurdcatione of the ACM 1997. • ^8 T e n m y th s o f m ultim odal in tera ctio n Sharon Oviatt CACM, Comrnucications of the ACM. 1999. • & A ctualizing organizational m em o ry w ith info rm atio n sy ste m s Brie W . Stan, Vladimir Zv/ass, E3R , Information Systems Research. 1995. « 81 L e s s o n s fro m th re e y e a rs o f W eb developm ent Alan R D em is. CACM, Communications of the ACM. 1999. • ^ E sse n tia l b u sin e ss o b je c t design David OiiSbraad. CACM, Communications of the ACM. 2900. 2. For your research project • D e stin atio n inform ation sy stem s- D esig n is s u e s and d irec tio n s Hone^MeiChen. Paidine j Sheldon. JM 1S, Journal of Manasonaent Information Systems. 1997. • ^ A chieving b o tto m -lin e im p ro v em en ts w ith E n te rp rise F ram ew o rk s David S lla im , MohamedBFavad CACM, Comtrtunicati&na of the~ACM.~1993. a 1 T h e e ffe c ts o f tim e p re s s u re and co m p lete n ess of info rm atio n on decision m aking Ntv Ahituv, Masid Isbaiia. Aviem Sella. JM 1S. Journal of Management MormaUon Systems . 1990. • H A th e o ry o f decision su p p o rt s y s te m design f o r u s e r calibration Georse M. Kasper. ISR, Information. Systems Research, 1996. £ jb « * ItiKfiBiiliMgll tt C. Questionnaire after the experiment (Research Papers) LUBES Me l_d* ■ 'rav i«JU Holp ^ " * ■ r * . ~ . j j ,d s ' H -s j- J . V - J ol . J 9 » S Off ReTttsn Srtcirh* f ^ i b s lit Mart- f-nn* tcf* l>m *» Afrir?« jf t] hi.tp://cifcranum.u$c.edu/c^dV?showfcrnTi=Cl _t' B f ”3 < r 5 E r T h a n k y o u ! Please answer the following questions. All information collected here will be analyzed anonymously and used only for research purposes. f C heck h ere if y ou would tike to g e t an u p d a te d list o f recom m endations by e-mail. How long h av e you w orked in IS a r e a .. (o r re la te d a re a ) a fte r y our m a ste rs' j . years d egree? How fong h av e you w orked in th e ............. ............. r e s e a r c h p r o je c t for w hich you j y e a rs j m onths e v a lu a te d th e p ap ers? W hat is your acad em ic position? } Select one 3 How w ere th e recom m ended p ap ers in For Your G e n e ra l R e s e a rc h term s o f th e following criteria? Topics <*• Excellent Poor -> Providing now p e rsp e c tiv e s f i f j f j f i f s •"« f i For Your S p e c ific R e s e a rc h P roject Excellent Poor •-> r , r j r 3 r* cs c6 c 7 C i < * 3 C 3 Fj r, r 7 W as th e sy stem e a sy to use? <—E&cy D ifficult--> r i C j f j r< c5 r« r? If you have any other comments, please type in the box below. 888 iK S ® 3 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. D. The soliciting e-mail message sent to the subject pool From: II Ira To: **** Subject: Recommendations for research papers Dear **** I would like to cordially invite you to participate in an experiment that tests strategies for recommending research articles to researchers based on their particular interests. I found your e-mail address in the IS World faculty directory and thought that this experiment may be of interest to you and that you could potentially find it useful for your own research as well. I have already prepared an account in the system for you. You can participate in the experiment by visiting the URL below and follow the instructions. Since your account has already been established, you will be able to log on automatically for the experiment. The entire experiment should not take more than about 10 minutes of your time. http://cybrarium.use.edu/user=ID&password=psswd&CF I am working on a research project that examines various aspects of collaborative filtering technology in knowledge- intensive items such as research papers. As you may already know, the collaborative technology searches for people who have similar interests as you, and generates recommendations for you based on their evaluations or opinions. The system for this experiment was implemented on the top of a server, Cybrarium, which is a collaborative infrastructure for the creation, integration and dissemination of research knowledge in Information Systems. In addition to the information in the Cybrarium, abstracts of the research papers from 5 IS journals (ISR, MISQ, JMIS, CACM, and Management Science) for the past 8-10 years were added for this experiment. You need to select 10 or more papers and enter your evaluations based on the relevancy of the papers to your research. Then, the system will give you a list of recommended papers that match your interests. When you logon to the server next time, please use the following ID and password. You can change your password once you logon to the system. ID = ID Password = psswd Any information obtained in connection with this study and that can be identified with you will remain confidential as required by law. You may stop, without any penalty, at any point of the survey if you so desire. If you have any questions or comments regarding the study, please send an e-mail to one of the following investigators. Thank you for your time. 153 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. II Im Doctoral Candidate Department of Information and Operations management Marshall School of Business University of Southern California E-mail: iim@rcf.usc.edu Phone: (213)740-7283 Alexander Hars Assistant Professor Department of Information and Operations management Marshall School of Business University of Southern California E-mail: hars@bus.usc.edu Phone: (213)740-0186 154 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. E. Screen Shot of the Simulation Program and an Example of Simulation Output 1. Screen Shot of the Simulation Program .TtTTfYllfMi IM S ,, . ■ * '.a - / - Sue 1 4 1 0 0 59 40 0.012 5 1 0.108 \ Simulation Output - r=s“ — 3 X \ / Id • s Holdo • Setting • ^ ^BoldwtQset S et f5 * User iterations JlO P" ATPosrtteUsers M i ^ i ’ * ^ ~ J 1 ji Ite&tafe f r 7 ost of P * '~ I f otRum Jio“ * HojJe | T "* Reference WmwB ^ F r c m pi ~ \ * ' Group Size tm M m m r n m s i 1 0 ,1 0 1 s$*w~~y V r" r„ default Value & ndom Selection f ll-sl'i.ij'll.-CJlji.-.'l".iVii ’ H '. :;7 V '.m ,V.. j-l.i.’ ? . 2. An Example ot Simulation Output 1.116 1.60C 0.196 0.635 0.196 0.635 1.125 Size 1 2 41 0.141 0.1 42 0.211 0. 99 100 RGS 0.078 0.000 1005 , 045 3 4 5 CF-MSE Avg-M£fe CF-MAD Avg-MkD CF-Cov Avg-Cov 0.203 0.137 0.289 1.869 4.1687 1.077 1.613 1 0.192 0.639 0.242 0.087 0.125 1.995 4.05/ 1.098 1.577 \ 0.208 0.649 0.101 -0.047 0.068 2.147 A.309 0.146 0.139 -0.009 2.145|Z3.488 HOSize HISize Ulter liter Siz^'From To StepNRuns M 0 1 5 3 10 20 ) 264 Ml 100 1 J/5 1 CF-MSE, Avg-MSE : MSE of recommendations by CF and ‘average of everybody’ CF-MAD, Avg-MAD : MAD of recommendations by CF and ‘average of everybody’ CF-Cov, Avg-Cov : Coverage of CF and ‘average of everybody’ 155 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. F. Source code of the collaborative filtering engine The collaborative filtering engine developed for this study consists of 5 modules. 1. CFService.pas - Appliation (main) module 2. Main.pas - Module that has a form displaying the status of the program 3. CFCOM.pas - Module that dispatches the requests to the COM interface. 4. CFService_TLB.pas - A Type Library module for COM interface, which was generated by Delphi 5. DataModule.pas - Core module which has objects and functions for collaborative filtering and database access. 1. CFService.pas program CFService; uses Forais, Main in 'Main.pas' {Forml}, CFService_TLB in 'CFService_TLB.pas 1, DataModule in 'DataModule.pas' {DataModulel; TDataModule}, CFCOM in 'CFCOM.pas'; {$R *.TLB} {$R *.RES} begin Application. Initialize, - Application.Title := 'Collaborative Filtering Engine'; Application.CreateForm(TForml, Forml); Application.CreateForm(TDataModulel, DataModulel); Appli cation.Run; end. 2. Main.pas unit Main; interface uses Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs, Db, DbTables, DataModule, StdCtrls, ExtCtrls; type TForml = class(TForm) Labe11: TLabel; edUpdate: TEdi t; edRecomm; TEdit; Label2: TLabel; Label4: TLabel; edLast: TEdit; Timerl: TTimer; chkStop: TCheckBox; edMinutes; TEdit; Label5; TLabel; procedure TimerITimer(Sender: TObject); procedure chkStopClick(Sender: TObject); 156 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. procedure edMinutesChange(Sender: TObject) private { Private declarations } public { Public declarations ) end; var Forml: TForml; implementation {$R *.0FM) procedure TForml.TimerlTimer(Sender: TObject); begin if (chkstop.Checked) then begin inc(TimeCount) ; if (TimeCount > TimeLimit) then // close itself if idle more than the limit Application.Terminate; end; end ; procedure TForml.chkStopClick(Sender: TObject); begin if (chkstop.Checked) then begin edMinutes.Enabled := true; edMinutes.Color : = clWindow; if (edMinutes.Text = '') then TimeLimit : = 0 else TimeLimit : = StrToInt(edMinutes.Text); TimeCount := 0; end else begin edMinutes.Enabled := false; edMinutes.Color := clBtnFace; end; end; procedure TForml.edMinutesChange(Sender: TObject); begin Forml.chkStopClick(Sender); end; end. 3. CFCOM .pas unit CFCOM; interface uses SysOtils, ComObj, ActiveX, CFService_TLB, StdVcl, Forms, DataModule, Main; type TCF = class(TAutoObject, XCF) protected function UpdateEval(UserlD, ItemID, Mode, Evaluation: Integer): Integer; safecall; function GetRecomm(UserID, Mode: Integer): OleVariant; safecall; function Remove(UserlD, Flag: Integer): WordBool; safecall; function CheckN(DserID: Integer; Mode: Integer): Integer; safecall; 157 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. { Protected declarations } end; implementation uses ComServ; function TCF.UpdateEval(UserlD, ItemlD, Mode, Evaluation: Integer): Integer; var i, Flag: integer; begin TimeCount := 0 ; Flag := 0; if ((UserlD < 1000000) and (UserlD > 1000)) then begin Flag := Neutral; UserlD := UserlD - 1000; end; if (Users[Mode].GetUserlndex(UserlD) = 0) then // a new user begin for i : = 1 to NofModes do Users[i].AddUser(UserlD); end; Result := DataModulel.UpdateEvaluation(UserlD, ItemlD, Mode, Evaluation, DataModulel.GetDatal, Flag); inc(NUpdate); Forml.edUpdate.Text := IntToStr(NUpdate); Forml.edLast.Text := IntToStr(UserlD); end; function TCF.GetRecomm(UserID, Mode: Integer): OleVariant; var i, Userlndex: integer; begin TimeCount := 0; Userlndex := Users[Mode].GetUserlndex(UserlD); if ((Userlndex > 0) and (Matrices[Mode].UpdateSI(Userlndex, Mode, Correlations, DataModulel.GetDatal))) then begin Application.ProcessMessages; PaperList.Initialize; if (Estimates.Calculate(Userlndex, Mode, MinRef, MaxRef, RefRatio, MediumF)) then begin i := ArrayLowBound; while ((i <= Estimates.Counter) and (i <= ArrayHighBound)) do begin PaperList.Data[i] := Estimates.ItemlD[i]; inc(i); end; Result := PaperList.ToVariant; end; end; inc(NRecomm); Forml.edRecomm.Text := IntToStr(NRecomm); Forml.edLast.Text := IntToStr(UserlD); [************* i f * * fQj- test Forml.Memol.Lines.Clear; i : = 1 ; while ((i <= 20) and (i <= Estimates.Counter)) do begin str(Estimates.EEvaluation[i]:8:3, temp); Forml.Memol.Lines.Add(IntToStr(Estimates.ItemlD[i]) + #9 + temp); inc(i); end; ****************for test} end; 158 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. function TCF.Remove(UserlD, Flag: Integer): WordBool; begin TimeCount := 0 ; Forml.edLast.Text := IntToStr(UserlD); Result := DataModulel.RemoveUser(UserlD, Flag, DataModulel.GetDatal); end; function TCF.CheckN(UserlD: Integer; Mode: Integer): Integer; begin TimeCount := 0; Result := DataModulel.CheckNumber(UserlD, Mode); end; initialization TAutoObjectFactory.Create(ComServer, TCF, Class_CF, ciMultilnstance, tmApartment); end. 4 . CFService_TLB.pas unit CFService_TLB; /j * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * jj // WARNING / / --------- // The types declared in this file were generated from data read from a // Type Library. If this type library is explicitly or indirectly (via // another type library referring to this type library) re-imported, or the // 'Refresh' command of the Type Library Editor activated while editing the // Type Library, the contents of this file will be regenerated and all // manual modifications will be lost. j j j j II PASTLWTR : $Revision: 1.88 $ // File generated on 5/31/00 1:38:20 PM from Type Library described below. jj * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * jj II NOTE: // Items guarded by $IFDEF_LIVE_SERVER_AT_DESIGN_TIME are used by properties // which return objects that may need to be explicitly created via a function // call prior to any access via the property. These items have been disabled // in order to prevent accidental use from within the object inspector. You // may enable them by defining LIVE_SERVER_AT_DESIGN_TIME or by selectively // removing them from the $IFDEF blocks. However, such items must still be // programmatically created via a method of the appropriate CoClass before // they can be used. jj * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * jj II Type Lib: C:\Imil\D_files\Cf-Main\CFService.tlb (1) // IIDXLCID: {53 6C15B4-099F-11D4-A5F5-0020AFE4AAD6}\0 // Helpfile: // DepndLst: 11 (1) v2.0 stdole, (D:\WINNT\System32\Std01e2.Tlb) // (2) v4.0 StdVCL, (D:\WINNT\System32\STDVCL40.DLL) jj * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * jj {$TYPEDADDRESS OFF} // Unit must be compiled without type-checked pointers, interface uses Windows, ActiveX, Classes, Graphics, OleServer, OleCtrls, StdVCL; II GUIDS declared in the TypeLibrary. Following prefixes are used: // Type Libraries : LIBID_xxxx 11 CoClasses : CLASS_xxxx // DISPInterfaces : DIID_xxxx R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. // Non-DISP interfaces: IID_xxxx j j i t * - * * * * * * * * * * * * ' * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / j const // TypeLibrary Major and minor versions CFServiceMajorVersion = 1; CFServiceMinorVersion = 0; LIBID_CFService: TGUID = '{536C16B4-099F-11D4-A5F5-0020AFE4AAD6}'; IID_ICF: TGUID = '{536C16B5-099F-11D4-A5F5-0020AFE4AAD6}1; CLASS_CF: TGUID = '{536C16B7-099F-11D4-A5F5-0020AFE4AAD6}'; type j j * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * j j // Forward declaration of types defined in TypeLibrary j j * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / j ICF = interface; ICFDisp = dispinterface; j j . A - * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * - * * * * * - * * * * * * * * * * * * * * * * * * * * * * * * j j // Declaration of CoClasses defined in Type Library // (NOTE: Here we map each CoClass to its Default Interface) CF = ICF; jj ********************************************************************* jj II Interface: ICF // Flags: (4416) Dual OleAutomation Dispatchable // GUID: {536C16B5-099F-11D4-A5F5-0020AFE4AAD6} Ij *********************************************************************jj ICF = interface(IDispatch) ['{53 6C16B5-099F-11D4-A5F5-0020AFE4AAD6}'] function UpdateEval(UserlD: Integer; ItemlD: Integer; Mode: Integer; Evaluation Integer): Integer; safecall; function GetRecomm(UserlD: Integer; Mode: Integer): OleVariant; safecall; function Remove(UserlD: Integer; Flag: Integer): WordBool; safecall; function CheckN(UserlD: Integer; Mode: Integer): Integer; safecall; end ; jf *********************************************************************jj II Displntf: ICFDisp // Flags: (4416) Dual OleAutomation Dispatchable // GUID: {536C16B5-099F-11D4-A5F5-0020AFE4AAD6} ji *********************************************************************jj ICFDisp = dispinterface ['{53 6C16B5-099F-11D4-A5F5-0020AFE4AAD6}'] function UpdateEval(UserlD: Integer; ItemlD: Integer; Mode: Integer; Evaluation Integer): Integer; dispid 1; function GetRecomm(UserlD: Integer; Mode: Integer): OleVariant; dispid 2; function Remove(UserlD: Integer; Flag: Integer): WordBool; dispid 3; function CheckN(UserlD: Integer; Mode: Integer): Integer; dispid 4; end ; jj *********************************************************************jj II The Class CoCF provides a Create and CreateRemote method to // create instances of the default interface ICF exposed by // the CoClass CF. The functions are intended to be used by // clients wishing to automate the CoClass objects exposed by the // server of this typelibrary. CoCF = class class function Create: ICF; class function CreateRemote(const MachineName: string): ICF; end; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. / / OLE S e r v e r P r o x y c l a s s d e c l a r a t i o n / / S e r v e r O b j e c t : TCF // Help String : CF Object // Default Interface: ICF // Def. Intf. DISP? : No // Event Interface: // TypeFlags : (2) CanCreate { $ IFDEF LI VE_SERVER_AT_DESIGN_TIME } T C F P r o p e r t i e s = c l a s s ; { $ENDIF} TCF = c l a s s ( T O l e S e r v e r ) p r i v a t e FIntf: ICF; {$IFDEF LIVE_SERVER_AT_DESIGN_TIME} F P r o p s : T C F P r o p e r t i e s ; function GetServerProperties: TCFProperties; {$ENDIF} function GetDefaultlnterface: ICF; protected procedure InitServerData; override; public constructor Create(AOwner: TComponent); override; destructor Destroy; override; procedure Connect; override; procedure ConnectTo(svrlntf: ICF); procedure Disconnect; override; function UpdateEval(UserlD: Integer; ItemlD: Integer; Mode: Integer; Evaluation: Integer): Integer; function GetRecomm(UserlD: Integer; Mode: Integer): OleVariant; function Remove(UserlD: Integer; Flag: Integer): WordBool; function CheckN(UserlD: Integer; Mode: Integer): Integer; property Defaultlnterface: ICF read GetDefaultlnterface; published { $ IFDEF LIVE_SERVER_AT_DESIGN_TIME } property Server: TCFProperties read GetServerProperties; {$ENDIF} end; {$ IFDEF LIVE_SERVER_AT_DESIGN_TIME} / / OLE S e r v e r P r o p e r t i e s P r o x y C l a s s / / S e r v e r O b j e c t : TCF // (This object is used by the IDE's Property Inspector to allow editing // of the properties of this server) j j * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * / j TCFProperties = class(TPersistent) private F S e r v e r : TCF; function GetDefaultlnterface: ICF; constructor Create(AServer: TCF); protected public property Defaultlnterface: ICF read GetDefaultlnterface; published end; {$ENDIF} procedure Register; implementation uses ComObj; class function CoCF.Create: ICF; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. begin Result := CreateComObject(CLASS_CF) as ICF; end; class function CoCF.CreateRemote(const MachineName: string): ICF; begin Result := CreateRemoteComObject(MachineName, CLASS_CF) as ICF; end; procedure TCF.InitServerData; const CServerData: TServerData = ( ClassID: 1{536C16B7-099F-11D4-A5F5-0020AFE4AAD6}1; IntfllD: '{53 6C16B5-099F-11D4~A5F5-0020AFE4AAD6}' ; EventUD: LicenseKey: nil; Version: 500); begin ServerData := SCServerData; end; procedure TCF.Connect; var punk: IUnknown; begin if FIntf = nil then begin punk := GetServer; Fintf:= punk as ICF; end; end; procedure TCF.ConnectTo(svrlntf: ICF); begin Disconnect; FIntf := svrlntf; end; procedure TCF.Disconnect; begin if Fintf <> nil then begin Fintf - . = nil; end; end; function TCF.GetDefaultlnterface: ICF; begin if Fintf = nil then Connect; Assert(Fintf <> nil, 'Defaultlnterface is NULL. Component is not connected to Server. You must call ''Connect'' or ''ConnectTo'' before this operation'); Result := Fintf; end; constructor TCF.Create(AOwner: TComponent); begin inherited Create(AOwner); {$ IFDEF LIVE_SERVER_AT_DESIGN_TIME} FProps := TCFProperties.Create(Self); {$ENDIF) end; destructor TCF.Destroy; begin {$IFDEF LIVE_SERVER_AT_DESIGN_TIME} FProps.Free; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. {$ENDIF} inherited Destroy; end; { $ IFDEF LI VE_SERVER_AT__DES IGN_TIME} function TCF.GetServerProperties: TCFProperties; begin Result := FProps; end ; {$ENDIF} function TCF.UpdateEval(UserlD; Integer; ItemlD: Integer; Mode: Integer; Evaluat' Integer)• begin Result end; function begin Result end; function begin Result end; function begin Result end; { $ IFDEF LIVE_SERVER_AT_DESIGN_TIME > constructor TCFProperties.Create(AServer: TCF); begin inherited Create; FServer := AServer; end; function TCFProperties.GetDefaultlnterface: ICF; begin Result := FServer.Defaultlnterface; end; {$ENDIF} procedure Register; begin RegisterComponents('Servers',[TCF]); end ; end. 5. DataModule.pas unit DataModule; interface uses Windows, Messages, SysUtils, Classes, Graphics, Controls, Forms, Dialogs, Db, DBTables; type TReferenceGroup = Array [1..2000] of integer; f := Defaultlnterface.UpdateEval(UserlD, ItemlD, Mode, Evaluation); TCF.GetRecomm(UserlD: Integer; Mode: Integer): OleVariant; := DefaultInterface.GetRecomm(UserID, Mode); TCF.Remove(UserlD: Integer; Flag: Integer): WordBool; := Defaultlnterface.Remove(UserlD, Flag); TCF.CheckN(UserlD: Integer; Mode: Integer): Integer; := Defaultlnterface.CheckNfUserlD, Mode); R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. TSIData = Array [1..2000] of double; TEstimateData = Array [1..5000] of double; TltemList = Array [1..5000] of integer; TDAInteger = Array of Integer; TDADouble = Array of Double; TSortCompareDouble = function (Iteml, Item2: Double): Integer; TSortComparelnteger = function (Iteml, Item2: Integer): Integer; TFunction = (NarrowF, MediumF, BroadF); TSI = (Correlations, EDistances, ADistances, BayesianP); TUser = class UserlD: integer; ItemlD: TDAInteger; Evaluation: TDAInteger; AvgEvaluation: Double; Counter: integer; private procedure ResetCounter; public procedure Sort; function AddItem(IID, Eval: integer):Boolean; function Getltemlndex(IID: integer):integer; function GetDeviation(Itemlndex: integer): double; procedure UpdateAverage; constructor Create(UID: integer); destructor Destroy; end; TAllUsers = class User: Array [1..2000] of TUser; UserCounter: integer; public function GetUserlndex(UID: integer): integer; function AddUser(UID: integer): integer; procedure RetrieveAllUsers(Mode: integer); procedure DeleteUser(UID: integer); constructor Create(Mode: integer); destructor Destory(Mode: integer); end; TPairData = class ItemlD: TDAInteger; Evaluation!: TDAInteger; Evaluation2: TDAInteger; Userlndexl: integer; Userlndex2: integer; Counter: integer; private procedure ResetCounter; public procedure GetPairData(Userl, User2: TUser; CurrentUserlndex, OtherUserlndex: integer); constructor Create; destructor Destroy; end ; TEstimatedEvaluation = class ItemlD: TDAInteger; UserCounter: TDAInteger; EEvaluation: TDADouble; Counter: integer; private procedure ResetCounter; procedure Addltems(IID: integer); public function Getltemlndex(IID: integer): integer; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. function CalculateESI(CurrentUserlndex, ItemlD, Mode: integer; ReferenceGroup: TDAInteger; UserSI: TDADouble; RefGroupSize: integer; FunctionType: TFunction): double; function Calculate(CurrentUserlndex: integer; Mode, MinRef, MaxRef: integer; RefRatio: Double; FunctionType: TFunction): boolean; procedure Sort; constructor Create; destructor Destroy; end; TSIMatrix = class Data: TSIData; private procedure Initialize; function CorrelationsSI(PairData: TPairData):double; function EDistancesSI(PairData: TPairData):double; function ADistancesSI(PairData: TPairData):double; function BayesianPSI(PairData: TPairData):double; function CalculateSI(PairData: TPairData; IndexType: TSI):boolean; public function UpdateSI(Userlndex: integer; Mode: integer; IndexType: TSI; Queryl: TQuery): Boolean; constructor Create(Mode:integer); end; TDataModulel = class(TDataModule) GetDatal: TQuery,- GetData2: TQuery; Sessionl: TSession; CF: TDatabase; Tablel: TTable; procedure DataModuleCreate(Sender: TObject); procedure DataModuleDestroy(Sender: TObject); public function UpdateEvaluation(UserlD, ItemlD, Mode, Evaluation: integer; AQuery: TQuery; Flag: integer): integer; function RemoveUser(UserlD, Flag: Integer; AQuery: TQuery): Boolean; function CheckNumber(UserlD, Mode: integer): integer; end ; {The following two types are for data transfer between server and client) TData = Array [1..20] of integer; TPaperList = class Data: TData; public procedure Initialize; function ToVariant: OleVariant; function FromVariant(VData: OleVariant): integer; constructor Create; end; const MaxUserSize : integer = 2000; MaxItemSize : integer = 5000; MinUsers : integer = 2; recommendations Minltems : integer = 2; generating recommendations MinRef : integer = 40; MaxRef : integer = 100; Increaselnterval : integer = 30; RefRatio : double = 0.2; group NullEvaluation : integer = 9; NeutralEvaluation : integer = 3; Neutral : integer = 1; NullSI : double = 0.0; 165 / / m in im u m n u m b e r o f u s e r s f o r g e n e r a t i n g / / m in im u m n u m b e r o f common i t e m s f o r / / m in im u m r e f e r e n c e g r o u p s i z e / / m axim um r e f e r e n c e g r o u p s i z e / / i n c r e a s e i n t e r v a l f o r d y n a m i c a r r a y / / r e f e r e n c e g r o u p s i z e r e l a t i v e t o t o t a l u s e r R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. NofModes : integer = 2; SIMatrixFileName : Array [1..2] of string = ('D:\SIMatrixl.dat1, 'D:\SIMatrix2.dat’); IndexType : TSX = Correlations; Delimeter : string = ' ,.;‘; S N u l l E v a l u a t i o n ; s t r i n g = ' 9 1; SNullSI : string = '0.0'; CfPath : string = 'D:\Inetpub\wwwroot\main'; RefreshOser : integer = 1; ArrayLowBound : integer = 1; ArrayHighBound : integer = 20; v a r DataModulel: TDataModulel; PairData: TPairData; E s t i m a t e s : T E s t i m a t e d E v a l u a t i o n ; Matrices: Array [1..2] of TSIMatrix; // number of modes Users; Array [1..2] of TAllUsers; A S e s s i o n : T S e s s i o n ; PaperList; TPaperList; NUpdate, NRecomm, NSurvey: integer; T i m e C o u n t : i n t e g e r ; TimeLimit : integer; // time limit for being idle (10 minutes) procedure Quicksort(var SortList; TDADouble; var Listl: TDAInteger; L, R: Integer; SCompare: TSortCompareDouble); overload; procedure Quicksort(var SortList: TDAInteger; var Listl: TDAInteger; L, R: Integer SCompare: TSortComparelnteger); overload; function CompareDouble(Iteml, Item2: Double): Integer; function Comparelnteger(Iteml, Item2: Integer): Integer; i m p l e m e n t a t i o n uses Main; {$R *.DFM> function TDataModulel.UpdateEvaluation(UserlD, ItemID, Mode, Evaluation: integer; AQuery: TQuery; Flag: integer): integer; var SUserlD, SItemID, SMode, SEvaluation, ASQL: string; UserIndex, Itemlndex: integer; begin { update RawData } Userlndex := Users[Mode].GetUserlndex(UserlD); if (Userlndex = 0) then //a new user begin Userlndex := Users(Mode]-AddUser(UserlD); end; if (Userlndex > 0) then // user successfully created begin Itemlndex := Users[Mode].User[Userlndex].Getltemlndex(ItemID); if (Itemlndex = 0) then begin if (Users [Mode] .User [Userlndex] . AddltemdtemID, Evaluation)) then Result := Users[Mode]-User[Userlndex].Counter; end else begin Result := Users[Mode].User[Userlndex].Counter; end; Users[Mode].User[Userlndex].UpdateAverage; // re-calculate average evaluation end; { update the database } SUserlD := IntToStr(UserlD); SItemID := IntToStr(ItemID); SMode := IntToStr(Mode); SEvaluation := IntToStr(Evaluation); R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. with AQuery do begin Close; if (Flag = Neutral) then // adding neutral users, make it faster begin ASQL := 'insert into Evaluation (DserlD, ItemID, EMode, Evaluation)'; ASQL := ASQL + ' v a l u e s ( ' + S U s e r l D + ' , 1 + S I t e m I D + 1 , ' + SM ode + ' , 1 S E v a l u a t i o n + ')' ; SQL.Clear, - SQL.Add(ASQL); try ExecSQL; except end; end else begin ASQL := ‘select UserlD from Evaluation'; ASQL := ASQL + ' where (UserlD = ' + SUserlD; ASQL : = ASQL + ' and ItemID = ' + SItemID; ASQL := ASQL + ' and EMode = 1 + SMode +')'; SQL.Clear; SQL.Add(ASQL); O p e n ; if (IsEmpty) then // there is no evaluation on this item begin Close,- ASQL := 'insert into Evaluation (UserlD, ItemID, EMode, Evaluation)'; ASQL := ASQL + ' values (' + SUserlD + ', ' + SItemID + ', ' + SMode + ', SEvaluation + ')'; SQL.Clear; SQL.Add(ASQL) ; try ExecSQL; except end; end else begin // there is an evaluation for this item already Close; ASQL := 'update Evaluation set Evaluation = ' + SEvaluation; ASQL := ASQL + ' where (UserlD = ' + SUserlD; ASQL := ASQL + ’ and ItemID = ' + SItemID; ASQL := ASQL + ' and EMode = ' + SMode + ')'; SQL. Clear; SQL.Add(ASQL); try ExecSQL; except end; end; Close; end; //of non-neutral end; // of with TQuery end; function TDataModulel.RemoveUser(UserlD, Flag: Integer; AQuery: TQuery): Boolean; var i: integer; begin Result := false; with AQuery do // delete from database begin Close; SQL.Clear; SQL.Add('delete from Evaluation'); SQL.Add(' where UserlD = ' + IntToStr(UserlD)); try ExecSQL; except R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. end; Close; SQL.Clear; SQL.Add('delete from Users'); SQL.Addl1 where UserlD = ' + IntToStr(UserlD)); try ExecSQL; except end; Result ;= true; Close; end; if (Flag = RefreshUser) then begin for i := 1 t o NofModes do // delete from data begin Users[i].DeleteUser(UserlD); end; end; end; function TDataModulel.CheckNumber(UserlD, Mode: integer): integer; var Userlndex: integer; begin Result := 0; Userlndex := Users[Mode].GetUserlndex(UserlD); if (Userlndex > 0) then // user exists begin Result := Users[Mode].User[Userlndex].Counter; end; end; function CompareDouble(Iteml, Item2: Double): Integer; begin if (Iteml < Item2) then Result := 1 else if (Iteml > Item2) then Result := -1 else Result := 0; end; function Comparelnteger(Iteml, Item2: Integer): Integer; begin if (Iteml < Item2) then Result := 1 else if (Iteml > Item2) then Result := -1 else Result := 0; end; procedure Quicksort(var SortList: TDADouble; var Listl: TDAInteger; L, R: Integer; SCompare: TSortCompareDouble); var I, J: Integer; P, T: Double; T1: integer; begin if ((L < Low(SortList)) or (R > High(SortList))) then begin Application.MessageBox('The array index is out of range', 'Sort Error', MB_OK + MB_DEFBUTT0N1); end else begin repeat I := L; J := R; P := SortList[(L + R) shr 1]; 168 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. repeat while SCompare(SortList[I], P) < 0 do Inc(I); while SCompare(SortList[J], P) > 0 do Dec(J); if I <= J then begin T := SortList[I]; SortList[I] := SortList[J]; SortList[J] := T; T1 := Listl[I]; Listl[ I ] := Listl[J]; Listl(J] := Tl; Inc(I) ; Dec(J); end; until I > J; if L < J then Quicksort(SortList, Listl, L, J, SCompare); L := I ; until I >= R; end ; end; procedure Quicksort(var SortList: TDAInteger; var Listl: TDAInteger; L, SCompare: TSortComparelnteger); var I , J: Integer; P, T: Integer; Tl: integer; begin if ((L < Low(SortList)) or (R > High(SortList))) then begin Application.MessageBox(1 The array index is out of range', 'Sort Error', MB_OK + MB_DEFBUTT0N1); end else begin repeat I := L; J := R; P := SortList[(L + R) shr 1]; repeat while SCompare(SortList[I], P) < 0 do Inc(I) ; while SCompare(SortList(J], P) > 0 do Dec(J); if I <= J then begin T := SortList[I]; SortList[I] := SortListlJ]; SortList[J] := T; Tl := Listl[I]; Listl[I] := Listl[J]; Listl[J] := Tl; I n c ( I ) ; D e c (J ) ; end; until I > J; if L < J then Quicksort(SortList, Listl, L, J, SCompare); L := I; until I >= R; end; end; constructor TUser. Create (UID: integer) , - begin R: I n t e g e r ; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. inherited Create; R e s e t C o u n t e r ; UserlD := UID; end; d e s t r u c t o r T U s e r . D e s t r o y ; begin I t e m I D := n i l ; Evaluation := nil; inherited Destroy; end; procedure TUser.ResetCounter; begin Counter : = 0; I t e m I D := n i l ; Evaluation := nil; end; function TUser.GetItemIndex(IID: integer):integer; var i : integer; begin {This function accepts an Item ID and returns the corresponding index number in the SI table} Result := 0; for i := 1 to Counter do begin if (ItemID[i] = IID) then begin Result := i; exit; end; end ; end; function TUser.GetDeviation(Itemlndex: integer): double; begin if ((Itemlndex > 0) and (Itemlndex <= Counter)) then Result := Evaluation[Itemlndex] - AvgEvaluation else Result := 0; end; procedure TUser.UpdateAverage; var Sum: double; j: integer; begin Sum := 0; for j := 1 to Counter do // calculate sum of evaluations Sum := Sum + Evaluation[j]; if (Counter > 0) then AvgEvaluation := Sum/Counter; end; procedure TUser.Sort; begin if (Counter > 1) then Q u i c k s o r t ( I t e m I D , E v a l u a t i o n , 1 , C o u n t e r , C o m p a r e l n t e g e r ) ; // sort items according to the item ID end; function TUser.AddItem(IID, Eval: integer):Boolean; begin {This function adds a new user to the similarity index table} if (Counter >= MaxItemSize) then 170 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. begin R e s u l t := f a l s e ; Exit; end else begin inc (Counter, 2); SetLength(IteraID, Counter); SetLength(Evaluation, Counter); dec(Counter); ItemID[Counter] ;= IID; Evaluation[Counter] := Eval; Result := true; end ; end; constructor TAllUsers.Create(Mode: integer); begin inherited Create; RetrieveAllUsers(Mode); end; destructor TAllUsers.Destory(Mode: integer); var i: integer; begin for i := 1 to UserCounter do User[i].Free; inherited Destroy; end; function TAllUsers.GetUserIndex(UID: integer): integer; var i: integer; begin Result := 0; for i := 1 to UserCounter do begin if (User[i].UserlD = UID) then begin R e s u l t := i ; E x i t ; end ; end ; end; function TAllUsers.AddUser(UID: integer): integer; begin Result := 0; if (UserCounter < MaxUserSize) then begin inc(UserCounter) ; User[UserCounter] := TUser.Create(UID) ; Result := UserCounter; end ; end; procedure TAllUsers.DeleteUser(UID: integer); var UserToDelete: TUser; Userlndex: integer; begin Userlndex := GetUserlndex(UID); if (Userlndex > 0) then begin UserToDelete := User[Userlndex]; User[Userlndex] := User[UserCounter]; dec(UserCounter); UserToDelete.Free; end; end; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. procedure TAllUsers.RetrieveAllUsers(Mode: integer); var Ulndex, i: integer; begin UserCounter := 0 ; with DataModulel.GetDatal do begin Close; SQL.Clear; SQL.Addl’ SELECT userid, itemid, evaluation FROM evaluation'); SQL.Add(' WHERE emode = ' + IntToStr(Mode)) ; Open; while (not Eof) do // store all data begin Ulndex := GetUserlndex(FieldByName('UserlD').Aslnteger); if (Ulndex = 0) then Ulndex := AddUser(FieldByName('UserlD').Aslnteger); if (Ulndex > 0) then begin User[UIndex].AddItem(FieldByName('ItemlD').Aslnteger, FieldByName('Evaluation').Aslnteger) end; Next ; end; end ; for i := 1 to UserCounter do begin // calculate average evaluation User[i].UpdateAverage; end; end; constructor TPaperList.Create; begin inherited Create; I n i t i a l i z e ; end; procedure TPaperList.Initialize; var i: integer; begin for i := ArrayLowBound to ArrayHighBound do Data[i] := 0; end ; function TPaperList.ToVariant: OleVariant; {This function converts an array of integer into variant (reverse of FromVariant} var p: pointer; begin Result := VarArrayCreate([ArrayLowBound, ArrayHighBound], Varlnteger); p := VarArrayLock(Result); Move(Data, pA, sizeof(TData)); VarArrayUnlock(Result); end; function TPaperList.FromVariant(VData: OleVariant): integer; var p: pointer; {This function converts a variant into an array of integer (reverse of ToVariant) Also, it returns the maximum index of none-zero element} begin p := VarArrayLock(VData); Move(pA, Data, sizeof(TData)); VarArrayUnlock(VData); Result := ArrayLowBound; while ((Result <= ArrayHighBound) and (Data[Result] <> 0)) do begin i n c ( R e s u l t ) ; e n d ; R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. dec(Result); end; function SlWeight(OneSI: double; FunctionType: TFunction): double; begin {This fuction calculates similarity index weights) case FunctionType of II Whether the type is narrow, medium, or broad reference group NarrowF : begin if (OneSI < 0) then Result := - sqr(OneSI) else Result := sqr(OneSI); end; MediumF : begin Result := OneSI; end; BroadF : begin if (OneSI < 0) then Result := -sqrt(-OneSI) else Result ;= sqrt(OneSI); end else begin Result := 0; end; end; end ; procedure TPairData.ResetCounter; begin Counter := 0 ; end ; constructor TPairData.Create; begin inherited Create; I t e m I D := n i l ; Evaluationl := nil; Evaluation2 := nil; ResetCounter; end; destructor TPairData.Destroy; begin I t e m I D := n i l ; Evaluationl := nil; Evaluation2 := nil; inherited Destroy; end; procedure TPairData.GetPairData(Userl, User2: TUser; CurrentUserIndex, OtherUserlndex: integer); var Nl, N2: integer; begin {This function collects and cleans the evaluations of two users. The pair data are used to calculate similarity index for two users. The data should be sorted by the Item ID) R e s e t C o u n t e r ,- Userlndexl := CurrentUserlndex; Userlndex2 := OtherUserlndex; Userl.Sort; User2.Sort; Nl := 1; N2 := 1; while ((Nl <= Userl.Counter) and (N2 <= User2.Counter)) do // while there are some items left 173 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. begin if (Userl.ItemlD[Nl] = User2.ItemID[N2]) then // if a common item is found begin inc(Counter, 2); SetLength(ItemlD, Counter); SetLength(Evaluationl, Counter); SetLength(Evaluation2, Counter); dec (Counter) , - ItemlD[Counter] := Userl.ItemlD[Nl]; Evaluationl[Counter] := Userl.Evaluation[Nl]; Evaluation2[Counter] := User2.Evaluation[N2]; inc(Nl); inc(N2); end else if (Userl.ItemlD[Nl] > User2.ItemlD[N2]) then // proceed to find common items begin { inc(Counter, 2); SetLength(ItemlD, Counter); SetLength(Evaluationl, Counter); SetLength(Evaluation2, Counter); dec(Counter); ItemlD[Counter] := Userl.ItemlD[Nl]; Evaluationl[Counter] := Userl.Evaluation[Nl]; Evaluation2[Counter] := NeutralEvaluation; } inc(Nl); end else begin { inc(Counter, 2); SetLength(ItemID, Counter); SetLength(Evaluationl, Counter); SetLength(Evaluation2, Counter); dec(Counter); ItemlD[Counter] := User2.ItemlD[N2]; Evaluationl[Counter] := User2.Evaluation[N2]; Evaluation2[Counter] := NeutralEvaluation; } inc(N2); end; end ; end; procedure TEstimatedEvaluation.ResetCounter; begin Counter := 0 ; ItemlD := nil; UserCounter := nil; EEvaluation ;= nil; end; constructor TEstimatedEvaluation.Create; begin inherited Create; ResetCounter; end; destructor TEstimatedEvaluation.Destroy; begin ResetCounter; inherited Destroy; end; procedure TEstimatedEvaluation.Addltems(IID: integer); begin if (Counter < MaxItemSize) then begin inc(Counter, 2); SetLength(ItemlD, Counter); SetLength(UserCounter, Counter); 174 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. SetLength(EEvaluation, Counter); dec(Counter); ItemlD[Counter] := IID; UserCounter[Counter] := 1; end; end; function TEstimatedEvaluation.GetltemlndexdID: integer): integer; var i: integer; begin Result := 0; for i : = 1 to Counter do begin if (ItemID[i] = IID) then begin Result :— i ; Exit; end ; end ; end; function TEstimatedEvaluation.CalculateESI(CurrentUserlndex, ItemlD, Mode: integer; ReferenceGroup: TDAInteger; UserSI: TDADouble; RefGroupSize: integer; FunctionType: TFunction): double; var i, NofUsers, OtherUserlndex, Itemlndex: integer; Sum, Weight, WeightSum: double; begin {This function calculates estimated Sis} Result := NullSI; SetLength(EEvaluation, Counter+1); // set length of dynamic array Sum := 0; WeightSum := 0; NofUsers := 0; for i := 1 to RefGroupSize do // for all other users begin OtherUserlndex := ReferenceGroup[i] ; if ((OtherUserlndex <> CurrentUserlndex) and (UserSI[i] <> NullSI)) then // exclude self and users with no SI with current user begin Itemlndex := Users [Mode] .User [OtherUserlndex] .Getltemlndex(ItemlD) , - if (Itemlndex > 0) then // if this user has evaluated this item begin Weight := SlWeight(UserSI[i], FunctionType); Sum := Sum + Users[Mode].User[OtherUserlndex).GetDeviation(Itemlndex)*Weight; WeightSum := WeightSum + Weight; inc(NofUsers); end ; end end; if (NofUsers > 0) then Result := Users[Mode].User[CurrentUserlndex].AvgEvaluation + Sum/NofUsers; end , - function TEstimatedEvaluation.Calculate(CurrentUserlndex: integer; Mode, MinRef, MaxRef: integer; RefRatio: Double; FunctionType: TFunction): boolean; var ReferenceGroup: TDAInteger; UserSI: TDADouble; RefGroupSize, AnItemID, Anltemlndex : integer; i, j: integer; // temp: string; begin {This function is the main function for calculating estimated evaluations} j := 0; ReferenceGroup := nil; UserSI := nil; 175 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. for i := 1 to Users[Mode].UserCounter do begin if (i <> CurrentUserlndex) then // exclude self begin inc(j, 2) ; S e t L e n g t h ( R e f e r e n c e G r o u p , j ) ; SetLength(UserSI, j ) ; dec(j ) ; ReferenceGroup[j] := i; UserSI[j] := Matrices[Mode].Data[i]; end; end; Quicksort (UserSI, ReferenceGroup, 1, j, CompareDouble) // sort users according to the SI A p p l i c a t i o n . P r o c e s s M e s s a g e s ; RefGroupSize := round(Users[Mode].UserCounter*RefRatio); // set up reference group size if (j <= MinRef) then // total number users is less than minimum referenc group size RefGroupSize := j else if (RefGroupSize < MinRef) then // check minimum reference group size RefGroupSize := MinRef else if (RefGroupSize > MaxRef) then // check maximum reference group size RefGroupSize := MaxRef; R e s e t C o u n t e r ; for i := 1 to Users[Mode].UserCounter do // add items to calculate estimated evaluations begin for j := 1 to Users[Mode].User[i].Counter do begin AnItemID := Users[Mode].User[i].ItemlD[j]; // an item evaluated by the user if (Users[Mode].User[CurrentUserlndex].Getltemlndex(AnItemID) = 0) then // check if this item is the item that current user has not evaluated begin Anltemlndex := Getltemlndex(AnItemID); // item index of the item in TEstimatedEvaluation if (Anltemlndex = 0) then // the item is not in the list of Estimates Addltems(AnItemID) e l s e inc(UserCounter[Anltemlndex]); e n d ; end; end; R e s u l t := t r u e ; if (Counter > 0) then begin for i := 1 to Counter do begin if (UserCounter[i] >= MinUsers) then begin EEvaluation[i] := CalculateESI(CurrentUserlndex, ItemID[i], Mode, ReferenceGroup, UserSI, RefGroupSize, FunctionType); end else begin EEvaluation[i] := NullSI; end; Application.ProcessMessages; end; Sort; end e l s e b e g i n R e s u l t ;= f a l s e ; 176 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. ReferenceGroup := nil; UserSI := nil; exit; end; i := 1 ; Forml.Memol.Clear ; while (i <= Counter) do begin str(EEvaluation[i] :8 ; 3, temp); Forml.Memol.Lines.Add(IntToStr(ItemlD[i]) + #9 + temp); inc (1) ; end; tgsfc} ReferenceGroup := nil; UserSI := nil; end; procedure TEstimatedEvaluation.Sort; begin if (Counter > 0) then Q u i c k s o r t ( E E v a l u a t i o n , I t e m l D , 1 , C o u n t e r , C o m p a r e D o u b l e ) ; end ; constructor TSIMatrix.Create(Mode:integer); begin inherited Create; Initialize; end; procedure TSIMatrix.Initialize; var i: integer; begin for i := 1 to MaxUserSize do Data[i] ;= NullSI; end ; {function TSIMatrix.LoadFromFile(FileWame; string); boolean; var F: File; UserlDSize, NumRead: integer; begin Result := false; UserlDSize := SizeOf(UserlD); AssignFile(F, FileWame); {$ !-> // Reset(F,1) ; { $ 1 + } { if (IOResult = 0) then begin BlockReadfF, UserlD, UserlDSize, NumRead); if (NumRead <> UserlDSize) then exit; BlockRead(F, Counter, SizeOf(Counter), NumRead); if (NumRead <> SizeOf(Counter)) then exit; CloseFile(F); Result := true; end; end; function TSIMatrix.SaveToFile(FileName: string): boolean; var F: File; UserlDSize, NumWritten: integer; begin Result := false; 177 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. U s e r l D S i z e := S i z e O f ( U s e r l D ) ; A s s i g n F i l e ( F , F i l e N a m e ) ; {$!-> / / R e w r i t e (F , 1 ) ; {$ 1+} { i f ( I O R e s u l t = 0) t h e n b e g i n T r y B l o c k W r i t e ( F , U s e r l D , U s e r l D S i z e , N u m W r i t t e n ) ; i f ( N u m W r i t t e n <> U s e r l D S i z e ) t h e n e x i t ; B l o c k W r i t e ( F , C o u n t e r , S i z e O f ( C o u n t e r ) , N u m W r i t t e n ) ; i f ( N u m W r it te n <> S i z e O f ( C o u n t e r ) ) t h e n e x i t ; F i n a l l y C l o s e F i l e ( F ) ; R e s u l t ;= t r u e ; E n d ; e n d ; e n d ; function TSIMatrix.CorrelationsSI(PairData: TPairData):double; var meanl, mean2: double; diffl, diff2: double; suml, sum2, sum3: double; i : integer; begin {This function calculates correlations, one of the similarity indices) if (PairData.Counter < Minltems) then begin Result := NullSI; exit; end else begin suml := 0; Sum2 := 0; for i := 1 to PairData.Counter do begin suml := suml + PairData.Evaluationl[i]; // summing up sum2 := sum2 + PairData.Evaluation2[i]; end ; meanl := suml / PairData.Counter; // averages mean2 := sum2 / PairData.Counter; suml := 0; sum2 ;= 0; sum3 := 0; for i := 1 to PairData.Counter do begin diffl := PairData.Evaluationl[i] - meanl; diff2 := PairData.Evaluation2[i] - mean2; suml := suml + sqr(diffl); sum2 := sum2 + sqr(diff2); sum3 := sum3 + diffl*diff2; end ; if ((suml > 0) and (sum2 > 0)) then // ordinary Result := sum3/(sgrt(suml)*sqrt(sum2)) else if ((suml > 0) or (sum2 > 0)) then // either one of users have same evaulations across items Result := 0.01 else begin // both users have same evaluations across items Result := 1/(abs(meanl-mean2)+1); end; end; 178 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. end; function TSIMatrix.EDistancesSI(PairData: TPairData):double; var i ; integer; sum : double; begin {This function calculates Euclidean distance, one of the similarity indices} if (PairData.Counter < Minltems) then EDistancesSI := NullSI else begin sum := 0; for i := 1 to PairData.Counter do begin sum := sum + sqr(PairData.Evaluationl[i] - PairData.Evaluation2[i]) end; EDistancesSI := sqrt(sum); end; end ; function TSIMatrix.ADistancesSI(PairData: TPairData):double; var i : integer; sum : double; begin {This function calculates absolute distances, one of the similarity indices} if (PairData.Counter < Minltems) then ADistancesSI := NullSI else begin sum := 0; for i := 1 to PairData.Counter do begin sum := sum + abs(PairData.Evaluationl[i] - PairData.Evaluation2[i]) end; ADistancesSI := sum/PairData.Counter; end; end; function TSIMatrix.BayesianPSI(PairData: TPairData):double; begin {This function calculates Bayesian probability, one of the similarity indices} if (PairData.Counter < Minltems) then BayesianPSI := NullSI else begin BayesianPSI := 0.1; end; end; function TSIMatrix.CalculateSI(PairData: TPairData; IndexType: TSI):boolean; begin {This function calculates a similarity index for a pair of users} case IndexType of Correlations: begin Data[PairData.Userlndex2] := CorrelationsSI(PairData); end; EDistances: begin Data[PairData.UserIndex2] := EDistancesSI(PairData); end ; ADistances: begin Data[PairData.Userlndex2] := ADistancesSI(PairData); end; BayesianP: begin Data[PairData.Userlndex2] := BayesianPSI(PairData); end; 179 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. end ; if (Data[PairData.Userlndex2] = NullSI) then Result := false else Result : = true; end; function TSIMatrix.UpdateSI(Userlndex: integer; Mode: integer; IndexType: TSI; Queryl: TQuery): Boolean; var i, NofUsers: integer; begin (This function is the main function for updating similarity index It updates the specified users' similarity indices} Result := false; NofUsers := 0; for i := 1 to Users[Mode].UserCounter do // repeate for all users begin if (i <> Userlndex) then // exclude self begin PairData.GetPairData(Users[Mode].User[Userlndex], Users[Mode].User[i], Userlndex, i); if (CalculateSI(PairData, IndexType)) then inc(NofUsers); end; if (NofUsers >= MinUsers) then Result := true; end; end; procedure TDataModulel.DataModuleCreate(Sender: TObject); var i : integer , - begin {This procedure initializes settings} ASession := TSession.Create(nil); with CF do begin Parains . Values [ ' USER NAME ' ] : = 1 CF' ; Params.Values[1 PASSWORD'] := 1CF•; LoginPrompt := False; Connected := true; end; Estimates := TEstimatedEvaluation.Create; for i := 1 to NofModes do begin Matrices[i] := TSIMatrix.Create(i); Users[i] := TAllUsers.Create(i) ; end; PairData := TPairData.Create; PaperList := TPaperList.Create; NUpdate := 0; NRecomm : = 0 , - NSurvey := 0; Forml.chkStopClick(Sender) ; end; procedure TDataModulel.DataModuleDestroy(Sender: TObject); var i: integer; begin for i := 1 to NofModes do begin Matrices[i].Free; Users[i].Free; end; Estimates.Free ; PairData.Free ; PaperList.Free; CF.Close; 180 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission. ASession.Close; ASession.Free; end; e n d . 181 R eproduced with perm ission of the copyright owner. Further reproduction prohibited without perm ission.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

A flexible framework for replication in distributed systems

PDF

Conflict identification and resolution for software attribute requirements

PDF

Combining compile -time and run -time parallelization

PDF

An empirical investigation of everyday low price (EDLP) strategy in electronic markets

PDF

Information systems capabilities and indirect relational value: Sustaining networked organizations

PDF

Defining the advice -seeking request in electronic solicitations in boundary spanning environments

PDF

Grid workflow: A flexible framework for fault tolerance in the grid

PDF

A divide-and-conquer computer

PDF

IT -enabled dynamic capabilities in new product development: Building a competitive advantage in turbulent environments

PDF

Collaborative negotiation for early stage parametric design of mechanical systems

PDF

A unified approach to the construction of correct concurrent programs

PDF

Extending the COCOMO II software cost model to estimate effort and schedule for software systems using commercial -off -the -shelf (COTS) software components: The COCOTS model

PDF

An algebraic view of protection and extendibility in abstract data types

PDF

An implicit-based haptic rendering technique

PDF

Agile COCOMO-II

PDF

Building a biological hybrid biosensor using nonlinear system analysis of the CA1 hippocampal neural network

PDF

A typology of corporate environemental strategy and its driving factors in jultinational corporations

PDF

Functional testing of constrained and unconstrained memory using march tests

PDF

Beyond binary theories to agile learning: An empirical investigation of learning methods in warranty management.

PDF

"Pay no attention to those men behind the curtains": An ethical examination of Los Angeles charter reform activities 1996--1999 by use of crisis management

Asset Metadata

Creator Im, Il (author)

Core Title Augmenting knowledge reuse using collaborative filtering systems

School Graduate School

Degree Doctor of Philosophy

Degree Program Information Systems

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag business administration, management,Computer Science,OAI-PMH Harvest

Language English

Contributor Digitized by ProQuest (provenance)

Advisor [illegible] (committee chair), Andrews, John D. (committee member), El Sawy, Omar A. (committee member), Horowitz, Ellis (committee member)

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c16-107511

Unique identifier UC11327656

Identifier 3027730.pdf (filename),usctheses-c16-107511 (legacy record id)

Legacy Identifier 3027730.pdf

Dmrecord 107511

Document Type Dissertation

Rights Im, Il

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the au...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus, Los Angeles, California 90089, USA