Chen, Y., Kiesler, S., Konstan, J., Kraut, R., Resnick, P., Reidl, J., & Terveen, L. (2003). Designing on-line communities to enhance participation -- bridging theory and practice. A proposal to the National Science Foundation.

Discussion leader/summarizer: CaRichardson

Research Question:

Can social and economic theories of voluntary contributions be applied to the design of on-line communities thereby increasing participation.

Key Points/Claims/Methods

This is the full text of an NSF proposal for a complex series of field experiments to test various principles of on-line community design to enhance participation. As this is the grant proposal, it does not contain any research results. However, it does provide a detailed background section on current state of knowledge about voluntary contribution with respect to on-line communities. Furthermore it is a relevant example of a federally funded on-line communities research proposal.
Pages 11 and 12 provide an excellent overview of the proposed sequence of experiments. In particular, it describes the 12 modifiable design factors that will be manipulated in experimental field studies to determine the impact of each factor on participation. The 12 factors include 1) group size, 2) group attractiveness, 3) attractiveness of members, 4) task attractiveness, 5) importance of own contribution, 6) expected group performance, 7) identifiability, 8) expected repeated interaction, 9) visible economy, 10) cost of membership, 11) payment for contribution, and 12) award status.
Two different existing on-line communities will be used for the experiments, Movielens and the CHI reviewing group. Human subjects protection and ethical issues are briefly described for some of the proposed experiments. Some attention is paid to the coordination and management of this complex multi-center proposal.

Critique

I have already declared my strong conflicts of interest with respect to one of the authors of this proposal so take everything I say here with a grain of salt. First, let me just say that I think this is a brilliant proposal. I will mention two things that seemed to be missing:

1. Attention to measures of various constructs. For example, the sentence describing how results of the CHI reviewing experiments will be evaluated “…results in enhanced participation and better reviewing results as judged by the chairs or the authors themselves” (p. 12) seems kind of vague.
2. Normative feedback. Normative feedback means giving participants information about where they rank with respect to others in the community with respect the the behavior you are trying to encourage. For example, with CHI reviews, you could rank reviewers by how many reviews they had completed and then let them know where they rank with respect to other community members. Normative feedback, sometimes called a “physician report card” has been shown to be just about the only thing that can change physician behavior with respect to preventive interventions like vaccinations and cancer screening. It is also one of the most powerful motivators to change patient self-care behavior. For example, on the sportbrain.com website, participants can join a group and then within the group they can see where they rank with respect to how many steps the have taken. These “league tables” are very popular and participants claim they are motivating, however, there is no empirical evidence that I know of that league tables actually significantly increase walking. You could say that normative feedback fits into the twelfth “award status” factor, but I actually think it is something different. I don’t know anything about the social theory of normative feedback but I’m guessing that a lot is known about this.
3. Public v. Private goods. The grantees claim that the research questions will be addressed using theories from psychology, economics, and sociology and that the self-interest postulate underlies all theories. One key theory that they all make use of is the public goods theory. This theory explains what the economists call free-riding and what the psychology call social-loafing. The grant proposal has argued that much online contribution is public good. However, they seem to have overlooked the transformation of public goods into private goods. In a case study, Chiao (2003) has argued that the open source movement has undergone a similar change as the Chinese economy but at an accelerated pace due to a lack of enforcement on the Internet. He argues that the lack of enforcement diminishes the involuntary contribution of public goods so private goods are formed instead through the process of separating legal and economic property rights. [BenjaminChiao]

I really thought that the ideas presented in this grant proposal were very unique and quite interesting. I would very much like to read more about where this project has gone... [good, we'll talk about one paper that came out of this proposal later in the course - DerekHansen] After reading the proposal, I kept coming back to one factor that I couldn't help but wonder about. I would be very interested to know how peoples perceptions of their ability to contribute good information would have an impact on contributions. I guess this would be sort of an online posting self efficacy. This is very closely related to the importance of contribution factor outlined in the proposal, but I don't think that they are one in the same. From the way I understood it, importance of contribution refers to how important I feel that my contribution is to the online discussion. This self efficacy factor I'm thinking of relates to my perceptions of how able I am to contibute to the conversation... does this make sense? [LaurieBuis]

One of the additional readings in this course (Karau1993) discusses the idea that LauriBuis raises. The articles summarizes a number of social psychology studies on social loafing and proposes a general model (the Collective Effort Model CEM) that can be used to understand varous factors that determine why people do or don't contribute to public goods (see pages 664-6 especially). One of the key factors that determines how much someone contributes is called expectancy - the degree to which high levels of effort are expected to lead to high levels of performance. I believe this is what you have described above, and I also believe that this is more than just how unique your contribution is to the whole. It is really about the expected value of your contribution. [DerekHansen]

Connections with other readings, ideas, etc.

From Theory to Practice

I find the lessons I have learned from Psych stats classes very much relevant to the difficulties this online community design project may face along the way in their field studies. Although I strongly agree that "identifying key factors drawn from theory is an essential step" (p. 4) for design decisions and field studies, we also have to carefully examine on what assumptions or conditions the theory is based. Theoriests tend to highlight only a few (or few) main effects explaining social or psychological phenomena among others in order to generalize their findings. Generally, experiments are designed to maximize the differences between the control group and experimental groups in a simple form (a limited number of variables controlled.) That's why, very often, it is difficult to directly apply theories to system design because life is not that simple. There are many factors that may influence hypothses in the field study; unexpected interactions between ignored main effects, the effects of unexpected covariates, or the contextual use of system, etc. My point is that it is a very difficult misson to bridge theory and practice and we should thoroughly identify what key design factors missed to be addressed in theories through iterative tests. [YongKim]

From Online to Offline

I think there may be two bridges we need to eventually cross: one between theory and practice, which the proposal addresses very well, i.e., can the social and economic theories of contribution based on physical interactions apply to the interactions of online communities? A second bridge may exist between the social capital that we accrue online and our social capital or sense of well-being offline. Will membership in a newly-enriched online community motivate me to become more involved in my local community? Will I be more likely to join the local choir if my online glee club sees more meaningful contributions and less flaming? Or along the ideas of Tom Malone, will it make me happier?

I think Caroline's research is a good example of this (crossing the boundary between on- and off-line), and Bob Kraut's HomeNet studies may have said something along these lines... [KathyLee]

Barry Wellman has also written extensively on this topic [DerekHansen]

Powazek

I thought this reading linked most tightly with the PowazekChapter2 and not as well with the other readings from 884. The two other 884 readings seemed to be very abstract to me but the Powazek book seems to provide more practical design advice much of which reflects the factors in this proposal.

Questions

What's the difference between high- and mid-level theories? Is it just the operationalization of concepts? [KathyLee]

I don't think there is any clear-cut definition, but as I understand it high-level theories have broader application than mid-level theories and are less domain-specific. An example of a high-level theory would be social loafing (or the free-rider problem), a theory that is used in many disciplines and places. Other thoughts and examples? [DerekHansen]

References

Benjamin Hak-Fung Chiao, "An Economic Theory of Free and Open Source Software: A Tour from Lighthouse to Chinese-Style Socialism", Proceedings of the International Conference on Open Source 2003. Archived at the MIT Free/Open Source Research Community. Download here: http://opensource.mit.edu/papers/chiao.pdf∞.