Paper Analysis: “Resouorces and Tools for Evaluation of Online Communities of Practice” (Part 1)


Connected Educator, & Summit, P. (2011). Resources and Tools for Evaluation of Online Communities of Practice, 1–23. Retrieved from


Purpose and goals of evaluation:

Good evaluation can:

  • save community leaders time
  • help gain additional resources for their work
  • help allocate time and other resources most effectively
  • enable the community to better engage members and provide resources and activities that best meet their needs

Targeted evaluation techniques allow leaders to accomplish the following:

  • gauge the health of their communities
  • determine which parts of the communities are functioning well and which are underperforming
  • learn how to meet the needs of different types of participants and how to attract and retain more of them
  • track and report what is happening to their communities over time including content, activities and technical needs
  • demonstrate the impact of the community on attitudes, on practice and on student learning
  • compare their communities to others reaching the same or similar audiences
  • make a better objective case of value for funders

Evaluation plans should help community leaders to answer the following questions:

  • Is the community serving its intended purpose and audience?
  • Is knowledge around a domain and related practice emerging?
  • Do members’ interactions have continuity and depth? Are members engaged in productive, ongoing interactions?
  • Are collaborative activities emerging?
  • Is participants’ involvement in the community affecting their professional practices and student learning?

E.g. of a combination of evaluation methods:

  • Focus Groups to uncover ideas or concerns to test
  • Online Surveys to more broadly test member attitudes about those concerns or ideas
  • Community Content Analysis to see whether community members’ self-reported attitides are reflected in their actual on site behaviour

Questions that can be answered using basic community evaluation approaches:

  • What are community members doing in the community?
  • What are the popular trends in posts, blogs and forums?
  • What resources are being used?
  • What are the ongoing practices and processes that contribute to the “life” of the community and keep members engaged?
  • How is knowledge being shared within the community? Beyond the community?
  • Are leaders or roles emerging in the community? In what ways? How are they being cultivated?
  • How are members being supported in the community?
  • How are members contributing? Posting? Replying? (When? How often?)
  • What are the prevalent patterns of interactions?
  • How much of members’ online time is spent connecting to others in the community?
  • What are members technical issues?
  • What are members FAQs?

Site Analytics

  • evaluate traffic and popularity trends in their online communities of practice
  • free analytic applications include
    • Google Analytics
    • Crazy Egg
    • Firestats
    • Yahoo! Web Analytics
  • extended picture of trends in the community over time
  • analytic programmes can capture:
    • total number of topics reated
    • total number of blog posts
    • total nuber of photos and videos uploaded
    • total number of messages posted
    • total quantities of other forums user generated content
    • total participation in site polls and surveys
  • they can provide some useful overall information
    • total visits to the site
    • unique visitors
    • repeat visitors
    • number of registered participants/numbers
    • conversion rate – percentage of unique visitors who become registered members
    • total page views
    • average page views per visit
    • average time per visit or session
    • top landing pages, most requested pages, top paths
    • bounce and exit rates
    • top and total referrers
  • this allows the community leaders answer questions about the community’s performance over time:
    • What is the average number of new topics? (in the last 30-60 days)
    • What is the proportion of new topics that gets 10+ replies?
    • What proportions of new topics or new blog posts are un-responded or un-commented on?
    • are the numbers of participants in new site polls/surveys growing?
    • what is the average new topoic/reply/blog post created per member?
    • what is the proportion of grouops who have or have not posted in the last six weeks/months etc?
    • What proportion of groups have fewer than 5 or 10 members?

Online Surveys

Advantages of online surveys:

  • Direct Query
  • Segmentation
  • Broader Reach
  • Anonymity

Questions that online surveys can help to answer:

  1. What new features, functions or activities would be of greatest interest to community members?
  2. How has the community helped members solve problems of practice?
  3. How can the community attract new members and encourage existing members to participate more actively?
  4. What are the strengths and weaknesses of the community for different types of participants?
  5. What technical difficulties or issues are members experiencing?

Effective Online Surveys:

  • Survey length:
    • 5-10 minutes to complete
    • test it first with a few colleagues
  • Setting Expectations:
    • Tell them upfront how long it is likely to take
    • have some form of progress meter on each page so they know how much they have done and how much they have yet to complete
  • Varying Question Types:
    • use a variety of question styles
  • Multipoint Scales:
    • ask the participant to rate something or agree with a statement on a 1-5 or 1-10 scale
  • Skip Logic:
    • automatically routes specific respondents past questions that don’t apply to them
    • this increases the scope and depth of what the community survey can cover
  • Randomization:
    • randomizing the order or responses/choices a respondent sees for each question is often an important way to reduce order bias
  • Choices + “other” vs. open ended:
    • open ended question limit how many questions the community can be asked because they take longer to answer
    • only offer an open-ended option if there is a concern that a choice has been missed
  • Soliciting Interest in follow-up surveys:
    • at the end of the survey ask would they like to be part of a follow-up survey or an ongoing online panel for the community
  • Providing Incentives:
    • consider awarding each survey participant with community points or badges that hold significance within their community
    • or put all participants into a draw where they may be randomly selected to win a voucher or small prize
  • Number of respondents:
    • the larger the number of participants, the easier it is to compare findings
  • Representativeness:
    • start with screeners – questions about demography, education role, community role and community use to see how different kinds of members answer other survey questions is important
  • Sample Size:
    • Looking at 50-100 posts, topics, messages per month can provide potentially uselful information




Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s