Connected Educator, & Summit, P. (2011). Resources and Tools for Evaluation of Online Communities of Practice, 1–23. Retrieved from http://lincs.ed.gov/professional-development/resource-collections/profile-535
Purpose and goals of evaluation:
Good evaluation can:
- save community leaders time
- help gain additional resources for their work
- help allocate time and other resources most effectively
- enable the community to better engage members and provide resources and activities that best meet their needs
Targeted evaluation techniques allow leaders to accomplish the following:
- gauge the health of their communities
- determine which parts of the communities are functioning well and which are underperforming
- learn how to meet the needs of different types of participants and how to attract and retain more of them
- track and report what is happening to their communities over time including content, activities and technical needs
- demonstrate the impact of the community on attitudes, on practice and on student learning
- compare their communities to others reaching the same or similar audiences
- make a better objective case of value for funders
Evaluation plans should help community leaders to answer the following questions:
- Is the community serving its intended purpose and audience?
- Is knowledge around a domain and related practice emerging?
- Do members’ interactions have continuity and depth? Are members engaged in productive, ongoing interactions?
- Are collaborative activities emerging?
- Is participants’ involvement in the community affecting their professional practices and student learning?
E.g. of a combination of evaluation methods:
- Focus Groups to uncover ideas or concerns to test
- Online Surveys to more broadly test member attitudes about those concerns or ideas
- Community Content Analysis to see whether community members’ self-reported attitides are reflected in their actual on site behaviour
Questions that can be answered using basic community evaluation approaches:
- What are community members doing in the community?
- What are the popular trends in posts, blogs and forums?
- What resources are being used?
- What are the ongoing practices and processes that contribute to the “life” of the community and keep members engaged?
- How is knowledge being shared within the community? Beyond the community?
- Are leaders or roles emerging in the community? In what ways? How are they being cultivated?
- How are members being supported in the community?
- How are members contributing? Posting? Replying? (When? How often?)
- What are the prevalent patterns of interactions?
- How much of members’ online time is spent connecting to others in the community?
- What are members technical issues?
- What are members FAQs?
- evaluate traffic and popularity trends in their online communities of practice
- free analytic applications include
- Google Analytics
- Crazy Egg
- Yahoo! Web Analytics
- extended picture of trends in the community over time
- analytic programmes can capture:
- total number of topics reated
- total number of blog posts
- total nuber of photos and videos uploaded
- total number of messages posted
- total quantities of other forums user generated content
- total participation in site polls and surveys
- they can provide some useful overall information
- total visits to the site
- unique visitors
- repeat visitors
- number of registered participants/numbers
- conversion rate – percentage of unique visitors who become registered members
- total page views
- average page views per visit
- average time per visit or session
- top landing pages, most requested pages, top paths
- bounce and exit rates
- top and total referrers
- this allows the community leaders answer questions about the community’s performance over time:
- What is the average number of new topics? (in the last 30-60 days)
- What is the proportion of new topics that gets 10+ replies?
- What proportions of new topics or new blog posts are un-responded or un-commented on?
- are the numbers of participants in new site polls/surveys growing?
- what is the average new topoic/reply/blog post created per member?
- what is the proportion of grouops who have or have not posted in the last six weeks/months etc?
- What proportion of groups have fewer than 5 or 10 members?
Advantages of online surveys:
- Direct Query
- Broader Reach
Questions that online surveys can help to answer:
- What new features, functions or activities would be of greatest interest to community members?
- How has the community helped members solve problems of practice?
- How can the community attract new members and encourage existing members to participate more actively?
- What are the strengths and weaknesses of the community for different types of participants?
- What technical difficulties or issues are members experiencing?
Effective Online Surveys:
- Survey length:
- 5-10 minutes to complete
- test it first with a few colleagues
- Setting Expectations:
- Tell them upfront how long it is likely to take
- have some form of progress meter on each page so they know how much they have done and how much they have yet to complete
- Varying Question Types:
- use a variety of question styles
- Multipoint Scales:
- ask the participant to rate something or agree with a statement on a 1-5 or 1-10 scale
- Skip Logic:
- automatically routes specific respondents past questions that don’t apply to them
- this increases the scope and depth of what the community survey can cover
- randomizing the order or responses/choices a respondent sees for each question is often an important way to reduce order bias
- Choices + “other” vs. open ended:
- open ended question limit how many questions the community can be asked because they take longer to answer
- only offer an open-ended option if there is a concern that a choice has been missed
- Soliciting Interest in follow-up surveys:
- at the end of the survey ask would they like to be part of a follow-up survey or an ongoing online panel for the community
- Providing Incentives:
- consider awarding each survey participant with community points or badges that hold significance within their community
- or put all participants into a draw where they may be randomly selected to win a voucher or small prize
- Number of respondents:
- the larger the number of participants, the easier it is to compare findings
- start with screeners – questions about demography, education role, community role and community use to see how different kinds of members answer other survey questions is important
- Sample Size:
- Looking at 50-100 posts, topics, messages per month can provide potentially uselful information