Second Life Resident Answers official forum gets, on average, two attempts to run a survey per week. Is it a good idea? Oh how wrong it is, let me count the ways…
- For survey results to actually mean anything, you need to know just how many people could have responded but did not — this lets you tell just how reliable your results are, statistically. In Second Life you do not. The actual number of active users is not released by Linden Labs and not obtainable without their assistance. The online count is not clearly indicative of the number of active users because of large populations of bots, used to inflate traffic statistics, and the amount of bots actually out on the grid is a very interesting research question.
- In case of Second Life, forum population is a minority, consisting of largely vocal and active residents, and it’s attitudes and preferences as well as any measurements taken from them have a certain likelihood of not reflecting the same about Second Life as a whole at all. It could be argued that they’re the ones that matter, and that is probably true, however, if this is not taken into account, the result of any survey is essentially void.
- The particular deeply invested residents inhabiting the forums have a well-defined, complex and diverse culture of their own, with it’s own slang and cultural standards, overlaid and mixed with national and professional subcultures. Finding questions that would be meaningful to them and properly wording them is it’s own challenge that requires a researcher to be determined. The approach of addressing them as an average consumer, or average American, or any other kind of average simply doesn’t work.
- There is no reliable way to guard against deliberate sabotage of an internet survey, particularly with a group that is on average more technologically qualified than much of the rest of the Internet and is hostile to surveys in general because they often treat them as deviants.
- Voluntary surveys are in general an iffy research method for reasons described elsewhere. It works if your respondents don’t recognise themselves as part of a group with defined interests and goals. Otherwise, they will fudge it to the best of their ability, see Hawthorne effect.
That is not to say a survey is completely unusable, but unless all these limitations are understood and taken into account for, it’s pure and simple hogwash.