Survey numbers don’t add up

May 29, 2014 09:46 AM

Re: District of Sechelt satisfaction survey


In response to Christine Wood’s summary in the May 16 Coast Reporter, I read the full report on the satisfaction survey of 300 Sechelt residents.

This survey was contracted by the District of Sechelt to Innovative Research Group in Toronto. With a PhD in research/evaluation methods, I found the methodology to be unsound, with so many basic errors in analysis and reporting that it is difficult to have much confidence in the findings.

First, many survey questions are poorly worded, too long, contain too much information, and/or are inconsistent with the response choices. It is hard to imagine how anyone could listen to some, let alone all, of these questions, and their response choices, over the phone without falling asleep.

Second, we do not know the response rate. How many people responded to each survey question (N=?)? How many people chose each response option (N=?)? This information needs to be provided for every survey question. Reporting percentages alone is not good enough.

Third, no statistically significant differences, p-values, or effect sizes are reported, so any group differences, e.g. across years, may be insignificant or due to chance.

As a result, conclusions like “More are satisfied with efforts to reduce traffic congestion than dissatisfied,” actually tell us very little.
Fourth, a regression model might be more useful than an “imputed importance” matrix without Ns.

Finally, I have a few questions: How much was spent on this report? Why are our tax dollars being spent to purchase a report of this quality from a firm in Toronto?

Valerie Ruhe, Sechelt 

© Copyright 2015 Coast Reporter


NOTE: To post a comment you must have an account with at least one of the following services: Disqus, Facebook, Twitter, Google+ You may then login using your account credentials for that service. If you do not already have an account you may register a new profile with Disqus by first clicking the "Post as" button and then the link: "Don't have one? Register a new profile".

The Coast Reporter welcomes your opinions and comments. We do not allow personal attacks, offensive language or unsubstantiated allegations. We reserve the right to edit comments for length, style, legality and taste and reproduce them in print, electronic or otherwise. For further information, please contact the editor or publisher, or see our Terms and Conditions.

comments powered by Disqus

Email to a Friend



Should BC Ferries be returned to a Crown corporation?

or  view results

click here to read the editorial

Popular Letters