Beverly Hills Community Survey: Where’s the Validity? [Updated]

Bevery Hils resident survey 2013 by True North ResearchWe were surprised to see a question about Santa Monica Boulevard bicycle lanes in the city’s recently-released resident survey (right). We got the heads-up from resident and cyclist Mel Raab; he eagle-eye spotted the single question buried deep into a too-lengthy questionnaire about city services. “Asked alone after some mind-numbing questions,” Mel said, “is whether I’d be in favor of losing 3′ to 5′ of Santa Monica for a bike lane.” Mel rightly wondered if that was the most constructive way of soliciting feedback about bike facilities in Beverly Hills. We agree.

When the draft “Resident Satisfaction Survey” questionnaire was presented to Council in study session recently, we noticed the question that addressed active mobility in Beverly Hills. We thought it a poor way to gauge public interest in transportation improvements that could encourage cycling, but we didn’t get a chance to provide feedback because the link to an online survey had already gone out. That’s when Mel contacted us.

Here’s how the only question about alternative means of mobility was presented to the survey-taker:

Resident satisfaction survey bicycle lanes questionThere are several problems with posing the question in this way. For one thing, the topic of cycling is introduced out of the blue. Cycling is a form of transportation as well as a leisure and sporting pursuit, but this question finds no home in a survey subsection on transportation or recreation. It is tucked into a ‘miscellaneous’ section – out of context. For the typical Beverly Hills respondent unaccustomed to thinking about cycling in any form, why should they care?

Second, this question about cycling is framed in a negative way. For why else would the survey designers condition respondents to think of bicycle-friendly improvements in terms of sacrifice? Indeed it seems designed not so much to gauge support for cycling as much as summon opposition to losing green space.  And given it’s the only question about cycling on this survey, why tie support for bicycle infrastructure generally to a tradeoff? Again so we’re back to the context problem: why would anyone trade away parkland or any other public good for some special interest giveaway like a bicycle lane?

After all, no question asks a respondent to trade pedestrian safety for diminished traffic flow, say. No question suggests that sidewalk or crosswalk improvements need come at the expense of some community amenity. For reasons unknown to us, city transportation staff can’t acknowledge that cycling is transportation. That’s why bicycle racks are viewed as a giveaway to cyclists and no sign hangs in Beverly Hills to alert motorists to their obligation to safeguard a cyclist’s safety.

Last, this question requires a binary answer: yes or no. There’s no opportunity for the respondent to communicate the strength of her preference either way, much less to suggest how she would prioritize investments in cycling infrastructure relative to anything else. (Most survey questions here ask the respondent to reflect on the strength of her  preferences.)

Constructing the Question to Solicit the Desired Response?

Without context and with a negative frame around cycling, the question “Would you trade parkland for bicycle lanes?” is likely to solicit a “No, don’t add bike lanes.” Is that the answer the city seeks? Since we first started jawboning with transportation staff about our outdated bike plan and bike facilities, they’ve bobbed and weaved about bicycle lanes for Santa Monica. There’s no interest in widening the boulevard, we’re told. And staff has always mentioned putting a westbound lane on another corridor, namely Carmelita (until residents came out against that plan last year). Anything to not have to make a case for a few extra feet for the regional ‘backbone’ bike route long missing from the Westside.

So it’s not surprising that the question was offered without context, and then biased by the addition of this phrase to the ‘no’ answer: “Keep green space.” Heck, aside from a few bike advocates, who would elect to trade away greenspace without understanding the benefit of a bicycle lane?

Yet it matters because findings from this key question will likely inform policymakers as they evaluate Santa Monica Boulevard reconstruction design alternatives. And the respondents’ preference will undoubtedly make its way into discussion during the SM Blvd blue-ribbon committee meetings this fall. With so much on the line, we should ask, Who worded the question this way? The obvious answer is the consultant, True North Research, but this hired gun specializes in public-sector temperature-taking and, as we know (and they know), how a question is worded has a great effect on responses. Very likely city staff discussed the wording of the question with them.

Are we reaching here? Take a counter-example. The preceding question (#13) in the ‘traffic’ section gets it right. Note the introduction that frames the issue:

Resident satisfaction survey question 13The introduction establishes a context for the question, “Are there additional actions that the city should be taking….?” And it doesn’t beg a binary response. Indeed this question is open-ended: Tell us what you wish, the city asks the respondent. So it seems that on issues of traffic management the city is all ears; but about active-transportation bike-friendly actions, maybe not so all-ears.

It’s curious because the stuff of a proper introduction to Q14 is there: the city can point to its own meager effort under the Pilot program to argue that bike improvements have been made. Crescent and Burton Way now both have several segments of bicycle lane, for example, so why not introduce the question about bicycle lanes this way: “The City has taken some actions to improve the safety of cyclists…” and then get on to the question about support for bicycle lanes on Santa Monica Boulevard.

Were the survey designers interested in an accurate and informed reflection about infrastructure like bicycle lanes that protects cyclists and encourages cycling, they could have phrased this question very differently.

The crux of the problem with survey question #14 is validity. Simply put, is the question measuring what we want to measure? Or is it designed to elicit a negative response about bicycle lanes? (A related issue is to what extent the responses are representative of the entire city. Our household didn’t receive a letter with a survey web link. How many did? How were they chosen?)

How Could Q14 Have Been Worded?

We would suggest a multiple-part question that properly frames the issue in terms of mobility, then asks generally about respondent support for cycling infrastructure, and then finally moves on to introduce bicycle lanes and asks about support for incrementally expanding Santa Monica Boulevard to accommodate dual bicycle lanes.

For example, here is a preface that suggests cycling is a legitimate form of transportation and that the city bears responsibility to plan for safe cycling (no differently than we do for driving and walking). Here we also ask about overall support:

14a. Beverly Hills city plans identify multimodal mobility as a policy goal, with an emphasis on encouraging walking, cycling and use of public transit. How strongly do you feel about the city creating additional facilities to improve cycling circulation?

Response choices could be presented in Likert-scale fashion: very strongly, somewhat strongly, not very strongly, not strongly at all. A follow up question could address bicycle lanes generally:

14b. Active transportation advocates say separate, on-street bicycle lanes increase protection for cyclists while encouraging less-experienced cyclists to ride more. Would you support the creation of additional bicycle lanes in Beverly Hills?

And then finally, with the mobility context established and the respondent conditioned to think of bicycle lanes as a transportation measure, she is asked about the specific tradeoff as stated in Q14:

14c. “…Would you support adding bike lanes to Santa Monica Boulevard if it requires removing 3 to 5 feet of green space along Beverly Gardens Park?

In this survey, we don’t see respondents asked about investments in other city services at the expense of the city’s signature park. So why attach baggage to a pro-safety improvement like bicycle lanes? Moreover, as important as mobility and safety are to our community, this survey asks for no feedback about how our city could better protect pedestrians and cyclists. Of the 23 city services we ask respondents to evaluate, “managing traffic congestion” makes the cut but not any service related to people circulation.

We can and should do better. With the public outreach portion of the Santa Monica Boulevard reconstruction process underway this fall, policymakers need all of the unbiased and valid feedback from the public they can get. That not only includes the cycling community, but all of us interested in safe streets, safe crossings, and yes, safe conditions for cyclists and motorists alike. Unfortunately, this survey simply doesn’t provide that feedback.

Update: The results are in! According to the staff report: “There is a slight advantage (49%) in those who prefer to keep the 3 to 5 feet of green space along Santa Monica Blvd. rather than add bike lines (44%).” The surprise is that 44% of Beverly Hills residents actually spoke up in favor of bike lanes given how this question was presented.