This is “How Do We Know What Worked? Evaluating Ad Executions”, section 11.2 from the book Advertising Campaigns: Start to Finish (v. 1.0). For details on it (including licensing), click here.

For more information on the source of this book, or why it is available for free, please see the project's home page. You can browse or download additional books there. To download a .zip file containing this book to use offline, simply click here.

Has this book helped you? Consider passing it on:
Help Creative Commons
Creative Commons supports free culture from music to education. Their licenses helped make this book available to you.
Help a Public School
DonorsChoose.org helps people like you help teachers fund their classroom projects, from art supplies to books to calculators.

11.2 How Do We Know What Worked? Evaluating Ad Executions

Learning Objectives

After studying this section, students should be able to do the following:

  1. Summarize how advertisers evaluate ad executions.
  2. Explain how copy research is conducted.
  3. Illustrate how pretesting and posttesting of advertisements takes place.

Recall and Recognition

Executing advertising effectively requires that consumers notice the ad, recall the brand, and remember it favorably when they make a purchase decision. RecallViewers can remember and retell the specific marketing messages to which they were exposed. means that viewers can remember and retell the specific marketing messages to which they were exposed. RecognitionViewers recognize the brand or message when they see or hear it again. means they recognize the brand or message when they see or hear it again, even if they can’t recite it from memory.

Because marketers pay so much money to place their messages in front of consumers, they are naturally concerned that people will actually remember these messages at a later point. It seems that they have good reason to be concerned. In one study, fewer than 40 percent of television viewers made positive links between commercial messages and the corresponding products, only 65 percent noticed the brand name in a commercial, and only 38 percent recognized a connection to an important point.“Only 38% of T.V. Audience Links Brands with Ads,” Marketing News, January 6, 1984, 10.

Ironically, we may be more likely to remember companies that we don’t like—perhaps because of the strong negative emotions they evoke. In a 2007 survey that assessed both recall of companies and their reputations, four of the ten best-remembered companies also ranked in the bottom ten of reputation rankings: Halliburton Co., Ford Motor Co., General Motors Corp., and Exxon Mobil Corp. In fact, Halliburton, with the lowest reputation score, scored the highest media recall of all the sixty companies in the survey.Ron Alsop, “News, Ads Shape Corporate Images,” Wall Street Journal Online, January 31, 2007, http://online.wsj.com, by paid subscription (accessed October 15, 2007).

Metrics related to recall and recognition ignite controversy even among agencies themselves. For example, Carat Insight uses recognition techniques rather than recall. Mary Jeffries, the agency’s head of evaluation, explains: “Most research techniques have relied on consumers’ ability to remember advertising messages and they then use this as a proxy for effectiveness. This means that media such as radio, outdoor, press, cinema and online suffer terribly, as they do not get recalled. Our belief is that ads can work even if you can’t spontaneously recall them. This is why [we] use a recognition technique, which is a more accurate measure of likely exposure to advertising than recall.” Carat Insight provides a service it calls integrated communications evaluation (ICE), which uses recognition techniques and statistical modeling to identify the relationship between media channels and creative executions.“Marketing League Table,” Marketing, September 5, 2007, 35.

In contrast, Intermedia Advertising Group is a research firm that measures advertising effectiveness by monitoring the TV-viewing population’s ability to remember an ad within twenty-four hours. The firm assigns a recall index to each ad to indicate the strength of the impact it had. In one recent year, while ads with well-known celebrities like Britney Spears, Austin Powers, and Michael Jordan had very high recall rates, three of the top five most remembered ads starred another (and taller) celebrity: Toys “R” Us spokesanimal Geoffrey the Giraffe.Vanessa O’Connell, “Toys ‘R’ Us Spokesanimal Makes Lasting Impression: Giraffe Tops List of Television Ads Viewers Found the Most Memorable,” Wall Street Journal Interactive Edition, January 2, 2003.

Under some conditions, these two memory measures tend to yield the same results, especially when the researchers try to keep the viewers’ interest in the ads constant.Richard P. Bagozzi and Alvin J. Silk, “Recall, Recognition, and the Measurement of Memory for Print Advertisements,” Marketing Science 2 (1983): 95–134. Generally, though, recognition scores tend to be more reliable and do not decay over time the way recall scores do.Adam Finn, “Print Ad Recognition Readership Scores: An Information Processing Perspective,” Journal of Marketing Research 25 (May 1988): 168–77. Recognition scores are almost always better than recall scores because recognition is a simpler process and the consumer has more available retrieval cues.

Both types of retrieval play important roles in purchase decisions, however. Recall tends to be more important in situations in which consumers do not have product data at their disposal, so they must rely on memory to generate this information.James R. Bettman, “Memory Factors in Consumer Choice: A Review,” Journal of Marketing (Spring 1979): 37–53. On the other hand, recognition is more likely to be an important factor in a store, where retailers confront consumers with thousands of product options (i.e., external memory cues are abundantly available), and the goal is simply to get the consumer to recognize a familiar package.Mark A. Deturck and Gerald M. Goldhaber, “Effectiveness of Product Warning Labels: Effects of Consumers’ Information Processing Objectives,” Journal of Consumer Affairs 23, no. 1 (1989): 111–25.

SS+K Spotlight

SS+K and msnbc.com also wanted to be able to measure the effects of the first effort. All parties agreed that given the size of the audience and the budget, the expectation was not to convert a huge number of people but rather to articulate the brand to the target audience. Michelle Rowley and John Richardson led the research effort by enlisting a firm called Russell Research to conduct surveys before the launch and then again after the launch to be able to understand any changes. Catherine Captain’s research background also came in quite handy here, as all agencies worked together to set up the appropriate parameters. We’ll reveal the results of this research in Chapter 14 "ROI: msnbc.com Decides if the Campaign Worked".

The Stopping Power of Creative Ads: Are They Effective, or Just Cool?

Other agencies maintain that above all, the ad must get noticed. And very creative ads do get noticed—they break through the clutter. Ads that win creative awards have twice the “stopping power” of regular non-award-winning ads. They get your attention. Moreover, award-winning ads create buzz. Even after two decades, people still talk about Apple’s “1984” ad.

Video Highlight

Apple’s “1984” Ad

(click to see video)

This commercial, which aired during the 1984 Super Bowl, is an example of a breakthrough creative message.

But, although they are more entertaining, creative ads also can confuse the very people they’re intended to persuade. Sometimes a clever ad can be too hip for its own good. Research on award-winning ads finds that consumers are more likely to say “I couldn’t tell what that brand had to do with what was said and shown.” This means that executions may require tweaks (which copy testing can guide) so that the ads are able to generate sales for the brand as well.Charles Young and Larry Cohen, “Creative Awards vs. Copytesting,” Quirk’s Marketing Research Review, April 2004.

Cheer up: advertisers do not have to simply sit back and hope for the best. By being aware of some basic factors that increase or decrease attention, they can take steps to increase the likelihood that product information will get through. An advertiser who wants to wake people up can:Parts of this section are adapted from Michael R. Solomon, Consumer Behavior: Buying, Having and Being, 8th ed. (Upper Saddle River, NJ: Prentice Hall, 2009). Cf. also David W. Stewart and David H. Furse, “Analysis of the Impact of Executional Factors in Advertising Performance,” Journal of Advertising Research 24 (1984): 23–26; Deborah J. MacInnis, Christine Moorman, and Bernard J. Jaworski, “Enhancing and Measuring Consumers’ Motivation, Opportunity, and Ability to Process Brand Information from Ads,” Journal of Marketing 55 (October 1991): 332–53.

  • Use novel stimuli, such as unusual cinematography, sudden silences, or unexpected movements. When a British online bank called Egg Banking introduced a credit card to the French market, its ad agency created unusual commercials to make people question their assumptions. One ad stated, “Cats always land on their paws,” and then two researchers in white lab coats dropped a kitten off a rooftop—never to see it again (animal rights activists were not amused).Elaine Sciolino, “Disproving Notions, Raising a Fury,” New York Times on the Web, January 21, 2003, http://www.nytimes.com/2003/01/21/business/media/21ADCO. html?ex=1234501200&en=eafd1f9635946454&ei=5070 (accessed February 10, 2009).
  • Use prominent stimuli, such as loud music and fast action, to capture attention. In print formats, larger ads increase attention. Also, viewers look longer at colored pictures than at black-and-white ones.

New Ideas Support New Brand Launches

Attention-getting ads are particularly valuable when the communication objective is to help launch a new brand by boosting awareness and generating buzz. Apple’s “1984” ad is a case in point; the classic spot elevated the Apple brand from simply a utilitarian message (how a computer makes you productive) to an icon representing an attitude and point of view. Before the breakthrough “1984” ad, Apple’s TV commercials used slice-of-life and problem-solution frameworks. The “1984” commercial—shown only during the Super Bowl—created huge buzz for its allegory and cinematic distinctiveness. It created a position for Apple as revolutionary, liberating—much more strongly than a recounting of Macintosh’s user-friendly features would have done.

Copy Research

Copy researchEvidence that an ad gets the audience’s attention and delivers a message; effectiveness is judged in pretesting and posttesting and is gauged by attention, branding, and motivation. provides evidence that your ad gets the audience’s attention and delivers a message that motivates the consumer to consider buying your product or service. The overall effectiveness of an ad is a combination of three variables:

  1. Attention: Entertainment value is a major predictor of attention-getting power, but if consumers don’t see the connection of the ad to the brand, the ad won’t lead to a sale.
  2. Branding: Communicating an idea or feeling that the consumer already has about the brand confirms the value of reminder advertising. Even better is advertising that communicates a new idea or a new feeling, but one that still fits the brand in the eyes of the consumer. This kind of advertising helps consumers to see the brand in a new light, to think about it in a new way.
  3. Motivation: Finally, an effective ad makes the viewer want to take action and buy the product. Pretesting asks the test subject whether they are more likely to buy the product now or in the future.

Pretesting and Posttesting

Copy research involves two phases: pretesting and posttesting. PretestingIn copy research, consumer testing that takes place before the campaign starts, aimed at identifying any weaknesses and ensuring that the right message is coming across. takes place before the campaign starts. PosttestingIn copy research, consumer testing that takes place after the campaign, aimed at evaluating the effectiveness of the copy in communicating its message. takes place after the campaign, to evaluate the effectiveness of the copy in communicating its message.

The idea behind pretesting is to verify that the product claims and technical aspects of the ad resonate with the target audience. Pretesting also identifies weak spots within an ad campaign. Pretesting can also be used to edit a longer commercial into a shorter one, or to select images from the spot to use in an integrated campaign’s print ad.Charles E. Young, The Advertising Research Handbook (New York: Ideas in Flight, 2005), 27–30. This process often involves asking consumers to place the ad into one of these categories based on their perception of the brand:

  • Reinforcement: The ad fits the way I already think and feel about the brand.
  • Augmentation: The ad gives me a new idea or feeling toward the brand, and I can see how it fits the brand.
  • Dissonance: The ad does not fit the brand at all.Charles E. Young, “Finding the Creative Edge: Research as Flow,” Admap, December 2006.

Copy Testing: Comparing Appeals

A related term, copy testingTesting that is done before launching the campaign, comparing one type of execution over another, or one kind of product feature, benefit, or price over another; it aims to fine-tune the ad., refers to testing one type of execution over another, or one kind of product feature, benefit, or price over another. Copy testing is done before launching the campaign to fine-tune the ad to be most effective.

Copy research involves assessing that the consumer noticed the ad, was able to recall the brand name, learned something about the brand, and became favorably disposed to trying or buying the product. Companies like Ameritest, Anderson Analytics, and Millward Brown specialize in providing copy testing and related research to ad agencies and advertisers.

The Starch testThe product of a research service founded in 1932 by Daniel Starch; a widely used commercial measure of advertising recall for magazines that provides scores on a number of aspects of consumers’ familiarity with an ad., the product of a research service founded in 1932, is a widely used commercial measure of advertising recall for magazines. This service provides scores on a number of aspects of consumers’ familiarity with an ad, including such categories as “noted,” “associated,” and “read most.” It also scores the impact of the component parts of an overall ad, giving such information as “seen,” for major illustrations, and “read some,” for a major block of copy.Adam Finn, “Print Ad Recognition Readership Scores: An Information Processing Perspective,” Journal of Marketing Research 25 (1988): 168–77. Factors such as the size of the ad, whether it appears toward the front or the back of the magazine, if it is on the right or left page, and the size of illustrations play an important role in affecting the amount of attention readers give to an ad.

Dig Deeper

Believe it or not, only 7 percent of television viewers can recall the product or company featured in the most recent television commercial they watched. This figure represents less than half the recall rate recorded in 1965. We can explain this drop-off in terms of such factors as the increase of thirty- and fifteen-second commercials and the practice of airing television commercials in clusters rather than in single-sponsor programs.“Terminal Television,” American Demographics (January 1987): 15.

Television commercials tell a visually compelling story with moving pictures. During a TV commercial, the audience’s feelings change as they move through the film. Copy research company Ameritest calls this the “flow of emotion” and uses it as a measurement device based on frame-by-frame testingTechnique of showing consumers a deck of photographic images—created by grabbing key frames from the commercial—that represent the visual content of the ad and having them sort by negative to positive feelings and how much it made them think about the brand.. This technique involves taking a deck of photographic images—created by grabbing key frames from the commercial—that represent the visual content of the ad. Consumers sort the images into a one-to-five scale from “very negative” to “very positive” feelings. The number of frames in a test varies with the visual complexity of the ad rather than its length. A typical thirty-second commercial will break down into about ten to thirty frames for viewers to evaluate. The resulting sort by the consumer shows how (or whether) their emotional response changed during the commercials. Frames can also test whether the commercial prompted the viewer to think about the brand (on a one-to-five scale from “did not make me think” to “made me think a lot”).Charles E. Young and John Kastenholz, “Emotion in TV Ads,” Admap, January 2004.

Creativity versus Safety

Many creatives believe that testing a campaign will drain the creativity from the campaign—that the only messages audiences will “approve” will be those that are safe and predictable, and hence, boring. Advertising legend David Ogilvy, however, disagreed. Near the end of his career he commented, “Most creative people detest research, and I’ve never understood why.…In my day, I used research very often to give me the courage to run campaigns that were risky.”

In fact, copy research can actually give you the evidence to go with a radical or risky idea that company executives might not have approved otherwise. Boring ads that don’t tell the consumer anything new aren’t very effective. The most effective ads are those that stretch the meaning of the brand in the mind of the consumer. That is, the consumer learns something new about the brand, or the ad pushes the frontier of the brand. An effective ad is neither too far removed from the brand nor too staid.

Key Takeaway

The harsh reality is that consumers don’t remember the large majority of advertising messages they see or hear. And if they do recall an ad, this doesn’t mean they’ll associate the image with the brand. Even slight differences in the elements of an ad influence its effectiveness (for example, the colors or fonts in a print ad). Careful pretesting increases the odds that a message will accomplish its objective. Copy research provides evidence that your ad gets the audience’s attention and delivers a message that motivates the consumer to consider buying your product or service. Advertisers typically try to determine if people can recall an ad’s contents, or at least recognize it when they see or hear it again. Both measures have their supporters; overall recognition is used more widely. Novel or innovative ads are most effective when the objective is to create buzz or brand awareness, but more straightforward executions do a better job when the objective is to deliver information or move consumers from one well-known brand to another.

Exercises

  1. Define recall and recognition.
  2. Explain how the stopping power of creative ads can be increased.
  3. List and describe the three variables that contribute to overall ad effectiveness.
  4. Explain pretesting and posttesting.
  5. Describe how a Starch test is conducted.