Obtaining Feedback with Qualitative Research

Qualitative researchInvolves interactions, which by their very nature are subject to interpretation and, as a result, are less reliable and statistically valid. involves investigative methods that cross subjects and academic disciplines to gain in-depth information. If quantitive research explores “what,” qualitative research explores “how” and “why.” From interviews to focus groups, many of the face-to-face strategies used to gather information are qualitative in nature.

You have five senses, and you may be able to distinguish between sweet and salty foods, but can you describe what you taste and smell? Let’s say you work for a vineyard, and have been tasked to write a paragraph describing a new wine. Could you? Capturing fine data points and representing them in words and symbols can be a significant challenge for researchers. When testing the wine with a focus group, you might want information on how it is perceived, and the responses may be varied and unusual. What do you do with the information you gather? You may be able to identify trends among the varied responses, and create groups that indicate a woody or earthy flavour, but numbers will fail to capture the nuances of flavour and body of the wines in the information.

Some information—like the way consumers characterize the taste of a wine—is a challenge to obtain, and qualitative research often serves well in this capacity. If quantitative research handles large audiences well, qualitative research allows for in-depth interpersonal interviews that produce rich and meaningful results. The information may not be as reliable, and your ability to produce the same results over time may be limited, but humans are emotional, irrational, and unpredictable. They are also, each in his or her own way, unique. As you increase the level of perspective in terms of abstraction, all humans may eventually come to look similar, even the same. We all possess some similar characteristics, such as the use of language, or the composition of our bodies. But when you look more closely, you see the diverse range of languages, and learn that not everyone has 206 bones in an adult body. Between these two views we find the range of information that quantitative and qualitative research attempt to address.

Suppose we want to determine who has greater lifetime risk of developing heart disease, a man or a woman? If we are talking about an individual man and an individual woman, our answer might be quite different from what it would be if we were talking about men in general versus women in general. A survey may work well to capture the data about men versus women, but a face-to-face interview with a man and a woman will allow for interaction, follow-up questions, and a much better picture of the question: between this individual woman and this individual man, who is more likely to be at risk? The risk and protective factors we learn from broad research projects involving thousands of subjects have value, but there are times when a broad brushstroke will fail to capture the fine data that is needed or desired.

Imagine that you are involved with a direct observation of buying behaviour by reviewing video recordings of security cameras that clearly show your company’s product in relation to other products on the shelf. You may find, particularly after a review of the literature, that product placement makes a significant impact on purchase decisions. In addition, you may be involved with some level of participation in the setting. Serving as a participant observer means you are part of the process, involved in action, and not separated from the interaction. You look at the sales experience through the eyes of a participant, and view others through the eyes of an observer. You may find that interviews and focus groups serve to teach you more about your audience, but may also find that others have conducted similar interviews and learn from their findings.

As a business writer, you should be familiar with qualitative research and its relative strengths and weaknesses. You may use some of its techniques to gather information about your audience, may cite research that involves qualitative methods, and may utilize its strategies with an audience post document, product, or service.

Obtaining Feedback with Quantitative Research

Quantitative researchInvestigation and analysis of data and relationships between data that can be represented by numbers. involves investigation and analysis of data and relationships between data that can be represented by numbers. It is often used to test a hypothesis, and normally involves large volumes of data. Where a qualitative research project may involve a dozen interviews, a quantitative one would involve hundreds or thousands. Since each interview carries a cost—and a thousand or ten thousand interviews may exceed the research budget of your organization—a more cost-effective alternative must be found. By limiting the number of questions and limiting the ways in which participants can respond, the data can be gathered at a lower cost with often a higher level of statistical validity.

In qualitative research, you may ask an open-ended question like “What does the wine taste like?” In quantitative research, you may limit the response options: “Does the wine taste (a) woody, (b) fruity, or (c) both?” You may find that 90 percent of respondents indicate answer (c); you can represent it with numbers and a graph, but it may not serve your investigation the way you planned.

Research methodologiesThe methods used in investigation. involve examining and evaluating the methods used in investigation or soliciting feedback. They are used to address and improve poorly worded questions, and to help the investigator match the research goal to the method. Quantitative research serves us well when we ask, does vitamin C, taken at a dosage of 500 mg daily for five years, lower the incidence of the common cold? We could track a thousand participants in the study who provide intake prescreening information, confirm daily compliance, and participate in periodic interviews. We also know that part of our group is taking a placebo (sugar pill) as part of the requirements of a double-blind study. At the end of the term, we have certain numbers that may be able to indicate the degree to which vitamin C affects the incidence rate of illness.

Figure 8.3

Audience research is critical to success.

© 2010 Jupiterimages Corporation

Advertisers often conduct research to learn more about preference and attitudes, two areas that are not easily captured. Sometimes preference studies use Likert scales, which give respondents a preset scale to rate their answers. An example of a Likert item might be, “Please indicate to what degree you agree or disagree with this statement: I enjoy drinking brand X wine. Do you (1) strongly agree, (2) agree, (3) neither agree nor disagree, (4) disagree, or (5) strongly disagree?”

There is a tendency for some attitudinal and preferential research that may be more accurately described as qualitative, to be described in numerical terms. For example, you have probably heard the claim that “four out of five dentists prefer brand X,” when in itself, the number or representation of preference is meaningless. As an astute business writer, you will be able to understand pre (before) and post (after) document, product, or service research investigations and distinguish between the two main approaches.

What Is Validity?

How do you know the results presented in a study or article have value? How do you know they are valid? ValidityThe extent to which a point, concept, conclusion, or indication of measurement is accurate based on rigourous examination. involves the strength of conclusions, inferences or assertions. Thomas Cook and Dan CampbellCook, T. D., & Campbel, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago, IL: Rand McNally. indicate that validity is often the best available approximation of the truth or falseness of an inference, proposition, or conclusion. Readers want to know that your information has value and that there is confidence in its points, supporting information, and conclusions. They want to know you are right and not making false statements.

One way you can address the value of validity is to cite all your sources clearly. As a writer, you may certainly include information from authorities in the field when the attribution is relevant and the citation is clear. Giving credit where credit is due is one way to make your information more valuable, and by referencing the sources clearly, you enable the reader to assess the validity of the information you have provided.

Does all feedback have validity? Just as there are many threats to validity in research applications, you cannot always be sure that the feedback you receive is accurate or truthful. Have students ever evaluated professors negatively because of the required work in the course? Of course. In the same way, some readers may have issues with the topic or your organization. Their feedback post may be less than supportive, and even openly hostile. Assess the validity of the feedback, respond with professionalism at all times, and learn how to let go of the negative messages that offer little opportunity to improve understanding.

What Is Reliability?

ReliabilityThe extent to which an assessment instrument measures the same way each time given the same variables, circumstances, or conditions. is the consistency of your measurements. The degree to which an instrument gives the same measurement each and every time with the same subjects, in the same context, is a measure of its reliability. For example, if you took your temperature three times within fifteen minutes, and your thermometer gave a different reading each time—say, 37.4, 37.8, and 37.0 —you would conclude that your thermometer was unreliable.

How does this apply to feedback in business writing? Let’s say you have three sales agents who will complete follow-up interactions with three customers after you have sent a report to each customer on their purchases to date with suggestions for additional products and services. All three sales agents have the same information about the products and services, but will they perform the same? Of course not. Each one, even if they are trained to stay on script and follow specific protocols, will not be identical in their approach and delivery. Each customer is also different, so the context is different in each case. As business professionals, we need to learn about our environment and adapt to it. This requires feedback and attention to the information in many forms. We need to assess the degree of strength or weakness of the information, its reliability, or validity, and be prepared to act on that information. Successful businesses, and by extension successful business communicators, recognize that communication is a two-way process in which we need to listen, learn, and respond to feedback. We need to meet and exceed the expectations of our customers.

Inter-rater reliabilityThe degree to which each evaluator evaluates the same in similar contexts. involves the degree to which each evaluator evaluates the same in similar contexts. One can think of a college essay, for example, to better understand this concept. Let’s say you write an essay on customer relationship management and submit it to the instructor of your business communication class. At the same time, you submit the same essay to your personal selling professor, and you submit a copy to your marketing professor. Will all three professors evaluate your essay the same? Of course not. They will each have their own set of expectations and respective disciplines that will influence what they value and how the evaluate. Still, if your essay is thoroughly researched, logically organized, and carefully written, each professor may give it a better than average grade. If this is the case, inter-rater reliability would indicate that you did a good job on the essay.

What Is Statistically Significant?

This is a research term that is often used and commonly misunderstood. Not every research finding is statistically significant, and many of those that are considered significant are only slightly more likely than pure chance. Statistically significant findingsConclusions that have a high level of reliability, in that if the same test is applied in the same context to the same subjects, the results will come out the same time and time again. are those that have a high level of reliability, in that if the same test is applied in the same context to the same subjects, the results will come out the same time and time again.Stone-Romero, E. F. (2002). The relative validity and usefulness of various empirical research designs. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psycology (pp. 77–98). Malden, MA: Blackwell. You may see a confidence level of +/– (plus or minus) three percentage points as a common statement of reliability and confidence in a poll. It means that if the poll was repeated, there is confidence that the results would be within three points above or below the percentages in the original results. When statements of statistical significance are made, you will know that it means a difference or a relationship was established with confidence by the study. That confidence gives the results credibility.