Goals

Abstract: The All the Arts for All the Kids Foundation is dedicated to increasing available arts resources for children in the K-6th grade range as well as providing grants for 7th and 8th grade teachers. Annually All the Arts for All the Kids holds a fund raiser known as Pins Pots and Possibilities. In service to All the Arts for All the Kids, Diligence Public Relations has been assigned the task of raising awareness for K-6th grade arts education programs, All the Arts for All the Kids and lastly their fund raiser Pins Pots and Possibilities, to local businesses in the downtown Fullerton area. As both a measure of the successful impact of Diligence Public Relations on local businesses as well as a tool for developing descriptive information on those businesses, Diligence Public Relations has developed a survey for participating businesses in the downtown Fullerton area. The Survey reflects a two-sample statistical approach towards estimating the impact of Diligence Public Relations work (i.e. practical significance) as well as evaluating the validity of that impact (i.e. statistical significance). The first sample was conducted before any work had been done on the part of Diligence Public Relations. The second sample was conducted after Diligence Public Relations campaign to raise awareness. A comparison of the results provides the framework for evaluating the impact of Diligence Public Relations efforts on the awareness for All the Arts for All the Kids on the local business of downtown Fullerton. Practical Significance: For the above mentioned survey there are three questions that ask a sampled business to evaluate their relative awareness of local school arts programs, the All the Arts for All the Kids Foundation and finally for Pins Pots and Possibilities, respectively. Prior to the public relations campaign a sample of businesses were asked to fill out a survey asking for some basic information about the business as well as the businesses level of awareness for local arts programs for kids. From henceforth this shall be known as the pre-campaign sample. After the campaign businesses were again sampled. A measure of the impact of the campaign is then simply a measure in the proportional increase of positive responses to the three previously mentioned questions. This is known as the practical significance of the impact of Diligence Public Relations campaign for All the Arts for All the Kids. Statistical Significance: Often it will be the case with two-sample tests that a difference will be detected. This is the same as saying there is some practically significant difference between one sample and the next. However, it is quite another thing to say that the cause for such difference is statistically significant. What it means to be statistically significant is that it is highly improbable that the differences between each given sample can be attributed to random chance. If discrepancies from one sample to another cannot be the result of randomness it follows logically that there must be some mechanism causing that change. If research is designed carefully it is possible to isolate that mechanism. For this particular research the initial goal is to establish that there is a difference between pre-campaign and post-campaign downtown Fullerton (practical significance). The next goal is to establish with some statistical credibility that the reason for that change was in fact the campaign and cannot be attributed to random chance processes (statistical significance).

Research Design
The Survey: The survey really serves two purposes. The first is to gather information on a sample of businesses that All the Arts for All the Kids would be interested in approaching for support. Of the eight survey questions five were designed for descriptive statistics. An example of a question designed primarily for descriptive statistics on the population of interest is as follows: Have you donated to a non-profit organization in the last year? Such information may be useful for future campaigns and future research projects but dont lend themselves much towards evaluating the impact of this particular campaign. The remaining three questions are designed specifically for inferential statistics. They stand to measure the effect of Diligence Public Relations campaign for this years Pins Pots and Possibilities, event. Sample Selection: Ideally two independent samples should be selected at random from a list of all potential businesses in the downtown Fullerton area that All the Arts for All the Kids is interested in reaching. This design was chosen over a paired sample test for one major reason. A paired sample test is when a sample of businesses is taken pre-campaign and asked to fill out a survey. Then post-campaign the same businesses would be asked to fill out the survey again. In this situation the simple act of sampling the first time around could impact their responses to the second round of surveying. In other words, if a representative from Diligence Public Relations surveys a business on behalf of All the Arts for All the Kids then that business has already had their awareness raised as a result of filling out the survey the first time around. It seemed this was an avoidable problem if a two sample test was chosen instead. Under a two sample framework two different samples of businesses are taken (it should be noted that there may be random overlap but for the most part different sets of businesses will be sampled before and after the campaign). Due to the limited size of the number of businesses simple random sampling would be a perfectly reasonable way to survey businesses and measure the effect of Diligence Public Relations campaign. However, in an effort to improve the effectiveness of the campaign Diligence Public Relations first used previously existing data to identify businesses that were more likely to be donors for All the Arts for All the Kids. This means that certain businesses were deemed more likely to be optimal candidates for campaign efforts based on past behavior of similar businesses. To reflect this greater allocation of campaign resource to certain businesses an unequally weighted probability sampling design was selected. What this means is that there was a higher probability that certain businesses were sampled than others. Also, on a practical note, if a business was sampled and it was in a shopping center or was highly localized to other businesses it was very practical to survey surrounding businesses as well. Typically this is known as cluster sampling though in this case it was more of a result of convenience than design. It should be noted that this does introduce an element of bias into the statistical process. This bias will be explained and then addressed later. After developing a sample list, individuals from Diligence Public Relations went to each of the sampled business and asked them to complete a survey. Sampling was to continue until roughly thirty sampled businesses were surveyed for both the pre-campaign and post-campaign samples. Bias: There are two main sources of bias that need to be identified and accounted for. First, not all businesses were given equal probability of being sampled. This was both by design and because of convenience. The impact of this bias is that certain businesses that were more likely to be affected by our campaign were more likely to be sampled as well. This may inflate the appearance of impact when the post-campaign sample is compared to the pre-campaign sample. However, under certain assumptions which seem fair to make, the impact of this bias should not be detrimental to our research goals. The second source of Bias comes from the fact that not all sampled businesses participated in filling out a survey. This is known as a self-selecting bias. Again under certain assumptions this bias can be discounted, but its existence should still be noted all the same. Assumptions Precluding Results and Analysis: The first source of bias was that businesses more likely to be campaigned were also more likely to be sampled. However, these businesses were more likely to be sampled both pre-campaign and post-campaign. Because these businesses were identified using data which

reflects past relationships with All the Arts for All the Kids they not only have a higher probability of high post-sample awareness but they also have a higher probability of high pre-sample awareness. The residual issue now is that overall awareness may be potentially over-estimated by our sample design but the impact of the campaign on the difference between pre-campaign and post-campaign local sampled businesses should only be minimally affected by the sampling design. The second source of bias was the self-selecting bias. This means that the samples were not random samples of all businesses All the Arts for All the Kids would ideally like to reach but rather were random samples of all such businesses that were willing to fill out a sample. The only way we can justify moving past this bias is to assume that businesses willing to fill out surveys are similar to businesses not willing to fill out surveys and that it is random which businesses are not willing to fill out surveys. That way we still have a pseudo-random sample of businesses that are willing to fill out surveys and the results from these businesses reflect the impact of the campaign on businesses that would not fill out surveys as well.

Results
Abstract: As noted earlier, there were three questions in which we asked a sampled business to evaluate their awareness of All the Arts for All the Kids. Those three questions were as follows: 1. 2. 3. How aware are of in class arts education programs provided to K-6th grade students in the Fullerton School District? How aware are you of All the Arts for All the Kids Foundation? How aware are you of the Downtown Fullerton event Pins Pots and Possibilities?

From henceforth these will be referred to as question 1, question 2 and question 3 respectively. The responses to these three questions were all of the following form: 1. 2. 3. 4. Not Aware Somewhat Aware Aware Very Aware

Recall that there are two goals of these questions. First the responses for the pre-campaign and postcampaign samples are to be used to establish the practical significance of the campaign. This is done simply by reporting the percentage of responses for each sample to each question. Thus,. for example, if the percentage of businesses who responded Not Aware to question 3 drops from 30% in the precampaign sample to 10% in the post-campaign sample then the measure of impact can be interpreted as a change of 20% in unawareness for Pins Pots and Possibilities. The second goal of the survey results is to establish some level of statistical significance. In response to the first goal a very descriptive approach was taken. In response to the second goal a much more statistically involved approach will be taken. First a set of hypothesis will be assumed. The null hypothesis quite literally will be that proportionally there is no difference between a response to a given question precampaign and post-campaign. Another way of stating this null hypothesis is that pre-campaign awareness and post-campaign awareness are not different from each other. The alternative hypothesis is that proportionally pre-campaign awareness is not the same as post-campaign awareness. At the end of this statistical test we will typically endorse one hypothesis or the other based on something called a p-value. A p-value very simply is the probability of getting the sample results that we did under the assumption that the null hypothesis is true. If that probability is too low then the logical conclusion is that assuming the null hypothesis is true was a bad assumption to begin with. Another literal definition of a p-value is that a p-value is the probability of our sample result occurring as a result of randomness if the null hypothesis is true. If the p-value is too low then it is more advisable to conclude that there is a mechanism causing the difference between the two samples and that difference is not caused by chance. In our particular case that mechanism is the campaign. The computation of a p-value begins with a discussion on the possible responses for a given question. For each of the three questions to be analyzed, there are four categorical responses coded 1-4. Because there are more than two categorical responses the situation is a multinomial distribution. The appropriate sampling distribution for finding a p-value when the data is multinomial is a chi-square distribution. Based on the computed chi-square value for each question a p-value can be computed using either a chi-square table or statistical software. Loosely translated a chi-square statistic measures how probable the average response differences are between the two samples if in fact they should have very similar responses (i.e. the null hypothesis is true.) It should be noted that typically any test with a p-value less than .05 is considered statistically significant. However, it is often the case in these kinds of tests that too much emphasis is placed on the .05 threshold and not enough emphasis is placed on a correct interpretation of a p-value and subsequently a responsible endorsement of a hypothesis. With that being said, the results of the survey for questions 1, 2 and 3 follow.

Question 1 Results: The first task is to evaluate the practical significance of Diligence Public Relations campaign on awareness of in class arts education programs provided to K-6th grade students. Table 1 reflects that relative percentages of response for each sample. Table 1: Contingency Table for proportional responses to question 1 Not Aware Pre-Campaign Sample (33 sampled) Post-Campaign Sample (29 sampled) 48.5% 24.1% Somewhat Aware 24.2% 27.6% Aware 21.2% 41.4% Very Aware 6.1% 6.9% Total 100% 100%

The results from table 1 tell a convincing story. As a practical measure unawareness of arts programs for K-6th graders dropped by a margin of 24.4% of the total sample after the campaign. Differences in the Somewhat Aware, and Very Aware, responses were marginal. Aware, responses saw a dramatic increase. It is safe to say that there was a very noticeable practical significance after the campaign. The second task is to evaluate the statistical significance of Diligence Public Relations campaign on awareness of in class arts education programs provided to K-6th grade students. The chi-square statistic for this analysis was 4.599 with three degrees of freedom. This yields a p-value of .0858. A premature conclusion would be that the p-value is above the statistical default threshold of .05 and that our results are not statistically significant. However, by literally interpreting the p-value it is easy to see that there is still weight and validity to be associated with our reported findings. With a p-value of .0858, the literal interpretation is that if in fact differences between the two samples were do to chance and not caused by efforts on the part of Diligence Public Relations then the probability of getting the sample results that we did is 8.58%. This is not a very high probability. There are only two mutually exclusive interpretations of the events at hand. Either there really was no mechanism causing the difference other than chance and something rare occurred (something that should only occur roughly one out of every twelve times) or there is a mechanism other than chance causing a difference. The latter is likely the more believable story and because the only difference between pre-campaign sampled businesses and post-campaign sampled businesses was the campaign itself it is a logical conclusion that the mechanism causing the change between samples was in fact the campaign. The logical progression of literally interpreting the p-value and then assessing the weight behind each of the mutually exclusive conclusions is analogous to raising the threshold level from .05 to .1 (a common practice in statistics). Question 2 Results: The results and analysis for questions 2 and 3 will be very similar to the analysis for question 1. The first task is description of the results via a contingency table followed by some explanation on the practical significance of the results

Table 2: Contingency Table for proportional responses to question 2 Not Aware Pre-Campaign Sample (33 sampled) Post-Campaign Sample (29 sampled) 72.7% 41.4% Somewhat Aware 15.2% 24.1% Aware 3.0% 31.0% Very Aware 9.1% 3.4% Total 100% 100%

Table 2 shows that responses for question 2 followed a similar trend for responses to question 1. The main difference between questions 1 and 2 is that the initial proportion of businesses that were aware of All the Arts for All the Kids is much lower than the initial proportion of businesses that were aware of arts education programs for K-6th graders. Nonetheless, the trends of significant decreases in unawareness (72.7% of the pre-campaign sample to 41.4% of the post-campaign sample) and well as significant increases in the Somewhat Aware and Aware lend themselves to the conclusion that the impact of Diligence Public Relations campaign is practically significant. An unusual feature of this sample was that 9.1% of businesses in the pre-campaign sample were Very Aware of All the Arts for All the Kids while only 3.4% of the post-campaign sample responded Very Aware, to question 2. Without some alternative reasonable explanation the simple conclusion that a few past donors were randomly sampled in the precampaign sample seems the most likely explanation of this oddity. The next item in this report is the discussion on the statistical significance of the findings for question 2. The calculated chi-square statistic for evaluating the null and alternative hypothesis was 11.5232 with three degrees of freedom. This yields to a p-value of .0043. This means that if differences really are results of randomness and not Diligence Public Relations campaign then the probability of differences between the sample being as profound as they are is .43%, which is truly too rare to even seriously consider the possibility of no mechanism other than chance accounting for discrepancy. The heavily likely conclusion is that the campaign has raised awareness. Question 3 Results: Table 3 contains the two sample results for question 3. Table 3: Contingency Table for proportional responses to question 3 Not Aware Pre-Campaign Sample (33 sampled) Post-Campaign Sample (29 sampled) 81.7% 58.6% Somewhat Aware 6.1% 10.4% Aware 6.1% 31.0% Very Aware 6.1% 0.0% Total 100% 100%

It is fairly evident that the trends from the analysis of question 2 continue in question 3. Again we see significant drop in proportion for the unawareness of the event Pots Pins and Possibilities, while at the same time we see moderate increases in Somewhat Aware, responses and a dramatic increase in the Aware, response. Even the quirky trend of Very Aware, responses decreasing continue. This

shouldnt seem too surprising as the same pre-campaign businesses who claimed to be very aware if All the Arts for All the Kids are probably also very aware of the annual fund raiser. Calculating the chi-square statistic will again lend itself towards a discussion on the statistical significance of our findings. The chi-square statistics for table 3 results was computed to be 8.705 with three degrees of freedom. This corresponds to a p-value of .01515. This means that the probability of getting the results that we did because of random chance alone is roughly 1.5% . Because this p-value is so low (much lower than the widely accepted threshold of .05) it is fair to conclude that the alternative hypothesis is true. This means that a conclusion which attributes the differences in awareness indicated by the two sets of sample responses to question 3 to the campaign efforts of Diligence Public Relations seems most appropriate.

Conclusions
The goal of the survey was two-fold. First, information on businesses was gathered towards the aim of future sampling and analysis. It has been documented that there were significant flaws in the research design which led to bias. With the help of the information gathered from this survey both a better research design and a better campaign strategy can be imagined for future services to the All the Arts for All the Kids Foundation. The second goal of the survey was to evaluate both the practical and statistical impact of Diligence Public Relations campaign for All the Arts for All the Kids. For all three areas of concern, it can be concluded that the campaign was a success in practically increasing awareness. If we lax the traditional statistical threshold of .05 to .1 it can also be concluded that the campaign was statistically significant in changing awareness for all three areas of concern.