Citation: Yang, L.-Q., Levine, E. L., Xu, X., & Lopez Rivas, G. E. (2009).  Surveying via the net vs. hard copy: A cautionary note. Ergometrika, 6(1), 20-39.

Surveying via the Net vs. Hard Copy: A Cautionary Note

INTRODUCTION

Internet-based surveys (IBS) have been applied more and more frequently in different fields due to their advantageous characteristics over traditional paper-and-pencil surveys (PPS), telephone surveys, interview-based surveys and even computerized surveys. For example, they are able to reach samples that are potentially more representative of the population at large, to provide a means for motivating participants (e.g., instant feedback), and to increase the degree of self-disclosure (Gosling, Vazire, Srivastava, & John, 2004; Hayslett & Wildemuth, 2004; Kiesler & Sproull, 1986; Krantz & Dalal, 2000). While it is clear that the Internet offers potentially significant advantages for research, there are also some concerns associated with its use, such as more potential technical accessibility problems and other technical difficulty for IBS than for PPS (e.g., Thompson, Surface, Martin, & Sanders, 2003), greater difficulty in protecting anonymity for IBS than for PPS (e.g., Simsek & Veiga, 2001), or less control over the sampling process for IBS than for PPS (e.g., Gosling et al., 2004; Hayslett & Wildemuth, 2004; Schonlau, 2004; Stanton & Rogelberg, 2001). Other than these concerns, there may be threats to reliability and validity of measures in IBS as well, which have yet to be adequately investigated (e.g., Buchanan, 2002; Schonlau, 2004). Unless researchers check the effectiveness of this survey mode or have access to research on this issue, the validity and generalizability of results from IBS as compared with results of other modes such as traditional PPS, is open to serious question (e.g., Stanton & Rogelberg, 2001).

Therefore, it is imperative to build a cumulative database that examines the quality of the data collected via Internet and to compare the effectiveness of IBS with that of other methods, especially the traditional PPS. There have been some studies examining the comparability of IBS and PPS (e.g., Buchanan & Smith, 1999; Hayslett & Wildemuth, 2004; Klassen & Jacobs, 2001; Smith & Leigh, 1997). For example, Buchanan and Smith (1999) administered Gangestad and Snyder’s (1985) revised self-monitoring questionnaire via both Internet and hardcopy. The comparison of 963 responses obtained from Internet and 224 responses from paper-and-pencil survey showed that the Internet-based version of the measure had similar psychometric properties to its conventional counterpart. Within the field of organizational research, there have been studies looking at the comparability of internet-based and paper-and-pencil instruments, such as cognitive ability tests and situational judgment tests used for personnel selection (e.g., Ployhart, Weekley, Holtz, & Kemp, 2003; Potosky & Bobko, 2004), personality tests (e.g., Meade, Michels, & Lautenschlager, 2007; Ployhart et al., 2003; Salgado & Moscoso, 2003), organizational performance and change (e.g., Church, 2001), employee attitudes (e.g., Cole, Bedeian, & Field, 2006; Stanton, 1998), and employee motivation (e.g., Yost & Homer, 1998). For instance, Cole, Bedeian, Field (2006) employed Bass and Avolio’s (2000) Multifactor Leadership Questionnaire, Riggs and Knight’s (1994) measure of collective-efficacy beliefs, Riordan and Weatherly’s (1999) measure of work-group cohesiveness, and Hollenbeck, Klein, O’Leary, and Wright’s (1989) scale of collective goal commitment in a survey across 50 countries via both IBS and PPS; they found strong evidence for the measurement invariance of these measures across IBS and PPS.

However, there has been little attention to the comparability of measures related to occupational stress, emotion at work, discretionary work behaviors such as organizational citizenship behavior, or counterproductive work behavior. The research on these topics has been rapidly growing in the past decade and presumably will continue growing in the future given their implications for employee motivation and health, and organizational performance and well-being (e.g., Brief & Weiss, 2002; Dalal, 2005; Gelfand, Erez & Aycan, 2007; Gilboa, Shirom, Fried, & Cooper, 2008; Hoffman, Blair, Meriac, & Woehr, 2007). In the context of the present research, it may be possible that employees feel more comfortable in responding to questions on the above relatively sensitive topics when they are provided with one survey mode than the other. For example, employees might be more willing to take an IBS than a PPS regarding the above topics and to reveal their emotions, stressful work situations, and discretionary (sometimes covert) behaviors at work, because they presume a higher degree of perceived anonymity offered by IBS (Gosling et al., 2004) than PPS (e.g., personal writing often is required). Or, compared to PPS, employees might perceive a lower degree of anonymity offered by IBS and so are less likely to respond to it. Assuming greater concern about anonymity online, reductions in response rate may be due to potential respondents’ unwillingness to reveal their feelings and behaviors at work because they are unsure about the protected nature of their responses online (Simsek & Veiga, 2001). Yet another possibility is lowered rates of response owing simply to lack of familiarity with Internet technology.

Our current studies contribute to the literature on the effect of survey mode in industrial and organizational survey research by including various measures on the above topics that are important for researchers and practitioners but have not been examined across the two survey media in the past. Another important aspect of this research that heightens its impact beyond that of past work is the incorporation of two separate looks at the key comparisons between IBS and PPS in two different cultures — China and the United States. In addition, a gap in past work is the lack of a true controlled experiment or quasi-experiment that keeps constant factors other than survey mode. One of the two studies reported here (Study 2) fills this gap. The pair of investigations reported here was designed to compare IBS (either e-mail-based or Web-based) and PPS (either mailed survey or survey via delivery/collection at survey sites). The comparison between survey modes was based on questionnaires dealing with the various organizational topics of actual and preferred work conditions (i.e., supervisory relationship and career advancement opportunities as potential stressors), job satisfaction, turnover intention, mental and physical well-being, affect, organizational citizenship behavior and counterproductive work behavior in our field studies. The studies reported here intend to extend our knowledge of the effect of survey mode in industrial and organizational survey research by including measures understudied in the literature and tapping various survey topics at the same time. The questionnaires administered also were of considerable length and similar to researchers’/practitioners’ typical survey practice (long surveys), another departure from much of the past research.

To reduce the potential criticism of IBS (especially that via public network) about self-selected bias (e.g., Gosling et al., 2004; Hayslett & Wildemuth, 2004; Simsek & Veiga, 2001; Stanton, 1998), we sent surveys to all the employees from 6 companies (or a whole branch of the company) in China in study 1, and selected our samples randomly from a population including all the faculty and staff in a large, southeastern, public university in the U.S. in Study 2. Response rate was of particular concern in comparing the two survey modes. But participants’ demographics, reliability of scales, and survey results were also of interest and were compared across these two survey modes in both of our studies. Although measurement invariance and comparative validity of our measures are also important issues in considering the equivalence of the two modes (e.g., Buchanan, Johnson & Goldberg, 2005; Buchanan & Smith, 1999; Cole et al., 2006; Vandenberg & Lance, 2000), our capacity to carry out such analyses in this research is limited owing to insufficient sample sizes and lack of accessibility to objective criterion data for the constructs we measured.

Response Rate

There have been some studies comparing the response rate of IBS and that of PPS (e.g., Hayslett & Wildemuth, 2004; Miller, Daly, Wood, Brooks, & Roper, 1996; Tse, Tse, Yin, Ting, & Hong, 1995; Weible & Wallace, 1998; Zhang, 2000). For instance, Zhang (2000) surveyed library and information science researchers about their use of electronic sources for their research via Web and also postal mail. He found that follow-up via postal mail generated a better wave of replies than that via e-mail. He suggested that “this group of respondents would have preferred a print copy of the questionnaire — they had ignored the three e-mail follow-ups before they replied to the survey in print format” (p. 63, as cited in Hayslett & Wildemuth, 2004). Hayslett and Wildemuth (2004) surveyed academic reference librarians via both IBS (including a Web survey announced by e-mail and a Web survey announced by mail) and PPS. They found that the overall response rate was 14 percentage points higher for the paper survey than for the Web survey, suggesting that PPS still holds the advantage in response rate over Web surveys.

Some Internet-based surveys have achieved response rate as high as 87 percent (Sproull, 1986), while others have been able to achieve only dismally low response rates (e.g., 6.0%, Tse et al., 1995). When the survey concerns topics likely to evoke strong feelings or topics sensitive to social desirability, the question of response rates may be especially important (e.g., Johnson, Woodley, & Reips, 2007; Records & Rice, 2006; Senn, Verberg, Desmarais, & Wood, 2000). The issue of whether IBS can achieve response rates that are as high as those seen in PPS has not yet been fully resolved (e.g., Hayslett & Wildemuth, 2004).

Demographics

A major factor affecting Internet-based surveys is that those who use the Internet have possibly different characteristics from the general population, which leads to the potential for sampling bias (e.g., Hayslett & Wildemuth, 2004; Pasveer & Ellard, 1998; Shaw & Davis, 1996; Walsh, Kiesler, Sproull, & Hesse, 1992). For example, some findings have shown that 70 percent of those aged 20-50 use the Internet while this rate drops dramatically among people in their 60’s and 70’s (Hayslett & Wildemuth, 2004; Loges & Jung, 2001). Although the overall gender gap among Internet users has essentially closed (National Telecommunications and Information Administration, 2000), there are still gender differences among particular age groups, with women aged 20-50 using the Internet more heavily than men, and men aged 60 and over using it more heavily than women (Cooper & Weaver, 2003; National Telecommunications and Information Administration & Economic and Statistics Administration, 2002). The literature has also suggested both family income differences and race-related differences in Internet use (Hacker & Steiner, 2002; National Telecommunications and Information Administration & Economic and Statistics Administration, 2002).

However, using a personality questionnaire, Gosling, Vazire, Srivastava and John (2004) investigated a large Internet sample (N = 361,703) collected from outofservice.com, a noncommercial and advertisement-free website. They found that the Internet samples were relatively more diverse with respect to gender, social-economic status, geographic region, and age, compared to a set of 510 published traditional samples.

These somewhat contradictory trends suggest that sample representativeness with regard to demographics of IBS respondents needs to be examined further, and compared with traditional PPS.

Psychometric Properties of Scales

Considering the potentially different composition of IBS samples, the unique environmental factors and nature of the testing environment (i.e., researchers have no control over the conditions under which an IBS is completed), the technological factors (e.g., different browser software packages to be used), and mischievous responding (i.e., respondents could answer the scales in different ways, Buchanan & Smith, 1999; Schmidt, 1997; Smith & Leigh, 1997) in Internet samples, it is critical to check the threats to reliability and validity of IBS data.

There have been a few studies comparing the reliability and validity of scales across Internet-based data and paper-and-pencil based data (e.g., Buchanan & Smith, 1999; Cole et al., 2006; Ritter, Lorig, Laurent, & Matthews, 2004). For instance, Ritter and his colleagues (2004) found that out of 16 instruments useful in evaluation of patient interventions, internal consistency reliability was similar for each instrument across the two survey modes, and Internet test-retest reliability was high at .8 or above for all the 16 scales.

To the best of our knowledge, no study has examined the comparability of reliability and validity of scales measuring occupational stress, affect, and discretionary work behaviors across IBS and PPS in the literature. Therefore, we explored the reliability and validity of these scales across these two survey modes in our studies, along with the degree of generalizability of our findings across two quite different cultural settings. We discuss issues associated with validity of IBS data in greater detail in the next section.

Survey Outcomes

It is instructive to examine the effect of the mode of survey administration on the responses given by the respondent and, thus, the results of survey. Naturally, researchers are interested in univariate results, though most researchers place a greater emphasis on relationships among variables (e.g., Pettit, 2002). We will compare both the univariate results and the bivariate results across the two survey modes.

Univariate results. There is some evidence showing that univariate results of psychological scales are consistent across the IBS and PPS data (e.g., Pettit, 2002; Ritter et al., 2004). For instance, Ritter and his colleagues (2004) found that none of the 16 instruments useful in evaluation of patient interventions showed significant differences in scores across the IBS and PPS when appropriate tests were used.

Bivariate results. Because researchers are more commonly interested in studying relationships between variables, it is important to determine whether relationships based upon IBS data are consistent with the relationships based upon PPS data. For example, Pettit (2002) found that mode of administration did not affect any of the intercorrelations between scores of specific scales (e.g., Computer Attitude Scale; Loyd & Gressard, 1984; Perfectionist Self-Presentation Scale; Hewitt & Flett, 1995; Marlowe-Crowne Social Desirability Scale; Marlowe & Crowne, 1960). However, we expect that our comparison of bivariate results for those understudied measures in organizational research (e.g., occupational stress or affect) will contribute to the degree of confidence held about the validity of IBS data in this field.

Given that the results of the comparisons mentioned above have not been replicated in terms of broad enough measures and topics, especially the measures of occupational stress, affect and discretionary work behaviors in organizational research and practice, no specific hypotheses will be generated for the current studies with respect to comparing the response rate, demographics, the reliability, the mean scores and intercorrelations of all scales across the two survey modes.

STUDY 1 METHOD

Participants and Procedure

We surveyed five companies in Beijing and one in Hubei province in the People’s Republic of China, including four state-owned businesses, one privately owned business and one joint venture in 2004. Two hundred and eighty-eight useable questionnaires in total were returned from 1745 distributed electronic questionnaires and 100 distributed paper questionnaires, yielding response rates of 11.7% for IBS and 83.0% for PPS.

Through the companies’ internal e-mail system, the Human Resources (HR) staff in those companies informed all the employees about a coming project for free job stress diagnosis an outside researcher initiated via forwarding an e-brochure from the researcher in charge. Personal feedback was promised within the following two months. To obtain feedback, participants in the PPS condition were instructed to create a unique code to be placed on the first page of the survey by participants themselves, and directly contact the researcher based on the code after the survey. For the IBS situation, participants were instructed to leave their e-mail addresses during the survey only if they wanted feedback. One week later the researcher in charge distributed electronic questionnaires (.exe file, with web interface used, and with the property of “save and resume at a later time”) via e-mails to all the employees in five of the six companies (or one of their branches), and gave out paper questionnaires to all the employees in one of the branches of the remaining company.

Specifically, for one of the five IBS companies, the e-mail from the researcher with brief survey instructions and the survey attachment (.exe file) was forwarded to all employees by one senior employee (the contact person of the researcher) upon the approval of the HR department of that company. For the other four companies, the e-mail from the researcher with the same content was forwarded to all employees by a certain HR staff (the contact person of the researcher). Confidentiality and the voluntary nature of participation were emphasized in that employees’ responses to the survey would be directly sent to the researcher in charge of the whole project. The researcher was not professionally related to any of the individual companies, and refusing to participate in the survey would not lead to any consequences. Two follow-up e-mails were sent as reminders to all the possible respondents in the electronic survey in an interval of one week before we stopped collecting data. For the paper survey, an e-version brochure about the project and the survey was sent out by the HR staff to all potential participants in a particular branch of a large company. One week after that, because of the feasibility issue (the internet firewall of the company blocked the researcher’s e-mail containing the survey (.exe) file), the hardcopy anonymous surveys were distributed at a certain location of the company by a researcher and employees were informed to drop the completed surveys into a locked box (only the researcher held the key) at the same location. The same researcher was arranged to collect the questionnaires from the locked box at a specific time in two weeks. The hardcopy and electronic questionnaires had exactly the same instructions and items. For both survey modes, researchers performed the role of external academic experts providing a free consulting service.

The final sample included 164 (56.9%) males and 124 (43.1%) females, with average age of 32.10 (SD = 7.36) and average job tenure of 3.74 years (SD = 4.23). Overall, 33.9% of the respondents worked in an information-technology (IT) company, 16.6% worked in an engineering company, 36.4% were from production companies and 13.1% worked in electronic commerce companies. The participants from one production company were surveyed via paper-and-pencil; those from two electronic commerce companies, another production company, one engineering company and one IT company were surveyed via Internet (e-mail).

Measures

Actual and Preferred Conditions at Work. Items from the career advancement subscale and supervisory relationship subscale of the Chinese version Occupational Stress Indicator-2 (OSI-2, Siu, Cooper & Donald, 1997; Williams & Cooper, 1996) were adapted to assess both the actual and preferred conditions at work. The actual conditions of career advancement were measured by 4 items with response choices ranging from 1 (Never) to 6 (Very frequently). An example item is “In my job, there are few opportunities for promotion.” The preferred conditions were measured with four corresponding items by changing “In my job” to “In my preferred job.” For example, “In my preferred job, there would be few opportunities for promotion.” Supervisory relationship was assessed with 6 items using the same response choices. An example item of the actual state is “In my job, there is little encouragement from superiors.” The preferred conditions were measured with 6 corresponding items substituting “In my preferred job” for “In my job.” In this current sample from both PPS and IBS, the internal consistency indices of the two actual scales were .87 and .80 and the two preferred scales were .89 and .85 for career advancement and supervisory relationship, respectively.

Mental well-being. The mental well-being subscale from Chinese OSI-2 (Siu, et al., 1997; Williams & Cooper, 1996) was used to measure general mental health. All 12 items used a 6-point Likert scale with response choices specific to each item. A sample item is “During an ordinary working day are there times when you feel unsettled and upset though the reasons for this might not always be clearly obvious?” Its response choices range from 1 (Frequently) to 6 (Never). This scale has been used for research cross-culturally and consistently demonstrated adequate reliability and evidence for construct validity in both the west and China (e.g., Siu, 2002; Williams & Cooper, 1998; Yang, Che, & Spector, 2008). The alpha coefficient of this scale .83 in the current whole sample.

Physical well-being. We used the 6-item physical well-being subscale from the Chinese OSI-2 (Siu, et al., 1997; Williams & Cooper, 1996), which assesses the physical state of health. The subscale has 6 items with 6 response choices ranging from 1 (Never) to 6 (Very frequently). A sample item is “Feeling unaccountably tired or exhausted.” This scale has also been established as a reliable measure cross-culturally, with evidence for construct validity in both the west and China (e.g., Lu, Kao, Cooper, & Spector, 2000; Siu, 2002; Williams & Spector, 1998). The alpha coefficient of this scale was .73 in the current whole sample.

Job satisfaction. The 12-item job satisfaction subscale from Chinese OSI-2 (Siu, et al., 1997; Williams & Cooper, 1996) was used to measure satisfaction with the job itself (6 items) and with the organization (6 items). This scale has also been established as a reliable measure cross-culturally with evidence for construct validity in both the west and China (e.g., Cooper & Williams, 1991; Lu et al., 2000; Williams & Cooper, 1998; Zhang, Yang, Xu, & Che, 2006). Each item has 6 response choices ranging from 1 (Very much dissatisfied) to 6 (Very much satisfied). Sample items are “The degree to which you feel you can personally develop or grow in your job.” and “The way changes and innovations are implemented.” The alpha coefficient of this overall scale was .89 in the current whole sample.

Turnover intention. A 3-item measure of turnover intention (Liang, 1999) was used with 6 response choices ranging from 1 (Disagree very much) to 6 (Agree very much). A sample item is “I often intend to leave the current organization.” This scale originated from Cammann, Fichman, Jenkins and Klesh’s (1979) scale of turnover intention and has been revised and established to be a reliable instrument in the PRC (e.g., Liang, 1999; Lu, 2001; Yang et al., 2008). The alpha coefficient of this scale was .85 in the current whole sample.

The survey had 71 items in total, including the above measures and demographic variables. Some examples of those demographics were gender, age, and tenure. All questions in both PPS and IBS were in the format of multiple choice except the ones about age and job tenure, which were open-ended and required participants' hand-written responses (for PPS) or typed responses (for IBS). Participants of PPS were asked to mark their responses directly on the hardcopy survey.

RESULTS

Response Rate

As shown in Table 1, the response rate using the PPS is much higher than that of the IBS. A chi-square test indicated that the IBS sample had a significantly lower response rate than that of the PPS sample (p < .01).

Demographics

As suggested by Table 2, IBS and PPS in Study 1 have similar gender composition and job level distribution. However, based on t-tests, the PPS respondents were significantly older on average (p < .05) and had significantly greater tenure on average than IBS (p < .01). Of course tenure is somewhat confounded with age.

Reliability

As indicated in Table 3, the reliability of the scales measuring the focal variables ranges from .70 to .90, for both IBS and PPS. In fact, most of the scales in IBS have slightly higher reliability than PPS, most likely due to its bigger sample than PPS. Specifically, all the scales have comparable reliability across survey modes except for actual career advancement and mental well-being where reliability was lower in the PPS sample than that in the IBS sample, based upon Lautenschlager’s (1989) ALPHATST.

Survey Outcomes

Univariate results. As shown in Table 3, most of the focal variables had similar means across IBS and PPS with few exceptions, based on t-tests after Bonferroni adjustment. Specifically, there were slightly fewer career advancement opportunities reported in the IBS sample than in the PPS sample, participants who responded to IBS seemed to prefer better supervisory relationships than their counterparts who responded to PPS, and participants reported slightly lower physical well-being in IBS than in PPS. The average effect size across all the variables was .02.

To examine the possibility of demographic variables as confounding variables for the effect of survey modality on the univariate results of the key study variables, regression analyses were conducted with each of the eight key variables (i.e., the two subdimensions of job satisfaction scale were not included) as dependent variable and survey modality as the predictor. The analyses were run once without and once with demographics1 (gender, age, or tenure) controlled for; one variable was controlled for at a time to preserve statistical power. Results from the regression analyses with demographic control variables were compared to those without control variables. Controlling for gender did not make a difference in the relationship between survey modality and the key variables. However, when we controlled for either age or tenure in examining the effect of modality on the key study variables, the difference of preferred supervisory relationship across survey modes lost its significance but the difference of turnover intention became significant across survey modes. Therefore, the differences in age and tenure across the IBS and PPS samples might have accounted for part of the effect of modality on some of the key variables (i.e., preferred supervisory relationship) but not for others (actual career advancement opportunities and physical well-being).

Bivariate results. A Z-test examination of each of the 45 pairs of independent interscale correlations (e.g., the correlation between actual career advancement opportunities and job satisfaction in IBS and that in PPS) was carried out to reveal any significant difference between these two survey approaches. Four out of the 45 z-tests were statistically significant (p < .05) after Bonferroni adjustment.

DISCUSSION

Data from Study 1 indicate a significant difference in response rate favoring PPS. It is possible that the participants from the particular production company welcomed our way of distributing the brochures about the project and the surveys at the work site. On the other hand, those potential participants from another five companies might have ignored our e-mails with e-brochures attached about the project and those e-mails with questionnaires attached due to too many junk e-mails every day, which may contribute to lower response rate in IBS than in PPS. Another possible explanation is different degrees of anonymity and confidentiality employees perceived in IBS vs. PPS. That is, employees in the five companies where IBS was conducted might have perceived lower anonymity and confidentiality because the researcher’s e-mails regarding the survey were forwarded by a staff inside the organization instead of directly from the researcher. In contrast, the employees in the company where PPS was conducted received hardcopy surveys distributed directly by the researcher from outside the company. In addition, different approaches the participants were instructed to use for obtaining personal feedback for their survey results (i.e., creating a unique code in PPS vs. leaving an email address in IBS) might have to some extent influenced their different perceptions of the survey anonymity, which then contributed to higher withdrawal from IBS (not completing the survey or not submitting it after completion) than from PPS (not completing the survey or not returning it to the locked box after completion).

On the other hand, the comparison of demographics, internal consistency reliability, univariate and bivariate results reveal relatively small differences across the two survey modes. The few significant differences in univariate results across survey modes may be partially accounted for by the industry/company-level difference of our IBS vs. PPS sample, and by the fact that PPS surveys were distributed in person. Specifically, given the consideration of feasibility, employees from one of the branches of a company in the production industry were surveyed via hardcopy, while employees from another five companies in four different industries (i.e., production, information technology, engineering and electronic commerce) were surveyed via e-mail. As a matter a fact, to explore the possible confounding of industry with the effect of modality on survey response rate and univariate results, we did supplementary analysis with only the subsamples from the production industry included. Specifically, data (N = 21) from the only one production company in IBS sample and the PPS data (N = 83) from another production company were compared. Their survey response rates were 5.3% and 83.0% respectively, yielding an even bigger difference between IBS and PPS. On the other hand, the comparisons of univariate results with these two samples (N = 21 and 83, respectively) indicated that none of the eight key study variables were significantly different across the two survey modes based on t-tests after Bonferroni adjustment. Therefore, with industry controlled (only the production industry included), the differences in univariate results across survey modes became nonsignificant although the difference in response rate remained (even somehow widened). Again, such a finding is tentative given the small sample sizes for both IBS and PPS and some other potential organization-level differences across these two subsamples (e.g., the company in IBS was private while the one in PPS was state-owned) in the supplementary analyses.

Differences may also have been due to demographic differences observed between respondents to PPS vs. IBS. For example one might expect that older and more senior employees tend not be concerned with supervisory relationship quality as much as younger or more junior employees because they may have developed other social support systems at work in addition to the one related to their direct supervisor. That might thereby have accounted for differences in preferred supervisory relationship between PPS respondents who were on average older and had longer tenure than IBS respondents.

The design of Study 2 controlled for the potential confounds (difference in survey administration procedures, industry, organization, and participants’ demographics) mentioned above. The samples of the same composition were randomly drawn from the same organization and randomly assigned to IBS or PPS, which offered protection against these threats to internal validity. In addition, Study 2 was carried out in another cultural context thus enhancing confidence, should outcomes be replicated, in the external validity of the results.

STUDY 2 METHOD

Participants

In total 142 participants provided useable responses (with more than half of the survey items filled in) to our survey. Ninety-two of them were from 800 surveys distributed to a randomly selected sample of employees at a large southeastern university in the U.S. via the Internet in the first wave of data collection. Our respondents included 29 male, 43 female and 20 unidentified. Forty-one were faculty, 26 were staff and 25 were unidentified. Three worked part-time, 68 worked full-time, and 21 of them did not report this information. Their average age was 47.25 (SD = 10.78) and average tenure was 12.21 years (SD = 9.26). One follow up e-mail was sent to all 800 potential respondents.

Given the low rate of return observed in this first wave, we wanted to determine if survey modality might have been responsible. In order to insure comparability between survey modes (and of course to increase our sample size) we decided to sample 200 randomly chosen faculty and staff via IBS and 200 randomly chosen faculty and staff via PPS.

Thirteen of the 142 useable responses were from a second randomly selected 200-case sample at the same university distributed via Internet in the second wave data collection, including 1 male, 10 female and 2 unidentified. Eight were faculty, 3 were staff and 2 were unidentified. They all worked full-time. Their average age was 37.55 (SD = 10.69) and average tenure was 3.97 years (SD = 2.34). To maintain comparability with our PPS administration no follow up e-mail was sent to this sample of 200.

Thirty-seven of the 142 useable responses were from 200 hardcopies distributed via inter-office mail to a third randomly selected sample at the same university simultaneously with the 200 described above, including 10 male and 27 female. Twelve of them were faculty, 19 were staff and 6 were unidentified. They all worked full-time. Their average age was 49.32 (SD = 10.20) and average tenure was 11.03 years (SD = 9.76). No follow up was sent to this set of potential respondents.

Measures

Using identical instructions, the IBS and PPS surveys included the following scales.

State-Trait Emotion Measure. Constructed by Levine and Xu (2005), this scale consists of five positive (Attentiveness/Energy, Attraction, Contentment, Joy, and Pride) and five negative (Anger Anxiety, Envy, Guilt/Shame, and Sadness) emotions. These specific emotions were culled from previous research as being particularly salient in work settings. Participants were asked to rate on a 10-point scale the extent to which they felt these emotions during their most recent work day (State) and generally (Trait). Responses were added up for each respondent across the five positive state and five negative state items to provide an indication of positive and negative state aspects of emotional climate (i.e., state positive affect, SPA and state negative affect, SNA). The same was done across the five positive trait items and the five negative trait items to provide an indication of positive and negative trait aspects of emotional climate (i.e., trait positive affect, TPA and trait negative affect, TNA). The Cronbach’s alpha of the subscales was .83 (SPA), .63 (SNA), .86 (TPA), and .65 (TNA) in the current whole sample, respectively.

Job Satisfaction Scale (JSS). Developed by Spector (1985), JSS has shown adequate reliability and validity in the previous literature. All items were rated on a 6-point scale ranging from 1 (Disagree very much) to 6 (Agree very much). Each of the nine work facets the scale covers (i.e., satisfaction with nature of work, pay, promotion, supervision, coworker, fringe benefits, contingent rewards, operating conditions, and communication) is assessed by four items. The responses were added up across items within each facet for each respondent. A sample item is, “I like doing the things I do at work.” The alpha coefficient of the subscales ranged from .61 to .90 in the current whole sample.

Scale of Counterproductive Work Behavior (SCWB). Counterproductive work behavior (CWB) was assessed using the 19-item CWB measure from Bennett and Robinson (2000). Participants were asked to rate on a 1-7 frequency scale ranging from 1 (Never) to 7 (Daily) to indicate how often they engage in certain behaviors. Sample items include “Taken property from work without permission” and “Made fun of someone at work.” The Cronbach’s alpha was .83 for CWB directed toward individuals (7 items) and .66 for CWB directed toward the organization (12 items) in the current whole sample.

Organizational Citizenship Behaviors (OCB). OCB was measured with Van Dyne and LePine’s (1998) scale that includes 7 items on helping and 6 items on voice behavior. Participants indicated how much they agree with the items on a 1-7 scale ranging from 1 (Strongly disagree) to 7 (Strongly agree). The Cronbach’s alpha was .83 for helping behaviors and .93 for voice behaviors in the current whole sample. Sample items include “Volunteers to do things for the organization” and “Help others in this organization learn about work.”

The survey had 102 items in total, including the above measures and demographic variables. Some examples of those demographics were gender, age, tenure, and job categories. All questions in both PPS and IBS were in the format of multiple choice except that the ones about age and job tenure, which were open-ended and required participants' hand-written responses (for PPS) or typed responses (for IBS). Participants of PPS were able to mark their responses directly on the hardcopy survey.

Procedure

In the first wave, 400 faculty and 400 staff were randomly selected from the whole name list of the university faculty and staff, and surveyed via the Internet. E-mails were sent to those 800 potential respondents invited to respond via the Web. The message in the e-mails explained the purpose of this anonymous survey and that an overall report based on the responses of all the participants would be provided for the HR department of that university in order to improve the work environment. At the end of the message, a link was provided that would direct participants to the website containing the questionnaire. A follow-up e-mail was sent to each of the potential respondents 3 weeks after the survey started. The first-wave data collection ended after another 3 weeks. In the second wave, 3 weeks after the first data collection, 100 faculty and 100 staff were randomly selected and surveyed via Internet, and the same number of cases were simultaneously surveyed via mail. The hardcopy version contained the exact same instruments and instructions as those included in the website, and was sent via interoffice mail together with a return envelope.

Both survey approaches took care to protect the anonymity of the respondents and emphasize the voluntary nature of the survey such that withdrawing from the survey would not lead to any consequences. The data collection of each wave lasted for around 6 weeks, and both surveys were conducted during the same semester. Across all three administrations care was taken to insure that no one in the population of interest was contacted twice.

As mentioned previously, a follow-up e-mail was included in the first wave of online data collection while no follow-up e-mail was included in the second wave of online data collection. Nevertheless, we viewed it as appropriate to combine the IBS data from these two-waves of data collection to contrast with the PPS data for key analyses. Our rationale was as follows: First, the follow-up in the first wave IBS only gained around 2 percent more of all the responses in that phase of data collection, which made the first-wave IBS similar to the second-wave IBS in the data-collecting process. Second, there were no significant differences in either demographics or focal variables that we measured across the two waves of online data collection. On the other hand, the second wave of IBS differed only in survey mode from the PPS administration and could serve as a more closely matched comparison group with PPS. Therefore, we also reported the data separately for each wave of the IBS as warranted.

RESULTS

Response Rate

The response rates and usable response rates of both the IBS data and PPS data are shown in Table 4.

As is shown in Table 4, although the first wave IBS data achieved a total response rate (15.8%) close to that of PPS data (19.5%), the useable response rates across both IBS samples are lower than that of the PPS. A chi-square test indicated that (combined) IBS sample has a significantly lower total response rate from that of the PPS sample (p < .05). These two (i.e., the combined IBS vs. PPS) samples also have significantly different usable response rates (p < .01). However, the most relevant comparison in terms of equivalent time of administration, number invited, and lack of follow up was between the second wave of IBS and PPS. Here the chi-square test indicated that second-wave IBS had a significantly lower response rate than PPS (p < .01). The same outcome was observed for useable response rate (p < .01).

Demographics

The demographics of both the IBS data and PPS data are shown in Table 5. As suggested by Table 5, the combined IBS sample has demographic composition very similar to PPS data. Chi-square tests of gender, job category and work schedule indicated that only job category differentiates the IBS sample and PPS sample (p < .05; i.e., combined IBS sample had 62.8% faculty while PPS sample had 38.7% faculty), while t-tests of age and tenure suggest that neither was significantly different across the two modes of administration.

Reliability

The reliability of all the scales we used in this study was generated both in the combined IBS sample and PPS sample, and is shown in Table 6.

As indicated in Table 6, most of the scales in this study have comparable reliability across the two survey modes except for satisfaction with fringe benefits where reliability was lower in the PPS sample than in the IBS sample, based upon Lautenschlager’s (1989) ALPHATST. Therefore, generally, the scales used in IBS have equivalent reliability to those used in paper-and-pencil based survey in this study.

Survey Outcomes

Univariate results. T-tests showed that there was no significant difference across survey modes for any of the focal variables after Bonferroni adjustment. Overall, the average effect size across all the focal variables is .006.

Bivariate results. A Z-test examination of each of the 195 pairs of independent inter-scale correlations (e.g., the correlation between recent positive affect and OCB total in IBS and that in PPS) was carried out to examine whether there were any significant differences between these two survey approaches. After Bonferroni adjustment, no more than a chance proportion (2.6%) of the 195 Z-tests were statistically significant at p < .05.

GENERAL DISCUSSION

The response rates from the IBS in Study 1 and both IBS and PPS in Study 2 were lower than the average response rate (M = 35.7%, SD = 18.8%) of typical surveys reported in 17 refereed management and behavioral science journals in 2000 and 2005 (e.g., Academy of Management Journal, Personnel Review; Baruch & Holtom, 2008). However, the two studies presented here might represent a type of organizational survey that tends to yield lower-than-average response rates, because in our studies there was little push from the organizational management; employees responded to the survey on a voluntary basis. At most, the researchers in charge of the surveys reported here tried to recruit respondents by raising their awareness of their own well-being and the necessity of improving their work environment.

Overall, both studies in two different cultural contexts found that the response rate or useable response rate was consistently lower in IBS than PPS. We used the same survey questionnaires and the same time frame for data collection in companies across IBS and PPS in Study 1, and applied exactly the same questionnaires and comparable survey procedures in a large U.S. university across IBS and PPS in Study 2, a carefully-designed experiment. The discrepancy in response rate across the two survey modes may indicate a substantial contrast in statistical power if limited resources can be used for participant recruitment, especially for academic research efforts. On the other hand, the demographic composition was mostly similar across IBS and PPS samples in both of our studies except for a few study-specific characteristics. The equivalence of scale reliability across IBS and PPS samples was also generally supported, with few exceptions. Finally, both univariate and bivariate results were generally comparable across the two survey modes in both of our studies, with stronger evidence from Study 2 than Study 1. These outcomes merit close attention by survey administrators because of the nature of the surveys, which were lengthy (similar to many surveys used in practice), included measures of occupational stress, affect at work, and discretionary work behaviors that are demonstrated to be important in the literature, and contained relatively sensitive questions and scales.

In terms of survey response rate, results across both our studies are consistent with some of the previous literature, especially recent studies of IBS (e.g., Hayslett & Wildemuth, 2004; Klassen & Jacobs, 2001); that is, results showed lower response rate in IBS than in PPS. Although some research in the literature comparing IBS and PPS found higher response rates for IBS than PPS (e.g., Mehta & Sivadas, 1995; Oppermann, 1995; Parker, 1992; Ritter et al., 2004), these studies generally recruited participants for both IBS and PPS from public web sites, search engines or specific online interest groups and so may tend to attract people with more computer or Internet savvy (e.g., Gosling et al., 2004; Hayslett & Wildemuth, 2004), which did not occur in our current studies. The discrepancy of response rate in IBS and PPS found in our studies may be due to participants’ different degrees of familiarity with each survey mode, different degrees of confidentiality and anonymity potential participants perceived of the two survey modes, or some other external factors operating differently for IBS and PPS samples. Buttressing the possibility that perceived differences in confidentiality and anonymity between IBS and PPS may have driven the outcomes, we noted in Study 2 that 17.6% of the combined IBS sample chose not to report their gender while nobody refused in the PPS sample. Overall, our results from both studies suggested that people, at least in our targeted population, tend to answer hardcopy surveys more than Internet-based surveys. Since participation in the surveys was voluntary, the differences in response rates we observed here could mean a substantial contrast across the two survey modes in relation to statistical power of the findings in future studies, especially academic studies in organizational settings.

As for the comparison of demographics across the IBS and PPS samples, we found significant differences in age and job tenure across IBS and PPS in Study 1. This was partially due to the procedure used for data collection. As mentioned previously, hardcopy questionnaires were only sent to one production company, while Internet-based surveys were administered to another five companies in four different industries in China. The difference in age and job tenure across our IBS and PPS could be partially the reflection of industry/company-level differences. In Study 2, which controlled for the potential confounds in Study 1, we identified an interesting difference in job category, i.e., faculty tended to respond more to questionnaires online while staff tended to respond more via the traditional paper-and-pencil mode. This could be accounted for by their difference in education level and experience with using the Internet, or their different levels of power and status in the organization; for example, perceived lower confidentiality in IBS than in PPS could have contributed to staff’s choice of PPS due to their lower levels of power or status in the organization, which may have created concerns about adverse consequences based on their responses. Future research needs to control for these potential confounding factors to clarify this issue.

With respect to psychometric properties of the scales we used, both studies generally supported the equivalence of scale reliability across IBS and PPS with few exceptions. In Study 1, all the scale reliabilities were above .70 in IBS and PPS although the measure of actual career advancement opportunities and that of mental well-being demonstrated significantly lower reliability in the PPS sample than in the IBS sample. This may partially reflect the difference in sample sizes of PPS and IBS. In Study 2, all the online scales achieved reliability above .60, while all the paper-and-pencil scales achieved reliability above .61, with only the measure of satisfaction with fringe benefits demonstrating significantly lower reliability in PPS sample than in IBS sample. This could partially be due to the small sample size of PPS (i.e., N = 37). Overall, it is reassuring that the evidence showing relatively equivalent reliabilities across IBS and PPS in our two studies is consistent with previous studies (e.g., Buchanan & Smith, 1999; Cole et al., 2006; Pettit, 2002; Ritter et al., 2004). However, the few differences that arose may suggest the need to consider type of measure as a moderator.

When it comes to results of the survey, general equivalence was found for both studies. In Study 1, the average effect size of survey approach across all the focal variables was small (effect size = .02), although t-tests identified three significant differences across survey modes in preferred supervisory relationship, actual career advancement opportunities, and physical well-being. Possibly, the differences in characteristics of industries, organization structure or policies between the company surveyed by hardcopy questionnaires and those companies surveyed by e-mail could have reflected on employees’ perception of potential stressors, and their well-being. Demographic differences may also have played a role. As a matter of fact, the supplementary analyses conducted in Study 1 suggested that the differences in respondents’ age and tenure across IBS and PPS might have to some extent accounted for their different levels of preferred supervisory relationship across the two survey modes. In Study 2, the effect size of survey administration mode was small for all the focal variables.

The bivariate results in both studies suggested that the relationships among the focal variables remained consistent across the two survey modes with only a few exceptions, which is parallel with the previous research (e.g., Pettit, 2002). This to some extent supports the comparability of the relationships among constructs identified by IBS data with that by PPS data, at least for the specific variables covered in our studies.

Power analysis (Murphy & Myors, 2004) showed that the power of null-hypothesis testing can reach .50 when population effect size (proportion of variance in the dependent variable explained by the linear model, or just by some particular predictor) is .01 or above and the power can reach .80 if population effect size is .03 or above, for our China sample size (N = 288) in Study 1. The analysis showed that the power of null-hypothesis testing can reach .50 when population effect size (correlation between survey mode and focal variable) is .03 or above and the power can reach .80 if population effect size is .05 or above, for our U.S. sample size (N = 142) in Study 2. Therefore, with our sample sizes, there is adequate power to detect potential differences across the two survey modes, with the assumption that the population effect size of survey mode is small.

In sum, the general equivalence of subject demographics, scale reliability, univariate and bivariate results of survey was demonstrated in a more solid way in our Study 2 than in Study 1. This is partially attributed to the better-manipulated design of Study 2 than that of Study 1. But, overall, our Study 1 and Study 2 data lead to a similar conclusion about lower response rates in IBS than PPS, and about general equivalence of subject demographics, scale reliability, univariate and bivariate results of survey data across these two survey modes with few exceptions. However, given the fact that one can never accept a null hypothesis of “no difference” (Cohen, 1994; Meehl, 1978), our interpretations regarding the comparability of IBS and PPS data in our current study should be viewed with caution.

Limitations and Implications

There were similarities between Study 1 and 2 that warranted our interpretation of their common findings. For example, Study 1 and 2 were both initiated by researchers outside the organizations or units where the surveys were conducted, and supported by the organizational management in a non-intrusive way, and the researchers were not professionally related to the organizations or units in any formal manner. Confidentiality was ensured in both studies by way of limiting the accessibility of the data to the researcher in charge. In addition, no monetary incentives were offered for the survey respondents in either of the two studies; the voluntary nature of the survey was emphasized in both studies.

However, we would like to point out the differences in survey administration procedures, potential differences in identifying respondents, sample characteristics, and survey settings (e.g., organizational characteristics) between these two studies, which to some extent could set the boundary conditions for us to interpret the findings from the current research. To be specific, the IBS in Study 1 was based on an .exe program delivered by e-mail, as opposed to the webpage-based survey in Study 2. That is, as opposed to those in Study 2, respondents in Study 1 had to go through an extra step to download the small program (160KB) and another extra step to send the survey response file (a 5-KB .txt file) directly to the researcher, although the interface of the program resembled that of a Web-based survey. Therefore, IBS was used as an umbrella term to include an e-mail-based survey approach (Study 1) and a Web-based survey approach (Study 2) albeit the link to the Web-based survey and basic survey instructions in Study 2 were also sent out via e-mails (comparable with Study 1 in that sense). Baruch and Holtom’s (2008) review of academic survey studies showed that e-mail-based surveys had somewhat higher average response rate that the Web-based ones, which parallels what we found in Studies 1 and 2, in the case of useable response rate since the Study 1 survey server only kept useable responses.

In addition, the PPS distribution in Study 1 involved contact between the researcher and the participants, and a locked box was used for the survey return. In Study 2, we used only inter-office mail within the organization where the survey was conducted. So, PPS was used as an umbrella term to include an in-person-distributed survey approach and an internal-mail-based survey approach (Study 2). Interestingly, Baruch and Holtom’s (2008) review indicated very similar average response rates for these two kinds of PPS.

As another note, the survey utilized in Study 2 was somewhat longer than that used in Study 1 given that the scales used in Study 2 were generally longer than those in Study 1; however, the survey length in both studies was typical for organizational surveys in academic studies. That the researcher in charge had access to the e-mail addresses of those sampled (Study 2) and those who responded (Study 1) may have been a deterrent for some potential IBS respondents, thereby contributing to the lower IBS response rates in Studies 1 and 2. However, the guarantee of confidentiality in Study 2 was monitored by the internal Institutional Review Board, and in Study 1 the researcher in charge was entirely external to the organizations studied. These considerations lead us to conclude that the danger of exposure was not a significant factor in choosing to respond.

Finally, we cannot completely rule out other differences in the procedures of survey administration which could have contributed to differences in response rates across the two studies for both IBS and PPS. The fact that personal feedback for the survey was promised upon participants’ request in Study 1 might have contributed to differences between their motives and those of Study 2 participants who may have been motivated by factors such as interests in the study topic or care for improving their work environment.

Regarding sample characteristics, it was possible that there were some differences between the two studies. For instance, respondents might have had different familiarity with prior surveys such that those in Study 1 might have had more regular surveys conducted by the organizational management themselves than those in Study 2. On the other hand, the respondents in Study 2 might have been surveyed more often and so more familiar with surveys than those in Study 1 because they, as employees in a research-oriented public university, might have served as research samples by multiple studies in the past. However, since we could not keep track of the number of studies in the past completed by members of our study populations, it was not possible for us to directly compare respondents’ familiarity with prior surveys across the two studies. With regard to survey settings, Study 1 involved more than one organization while Study 2 was conducted in only one organization.

The differences between the two studies discussed above might have accounted for the larger discrepancy between IBS and PPS in Study 1 than that in Study 2. These differences call for some caution in comparing their findings from these two studies, especially in interpreting the differences between their findings.

Despite the cautionary note due to the differences between the two studies and their relatively small sample sizes, this research has contributed to the literature on the effect of survey modality on variables of interest to organizational researchers. To be specific, this research examined the comparability of IBS data with PPS data in response rate, demographics, internal consistency (i.e., reliability) and results of a survey. The constructs we explored covered a broad range of those employed in organizational research: Actual and preferred work conditions (i.e., potential stressors), affect, job attitudes, discretionary work behaviors and well-being, some of which were understudied in terms of being compared across different survey modes in the literature. Practically, organizational researchers, who are particularly interested in surveying affect-related, discretionary work-behavior-related or occupational-stress-related topics without being officially and financially sponsored by the organizations, may be able to use either IBS or PPS due to the general comparability of participants’ demographic constitution and scale reliability shown by our studies. More importantly, our studies investigated not only the potential difference in univariate results but also those in bivariate results and alpha coefficients across IBS data and PPS data. The results supported the comparability of IBS data and PPS data more strongly than most of the previous studies (e.g., Hayslett & Wildemuth, 2004; Ritter et al., 2004). However, in particular where adequate response rate and sample size are necessary for academic research efforts in organizational settings, PPS may provide a higher yield than IBS, as suggested by our studies.

Another possibility, based on demographic differences in the survey respondents across survey modes in Study 1 and job type differences (faculty vs. staff) across survey modes in Study 2, is that the use of both modes in combination may produce a more diverse or representative sample of the population of interest than either alone. Comparisons of separate PPS and IBS samples with the combined sample relative to the population in Study 2 (no analysis was done for Study 1 due to the unavailability of population information) did not support this conjecture. Specifically, t-tests showed that in terms of age, the PPS sample, IBS sample, and the combined sample all represented the population they came from (i.e., non-significant difference in age between each sample and the population); chi-square tests indicated that the PPS sample's job category composition represented the population (non-significant test results) but it under-represented males (p < .05), the IBS sample's gender composition represented the population but it under-represented staff (as opposed to faculty; p < .01), whereas the combined sample under-represented males (p < .05) and staff (p < .01). But in our case sample sizes were small and the surveys were only based on academic research efforts. Future research might explore further the notion that complementary use of both modes will be better than either alone.

Although our studies are confined to a particular set of variables, our findings may extend those of previous research because both American and Chinese samples were included in our studies. Although self-selection bias cannot be ruled out completely in our studies, identical questionnaires were used for both IBS and PPS so that the results from IBS and PPS were comparable, and our conclusions based upon these comparisons may be seen as reliable.

However, future research is needed to investigate the mechanisms underlying the discrepancy of response rate across IBS and PPS. Motivation for filling in the survey may be different for IBS and PPS participants. For example, potential participants who receive e-mails about a potential survey may tend to disregard them as junk mails or stop filling in the survey prior to completing it due to a variety of reasons. On the other hand, potential participants who are contacted for hardcopy surveys may tend to feel obligated to finish the survey just because the hardcopy of the survey lies on their work desks, or because hardcopy survey is easy for them to fill in during lunch break, or due to other reasons. It will certainly be informative for organizational researchers to know more about how to motivate potential IBS or PPS participants to respond to surveys so as to increase the overall response rate, which may then decrease nonresponse biases in targeted samples. Another possible approach that could help future research address the discrepancy of response rate between IBS and PPS is to add some additional survey questions in each case to ask about respondents’ preference for IBS or PPS, their perceptions of survey confidentiality and anonymity, their motives of filling out the survey, and even their expectations about the possible use of the survey results.

It is also important for researchers to design more experimental and quasi-experimental studies in the future to examine whether participants have different response patterns to socially sensitive survey topics (e.g., income level; abusive supervision, e.g., Tepper, 2007; discretionary work behavior, e.g., Spector & Fox, 2002; violence, e.g., LeBlanc & Barling, 2005) in IBS vs. PPS by better controlling the sampling process and response-recording in both IBS and PPS. Particularly, the degree of social sensitivity (social desirability or undesirability) of survey topics should be defined and gauged from participants’ perspectives instead of researchers’ subjective judgment. Our Study 2 could have produced more informative results to address this point if larger sample sizes were obtained. Finally, it will be very interesting as well to look at whether the effect of survey modality on responses to sensitive topics could be different in different cultures (e.g., collectivistic vs. individualistic culture, tight vs. loose culture).

In sum, caution is still in order given response rate differences and the other differences, although relatively few, we observed in comparative data across two studies in two different settings. As well, future work that tests for measurement invariance, which our sample sizes in both studies were insufficient to test here, is needed for the kinds of constructs included in this research. More research will enable greater confidence in the choice of modes of survey administration in organizational research and practice. However, the consistency of results reported across two separate studies in two very different cultural contexts in this investigation warrants careful consideration by those involved in survey research, particularly in the area of occupational stress, affect at work, or discretionary work behaviors.

REFERENCES

Baruch, Y., & Holtom, B. (2008). Survey response rate levels and trends in organizational research. Human Relations, 61, 1139-1160.

Bass, B. M., & Avolio, B. J. (2000). Multifactor Leadership Questionnaire: Technical report, leader form, rater form, and scoring key for MLQ form 5x-short (2nd ed). Redwood City, CA: Mindgarden.

Bennett, R. J. & Robinson, S. L. (2000). Development of a measure of workplace deviance. Journal of Applied Psychology, 85, 349-360.

Brief, A. P., Weiss, H. M. (2002). Organizational behavior: Affect in the workplace. Annual Review of Psychology, 53, 279-307.

Buchanan, T., & Smith, J.L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.

Buchanan, T. (2002). Online assessment: desirable or dangerous? Professional Psychology: Research and Practice, 33, 148-154.

Buchanan, T., Johnson, J.A., Goldberg, L.R. (2005). Implementing a Five-Factor Personality Inventory for Use on the Internet. European Journal of Psychological Assessment, 21, 115-127.

Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1979). The Michigan Organizational Assessment Questionnaire. Unpublished manuscript, University of Michigan, Ann Arbor.

Church, A. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, 937-969.

Cohen, J. (1994). the earth is round (p<.05). American Psychologist, 49, 997-1003.

Cole, M.S., Bedeian, A.G., & Feild, H.S. (2006). The Measurement Equivalence of Web-Based and Paper-and-Pencil Measures of Transformational Leadership: A Multinational Test. Organizational Research Methods, 9, 339-368.

Cooper, J., & Weaver, K. D. (2003). Gender and computers: Understanding the digital divide. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers.

Cooper, C.L., & Williams, S. (1991). A validation study of the OSI on a blue-collar sample. Stress Medicine, 7, 109–112.

Dalal, R. S. (2005). A Meta-Analysis of the Relationship Between Organizational Citizenship Behavior and Counterproductive Work Behavior. Journal of Applied Psychology, 90, 1241-1255.

Gangestad, S. W. & Snyder, M. (1985). ‘To carve nature at its joints’: On the existence of discrete classes in personality. Psychological Review, 92, 317-340.

Gelfand, M., Erez, M., & Aycan, Z. (2007). Cross-cultural organizational behavior. Annual Review of Psychology, 58, 479-514.

Gilboa, S., Shirom, A., Fried, Y., & Cooper, C. (2008). A meta-analysis of work demand stressors and job performance: Examining main and moderating effects. Personnel Psychology, 61, 227-271.

Gosling, S.D., Vazire, S., Srivastava, & S., John, O.P. (2004). Should we trust Web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93-104.

Hacker, K. L., & Steiner, R. (2002). The digital divide for Hispanic Americans. Howard Journal of Communications, 13, 267-283.

Hayslett, M.M., & Wildemuth, B.M. (2004). Pixels or pencils? The relative effectiveness of Web-based versus paper survey. Library & Information Science Research, 26, 73-93.

Hewitt, P., & Flett, G. (1995). Perfectionistic Self-Presentation Scale. Unpublished test, York University, Toronto.

Hoffman, B. J., Blair, C. A. , Meriac, J. P., & Woehr, D. J. (2007). Expanding the Criterion Domain? A Quantitative Review of the OCB Literature. Journal of Applied Psychology, 92, 555-566.

Hollenbeck, J. R., Klein, H. J., O’Leary, A. M., & Wright, P. M. (1989). Investigation of the construct validity of a self-report measure of goal commitment. Journal of Applied Psychology, 74, 951-956.

Joinson, A., Woodley, A., & Reips, U. (2007). Personalization, authentication and self-disclosure in self-administered Internet surveys. Computers in Human Behavior, 23, 275-285.

Kiesler, S., & Sproull, L.S. (1986). Response effects in the electronic survey. Public opinion quarterly, 50, 402-413.

Klassen, R.D., & Jacobs, J. (2001). Experimental comparison of Web, electronic and mail survey technologies in operations management. Journal of Operations Management, 19, 713-728.

Krantz, J., & Dalal, R. (2000). Validity of Web-based psychological research. In Birnbaum, Michael H. Psychological experiments on the Internet (pp. 35-60). San Diego, CA, US: Academic Press.

Lautenschlager, G. J. (1989). ALPHATST: Testing for differences in coefficient alpha. Applied Psychological Measurement, 13, 284.

LeBlanc, M., & Barling, J. (2005). Understanding the many facets of workplace violence. In Fox, S.; Spector, P. E. (Eds.), Counterproductive work behavior: Investigations of actors and targets (pp. 151-174). Washington, DC, US: American Psychological Association.

Levine, E. L. & Xu, X. (2005). Development and Validation of the State-Trait Emotion Measure (STEM). Paper presented at the 20th Annual Conference of the Society for Industrial and Organizational Psychology, Los Angeles, US.

Liang, K. G. (1999). Fairness in Chinese organizations. Unpublished dissertation in Old Dominion University.

Loges, W. E., & Jung, J. Y. (2001). Exploring the digital divide: Internet connectedness and age. Communication Research, 28, 536-562.

Loyd, B.H., & Gressard, C. (1984). Reliability and factorial validity of computer attitude scales. Educational & Psychological Measurement, 44, 501-505.

Lu, J. (2001). The validation of a new job satisfaction scale. Unpublished master thesis, Institute of Psychology, Chinese Academy of Science.

Lu, L., Kao S.F., & Cooper C.L.; Spector P.E. (2000) Managerial stress, locus of control, job strain in Taiwan and UK: a comparative study. International Journal of Stress Management, 7, 209-226.

Marlowe, D., & Crowne, D. P. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349-354.

Meade, A.W., Michels, L.C., & Lautenschlager, G.J. (2007). Are Internet and Paper-and-Pencil Personality Tests Truly Comparable? An Experimental Design Measurement Invariance Study. Organizational Research Methods, 10, 430-448.

Mehta, R. & Sivadas, E. (1995). Comparing response rates and response content in mail versus electronic mail surveys. Journal of Market Research Society, 37, 429-39.

Meehl, P. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.

Miller, J., Daly, J., Wood, M., Brooks, A., & Roper, M. (1996). Electronic bulletin board distributed questionnaires for exploratory research. Journal of Information Science, 22, 107–115.

Murphy, K., & Myors, B. (2004). Statistical power analysis: A simple and general model for traditional and modern hypothesis tests (2nd Ed.). New Jersey/London: Lawrence Erlbaum Associates Publishers.

National Telecommunications and Information Administration (NTIA) & Economic and Statistics Administration (2000, October). Falling through the Net: Toward digital inclusion. A report on Americans’ access to technology tools. Washington, DC: Department of Commerce.

National Telecommunications and Information Administration (NTIA) (2002, July). A nation online: How Americans are expanding their use of Internet. Washington, DC: Department of Commerce.

Oppermann, M. (1995). Email surveys – potentials and pitfalls. Marketing Research, 7, 29-33.

Parker, L. (1992). Collecting data the e-mail survey. Training and Development, July, 52-54.

Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30, 309-313.

Pettit, F.A. (2002). A comparison of World Wide Web and paper-and-pencil personality questionnaires. Behaviors Research Methods, Instruments, & Computers, 34, 50-54.

Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733-752.

Potosky, D., & Bobko, P. (2004). Selection Testing via the Internet: Practical Considerations and Exploratory Empirical Findings. Personnel Psychology, 57, 1003-1034.

Records, K., & Rice, M. (2006). Enhancing Participant Recruitment in Studies of Sensitive Topics. Journal of the American Psychiatric Nurses Association, 12, 28-36.

Riggs, M. L., & Knight, P. A. (1994). The impact of perceived group success-failure on motivational beliefs and attitudes: A causal study. Journal of Applied Psychology, 79, 755-766.

Riordan, C. M., & Weatherly, E. W. (1999). Defining and measuring employees’ identification with their work groups. Educational and Psychological Measurement, 59, 310-324.

Ritter, P., Lorig, K., Laurent, D., & Matthews, K. (2004) Internet Versus Mailed Questionnaires: A Randomized Comparison. Journal of Medical Internet Research, 6, 29-35.

Salgado, J. F., & Moscoso, S. (2003). Internet-based personality testing: Equivalence of measures and assesses’ perceptions and reactions. International Journal of Selection and Assessment, 11, 194-205.

Schmidt, W.C. (1997). World-Wide Web survey research: Benefits, potential problems, and solutions. Behaviors Research Methods, Instruments, & Computers, 29, 274-279.

Schonlau, M. (2004). Will web surveys ever become part of the mainstream research. Journal of Medical Internet Research, 6, e-article #31 (1-5).

Senn, C., Verberg, N., Desmarais, S., & Wood, E. (2000). Sampling the reluctant participant: A random-sample response-rate study of men and sexual coercion. Journal of Applied Social Psychology, 30, 96-105.

Shaw, D., & Davis, C. H. (1996). The Modern Language Association: Electronic and paper surveys of computer based tool use. Journal of the American Society for Information Science, 47, 932–940.

Simsek, A. & Veiga, J.F. (2001). A Primer on Internet Organizational Surveys. Organizational Research Methods, 4, 218-235.

Siu O. l. (2002). Occupational stressors and well-being among Chinese employees: the role of organizational commitment. Applied Psychology: an International Review, 51, 527-544.

Siu, O.L., Cooper, C.L., & Donald, I. (1997). Occupational stress, job satisfaction, and mental health among employees of an acquired TV company in Hong Kong. Stress Medicine, 13, 99–107.

Smith, M.A. & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, and Computers, 29, 496-505.

Spector P.E. (1985). Measurement of Human Service Staff Satisfaction: Development of Job Satisfaction Survey. American Journal of Community Psychology, 13, 693-713.

Spector, P. E. & Fox, S. (2002). An emotion-centered model of voluntary work behavior: Some parallels between counterproductive work behavior (CWB) and organizational citizenship behavior (OCB). Human Resources Management Review, 12, 269-292.

Sproull, L. (1986). Using electronic mail for data collection in organizational research. Academy of Management Journal, 29, 159–169.

Stanton, J. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.

Stanton, J.M. & Rogelberg, S.G. (2001). Using Internet/Intranet Web Pages to Collect Organizational Research Data. Organizational Research Methods, 4, 200-217.

Tepper, B.J. (2007). Abusive supervision in work organizations: Review, synthesis, and research agenda. Journal of Management, 33, 261-289.

Tse, A.C.B., Tse, K.C., Yin, C.H., Ting, C.B., & Hong, W.C. (1995). Comparing two methods of sending out questionnaires: email vs. mail. Journal of Market Research Society, 37, 441-446.

Thompson, L.F., Surface, E.A., Martin, D.L., & Sanders, M.G. (2003). From paper to pixels: moving personnel surveys to the web. Personnel Psychology, 56, 197-227.

Vandenberg, R.J. & Lance, C.E. (2000). A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, 3, 4-70.

Van Dyne, L. & LePine, J.A. (1998). Helping and voice extra-role behaviors: Evidence of construct and predictive validity. Academy of Management Journal, 41, 108-119.

Walsh, J.P., Kiesler, S., Sproull, L.S., & Hesse, B.W. (1992). Self-selected and randomly selected respondents in a computer network survey. Public Opinion Quarterly, 56, 241-244.

Weible, R. & Wallace, J. (1998). The impact of internet on data collection. Market Research, 10, 19-23.

Williams, S., & Cooper, C.L. (1996). Occupational Stress Indicator: Version 2. RAD Ltd.: North Yorkshire.

Williams, S., & Cooper, C.L. (1998). Management of occupational stress: development of Pressure Management Indicator. Journal of Occupational Health Psychology, 3, 306-321.

Yang, L.Q., Che, H.S., & Spector, P.E. (2008). Job stress and well-being: An examination from the view of person-environment fit. Journal of Occupational and Organizational Psychology, 81, 567-587.

Yost, P. R., & Homer, L. E. (1998). Electronic versus paper surveys: Does the medium affect the response? Paper presented at the Annual Meeting of the Society for Industrial and Organizational Psychology, Dallas, TX.

Zhang, Y. (2000). Using the Internet for survey research: A case study. Journal of the American Society for Information Science, 51, 57–68.

Zhang, X.C., Yang, L.Q., Xu, X.F., & Che, H.S. (2006). The mechanisms of negative affectivity in stress process. Psychological science (in China), 29, 967-969.

FOOTNOTES

1 Job level was not considered as one variable to control because the uneven split among different job levels (e.g., only 10 participants at decision-making level but 111 at general employee level for IBS, with N = 5 and 30 respectively at those two levels for PPS) might limit the statistical power in our regression analyses.