Citation: Levine, E. L., & Sanchez, J. I. (2007).  Evaluating work analysis in the 21st century. Ergometrika, 4, 1-11.

Evaluating Work Analysis in the 21st Century

Introduction

The analysis of work, that is the decomposing of tasks, bundles of tasks or activities comprising positions, jobs, occupations, and even entire systems has been of great usefulness in education, ergonomics, industrial engineering, organizational development, and human resource management (Gael, 1988). The methods of analysis traditionally captured under the rubric job analysis have deep and well-established historic roots (Mitchell, 1988; Primoff & Fine, 1988). Despite the long standing tradition, and the demonstrated pragmatic benefits, job analysis has come under fire in recent years (Harvey, 1991; Morgeson & Campion, 1997; Sanchez & Levine, 1999b). The nature of these attacks is two-fold.  First, some have argued that job analysis is an obstacle to organizational innovation, because it creates artificial boundaries that interfere with the successful adoption of innovative management practices (Olian & Rynes, 1991).&nbs! p; Sanchez (1994) and Sanchez and Levine (1999a) have argued that this line of criticism is off target, in part because such criticisms are directed at obsolescent uses of job analysis information, not at the analysis of work per se. To deal with these criticisms they advocate the use of the term work analysis instead of the traditional term job analysis. The former term obviates the negative connotations associated with antiquated aspects of scientific management and smokestack industry, and conveys more accurately the full range of the applications or purposes that such analyses may serve.

A second stream of criticisms has argued that the quality of job analysis information is suspect, and that the information is inaccurate due to a variety of biases and cognitive limitations such as flaws in the judgments made by SME’s (see Morgeson & Campion, 1997, for a review). The term accuracy demands a consensually agreed upon “gold standard”, which a “fuzzy,” socially-constructed concept like the job does not permit. Sanchez and Levine (2000) maintained that this line of thinking is also unlikely to advance the practice of work analysis, because it conceives the analysis of work as a measurement instrument intended to capture a questionable “true” score.  Instead, they view work analysis as a set of tools intended to facilitate the inferences regarding the important work activities and work specifications that should be the basis for selection, training, compensation, and other human resou! rce management practices.  On the one hand we fully support studies, such as the recent meta-analysis on reliability of job analysis data by Dierdorff and Wilson (2003), aimed at improving the quality (reliability and validity) of work analysis data, as opposed to so-called accuracy. On the other hand in our view evaluation of work analysis data should scrutinize not only the psychometric quality of the data but also more importantly the consequences of work analysis in terms of human resource management programs and practices. This argument derives incremental strength from the notion that organizations in today’s global and hyper-competitive economy must justify costs and demonstrate the value added by programs such as work analysis. Only research directed toward consequential outcomes can provide the crucial information needed to meet these objectives.

Evaluations that examine the consequences of work analysis, however, face an important shortcoming:  the fact that the procedures and criteria used to turn work-analytic information into meaningful HRM practices have themselves been limited. One purpose of this paper is to explore the criteria and procedures used to evaluate work analysis methods and information. A key theme is the notion that validity, including consequential validity, as defined by neo-classical psychometric theory rather than so-called accuracy is the touchstone of such evaluations (Sanchez & Levine, 1999b). Messick (1995) maintained that consequential validity includes the impact on respondents of how their scores are used. Furthermore the current standards for educational and psychological tests (American Educational Research Association, et al., 1999) note that consequential evidence is one of the key elements under girding validity judgments. This notio! n is pivotal in work analysis, because the resulting descriptions of work are not directly used to make decisions about employees, but to inform the design of the practices and programs that will facilitate such decisions.  For instance, job descriptions are not directly used to select employees, but to inform the selection procedures. 

The consequences of work analysis, therefore, are not immediately seen on employees, but mediated by a series of transformation rules through which work-analytic data inform the practices and programs that follow from work analysis.  This places a premium on the manner in which consequences of work analysis programs may be assessed. Thus, our second purpose is to offer suggestions for needed research on the validity of work analysis. The paper begins with a discussion of an expanded notion of validity in the context of work analysis. Following this, illustrative changes in the work world are described, and how work analysis methods must evolve to maintain their validity for these purposes is depicted. Finally, suggestions are offered for future investigations of the validity of work analysis methods and information with special attention to how the military may contribute.

Validity of Work Analysis

Surprisingly little attention has been paid to the meaning of validity in the context of work analysis (Manson, Levine, & Brannick, 2000). Harvey (1991) in his extensive treatment of job analysis defines validity in terms of the operations used by some to assess a loosely defined notion of validity. He mentions such approaches as lie scales where items not in the job are included in questionnaires, comparisons of reported time spent against more objectively measured amounts of time spent on tasks, and task coverage judgments by subject matter experts of the extent to which the reported data from a work analysis capture the full extent of a job’s contents.  The validity of subject matter experts’ task coverage ratings, however, has been challenged elsewhere (Wilson, 1997). Morgeson and Campion (1997) speak to the absence of bias as an indicant of validity. Others have looked at inter rater agreement as a surrogate for validit! y, which is better conceptualized as an assessment of reliability rather than validity (Manson, et al., 2000). Levine, Ash, Hall, & Sistrunk (1983) in a study comparing work analysis methods finessed the issue by defining validity as the comprehensiveness and quality of information yielded by a work analysis as judged by job analysis experts.

How then should we approach the notion of validity in a work-analytic context?  First, we must divorce the notion of validity from accuracy. The latter term is only applicable when there is an acceptance of the existence of an objective reality that represents the work under study, and when we have established a standardized representation of that reality. Although this view is not shared by others (Harvey & Wilson, 2000), we argue that in the context of work analysis we are dealing with a social construction of work, and from a philosophical perspective the notion of an underlying objective reality independent of observers or judges is not tenable (Connell & Nord, 1996).

To better understand the notion of a job, one needs to examine its historical antecedents.  In fact, the conceptualization of the job title as a separate entity from the individual or incumbent holding such job title can be traced to the industrialization era, which led to the transformation of an economy organized around farmers and artisans to one organized around laborers.  The levels of mass production that characterized the industrial era created a need for division of labor, vertical integration, and economies of scale.  Under such circumstances, it made sense to conceptualize jobs as separate from individuals, because economies of scale dictated the need for large numbers of individuals performing the exact same functions.  In fact, multiple individuals are said to hold the same job title when they have similar responsibilities. Although it has been argued that the stable and long-lasting jobs of the past will no lon! ger be available in a rapidly changing business world (Bridges, 1994), the analysis of work remains pivotal for human resource management.

What is established via work analysis is a consensually shared percept or construction of what the work consists of, the environmental conditions under which the work occurs, and the attributes needed by the worker or by the elements of the working system. Asking whether or not this shared construction is accurate, therefore, is futile.  That is, the shared perception probably reflects the aggregate of behaviors displayed by job incumbents over time, but there is quite a bit of “legitimate” variability in those perceptions, and the aggregate is just a compromise intended to represent the manner in which an “average” job incumbent performs the duties of the job.

Second, given the conception of an intangible construct or bundle of constructs as the product of a work analysis, the notion of construct validity drawn from neo-classical psychometric theory is germane. Modern conceptions of validity are unitarian in nature, and allow for several types of evidence that provide insight into the extent of validity, the search for consequences of a construct’s impact being one such strategy (Society for Industrial and Organizational Psychology, 2003).

Third, validity is not a quality intrinsic to a psychological construct but instead must have relevance to some purpose (Guion, 1965; SIOP, 2003). This dictum applies as well to a percept labeled a task, a function or a job. The validity of a work analysis resides in its capacity to serve one or more purposes (Levine, Thomas, & Sistrunk, 1988).

Based on these considerations, we assert that the validity of a work analysis is most convincingly established when it is shown to be instrumental in accomplishing the objectives that gave rise to the analyses in the first place. Obviously, the analysis should successfully capture the job’s underlying constructs that are critical in such areas as selecting, compensating, training, or evaluating the performance of employees, but a successful analysis does not automatically guarantee that the programs developed as a result of the analysis will be successful. Indeed, a successful analysis should be accompanied by a series of transformation guidelines that facilitate its impact on the various HR programs, which it intends to inform.  Thus validity of a work analysis for a particular purpose is the extent to which the work analysis adds incrementally to the effectiveness or efficiency of individual- or system-level interventions that ! are derived from it.

Where does that leave us in regard to studies that deal with the quality of the information, even including the construct validity of scales used to document and quantify judgments of work components? As we have suggested earlier these are legitimate precursors and necessary to the establishment of the validity of a work analysis. Such research may be compared to the strategy of using evidence based on test content where the quality of the test, the care in its construction, its reliability, the comprehensiveness of its sampling of a domain of interest all create a presumption of validity. However, since we are dealing with a set of psychological constructs, i.e. constructs related to the work being accomplished, we must necessarily proceed to some estimation of the extent to which the constructs articulate with the purpose or purposes at hand.  Again, to make an analogy to the valid use of a test, the notion that a test validly measures a! construct such as anxiety must be buttressed by the judgment, based on data and theory, that the construct is a key contributor to performance or work attendance. Where work analysis is at issue often such judgments are not based on empirical data but on the reasoned degree of fit between the work analysis data and the purpose at hand. This judged degree of fit would be strengthened greatly by demonstrations that successful human resource programs have been derived from rigorous work analyses. Thus the validity of work analyses rests on not only whether work requirements are adequately identified, but also on a demonstration of the fact that work analyses accomplish the purposes they are intended to serve.

In this paper, as we contemplate a selected set of the changes taking place in the world of work, we rely on judgmental assessments of the extent to which work analyses articulate with the changes. However, our suggestions for future research offered in the last section focus directly on the need to establish a strong, empirically based track record under girding the validity of work analysis in terms of the (presumably beneficial) consequences of its use.
 
The Validity of Work Analysis in the 21st Century

Sanchez and Levine (1999a) outlined a select set of the changes taking place in many of today’s organizations. The potentially dramatic changes in military work brought about by revolutionary technology such as night-vision equipment, mobile communication devices, etc. clearly illustrate some of these changes.  The evolving role of the military in a post-Cold War era adds to the technological change: smaller and faster deployments, peace-keeping operations, and military support of drug trafficking control are examples of the new demands faced by the Armed Forces.

Using the notion of building blocks for job analysis methods provided by Levine (1983), we suggested in that paper the alternatives needed for work analysis descriptors and methods to maintain validity for the various purposes they must serve. First we briefly summarize the changes noted in that paper that have altered traditional patterns of work. Then we outline the manner in which work analysis may articulate with these changes, relying upon the more current treatment of Brannick and Levine (2002). Of course where jobs retain their traditional form, standard techniques of work analysis already developed will still be applicable.

Modern Patterns of Work

Increasingly employees are called upon to assume broadened responsibilities. Boundaries between jobs are becoming less and less distinct. In an era where self-directed work teams are widely used (e.g., Nygren & Levine, 1996), the ambiguity extends to the distinction between labor and management. Work is more dynamic and the pace of change in responsibilities seems to have accelerated. Networks of people and machines organized into work systems have gained in importance, which places a premium on interaction rather than individual performance. Often these interactions must take place across long distances and across national boundaries. The notion of accountability to customers and the building of customer loyalty have jointly become a sine qua non of organizational success. Nor is the military exempt from this concern with customer satisfaction. In an all-volunteer military, customer, i.e., recruit satisfaction is a critical concern.

The flattening of organizational hierarchies has also led to limited opportunities to build careers within organizations, and the turbulent, global economic forces facing us have mitigated in favor of short rather than long-term employment. Several of these factors have altered the nature and level of compensation provided to employees, and the manner in which compensation is set. In the face of the increasing diversity of our workplaces, cultural sensitivity and the management of conflict take on critical importance. Emotions and personality are now much more a center of attention as are values and attitudes, all of which are manifested behaviorally partly in the form of so-called contextual performance or citizenship (Borman & Motowidlo, 1993). Finally, cost containment within a mercilessly competitive environment has driven many staff functions out of organizations often to outside providers. Activities including but not limited to work ! analysis not considered part of the core elements of an organization’s competitive advantage are among such functions.

Necessary Responses in Work Analysis Methods

How must work analysis methods respond to maintain validity? We organize our suggestions into the five building blocks of such methods--types of data, sources of data, methods of collecting data, levels/units of analysis, and management of information that includes dissemination, storage and retrieval of information (Brannick & Levine, 2002).

Types of data. The emergence of teams and networks or systems of people and machines strongly mitigates in favor of team or system level descriptors in addition to descriptors applicable to individual performers. Clearly, an interdisciplinary focus involving such disciplines as industrial engineering, ergonomics and our discipline of industrial and organizational psychology is mandated. Mission and function analysis at the macro level, including allocation of functions to elements within a network, including robots, are called for in the service of such purposes as system design, job design, training and selection. (Harris, 1988; Levine & Baker, 1991; Nof, 1988; Price & Pulliam, 1988). Workflow analysis is likewise necessary (e.g., Hupp, Polak & Westgaard, 1995), and the dimension of time may be necessary for such purposes as workload assessment (Busby & Hutsell, 1988).

Broader and more organizationally relevant descriptors of personal attributes are likely to be helpful. Some refer to these as competencies. Sanchez and Levine (1999a) cite the example of customer service in a health setting. Attributes associated with activities that are described in traditional terms like taking vital signs and administering prescribed medications speak to specific knowledge and skill components. However, the manner in which the services are provided to the consumer layers on top of these a required emotional display and value orientation that would otherwise be lost in relying solely on the skeletal aspects of the tasks. Thus, even if we continue to rely on work activities as a descriptor type, we should insure that the statements of activities reflect these added aspects.  The set of attributes or competencies derived from the activities will then be more complete.  However, the validity of these broader descript! ors often referred to as competencies should not be taken for granted. The process of identifying competencies requires a sizeable inferential leap, because competency modeling often fails to account for the specific tasks performed at work (Schippmann et al., 2000). Lievens, Sanchez, and De Corte (2004) recently demonstrated that the validity of broadly scripted competencies can be enhanced when applying sound work analysis procedures such as providing subject matter experts with an opportunity to study task inventories.

Work taking place in the context of teams and more generally the increasingly interactive workplace raises the issue of all attributes associated with being a team player, and the lexicon to be used in work analysis must be more fully developed to accommodate these facets of work. Broader definitions of human attributes are likely also to be more serviceable in the face of dynamic changes in work, and the likelihood that an employee may be assigned to a variety of units within an organization.

At the opposite end of the spectrum, fine grained task analysis of complex tasks within complex environments may be necessary for training (Van Cott & Paramore, 1988), or cognitive modeling (Olson & Olson, 2003). Mental operations can be the subject of scrutiny even in team settings where shared mental models may facilitate team performance (Salas & Cannon-Bowers, 1997), and are critical for the understanding of team structure and function (Ilgen, Hollenbeck, Johnson, & Jundt, 2005).

In terms of scales to be used in quantifying judgments in a work analysis, there will be a need to do more customizing. For example, instead of the such traditional scales as time spent, organizations may need information on the extent to which functions are core or may be outsourced. Ratings of the impact of error on customer service should also prove useful.

Nor should we neglect the environment and the stressors and hazards it presents to the worker. Once again an interdisciplinary perspective is called for and could involve molecular analyses of physiological demands on the worker imposed by extreme work environments or mental workload assessment (Casali & Wierwille; 1988; Kamon, 1988; Sulsky & Smith, 2005). On the other end of the spectrum, identification of environmental conditions that represent threats to worker health and safety will continue to be a concern in light of the fast pace of work and the rate of change in responsibilities (Siegel, 1988). The salutary effects of worker control suggest that work analysis may be profitably directed toward increasing control while moderating unwarranted demands (Spector, 2006).

Sources of data. Where job holders have typically been the source of choice for work analysts, and immediate supervisors the checkpoint for an alternate perspective, new patterns of work demand great care in selecting sources of data and the types of data or judgments they are called upon to provide. Job incumbency is neither a necessary nor a sufficient condition for subject matter experts in work analysis (Sanchez, 2000). Targets or beneficiaries of work processes, including customers, recent military recruits and clients may be enrolled in focus groups to aid in the design of work processes. Future assignments not yet designed should benefit from the input of diverse representatives drawn from different functional areas that would eventually interface with the new workers. Computer simulations can be employed to model future work and provide indications of human attribute needs (Polito & Pritsker, 1988).

The nature of the data or judgments should likewise enter into the selection of sources. Incumbents for example often are unable to make judgments about constructs with which they may be unfamiliar or have an inappropriate frame of reference. Asking many types of incumbents to judge tolerance for ambiguity required in their setting is likely to be puzzling, as incumbents may lack an adequate standard of comparison regarding “normal” levels of ambiguity in other jobs.  Another example is asking incumbents about noise levels to which they have adapted over time. We can remember asking workers about noise levels in power plants or a ship’s engine room where the noise was deafening only to be informed that they did not find the noise levels out of the ordinary.

Estimates of difficulty of learning are often more informative when coming from trainers than incumbents.  However, current knowledge of the trainability of human attributes is limited and often times plagued by stereotypical conceptions and value judgments concerning the extent to which human nature is amenable to change.  The consequential validity approach may shed light on the validity of trainability judgments when such judgments are contrasted against learning criteria such as the extent to which trainees improve their pre- and post-training performance (Jones, Sanchez, Parameswaran, Phelps, Shoptaugh, Williams, & White, 2001).

Cross-domain investigations such as linkages between work activities and the human attributes required to perform such activities are still in their infancy. The identification of the human attributes or skill sets that underlie a task or a set of tasks calls for controlled experiments of the kind pioneered by Fleishman and his colleagues (Hogan & Fleishman, 1979), and also for field studies (Sanchez & Fraser, 1994).  However, researchers should approach the study of work activity-human attribute links with caution, because these modules of work-analytic knowledge are likely to change depending on the manner in which work activities are combined.  For instance, coordination and integration skills do not only follow from specific tasks, but from the number and manner in which tasks are combined and performed.  In this respect, estimates of the human attributes required by bundles of tasks are not necessarily equal to the s! um of the attributes required by each individual task, just as the whole is seldom equal to the sum of its parts.

Methods of data collection. As technology evolves the possibilities for greater efficiencies in data collection grow. Electronic performance monitoring offers a rich source of data. Reviews of phone calls recorded automatically for customer service representatives or telemarketers can be an alternate source of critical incidents which in our experience have been difficult for incumbents in many lines of work to generate. Surveys and interviews, the hallmark of traditional job analysis, can now be administered on-line or by telecommunication.

On the other hand the bandwidth limitations make this technology far from completely foolproof, as one of us recently discovered when conducting televideo interviews of professional employees. In too high a proportion of cases, picture quality was borderline, connections continually were broken or never made, and we had to resort to speakerphone interviews. These problems should subside as technology improves and becomes more user-friendly. Groupware will likewise facilitate the involvement of teams in an analysis either in real time or in nominal groups such as are used with the Delphi technique (Olson & Olson, 2003).

Technological warfare like the scenario under trial at Force 21 in Ft. Hood TX provides vivid examples of how technology facilitates the collection of valuable data regarding job requirements.  On-line mapping of the combat field and possibly in real time through the use of remote communication devices such as airborne drones and GPS devices is but one example of this potentially rich source of work analysis information.

When deciding between sources of data involving human judgment (i.e., subjective) vs. those involving “objective” data such as electronic records of activity “counters” and activity time, work analysts should remain mindful of the importance of collecting information about the incumbent’s subjective experience of work. Such data come by necessity from individual accounts that describe not only work, but also the emotions that surround it. Thus, open-ended formats such as interviews and surveys continue to be a useful tool in work analysis.

Where cost is an issue, recent research has shown that work analysis collected in face-to-face groups using carefully selected subject matter experts offer an acceptable alternative to large-scale surveys of incumbents. Levine, Maye, Ulm and Gordon (1997) report that judgmental data drawn from such groups exhibited good reliability and resulted in outcomes in the context of developing minimum qualifications across 14 diverse jobs. Levine and Sanchez (1998) analyzed this database further and found that group level characteristics, such as group size, did not affect the quality of data as reflected in such criteria as inter rater reliability. Some years ago Ash, Levine, Higbee, and Sistrunk (1982) compared survey data from incumbents with data derived from panels of SME’s and found close correspondence.

Access to electronically stored work information is another very exciting development. The O*NET an on-line data base implemented by the U.S. Department of Labor, will, if valid and up to date information is insured, be a worthy replacement for the Dictionary of Occupational Titles (Peterson, Mumford, Borman, Jeanneret, & Fleishman, 1998). Its widely applicable descriptors and ultimately its extensive coverage of occupations across the economy have the potential to render it a meaningful contributor to the validity of work analysis information, even though more detailed descriptions of work such as task statements may be demanded by interventions focused on single job titles within a specific organization.

Where future jobs and work are contemplated, expert systems embedded in computer models should provide useful inputs to an analysis (Coovert, Craiger, & Cannon-Bowers, 1996). Less technology based approaches should also be useful. Brainstorming and the creation of scenarios attempting to capture what a future work assignment and its environment will look like should be employed especially where an emerging technology that has not been used before is involved.

Levels/units of analysis. Given the increasing complexity of work and its context, valid work analysis is more likely to be multi-level in nature. As Levine and Baker (1991) have shown in testing a method for analyzing the work of teams, activities or tasks and personal attributes at the level of the individual team member, as well as team level work flow and analysis of work sequencing must both be present when the target of analysis is a network, team or system. The use of theoretically derived dictionaries of descriptors for various aspects of the analysis will both streamline and render more valid the information yielded by an analysis. Raymark, Schmit and Guion (1997) offer an example in the personality domain.

Management of Information(Dissemination, Storage, and Retrieval) (cf. Brannick & Levine, 2002). Information and data base management can be critical to the success of work analysis. Speed is often of the essence in modern organizations. Dissemination may be facilitated by means of current technology, although care must be exercised to insure that information and data are circulated only to proper sources, i.e., those with a need to know. Storage requires fail safe systems and suitable backup, and there might well be a function where the keepers of the information are notified automatically about the possible need to update the data. Retrieval, as is exemplified by the O*NET, should be feasible at any of multiple levels commensurate with the needs of those who may make inquiries of the database. In addition user-friendly linkages across databases should also be built in, so that such purposes as transitions of people from military t! o civilian occupations is facilitated.   

Suggestions for Future Investigations of the Validity of Work Analysis

In recent years there has been a proliferation of studies dealing with the internal quality of work analysis information (cf. for example Dierdorff & Wilson, 2003). Generally, it has been a matter of faith that once we have insured the quality of the data, good outcomes will follow. On the other hand the use of work analysis could be accused of being overhead on top of the overhead costs usually associated with staff functions like training. Absent compelling evidence, some may view work analysis as a mere adjunct feeding the security needs of those working in human resources or related disciplines, or worse as the organizational scientists’ answer to snake oil, a presumed cure-all with no real value.

Recognizing the dangers of using distal outcome data in evaluating the validity of work analysis, we recently proffered several suggestions for research that speaks to consequential validity (Sanchez & Levine, 1999b). As a starting point we identified four kinds of inferences supported by work analysis, derived and modified from a set suggested by Gatewood and Feild (1994). These are further modified here and include:

1. Derivation of worker or team member attributes such as KSAO’s from work activities or system/equipment specifications.

2. Derivation of organizational interventions based on work activities or human attributes, such as job or system design, selection programs or training programs.

3. Work performance indicators at the level of individuals, teams or systems inferred from work activities, sequences or measures of workload.

4. The effectiveness of organizational interventions inferred from comparisons with work performance indicators.

Perhaps the acid test of the validity of work analysis comes from the fourth member of the set, despite the possibility that intervening circumstances and variables could confound our ability to draw conclusions. Following Huselid (1995), we could quantify the extent to which formal work analysis, together with work-analytic procedures including the transformation guidelines through which analyses impact HR programs are involved in an organization’s operations and correlate this with indexes of organizational effectiveness.

Using a more microscopic approach, Levine, Ash and Bennett (1980) conducted a study in the context of selection that falls within the second alternative. They analyzed four jobs using four different job analysis methods, and had human resource specialists develop examination plans from the resulting reports. These exam plans were then evaluated by a separate group of experts. Results suggested that the Critical Incidents technique resulted in somewhat more favorable exam plans. Had they been able to validate the exams flowing from the exam plans against the performance measures derived from the same analyses, the study would have attained the standard set in the fourth listed member of the list.
In the context of training, we might establish three training programs-- one derived from a work analysis using fairly broad descriptors, one derived from general reviews of how to achieve performance results in a select set of dimensions, and one derived from an even more detailed task analysis. Performance measures derived from an integration of the broad and the fine-grained work analysis would serve as the criteria against which to validate the training programs. An alternate approach similar to the study by Levine et al. (1980) might have training contents judged by trainers and subject matter experts who were ignorant of whether work analysis played a role in their development of the training or the level of detail of the work analysis data.

Manson (2004) recently pioneered this new consequential validity model by examining the consequences of varying the amount of work-analytic information across four conditions based on the amount and specificity of information. Human resource professionals with personnel selection experience developed selection plans, and a separate panel of selection experts rated the quality of such plans. The results supported the collection of moderate and high-comprehensive information and the collection of specific rather than broad information. The use of complete task and KSAO inventories, however, did not lead to superior selection plans than did the use of top-10 task and KSAO inventories.  Therefore, the payoff from detailed work analyses appears to plateau after reaching an optimal “break-even” or equilibrium point balancing practicality and selection plan quality.

Another critical research approach might analyze different ways to make the inferential leaps from work analysis data to applications. For example, we could make empirical comparisons of validity between assessment batteries devised by close adherence to detailed work analysis data versus loose coupling of test types to the data. Or, we might conduct a cognitive task analysis to see how experts use work analysis information to formulate training programs as compared to novices, assuming that the training programs devised by experts are judged superior. These kinds of studies would enable the formulation of sorely needed guidelines for transforming work analysis data into applications. 

Studies like these if done systematically could help establish work analysis standards in addition to providing data about the value of work analysis. For example if broad descriptors worked equally well or better than the more labor-intensive, fine-grained units, work analyses could end at the broader level. Concerns such as cost and time to completion are paramount in today’s competitive environment. Similarly, such studies could determine whether the painstakingly crafted linkages between work activities and worker attributes add value (cf. for example Goldstein & Ford, 2002). Clearly the lack of clear standards, rules or tested theory in such aspects as how to make such inferences as are called for in work analysis weakens the case for fine-grained detail in the work-analytic process.

Mitchell (1988) has documented well the critical role played by the military in developing, improving, and applying work analysis methods. Its leadership would benefit from conducting the kinds of studies suggested here. The diversity of work types, available sample sizes and the wide range of interventions practiced in the services would add immeasurably to the rigor of the studies. It is our hope that the military will see fit to exercise the same level of commitment and leadership it has historically assumed in definitive studies of the role and validity of work analysis.

References 

American Educational Research Association, et al. (1999). Standards for educational and psychological testing. Washington DC: Author.

Ash, R. A., Levine, E. L., Higbee, R. H., & Sistrunk, F. (1982). Comparisons of task ratings from subject matter experts versus job incumbents. Paper presented at the annual meeting of the Southeastern Psychological Association, New Orleans, March.

Borman, W. C., & Motowidlo, S. J. (1993). Expanding the criterion domain to include elements of contextual performance. In Schmitt, N., & Borman, W. C. (Eds.) Personnel selection in organizations. San Francisco: Jossey-Bass, 71-98.

Brannick, M. T., & Levine, E. L. (2002). Job analysis: Methods, research and applications for human resource management in the new millennium. Thousand Oaks CA: Sage

Busby, R. E., & Hutsell, W. R. (1988). Workload analysis and personnel scheduling. In Gael, S. (Ed.). Op. Cit., 536-553.

Casali, J. G., & Wierwille, W. W. (1988). Mental workload assessment. In Gael, S. (Ed.). Op. Cit., 697-714.

Connell, A. F., & Nord, W. R. (1996). The bloodless coup: The infiltration of organization science by uncertainty and values. Journal of Applied Behavioral Science, 32, 407-427.

Coovert, M. D., Craiger, J. P., & Cannon-Bowers, J. A. (1996). Innovations in modeling and simulating team performance: Implications for decision making. In Guzzo, R. A., Salas, E., & Associates (Eds.). Team effectiveness and decision making in organizations. San Francisco: Jossey-Bass, 291-332.

Dierdorff, E. C., & Wilson, M. A. (2003). A meta-analysis of job analysis reliability. Journal of Applied Psychology, 88, 635-646.

Gael, S. (Ed.), Cornelius, E. T. (III), Levine, E. L., & Salvendy, G. (Associate Eds.) (1988). The job analysis handbook for business, industry, and government (2 Vols.). NY: Wiley.

Gatewood, R. D., & Feild, H. S. (1994). Human resource selection (3rd Ed.). Orlando, Dryden Press.

Goldstein, I. L., & Ford, J. K. (2002). Training in organizations (4th Ed.) Belmont CA: Wadsworth.

Guion, R. M. (1965). Personnel testing. NY: McGraw-Hill.

Harris, D. H. (1988). Network analysis. In Gael, S. (Ed.). Op. Cit., 672-682.

Harvey, R.J. (1991). Job analysis. In Dunnette, M.D., & Hough, L.M. (Eds.). Handbook of industrial and organizational psychology (2nd Ed., Vol. 2). Palo Alto, CA: Consulting Psychologists Press, 71-163.

Harvey, R. J., & Wilson, M. A. (2000). Yes Virginia, there is an objective reality in job analysis. Journal of Organizational Behavior, 21(7), 829-854.

Hogan, J. R., & Fleishman, E. A. (1979). An index of the physical effort required in human task performance. Journal of Applied Psychology, 64, 197-204.

Hupp, T., Polak, C., & Westgaard, O. (1995). Designing work groups, jobs, and workflow. San Francisco: Jossey-Bass.

Huselid, M. A. (1995). The impact of human resource management practices on turnover, productivity, and corporate financial performance. Academy of Management Journal, 38, 635-672.

Ilgen, D. R., Hollenbeck, J. R., Johnson, M. & Jundt, D. (2005). Teams in organizations: From input-process-output models to IMOI models. In S. T. Fiske, A. E. Kazdin, & D. L. Schacter (Eds.), Annual review of psychology, 56, 517-544

Jones, R. G., Sanchez, J. I., Parameswaran, G., Phelps, J., Shoptaugh, C., Williams, M., & White, S. (2001). Selection or Training? A Two-Fold Test of The Validity of Job-Analytic Ratings of Trainability. Journal of Business and Psychology, 15, 363‑389.

Kamon, E. (1988). Physiological basis for the design of work and rest. In Gael, S. (Ed.). Op. Cit., 715-730.

Lievens, F., Sanchez, J. I., & De Corte, W. (2004). Easing the inferential leap in competency modeling: The effects of task-related information and subject matter expertise. Personnel Psychology, 57, 881-904.

Levine, E. L. (1983). Everything you always wanted to know about job analysis. Tampa: Mariner.

Levine, E. L., & Baker, C. V. (1991). Team task analysis: A procedural guide and test of the methodology. Paper presented at the 6th Annual Conference of the Society for Industrial and Organizational Psychology, St. Louis, April.

Levine, E. L., Ash, R. A., & Bennett, N. (1980). Exploratory comparative study of four job analysis methods. Journal of Applied Psychology, 65, 524-535.

Levine, E. L., Ash, R. A., Hall, H. L., & Sistrunk, F. (1983). Evaluation of job analysis methods by experienced job analysts. Academy of Management Journal, 26, 339-348.

Levine, E. L., Maye, D. M., Ulm, R. A., & Gordon, T. R. (1997). A methodology for developing and validating minimum qualifications (MQs). Personnel Psychology, 50, 1009-1924.

Levine, E. L., Thomas, J. N., & Sistrunk, F. (1988). Selecting a job analysis approach. In Gael, S. (Ed.). Op. Cit., 339-352. 

Lievens, F., Sanchez, J. I., & De Corte, W. (2004). Easing the inferential leap in competency modeling: The effects of task-related information on subject matter expertise. Personnel Psychology, 57, 881-904.

Manson, T. M. (2004). Cursory versus comprehensive job analysis for personnel selection: A consequential validity analysis. Doctoral dissertation. Tampa, FL: University of South Florida.

Manson, T. M.,  Levine, E. L., & Brannick, M. T. (2000). The construct validity of task inventory ratings: A multitrait-multimethod analysis. Human Performance, 13(1), 1-22.

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from person’s responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

Mitchell, J. L. (1988). History of job analysis in the military. In Gael, S. (Ed.). Op. Cit., 30-37.

Morgeson, R. P., & Campion, M. A. (1997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82, 627-656.

Nof, S. Y. (1988). Job analysis for robots. In Gael, S. (Ed.). Op. Cit., 587-613.

Nygren, R., & Levine, E. L. (1996). Leadership of work teams: Factors influencing team outcomes. In Beyerlein, M., & Johnson, D. (Eds.). Advances in interdisciplinary studies of work teams: Team leadership. Greenwich CT: JAI Press, 67-105.

Olian, J. D, & Rynes, S. L. (1991). Making total quality work: Aligning organizational processes, performance measures, and stakeholders. Human Resource Management, 30, 303-333.

Olson, G. M., & Olson, J. S. (2003). Human computer interaction: Psychological aspects of the human use of computing. In S. T. Fiske, D. L. Schacter, & Zahn-Waxler, C. (Eds.), Annual review of psychology, 54, 491-516.

Peterson, N. G., Mumford, M. D., Borman, W. C., Jeanneret, P. R., & Fleishman, E.A. (Eds.)(1998). The occupational information network (O*NET). Washington, DC: American Psychological Association.

Polito, J., & Pritsker, A. A. B. (1988). Computer simulation and job analysis. In Gael, S. (Ed.). Op. Cit., 570-586.

Price, H. E., & Pulliam, R. (1988). Functional analysis and allocation of functions. In Gael, S. (Ed.). Op. Cit., 639-650.
Primoff, E. S., & Fine, S. A. (1988). A history of job analysis. In Gael, S. (Ed.). Op. Cit., 14-29.

Raymark, P. H., Schmit, M. J., & Guion, R. M. (1997). Identifying potentially useful personality constructs for personnel selection. Personnel Psychology, 50, 723-736.

Salas, E., & Cannon-Bowers, J. A. (1997). Methods, tools, and strategies for team training. In Quiñones, M. A., & Ehrenstein, A. (Eds.). Training for a rapidly changing workplace. Washington, DC: American Psychological Association, 249-280.

Sanchez, J. I. (2000). Adapting work analysis to a fast-paced and electronic business world. International Journal of Selection and Assessment, 8(4), 207-215.

Sanchez, J. I. (1994).  From documentation to innovation:  Reshaping job analysis to meet emerging business needs.  Human Resource Management Review, 4, 51-74.

Sanchez, J. I., Fraser, S. L. (1994). An empirical approach to identify job duty-ksa linkages in managerial jobs: A case example. Journal of Business and Psychology, 8(3), 309-325.

Sanchez, J. I., & Levine, E. L. (2000). Accuracy or consequential validity: Which is the better standard for job analysis data? Journal of Organizational Behavior, 21, 809-818.

Sanchez, J. I., & Levine, E. L. (1999a). Is job analysis dead, misunderstood, or both? New forms of work analysis and design. In Kraut, A. I., & Korman, A. K. (Eds.). Evolving practices in human resource management. San Francisco: Jossey-Bass, 43-68.

Sanchez, J. I. , & Levine, E. L. (1999b). Inaccuracy in job analysis data: How does one know? Paper presented as part of the symposium, “The validity of job analysis judgments,” chaired by Jeffrey A. Ryer, at the 14th Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta, April.

Sanchez, J. I., & Levine, E. L. (1998). Sources of inaccuracy in job analysis and suggestions for remediation. Paper presented as part of the symposium, “Job analysis inaccuracy: Cracks in the foundation of HR management?,” co-chaired by M. A. Campion, & F. P. Morgeson at the 13th Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, April.

Schippmann, J. S., Ash, R. A., Battista, M., Carr, L., Eyde, L. E., Hesketh, B., Kehoe, J., Pearlman, K., Prien, E., & Sanchez, J. I. (2000). The practice of competency modeling. Personnel Psychology, 53, 703-740.

Siegel, A. I. (1988). Identification of potential hazards and job stress. In Gael, S. (Ed.). Op. Cit., 796-803.

Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures ( 4th Ed.). College Park, MD: Author.

Spector, P. E. (2006). Industrial and organizational psychology: Research and practice. Hoboken NJ: John Wiley and Sons. 

Sulsky, L., & Smith, C. (2005).Work stress. Belmont CA: Thomson Wadsworth.

Van Cott, H. P., & Paramore, B. (1988). Task analysis. In Gael, S. (Ed.). Op. Cit., 651-671.

Wilson, M. A. (1997). The validity of task coverage ratings by incumbents and supervisors. Journal of Business and Psychology, 12, 85-95.

An abbreviated version of this paper was presented at the Job Analysis Workshop preceding the 14th International Occupational Analyst Workshop, San Antonio, April 4, 2005.