The field of Natural Language Processing (NLP) within computer science has developed methods for indexing, categorizing, summarizing, and interpreting large numbers of text documents. The testing industry has made use of these methods in the development of automated essay scoring engines (Shermis & Berstein, 2003). Recent research on item parameter estimation has also made use of NLP methods (Belov & Knezevich, 2008; Hall, 2008). However, outside of these areas, the testing industry has remained ignorant of the capabilities of NLP relative to our biggest assets—our item banks.
The testing industry works predominantly with large collections of text, and we have technology for organizing test items (content management tools). While we have moved away from shoe boxes for organizing test items, all the work done to review and evaluate item banks is performed by humans reading those items. This paper presents a proof-of-concept for the application of NLP techniques to several areas of test development and item bank management.
Theoretical Framework
Beginning in the 1950s, a new discipline arose in computer science devoted to make computers understand natural language, a language spoken or written by humans for general-purpose communication. The goal of Natural Language Processing is to convert samples of human language into more formal representations that are easier for computer programs to manage. Since the 1980s, research interest began to focus on systems that could deal with written language in paragraphs instead of with typed interactions by computer users. At the same time, with the idea of relaxing the goal to process every word of the input as deeply as necessary to produce an understanding of the sentence as a whole, researchers started to accept the value of “partial understand” of the sentence, considered more feasible and useful. (For more detail on the history of NLP, see Bates, 1995.)
more... white paper