NSF/JISC Repositories Workshop
Janet H. Murray, Georgia Tech
April 17, 2007
Download: PDF Version  WORD version

The advent of the computer has made possible an event that has happened only a few times in human history: the creation of a new medium of representation. The name computer is misleading in terms of the power of this change, since a machine that executes procedures and processes vast quantities of symbolic representation is not merely a bigger calculator. It is a symbol processor, a transmitter of meaningful cultural codes. The advent of the machinery of computing is similar to the advent of the machinery of the movie camera or the TV broadcast. The technical substrate is necessary but not sufficient for the process of meaning making. I call this process “Inventing the Medium” in a textbook on digital design that I am currently writing.

It is useful to think of a medium as having three characteristics: inscription, transmission, and symbolic representation. Inscription concerns the physical properties (the mark in the clay, the current through the silicon); transmission concerns the logical codes (ASCII, HTML, XML, JAVA). Representation is the trickiest part because it is a combination of logical and cultural codes. Representation relies on the negotiation of conventions of interpretation that set up the framework for receiving information. When these conventions coalesce into stable, complex, commonly recognizable units then we have genres.

Genres are partly composed of formats, patterns of organization that act as containers for information. A television sitcom exists within a format of a teaser, followed by 3 acts and a coda, with credits and commercial breaks punctuating these divisions. It also relies on more emotionally and culturally resonant and ambiguous patterns such as the bubbling boss, the demanding mother, the inept person, the insulting person, etc. We can make sense of a new story because we recognize these patterns through many different instantiations. Just as with the formatting patterns, the familiar fades into the background, directing our attention to the novel part of the message: the particular character. News stories, textbooks, scientific journal articles, statistical reports, also have genre conventions that allow us to ignore the routine information and foreground the novel information.

The digital medium is a capacious inscription technology with a wealth of formatting conventions and logical codes for reproducing legacy documents of many kinds. But it has inadequate native genre conventions to allow us to focus our attention appropriately and to exploit the new procedural and participatory affordances of the medium.  Instead, we rely on legacy conventions such as pie charts, headlines, even unordered lists. At the same time the encyclopedic capacity of inscription raises our expectations, creating what I have called the “encyclopedic expectation” that everything we seek will be available on demand.

Genre creation is how we use a new inscription and transmission medium to get smarter. For example, the printing press allowed us to put words on a page in a standardized manner and to distribute the words in multiple portable copies. This is the technical substrate. But the scientific treatise and the novel did not appear until two or three centuries later, because they required the invention of new representational conventions such as the smooth vernacular prose paragraph, the chapter title, the numbered page, the first person narrator, the philosophical essay.  The objects we now recognize as books make sense to us because they draw upon so many genre conventions.

The invention of the book increased our ability to focus our individual and shared attention, allowing us to sustain and follow an argument too long to state in oral form,  and to elaborate and examine an argument together across several books, across time and place. It led to the growth of domains of systematic knowledge, to shelves of books that relate to one another. We can understand one another across time and place when we refer to a domain of investigation because we have the shelf full of books to refer to.

The invention of a genre, therefore, is the scaffolding for shared knowledge creation, for extending the cognitive reach of human beings. It allows us to focus our attention together in order to create more complex media and more complex thinking.

Genre and Knowledge Creation in Digital Media

The computer is still in its early stages as a medium of representation and it has brought us a limited number of new symbolic genres. The most active genre design has been in the development of video games, which exploit the procedural and participatory power of the computer to create novel interaction patterns, new ways of acting upon digital entities and receiving feedback on the efficacy of one’s actions. Will Wright, the inventor of Sim City, the Sims, and other simulation games, has called computer games “prosthetics for the mind.”  His simulation worlds are perhaps the most successful implementations of the affordances that Seymour Papert first pointed out in (Papert 1999), the ability of computers to create worlds in which can ask “what if” questions, in which we can instantiate rule systems and invite exploratory learning. Sim City works as a resource allocation system in which we make decisions about zoning and power plants and watch a city grow according to the parameters we have chosen. Sim City is a toy but it uses some of the assumptions of professional urban planning simulations. Similar simulation systems are in use in scientific and social science contexts, and they are increasingly used to simulate emergent phenomena that could not be captured in any other way. These are specialized tools and they do not necessarily work across disciplines or related domains.

Tim Berners-Lee(Berners-Lee, Hendler et al. 2001) (Berners-Lee, Shadbolt et al. 2006) is the foremost advocate of a more powerful procedural genre. His vision of the semantic web would move from documents to databases, would give shared resources on web pages the coherent form of databases and would allow multiple procedures to be applied to these standardized data. The semantic web is the most ambitious vision of the development of large data resources into new knowledge. But in his recent reappraisal of the idea, Berners-Lee laments the reluctance of knowledge communities to come together to establish interoperability in the most trivial exchanges, and, more significantly, to do the hard work of inventing common vocabularies with which to tag information for common purposes.

Current strategies for sense-making of large data sources have limited success but point to the kinds of strategies that, over time, hold the promise of creating a richer shared representation.  Search engines still return much unnecessary information and miss key information; folksonomies provide uneven tagging of large resources. But to the extent that Google and Flickr and del.icio.us are useful to us it is because they leverage the efforts of many distributed annotators. Google owes its success to a key insight that the syntax of links is itself semantic; by using anchor text as a collectively created index to the web they gained a more reliable picture of what pages were about than others did by only relying on full text search.

As Bill Arms points out, Greg Crane has correctly identified the computer itself as the only possible reader of every page of a large archive. And in the case of images, we can’t even say that the computer has looked at them since our visual analysis tools are so limited. But that does not mean that the answer to understanding the data lies in teaching the computer new algorithmic tricks. Too much of what we need to understand we can only understand through the focused attention of a reliable collaborator. The computer cannot be that collaborator unless we provide semantic indexing structures similar to the anchor tag that Google has so richly exploited. 

I would suggest that the best way to do knowledge creation that exploits the vast new resources that are migrating into digital form and increasingly being “born digital,” we may not want to think of the process as “data-driven” since this can suggest a purely logical solution applied through automated approaches. Instead we should see the computer as a facilitator of a vast social process of meaning making.  This will have to mean something much better organized than folksonomy tagging, and will have to include ways of implementing collectively created rule sets as well as collectively created annotations. We need ways of creation simulations that interact with one another and also ways of sharing the task of annotating texts. Part of this process may be the creation of automated tools, but the tools can only implement shared understandings that arise among different communities. We cannot generate the rules from the data alone.

A modest, concrete beginning: Juxtapositions

Ted Nelson has long pointed to juxtaposition as a key underexploited affordance of digital environments. He finds the current World Wide Web inadequate largely because of its limited ability to allow a user to place one thing beside another, to compare versions side by side, or to bring together related instances of the same object. (see for example his recent Google talk, available on Google video for a remarkably lucid statement of his radical hypertext aesthetic.)

There are many tasks in the humanities that would benefit from more precise juxtapositions and from shared tools that invited distributed genre creation around juxtaposed objects. Perseus, the Oyez project and the Matrix project, represented include significant efforts in this direction. I think that film art offers a particularly appropriate opportunity to shape scholarly discourse in a way that produces new knowledge in the emerging context of digital films by supporting juxtapositions that capture knowledge that has not been apparent or representable before. My most recent project is The Casablanca Digital Critical Edition, a project funded by NEH and involving the AFI, Georgia Tech, and Warners Home Video, is an attempt to create a system that will allow scholars to bring together a classic American film with the originating play script, the shooting script, detailed production reports and memos, and expert commentary. It is meant to be a model of a common information design for American movies, which would allow studios to control copyright and scholars to have access to very precise, semantically segmented sequences with the same precision of reference as we expect to have over print materials.  Such a system would also afford preservation of the original context, which has also been instanced by Ted Nelson as a crucial component of a powerful hypertext system.(Murray 2005)  I am calling it a system because it involves a complex information design and authoring and display tools. But at the heart of it is the re-imagining of the knowledge genre of the variorum text, the critical edition, and perhaps the scholarly journal. It is a critical edition meant to live within the wider information landscape of a complete digital archive of films. I instance this as an example of an approach that I think would be productive in other disciplines as well. Scientists as well as humanists can approach knowledge-making as the invention of new conventions of representation that coalesce into new genres. I would suggest that these genres include tools for convening a community of experts around semantically segmented resources that afford juxtapositions across media and data types that have not been possible before.
 

Berners-Lee, T., J. Hendler, et al. (2001). "The Semantic Web." Scientific American(May 2001).           
Berners-Lee, T., N. Shadbolt, et al. (2006). "The Semantic Web Revisited." IEEE Intelligent Systems.
Murray, J. (2005). "Here's Looking at Casablanca." Humanities.
Papert, S. (1980). Mindstorms: Children, Computers, and Powerful Ideas. New York, Basic Books.