ScreenCrayons: Using Screen Captures for Annotation and Research ScreenCrayons: Using Screen Captures for Annotation and Research

by Taufer, Trent Alan

In a world full of digital information we should be able to easily collect, organize, annotate, and leverage information from many different sources. This should be easy to do and not interrupt our normal workflow. A system to support information collection and organization should be user-friendly and as unobtrusive as possible, while still allowing for flexible and intelligent annotation. It should also be able to leverage the inherent information content of a collection of annotated information. We present a system that will demonstrate how these ideas can come together to make information collection easier and more productive. The system facilitates collection, organization, and annotation of information using screen captures, and leverages the information content of the annotated collection to automatically summarize the information and find additional related information via searchable document repositories.

ScreenCrayons is a system for collecting annotations on any type of document or visual information from any application. The basis for the system is a screen capture upon which the user can highlight the relevant portions of the image. The user can define any number of topics for organizing notes. Each topic is associated with a highlighting "crayon." In addition the user can supply annotations in digital ink or text. Algorithms are described that summarize captured images based on the highlight strokes so as to provide overviews of many annotations as well as being able to "zoom in" on particular information about a given note and the context of that note. Annotations the user makes on the screen captures are automatically associated with regions of text that are then used to formulate queries to a search engine. The results of these queries are filtered and ranked based on their similarity to the original annotations. The system then presents links to these documents to the user. We also describe an experiment that shows that the documents found by the annotation system are generally found to be more relevant to a user's topic than the users own queries using the search engine.