what you do is what you see

Large Interactive Display Surfaces (LIDS)

Introduction

The large interactive display surfaces (LIDS) concept started with the creation of the "Whiteboard Paradigm". There were many available technologies that could be used as LIDS, however, most were prohibitively expensive, and many still did not support appropriate interaction styles. The goal of the LIDS research project has been to develop inexpensive technologies to use as displays, and investigate the interaction issues generated by their use. Furthermore work has gone into investigating potential uses for such technologies, and creating the software to support these uses.

 

The Whiteboard Paradigm

back to top
ordinary
    whiteboard

In personal computing, "direct manipulation" interfaces such as the desktop metaphor are used extensively. These interfaces are helpful to the new user; by providing familiar cues and a direct mapping from interface to action they allow a user to learn an interface very quickly. Furthermore as experience with computers has become widespread, developers can rely more on users knowing concepts like the "double-click" that are not immediately obvious from the interface.

When a group of users need to use a computer together though, this seemingly direct manipulation interface is shown not to be as direct as it initially seemed. An object on the screen is not moved directly "on-screen" but manpulated by moving the mouse which is next to the computer. This is fine for a single user — because they are moving the mouse, the manipulation feels direct. However, when someone not in control of the mouse has an interest in what is happening in the interface, mouse movements are too small and obscure, resulting in the need for a more direct manipulation.

The whiteboard paradigm of interaction is one possible answer to the lack of directness experienced by users when more than one person is using a traditional machine. With a whiteboard What You Do Is What You See (WYDIWYS), and therefore the entire interaction is transparent to everyone involved. When interacting with a computer within the whiteboard paradigm, interactions will be performed with a pen-like device in a manner paralleling a pen on an ordinary whiteboard, and the software will support whiteboard-style interactions. Advantages over using an ordinary whiteboard include being able to capture information to a hard disk for later use before clearing the screen, the ability to use traditional computer applications such as Microsoft PowerPoint™, and interoperability with other devices (such as handheld computers).

The main characteristics of the whitebaord paradigm of interaction are:

  • a shared visual space;

  • total transparency of interaction;

  • use of an existing medium for meeting support;

  • encouragement of spontaneous interaction.

These characteristics rely on the informality and ubiquity of ordinary whiteboards.

 

LIDS at Present

back to top

One of the goals of the LIDS project is to develop an affordable technology from readily available materials. Presently a LIDS package consists of:
  • a standard computer

  • a screen made of materials suitable for back projection

  • a projector

  • a "pen" package that can interact with the computer (e.g. the Mimio® pen)

All of these components are (comparatively) cheap and readily available. The Mimio® pen is an integral part of the LIDS research, at least here at Waikato, and some work has gone into researching how ordinary PC software can be operated with the pen.

The development of this hardware has allowed research into how the LIDS screen can be used for lecture capture and meeting support, these being two of the area where ordinary whiteboards are commonly used and where the advantages of the LIDS technology could be very beneficial. To this end, two major pieces of software have been developed:

LIDS under scrutiny
  • "gesture recognition" software that allows operation of Microsoft PowerPoint™ via pen input (including slide navigation and drawing on slides);
  • the "shadow software" for remote meetings, which given two LIDS screens with attached video camera will cast a "shadow" of the person standing in front of one screen onto the other. Combined with the actions from one screen (e.g. writing or sketching), appearing on the other, this promotes awareness amongst remote users.
the mimio pen

This software has already undergone some usability testing in University of Waikato Usability Laboratory.

 

Research Directions

back to top

There is a great potential for research with the LIDS technology, both in hardware and software directions. At present work is going on in four different institutions to a variety of ends. Major directions are:

  • Handwriting recognition

    LIDS has great potential for informal interaction by means of handwriting, rather than keyboard use. However, handwriting recognition software is still not accurate enough to allow interaction which is not error-prone and frustrating. Investigation into how best to avoid this frustration when using a LIDS screen is ongoing. One possible solution is to use the "Graffiti" input method, developed for the Palm®.

    Another area where the LIDS screen has great potential is in mathematical equations; at present these are tiresome to digitise. Investigation into how LIDS could allow these to be written naturally, and then convert them to some recognised format (such as Microsoft Word™) is ongoing.

  • LIDS as a software development tool

    Beryl Plimmer is investigating how interfaces drawn on the LIDS screen may be translated directly into Visual Basic. This would enable very rapid prototyping of interface ideas, even by interface designers without strong technical skills.

    John Grundy at the University of Auckland is investigating how the LIDS technology might be used as a software engineering tool, for example in capturing UML diagrams or use case scenarios. This works in with the use of LIDS for meeting support. This research may lead to LIDS technology being integrated with industry standard software, such as Rational Rose.

    The University of Waikato Usability Laboratory utilises the LIDS technology in collaborative design activities.

  • Hardware and technology development

    The LIDS screen is a technology which is still under development. There is great potential for investigation into both hardware and the underlying software technology employed by LIDS. Here at Waikato, an attempt is being made to write a Mimio® driver for Linux, and investigation into how to detect the use of more than one pen on the screen at one time is ongoing.

    At University of Auckland, Rick Mugridge is investigating the use of a "double-ended" pen, where both ends of the pen are detected by the software. This allows the angle of the pen to be used as an input; potential uses of this are LIDS-based calligraphy, and mode changes based on pen-angle.

    The Auckland University of Technology team are developing the LIDS hardware, including investigating packages of hardware for schools, and varying configurations of hardware for different environments. This team is examining new screen materials, different pen technologies, and projector requirements as part of their work.

    A LIDS screen in use

  • Use and usability
    LIDS in
	 use at open day

    There is huge scope for investigating possible uses and usability of the LIDS technology. These investigations might include traditional usability, sociological studies of how a LIDS affects group interactions, studies of the LIDS as a mechanism for e-learning... there are many possibilities.

    There is one study currently planned by the Auckland University of Technology in conjunction with experts in education: LIDS is schools. Schools will be provided with the technology and a list of possible ways to use it. Once the technology is in place, its use and reception will be observed.

  • Speech recognition and lecture capture

    Some work has gone into capturing the speech of a lecture, and using it to annotate Microsoft PowerPoint™ slides shown during a lecture. This is coupled with the PowerPoint™ control software described in The LIDS technology at present to allow lecturers to give lectures with the LIDS technology, and attach the speech they give to appropriate parts of the slideshow automatically. Research into how best to allow lecturers to edit the resulting annotations and speech is ongoing.

There are, of course, many other potential avenues of research with the LIDS screen. What is mentioned here is just some of the research currently being done.

 

Selected Publications

back to top

  • Apperley, M., Jansen, S., Jeffries, A., Masoodian, M., Paine, L., Rogers, B., and Voyle, T. (2002): "LLC Lecture Capture and Editing Tool for Online Course Delivery". Proceedings of E-Learn, World Conference on E-Learning in Corporate, Government, Healthcare and Higher Education, Montreal, Canada, October 15-19, p 1866-69.

  • Plimmer, B. and Apperley, M. (2002): "Computer Aided Sketching to capture preliminary design", in Porceedings of the Third Australasian Interface Conference, Melbourne, Australia.

  • Plimmer, B, and Apperley, M. (2002): "Freeform: An informal environment for interface prototyping", in Proceedings of the SIGCHI-NZ Symposium on Human Computer Interaction Hamilton, New Zealand, July 11-12 p 11-12.

  • Apperley, M. Dahlberg, B., Jeffries, A., Paine, L., Phillips, M. and Rogers, B. (2001): "Development and application of large interactive display surfaces", in Symposium on Human Computer Interaction Palmerston North, New Zealand, p 3-7.

  • Apperley, M. Dahlberg, B., Jeffries, A., Paine, L., Phillips, M. and Rogers, B. (2001): "Lightweight capture of presentations for review", in Volume 12 IHM-HCI Conference on Human Computer Interaction Lille, France, p 41-42.

  • Plimmer, B. and Apperley, M. (2001): "FreeForm: Informal Design on a large interactive display surface", in Symposium on Human Computer Interaction Palmerston North, New Zealand, p 81-83.

  • Plimmer, B. and Apperley, M. (2001): "From sketch to form on a large interactive display surface" in 14th Annual Conference of the National Advisory Committee on Computing Qualifications, Napier, New Zealand, p 371-374.

  • Apperley, M. and Masoodian, M. (2000): "Supporting collaboration and engagement using a whiteboard-like display", in Shared Environments to Support Face to Face Collaboration: A CSCW Workshop, Philadelphia, Pennsylvania, p 22-26.

For more publications from this group, see our publications page.

top

Valid HTML 4.01 Valid CSS