User Interface Challenges for Digital TV and Proposed Approaches To Address Those Challenges
Prepared for LG Electronics Research Center Of America, 26 May 1998, By John M. Kirkwood
© 1998 J. M. Kirkwood
A Short History of TV
As the television device grew from a modified oscilloscope through the black-and-white era to color and to large screen, it remained a one way medium to present content of the broadcaster’s choice to the “viewing audience”. The TV set, due to its low resolution, 30/60 Hz flicker, color aliasing and scan interlaced picture, was, and remains today, a rather unapproachable device in terms of viewing distance. Even the advertising content (jolting graphics, disorienting transitions, barking announcers and loud harsh sound) reflects the viewing distance that everyone finds normal.
For the above reasons and to
accommodate more than one viewer, the TV set today is viewed from a distance of
a few to ten or more times the diagonal screen measurement. The TV set is also
accepted as something that one does not control except for set-up and an
occasional change of channel or muting of commercials. The digital technology
of computers now being applied to the television set will solve the picture
deficiencies and produce a device that has the attributes of a TV set and a
computer monitor. As this change takes
place, the user interface (UI) of this transformed device must accommodate
various social aspects of the TV including viewing distances and control
approaches.
The challenge
The challenge of taking the television set and its user interface into the digital future is, then, at least as much social as technological. We, as designers and builders, will need to propose and execute insightful solutions that excite and entice the user to adopt a new paradigm for his relationship to the TV. The TV is no longer a “read only” device. The new paradigm will encompass the concept of user control of program content, add the idea of a variable viewing distance, include concepts of multiple sources for a single picture and multiple pictures for a single device. Because the digital TV can and will be connected to the internet, the concept of the user as communicator and broadcaster needs to be added to the paradigm. In time, because all information transfer will be digital, the TV will go from receive-only to one-on-one or one-on-many send and receive. The UI of a successful commercial television product must be SIMPLE, based on a paradigm that is easy to grasp, is easy to extend to new, and often unforeseen, technologies and is a delightful unburdening of the all-to-busy user. This is a social and technological challenge. It will not be easy. It will, however, be a joy to see it through the various steps of design and production and into the users hands. Let us now look at some of the tools and concepts that can be used to take us there.
The User Interface as a Social Tool
Throughout history the UI of horses, cars, airplanes, typewriters, computers and so forth, have conformed to certain conventions or standards. Usually these were not conceived as a complete design at the outset but grew a bit here and were improved some there. A notable negative example we all live with today is the “QWERTY” key layout for computers and other devices. The first key layout was too easy for the user and caused problems for the technology of that day. The QWERTY layout was conceived to thwart any attempts at speed typing. Very often in the past the user interface has suffered to make up for crude mechanisms that required large control forces from the user. The automobile steering wheel of today might have long ago been replaced by the stick or yoke used in airplane control if wheels were as easy to move as ailerons. Even though we now have the amplifiers or servo drivers necessary to control an automobile with a low-force stick, social forces require that we continue with the steering wheel.
A positive example is that of the point and click computer user interface invented at Xerox Palo Alto Research Center (PARC) and brought to commercial viability by Apple Computer Co. in its Lisa and Macintosh computers. Belittled at first as a toy and put down as not worthy of “real” computers, the Macintosh user interface brought a welcome relief from the enigmatic “command line” UI. The “desktop” paradigm that it employed has been refined and improved by apple since it was first seen on the “Mac” yet remains unchanged in its foundation concept. Widely copied by other computer operating system makers including Microsoft, Sun and Silicon Graphics and utilized by application designers for these machines, the point and click user interface is today, on more than 95 percent of home and business computers.
The pioneering UI designed for digital TV will undoubtedly set a precedence and provide a standard for a long time to come. Today, we are unfettered by much of the mechanical baggage of the past and are empowered with swift and sensitive technologies to build an exemplary user interface. If the UI is done right, it will be a thing of beauty and facility working tirelessly to convey the best information in the best format to the right audience/user in a timely manner. That done, the interface system will return information to the creator(s) on who the audience was, when they witnessed the presentation, what they thought of it, and so forth. If it is poorly conceived, its deficiencies may drag on for decades or, it may mercifully fail as a commercial venture. Can we succeed? Yes. The question is not so much one of if, but one of what, when and in what order. The plan needs to be one of years if not decades, envisioned in such a way that the products brought to market to meet today’s needs dovetail into the concepts of the next phase as new technologies become available. This paper deals with the proposed user interface as a unified concept with some necessary attention to the interim steps.
Necessary Technologies for The DTV UI
The Internet
The Internet, and the digital data transfer network that it will evolve into, will provide the driving force behind the digital television user interface development. Without this bi-directional link to all the other digital television users, the control devices of today might suffice for some time to come. With it we have a new paradigm of global dialog among all who care to join. The Net and digital TV will bring anyone and everyone from all places and time zones face to face. The user interface challenge will also be one of real-time multilingual dialog among contrasting cultures.
Processing Speed
The synergism of the UI, the software, and the hardware will not be born without the immense processing power of digital devices now in the planning stages. The necessary optical fiber backbones and pipelines, digital servers, switches, and hubs will soon be in place as Vice President Al Gore unveiled at the White House in April 98. Mr. Gore said that the new fiber-optic network, to be operational by the end of 1998, will be able to transmit the entire contents of the Encyclopedia Britannica in one second. Also at the meeting, Joseph Nacchio, chief executive officer for Qwest, said that improvements in networks may soon lead experts to measure transmission speeds in LoCs, (the number of times the content of the Library of Congress can be transmitted per second). By all accounts, the necessary network speed will be available.
The Digital TV Screen
The Cathode Ray Tube (CRT) and the Liquid Crystal Display (LCD) now widely used for computer and TV picture presentation may soon give way to display panels based on other technologies such as the Electrocrystalochromic technology disclosed by Mr. Kirkwood in patent # 4,982,184,, and the hundreds of patents that reference it. The Electrocrystalochromic concept can be scaled from very small to very large, from grey scale to a color gamut beyond human discrimination and can scale up from a two dimensional flat picture to real-time three dimensional holographic images. (More details on this patent can be found at URL http:// www. patents. ibm. Com/ details?patent_number=4982184.) With each step forward, the data flow requirements will increase, keeping pace with advancing networking speeds. Ultimately, whole walls may present vistas reproducing all the sight and sound information (and perhaps other information later on) that is humanly perceptible, transmitted from any point in the universe to which man can send a probe.
The Proposed Digital Television User Interface
The proposed DTV UI paradigm can best be described in terms of capabilities rather than software, circuits or assemblies. The foundation of the UI is communication from the user to the DTV set by means of gesture, information gleaned from the DTV camera field of view, voice, and digital commands from a handheld electronic device. Data flow from the DTV to the user will be by means of images or sub-images on the DTV screen, images or sub-images on the handheld device screen, and audio output from both the DTV set and the handheld device. This is deliberately a very wide range of UI input and output data channels. Each enhancement of the UI over its useful life will lead to improvements in how each data channel is used. For example, in its first public release, the camera image might only be used to sense large motions while in later releases the camera image might be used to detect facial expressions.
Sight
Digital cameras and camcorders are now becoming more commonplace in the consumer market and prices are approaching those of high end consumer models. The DTV UI must accept input from a live camera as well as from recorded media. Including a camera in the purchase price or recommending a certain model and including an IEEE 1394 (“Fire Wire”) and Universal Serial Bus (USB) connection ports as standard on the DTV unit will make future upgrades in the equipment and the UI simple and intuitive. The camera may be on when the UI is being used even though no picture is being recorded or transmitted to let the DTV unit be aware of its surroundings. This awareness will begin at a very basic level and improve over time as technological advances permit.
Hearing
The digital hearing abilities of the proposed UI will, at a minimum, equal the quality of the current compact disk recording technology and equal the voice recognition capabilities of current computers. As the audio channel is used more extensively for control during recording and multimedia integration, a headset with earphones and microphone might be used to keep the control and media channels separate.
The Kinesthetic Sense
A modest degree of kinesthesia is proposed for the DTV UI by incorporating micro-machined milti-axis accelerometers within the wireless pointing device. This device will function as a mouse when used on a work surface near the screen and, as it moves further from the screen, will function more like a laser pointer. Icons, text and the on-screen pointer will be enlarged in direct proportion to the user’s distance from the screen. The pointing device will transmit commands, nuances of feeling, and gestural information such as slow, fast, fluid, abrupt and chaotic by sensing all user hand motions of rotation and translation. These will then be interpreted and integrated into the program creation instructions for program content such as audio, video and scene transitions. By this means, the user will be able to roam freely and still maintain creative and effective control.
The Paradigm
The DTV UI software and hardware need to act as and become a trusted digital friend in the sense that the UI will assess the facts of any given situation, make decisions, and present them as possibilities to the user. At first, this assistance may be no more complicated than automatic spelling correction or audio and video synchronization during a multimedia transition. Yet, as processor speed increases and more data from more sources can be assessed, these agent decisions and recommendations will improve to the point that it may seem to the user that a human assistant is at work. Also, the UI will handle all aspects of software management including the replacement of outdated applications by downloading fresh copies from the internet, protection from all untoward changes in the system software and applications, resolution of software conflicts, general housekeeping, system optimization, and related duties. In the cases of serious malfunction, the unit may be repaired on site or in the shop, but under all other circumstances the user will concern himself with the DTV UI system complexities only somewhat more than one does with a telephone or automotive digital controller.
The following are qualities necessary for the DTV UI to approach, at first, and ultimately achieve, its paradigm goal. These will have modest beginnings but will mature with time as technology improves.
* Organization: This skill will keep the work space clean and free of viruses and conflicts, will keep all applications up to date, will keep things stored in logical (and intuitive for the user) places for easy retrieval.
* A good listener: Able to hear and act on verbal instructions, able to discriminate the flood of sounds (baby crying, music, voices, noise, etc.) and incorporate, into the multi-media content, the “mood” of relevant sounds.
* Quick to learn: With a nearly flawless memory.
* Able to associate commands that are not quite correct with the desired action.
* Expandable: into additional skills as science advances. A main underlying factor in this proposal is that a place needs to be reserved in the DTV UI software plan for all the proposed concepts of the DTV UI no matter how little of the concept can be incorporated at the outset. All concepts need a channel through which to act and be called to action.
* An eloquent speaker, able to dialog concisely.
We Drop in on the User
The following example dialog may help to clarify how the DTV UI works. Words in inverted italics are key to the UI understanding commands and speaking concisely. This is included in this paper in lieu of a functioning prototype and to bring into one section the many and varied concepts that compose this UI proposal.
In the user’s TV room, Chris and one of the later versions of the DTV UI are conversing.
Chris: “Hello, TV. This is Chris.”
The DTV UI recognizes the call to action, but before responding, checks the voice-print of these words with those on record for Chris. Finding that they match, the DTV UI replies in a permanently pleasant and optimistic mood.
UI: “Good morning, Chris”
Chris: “I’d like to send Paul … Paul … The one that lives in Toledo …”
UI: “Paul Worthington?”
Chris: “Yes, that’s right. I’d like to send him a video card of our trip to Japan last year.”
UI: “OK. I’ll have the rapid reviews running for you in a moment.”
As Chris watches the reviews, UI assesses Chris’s reactions and mood >from temperature, moisture and motion information transmitted by the hand held control.
UI: “You seem upbeat about this, Chris. Is it a happy occasion?”
Chris: “Yes, very much so. Paul is finally getting married and the happy couple will be honeymooning in Japan.”
UI: “Very nice. I have detected some out of focus or blurred frames in this series. If you want to keep any, just click while they are playing and I will save them. The rest I’ll dump. OK?”
Chris: “Fine.”
Chris watches and selects some to keep. UI expands some background detail to fill the gaps left by the deleted material while retaining audio continuity.
Chris: “We’d better delete that shot of Paul and his old girlfriend.”
UI: “I find six scenes of couples. Please pick the one you would like to delete by pointing to it on the screen.”
UI puts up six small still images on the screen and Chris picks, with the pointer, the one(s) to be deleted. Matching images of faces to names is not yet included in the UI’s list of capabilities.
UI: “There are some areas where there is only wind noise. I’ll run the Lute music you bought when you were in Japan behind the whole card and bring it up to cover the wind noise. OK?”
The UI is always positive and proactive, presenting a complete solution to a detected problem. A “No” from the user will elicit another candidate solution or the user can take control and create as he chooses.
Chris: “Yes, but don’t cover the music in the temple.”
Modifications to the proposed solution are carried out easily.
UI: “OK. When do you want to see the completed card?”
Chris: “Five thirty today will be fine.”
UI: “OK. Anything else?”
Chris: “Yes, today’s news and the price information on those system upgrades you found. And leave the commercials* in this time. There is supposed to be a funny one.”
UI: Here’s the news. I’ll be standing by.
*Soon, television will no longer be a “mass media” for those with DTV. Commercials, as we know them, will be completely filtered out of the occasional digitized analog feed unless asked for. The replacement for the overt hucksterism that passes for information today, can be seen on internet sites where pages with the most high quality information get the most “hits”. The majority of DTV programming will be available on demand from servers and therefore will be viewed at a time of the user’s choosing. Top quality programming will be watched repeatedly with the extraneous promotional material simply edited out. Poor quality material will just stay on the server, or better, never get made.
I hope this sample dialog and discussion has made clear where technology can, in fact IS, taking us, and how we can harness this technology with farsighted and insightful UI design.
Ten Steps to a New User Interface
The following steps lay out a procedure for developing the new user interface paradigm.
1. Brainstorm with the best technologists concerning when, how, in what order and at what cost the parts of a long-term DTV UI plan can be developed.
2. Select the number of and definitions for all the data channels that will be necessary to carry out the long-term plan.
3. Select a subset of items in # 2 that work well together for UI development in the near future. Make a list and check with the user community. Start hardware development.
4. Prototype several candidate user interfaces and check with the users.
5. Link the top UI contenders from step # 4 to code that will produce a working prototype. Test these with the users. Coordinate with hardware.
6. Set up a Web site with the necessary server side software on a very high speed intranet that can reach a representative sample of users. Distribute the client side software to the users. Test it. Change it. Debug it. Let the users use it. Listen to the feedback.
7. Write the real software with all the hooks required by step # 2. Keep it as simple and reusable as possible. Make it beautiful.
8. Bring the hardware and software together. Begin manufacturing.
9. Place the first production units with selected users without cost except that they would be required to use them, introduce the units to others and return or purchase the units when the test was complete. Give these users a good deal on the price in exchange for their contributions.
10. Market the unit. Enjoy the rewards. Continue the software and hardware development and watch the enduring positive changes it will have on our social structure.
Conclusion
The television set, though it has a somewhat longer history, and the computer have arrived at a digital crossroads in the here and now. The TV, with its multitude of program sources, is burdened with an outmoded analog recording and presentation format that degrades with every recording and editing generation. The computer, on the other hand, with a digital recording format that can be rerecorded or edited endlessly without degradation, is plagued with nearly incomprehensible system complexities and countless incompatible system and file formats. Will we turn the one way to a new computer system with an improved user interface and become multimedia “users”? Or, will we turn the other way with an ageless digital TV format to a whole new user paradigm and dialog into a television future? Either way, the user interface will be the decisively pivotal factor in an exciting new digital world.
NOTE, April 28 2019: When I have a chance, I will scan the 35mm slides that went with this presentation. The finer points of the picture map on the screen and the control unit will become apparent.
johnmkirkwood.org 