"All Authored Works" On-Line:
A Global Infrastructure for
Universal Access to Information.

http://www.ul.cs.cmu.edu/rr/aaw/aaw.html

Vision

• Reinvent the Information Source for the 21^st Century
• All "Authored Works" Online
• Over 100 Million documents
• All languages
• Anytime, anywhere, instantaneous
• on-line, global distributed

• Books
• UL Book Object
• Newspapers and Magazines
• The Washington Post
• Music
• Ode an die Freude - Beethoven's Ninth Symphony
• Concerto for Piano and Orchestra - Grieg's Piano Concerto in A minor
• Contemporary Music
• Movies and Videos
• The Hubble Space Telescope
• Lectures
• Raj Reddy
• Works of Art
• Torner Collection of Illustrations
• Articles : Scientific, etc.
• Government rules and regulations

• Historical background
• Current Efforts
• CMU On-Line Books Page
• National Digital Library Program
• D-Lib Homepage
• Digital Libraries Resource Page
• Texas U. Building Digital Libraries
• U. of Michigan Digital Library

• www.ul.cs.cmu.edu/search/
• Browsing and Searching
• Books
• Music
• Art
• Video
• Content based access
• Meta Data
• promotional material
• reviews, summaries, table of contents
• individual chapters

The Copyright Problem

• Out of copyright
• All authored works pre 1920
• All works not re-registered pre 1971
• Out of print books : purchase electronic rights
• Uncopyrighted documents
• All government reports
• Encourage web publishing industry : E-Commerce
• Each publisher has their own webservers

The Scanning Problem

• Cut up the book and scan (TIFF files)
• 600 dpi, 20-100 pages per minute
• The precious document problem
• OCR using Xerox Textbridge (RTF files)
• 1% error rate : 10-15 words wrong per page
• Ultimate goal : < 3 errors per book (.01%)
• Manual proofing and correction
• Create an inverted index
• Convert to Web format (HTML files)
• Cost per book : $100 - $500

The Access Control Problem

• Customized Secure Plug-in with Access Control Privileges
• Read once or many times?
• For one week, one year, forever?
• Access by one, by whole family, by whole class

The Very Large Database Problem

• 100,000,000 documents : 30 to 300 GB
• 500 pages / 2000 characters per page
• 300 KB per book compressed (HTML)
• 3 MB per book compressed (TIFF + HTML)
• Audio and video : 1 to 2 orders of magnitude larger
• Managing and querying data using SQL
• Object based representation of documents

The Scaling Problem

• 100,000,000 documents on line by the end of 2020
• 5,000,000 per year
• 20,000 every day for 20 years
• GPO publishes 40K documents(6M pages) a year
• Over 1M Open Source documents / day
• NSPCC produces 743 pieces of paper per acquisition
• Manuals for M1A1 tank 10 feet high!
• Government Policy and regulations in millions of pages?

The On-Line Reading Problem

• Do people want to read books online?
• Search and selection tools
• Flexibility and Extensibility
• Metadata schemas
• Access Management policies
• Version control
• End user initiated changes
• Online purchasing of documents
• Replication and mirroring of the database
• Translation assistants
• Summarization and Annotation tools

The Standards Problem

• What is the representation of the book?
• How are chapters, sections, pages represented?
• SGML or HTML?
• Scanning resolution
• 600 DPI?
• Color / greyscale?
• Storage formats?
• TIFF, RTF, HTML, all of the above?
• The Naming Problem
• URLs, URNs
• The User Interface for Access

The Coordination Problem

• Build upon existing initiatives
• Create an International Coordination Council under NSF
• Education and training of digital library professionals
• Avoiding duplication of effort
• 1000 Expert committees
• 1000 Volume / each sub area

The Selection Problem

• What books, music and movies?
• What areas?
• Which authors?
• Which order?
• Who decides?
• The International Problem
• Multilingual documents
• The Censorship Problem

• Technology vs. Content
• Selective access vs. Universal access
• Content vs. Catalog cards
• Isolated, specialized vs. International, coordinated

• International Coordination
• Universal Library Consortium
• Send pointers for content sites to webmaster@ul.cs.cmu.edu
• Standards
• HTML vs. PDF
• Multiple Representations of Audio and Video
• Translation Assistants

The Funding Problem

• Phase 1 :
• one million books online
• ten thousand musical recordings online
• one hundred movies and videos online
• other specialized collections as they become available
• Estimated cost : 100 to 500 million dollars

Do We Need Traditional Libraries?

• Assume all digital library
• No space needed for book stacks
• Still need: Selection and acquisition
• Free access to documents
• Using Access Control software
• If a library buys 3 copies
• Only 3 can be checked out for specified period after which access is denied
• Fee-Based-Access facilitator role
• When a book is not in the library
• User can buy a copy using a book vending machine
• Every library would have a book-vending machine
• Library would get, say, 10% commission

Do We Need Librarians?

• New acquisition process
• Site licenses for journal/information tools
• acquire new digital-back books only upon request
• instantaneous acquisition
• possibly subsidized in part by the first user(s)
• Digital Reference Librarians
• Information glut competition for human attention
• Need for a new type of an expert:
• Internet, computers and software tools
• data sources on the net
• search, selection and transaction methods
• New roles for current service providers
• Some jobs may not be needed (Which?)
• New Jobs created (What roles?)
• Computer systems for libraries : Support

Do We Need Publishers?

• Publishers needed as editors
• Expertise in content, presentation, editing, and distribution
• Information Explosion
• Need publishers to provide high quality distillation
• Manage relationships between authors and consumers
• Contractual
• Intermediaries such as Booksellers and Printers
• Access to documents
• Publishers start selling Digital-back books with access control software
• Need to define new roles for current publishers
• Some jobs may not be needed (Which?)
• New Jobs created (What roles)
• Computer systems for Publishing : Support

Do We Need Printers?

• Printers are Logical Choice for Service
• Digital form layout
• Craftsmanship
• High quality digital acquisition, color matching, proofing
• Assume Print on Demand
• Individual or low volume orders for high quality printing
• Diversity : e.g., picture/font/layout/language changeouts
• Orders over Internet
• Localization and order control
• Prepare new printable digital-back books
• Need to define new roles for current service providers
• Some jobs may not be needed (Which?)
• New Jobs created (What roles)
• Computer systems for printers : Support

The Pilot Project UL-2000

• 1% of the ultimate collection
• 1M books
• 100K rare books
• 10K musical recordings
• 1K artworks
• 100 movies
• 100 public lectures

Role of Foundations

• Provide seed funding for scanning and conversion
• Support expert committees to determine what content should be online
• Support international activities in third world countries

Role of Government

• International coordination
• G7, Science and Technology Policy
• Sponsor research
• 0.01% OCR
• Secure Survivable Distributed Very Large Databases
• Precious documents
• Authentication
• Ease of use
• Browsing response time
• Translation assistants

Role of Industry

• Establish standards
• Support Digital Library Resource Centers