Menus Subscribe Search

How to Hold a World of Tweets

• April 23, 2010 • 4:04 PM

The U.S. Library of Congress is blazing a trail in determining how to store an ever-expanding trove of information that never had physical form.

After the Library of Congress announced last week that it had acquired the rights to Twitter’s entire archive for preservation as a forever testament to how Americans lived and communicated in the early 21st century, the Idea Lobby called up to ask if we could come by and see, well, where they were going to put it. Would tweets be housed amid Thomas Jefferson’s 6,487 tomes? Or on the shelf beneath the Federalist Papers?

“There isn’t really much to see,” conceded Beth Dulabahn, the library’s director of integration management. Sure, there’s a server that physically houses the LoC’s digital collection, although it looks pretty much like any other server.

But, on both a practical and philosophical level, micro-blogging’s arrival inside the nation’s oldest federal cultural institution raises a complex question, one Dulabahn and her colleague Martha Anderson happily accepted our inquiry to discuss.

How exactly do you preserve, archive and house for posterity things that don’t physically exist?

[class name="dont_print_this"]

Idea Lobby

THE IDEA LOBBY
Miller-McCune's Washington correspondent Emily Badger follows the ideas informing, explaining and influencing government, from the local think tank circuit to academic research that shapes D.C. policy from afar.

[/class]

The library has been mulling this question since 2000, when Congress created the National Digital Information Infrastructure and Preservation Program, which Anderson directs.

The library first began working in digital (and in other non-paper media like home movies) in the ’90s. At the time, it was trying to digitize physical objects unique to the LoC, like the Federalist Papers, to make them more accessible to people who weren’t old enough to get into the reading rooms or who lived too far away from Washington to visit. The material was then distributed by videodiscs to remote locations like schools.

“There was serious consideration about putting this stuff out on CDs,” Anderson said. “I just kind of cringe when I think about that – by this time, we would have millions of CDs.”

Around 1996, it dawned on the LoC that it could archive and share the artifacts over the then little-used Internet. Then came a dramatic shift in information outside the library: As the Web exploded in popularity, the LoC’s initiative shifted from digitizing physical objects to capturing born-digital objects that never existed in the physical in the first place.

“We’ve been reminded so much the last few years of all the things that didn’t even exist when our program began in 2000,” Anderson said. “There were no Google maps.”

“There was no Google!” Dulabahn added.

“There was no YouTube of course, there was no Facebook, no Twitter, no Flickr,” Anderson said. “All these things that are just commonplace today did not even exist when our program began. And we thought we had challenges then.”

The library spent the better part of a year trying to figure out how to capture video embedded in websites. Then, suddenly, came YouTube, and overnight the staff had to start all over again relearning the new universal format.

The library’s mandate — to chronicle information in the present to be available in perpetuity — bumps up against the very nature of the dynamic Internet. Digital photos today are created as JPEGs, but what happens when that format becomes obsolete, and 10 years from now, your computer can’t read it? If the library simultaneously preserves software such as a current JPEG reader, does it also need to preserve hardware, like a 2010 MacBook Pro?

And even if the library successfully captures everything — navigable links, text, images — on a website, what happens when that site is updated two hours, or two years, later?

“How often do you go out and try to grab them? What’s enough?” Dulabahn asked. “There are a lot of philosophical questions about, well, what are you actually trying to do? Are you trying to get blow-by-blow, or is once a month enough? You can’t make those decisions across the board; it’s not one size fits all.”

The LoC has focused on sites important for public policy, such as those related to elections and government, or major historic events like Sept. 11 or Pope John Paul II‘s death and succession. (Click here to tour WhiteHouse.gov on March 13, 2003, on the eve of the Iraq war.) Given its limited scope, and even with the coordinated efforts of national libraries across the globe, only a fraction of 1 percent of Web history must have been captured by now, right?

“A fraction at what point in time?” Dulabahn countered, reinforcing the impression that any true understanding of the topic might require a grasp of quantum mechanics.

In more measurable terms, it took the LoC two years to accumulate its first terabyte of Web data. In 2005, it began collecting a terabyte a month. Last spring, that became a terabyte a week. This year, Anderson expects to double that rate. The library isn’t necessarily collecting more websites; rather, the same number of sites today yields exponentially more content.

In total, the collection last year surpassed 3 billion individual “objects” like text files or images – “which was kind of horrifying to us,” Anderson said.

Because how do you sort through all that information to make it useful, and unbundle it from the format restrictions of this moment in time? All the typical standards for how to organize books don’t apply.

To a certain extent, the LoC and its numerous partner organizations are just trying to grab as much content as they can before it disappears, and they hope one day someone will know how to use it.

“This is a reversal of a model we’ve had for decades at the library, where basically we select at the front end,” Dulabahn said. “What’s left you catalogue, and you bind, and you move to the shelves. With this kind of voluminous digital content — like Web archives, things like Twitter archives — it’s a completely different model. You have a chance to get it; it’s probably your only chance looking across the spectrum of time. You have a very small window, so you get it and say, ‘Then we’ll figure out over time how to find the good stuff.’”

But just as they’re betting on future innovation in digital archiving — Web crawlers that can tackle Facebook and search functions that can index time — so too will digital information continue to evolve faster than its would-be preservers can keep up.

Twitter has created a revolution in communication that Dulabahn and Anderson could never have predicted 10 years ago. And so it’s possible something so new will arise in the future that it makes all of the LoC’s efforts at digital preservation obsolete.

“But I want to say this word of encouragement,” Anderson said. “Early video games are still alive — Pacman, those early Donkey Kongs — because people have cared enough about them to try to do something to keep them alive.”

People will also be motivated by more than just nostalgia as businesses become further invested in the enormous new marketplace of digital information.

Predicted Dulabahn: “I think we’ll be able to ride on the coattails of that.”

Emily Badger
Emily Badger is a freelance writer living in the Washington, D.C. area who has contributed to The New York Times, International Herald Tribune and The Christian Science Monitor. She previously covered college sports for the Orlando Sentinel and lived and reported in France.

More From Emily Badger

A weekly roundup of the best of Pacific Standard and PSmag.com, delivered straight to your inbox.

Recent Posts

September 1 • 6:00 AM

Why Someone Named Monty Iceman Sold Doogie Howser’s Estate

How unusual names, under certain circumstances, can score success.



August 29 • 4:00 PM

The Hidden Costs of Tobacco Debt

Even when taxpayers aren’t explicitly on the hook, tobacco bonds can cost states and local governments money. Here’s how.


August 29 • 2:00 PM

Why Don’t Men and Women Wear the Same Gender-Neutral Bathing Suits?

They used to in the 1920s.


August 29 • 11:48 AM

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.


August 29 • 10:00 AM

True Darwinism Is All About Chance

Though the rich sometimes forget, Darwin knew that nature frequently rolls the dice.


August 29 • 8:00 AM

Why Our Molecular Make-Up Can’t Explain Who We Are

Our genes only tell a portion of the story.


August 29 • 6:00 AM

Strange Situations: Attachment Theory and Sexual Assault on College Campuses

When college women leave home, does attachment behavior make them more vulnerable to campus rape?


August 29 • 4:00 AM

Forgive Your Philandering Partner—and Pay the Price

New research finds people who forgive an unfaithful romantic partner are considered weaker and less competent than those who ended the relationship.


August 28 • 4:00 PM

Some Natural-Looking Zoo Exhibits May Be Even Worse Than the Old Concrete Ones

They’re often designed for you, the paying visitor, and not the animals who have to inhabit them.


August 28 • 2:00 PM

What I Learned From Debating Science With Trolls

“Don’t feed the trolls” is sound advice, but occasionally ignoring it can lead to rewards.


August 28 • 12:00 PM

The Ice Bucket Challenge’s Meme Money

The ALS Association has raised nearly $100 million over the past month, 50 times what it raised in the same period last year. How will that money be spent, and how can non-profit executives make a windfall last?


August 28 • 11:56 AM

Outlawing Water Conflict: California Legislators Confront Risky Groundwater Loophole

California, where ambitious agriculture sucks up 80 percent of the state’s developed water, is no stranger to water wrangles. Now one of the worst droughts in state history is pushing legislators to reckon with its unwieldy water laws, especially one major oversight: California has been the only Western state without groundwater regulation—but now that looks set to change.


August 28 • 11:38 AM

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.


August 28 • 10:00 AM

The Five Words You Never Want to Hear From Your Doctor

“Sometimes people just get pains.”


August 28 • 8:00 AM

Why I’m Not Sharing My Coke

Andy Warhol, algorithms, and a bunch of popular names printed on soda cans.


August 28 • 6:00 AM

Can Outdoor Art Revitalize Outdoor Advertising?

That art you’ve been seeing at bus stations and billboards—it’s serving a purpose beyond just promoting local museums.


August 28 • 4:00 AM

Linguistic Analysis Reveals Research Fraud

An examination of papers by the discredited Diederik Stapel finds linguistic differences between his legitimate and fraudulent studies.


August 28 • 2:00 AM

Poverty and Geography: The Myth of Racial Segregation

Migration, regardless of race, ethnicity, gender, or sexuality (not to mention class), can be a poverty-buster.


August 27 • 4:00 PM

The ‘Non-Lethal’ Flash-Bang Grenades Used in Ferguson Can Actually Be Quite Lethal

A journalist says he was singed by a flash-bang fired by St. Louis County police trying to disperse a crowd, raising questions about how to use these military-style devices safely and appropriately.


August 27 • 2:00 PM

Do Better Looking People Have Better Personalities Too?

An experiment on users of the dating site OKCupid found that members judge both looks and personality by looks alone.


August 27 • 12:00 PM

Love Can Make You Stronger

A new study links oxytocin, the hormone most commonly associated with social bonding, and the one that your body produces during an orgasm, with muscle regeneration.


August 27 • 11:05 AM

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”


August 27 • 9:47 AM

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.


August 27 • 8:00 AM

A Skeptic Meets a Psychic: When You Can See Into the Future, How Do You Handle Uncertainty?

For all the crystal balls and beaded doorways, some psychics provide a useful, non-paranormal service. The best ones—they give good advice.


Follow us


Subscribe Now

Your Brain Decides Whether to Trust Someone in Milliseconds

We can determine trustworthiness even when we’re only subliminally aware of the other person.

Young, Undocumented, and Invisible

While young migrant workers struggle under poor working conditions, U.S. policy has done little to help.

Education, Interrupted

When it comes to educational access, young Syrian refugees are becoming a “lost generation.”

No, Smartphone-Loss Anxiety Disorder Isn’t Real

But people are anxious about losing their phones, even if they don’t do much to protect them.

Being a Couch Potato: Not So Bad After All?

For those who feel guilty about watching TV, a new study provides redemption.

The Big One

One in two full-time American fast-food workers' families are enrolled in public assistance programs, at a cost of $7 billion per year. July/August 2014 fast-food-big-one

Copyright © 2014 by Pacific Standard and The Miller-McCune Center for Research, Media, and Public Policy. All Rights Reserved.