Dotcom Chronicles (3): You Can't Set Up a System of Bidirectional Links

In the late fall of 1991, Tim Berners-Lee left CERN, the European particle physics laboratory in Switzerland, for a conference in San Antonio, and stopped on the way at Chapel Hill, North Carolina. He had written the first Web browser and the first Web server at CERN; the first piece in this series told that story. The Office of Information Technology at the University of North Carolina had a small group doing network services, and one of the people running it was Paul Jones. Jones recalled later that before the visitor arrived, he called a nearby computer science department, hoping to get an expert to come and have a look. The voice on the other end had heard of the project and was not impressed. It rattled off six objections; Jones did not write down who it was. The first four were technical: links in hypertext must be bidirectional, and the WWW’s are one way only; the servers know nothing of each other; there is no central node; the servers keep no state. The fifth was that whoever wrote this hypertext engine obviously did not understand SGML, and this HTML was done all wrong; that quarrel was the subject of the first side piece. The last: he is giving this away for free, which means it has no value at all, “We have our own version and a strong licensing partner.” Five objections said it was done wrong. The sixth said that even done right, it would be worthless.¹

When Berners-Lee arrived he pulled out a rewritable optical disk for the NeXT, with the demonstration pages he was going to use at the conference, loaded it onto Jones’s NeXT, and walked him through the Web. One link on the page went to a database in the next room. Click it, and the request went first to CERN in Switzerland, because the gateway program that translated Web requests for that database was at CERN, then back to North Carolina, and the results went back through Switzerland again. Looking something up in the next room crossed the Atlantic four times. I don’t think he was embarrassed by the route. To him this was the whole point of the demonstration: the page in Chapel Hill, the gateway in Geneva, the database next door, and one click reached all of them. Why the next room required a trip around the world was a matter of the gateway not yet being installed in North Carolina, not a matter of the Web. Jones did not take him to meet the person on the phone, and he most likely never knew the call had happened.¹

The six objections from that phone call in Chapel Hill, a passage from Paul Jones’s blog of 29 May 2013, recalling the late fall of 1991; the quotation is as written, the numbering is ours, and the typeface is modern. This is a recollection twenty-two years after the fact, and the six points are his paraphrase of an unnamed person on the other end of the line. Reproduced here as a historical excerpt.

In mid-December, Berners-Lee and his CERN colleague Robert Cailliau reached San Antonio. One of the database interfaces the demonstration relied on had been set up a few days earlier by the library at the Stanford Linear Accelerator Center (SLAC); the second piece was about that server. Hypertext ‘91, run by the Association for Computing Machinery (ACM), was the hypertext researchers’ own conference, and he was not one of them. He had finished a physics degree at Oxford in 1976, where he had built his first computer out of a soldering iron, a handful of logic gates and an old television. After graduating he spent several years in the south of England working on telecom equipment and printer software, and the job he took at CERN in 1984 was distributed real-time systems for data acquisition, where he designed a remote procedure call system that worked across different machines. Hypertext was a side interest until the proposal of 1989 made it his job.² He went in with some confidence. Jones remembered that before the trip, looking at the reaction on the newsgroups and the logs of his own server, he expected to get a good speaking slot. What he got was a poster and a demonstration slot. Mark Frisse, one of the reviewers, said years later that the decision was the committee’s, that the paper read like a document-writing tool rather than a hypertext system, and that the field was arguing at the time over whether links in such systems had to be bidirectional to keep citations intact; “if I remember correctly, Tim had a demo and really rocked the house.” What Berners-Lee himself said about it survives in one sentence of the project newsletter of 24 January: “The W3 demonstration had an enthusiastic reception at ‘HyperText’91’ in San Antonio, Texas.” The next sentence reports that Groff had presented W3 live at a workshop in France, and the one after that says a conference in Innsbruck in May had “asked to demonstrate, as well as present a paper.” The word poster does not appear. What I read in that newsletter is a man who did not intend to spend time on this: no podium here, an invitation elsewhere, so go elsewhere. The Web is not in the proceedings. Pages 39 to 50 hold a paper analyzing Xanadu. The name came up once in the second piece; it needs a little more here. Ted Nelson coined the word “hypertext.” By his own account he had the idea in 1960 as a graduate student at Harvard, and the word appeared in print in 1965, the year he read a paper at the ACM’s national conference titled “A File Structure for the Complex, the Changing, and the Indeterminate.” Xanadu was the system he had been designing since then, thirty years of it by 1991, and it did not yet exist. Not existing did not diminish its standing. Nobody in this field could talk about hypertext without it: Berners-Lee’s proposal of 1989 cited it, this conference formally accepted a paper analyzing it, and when he wrote an article introducing the Web in 1992 he singled it out again; all of this comes up below. Its later reputation and its ending get a piece of their own in this series. For this one, all you need to know is that Xanadu was what the experts had in mind when they said hypertext.³⁴

Reading this today, you might think a link is just an address: click and you jump, and if it breaks it breaks; is it worth taking this seriously? The hypertext researchers of 1991 did not see it that way, because links were the only structure hypertext had. A book has a table of contents, page numbers and an index. Hypertext has none of that, only the lines running from one passage of text to another, and the thing holds together only if the lines hold. What they meant by “bidirectional” was three things. A reader standing on a document should be able to look back and see who cites it; that was half the value of the “trails” Vannevar Bush imagined in 1945. When a document moves or changes, the links pointing at it should follow, or they break; Frank Kappe of the Graz University of Technology wrote to the Web project in June 1992 that tracing links backwards “is essential for maintaining the integrity of the hypertext.” And everything built on top of links, citations, annotations, Xanadu’s royalties, needs the link to still be there tomorrow. A link that records “from here to there” but not “who points here” was, in their eyes, a building put up on a foundation nobody had guaranteed.⁵

Berners-Lee knew the weight of these three things better than anyone, because he had done the first of them himself. In 1980 he spent six months at CERN as a contractor, and to keep track of the software that passed through his hands he wrote a program called ENQUIRE; the first piece told that story. In its manual, dated October 1980, every relationship between nodes comes in a pair: uses with used-by, includes with part-of, made with made-by, describes with described-by. Standing on any node, you could see who used it, and the manual says this is so that when you change one part you know what else is affected. That is exactly the backwards tracing Kappe would ask for twelve years later. The manual defines the “person” type of node as “those strange unreliable beings on which all systems eventually depend.” He did not know at the time that the word “hypertext” already existed; he said so himself in 1989. The program had one user, him, and was never published.⁶

The first objection from Chapel Hill and the reviewers’ worry in San Antonio point at the same thing: links go one way, and they can break at any time. So why did he do it anyway? And what took the place of the part he could not do? Berners-Lee answered on the mailing list in 1992 and 1993, more than once. He did not deny the charge, and reading his letters, I cannot see that it ever troubled him. People holding him to the standard of orthodox hypertext kept turning up through those two years, and the most complete version of that standard was called Xanadu.

“The Xanadu Project Excepted”#

What the experts wanted is laid out most fully in the paper that made it into the proceedings. The authors were Pamela Samuelson of the University of Pittsburgh School of Law and Robert Glushko of a Pittsburgh company called Hypertext Engineering, and they called Nelson’s Xanadu “by far the most ambitious proposal for a digital library and hypertext publishing environment”: a repository holding “all of the world’s literature” that was at once a writing environment, a publishing environment, a library and a bookstore. Readers pay by the byte, and ten to twenty percent goes to the author of what they read. To quote someone else’s passage you do not copy it; you embed those bytes in your own document, and when a reader reaches them, both authors get royalties. A link is itself a work, and when someone follows your link you get paid too. Under those rules, everyone quoted in this article would be owed money, every name in the footnotes. For any of this to hold, links cannot break: a published document cannot be withdrawn, because other people’s links and derivative works are built on it. The authors concluded that parts of this model of how authors and readers behave “may be unworkable.” The paper also noted that Xanadu was being commercialized by Autodesk.⁷

Page 39 of the proceedings: title, authors and abstract of Samuelson and Glushko’s analysis of Xanadu. Cropped from the first page of the PDF archived by Robert Glushko; the type is the original scan, not reset. © ACM / the authors; reproduced here as a historical excerpt.

Berners-Lee knew this design, and he was consistently polite about it. The proposal he wrote at CERN in 1989, covered in the first piece, did something the first piece did not mention: it credited the word “hypertext” to Nelson and cited his docuverse, the name Nelson gave to the imagined space holding all documents. In the spring of 1992, he and Cailliau and others introduced the Web in a journal article. The opening section was called “The Dream,” and it began with Vannevar Bush, the American science administrator who in 1945 imagined a machine linking a personal library by association, and then: “The Xanadu system is an ambitious distributed hypertext project.” The next section asked why, after so many years of hypertext and full-text retrieval systems, no global system existed. The answer was that three things were missing: a common naming scheme for documents, common network protocols, common data formats for hypertext. Then this sentence: “Most research in hypertext systems (the Xanadu project excepted) has focussed on the user interface and authoring questions, rather than the questions of wide-area and long-term distribution.” Singling out Xanadu was an admission that the other side was also thinking about wide-area distribution; the other systems had not even asked the question. They wanted different things, Xanadu wanted payment and royalties as well, but on the question of how documents scattered across the world’s machines get connected, they were two answers to the same problem. I suspect the courtesy was sincere. When a man leaves a kind word for a rival in the article introducing his own project, that is not manners; it is that he counts the rival as being in the same room, while everyone else in the room is thinking about something else.⁸

The Web project’s own mailing list was called www-talk, and the first time Xanadu’s name came up there, it was about money. At the end of June 1992, someone on the list proposed a “pay link” that would warn the reader of a charge before they clicked. Jean-François Groff, Berners-Lee’s colleague at CERN, replied that per-read charging had all sorts of problems, that this “was to be implemented in Xanadu as an answer to the Copyright issues,” and that anyone who wanted to know more should read the Samuelson and Glushko paper from San Antonio. “Was to be implemented” describes a design. At that point Xanadu’s per-read charging existed only on paper; on the Web side, even the paper version was still being argued over on the list.⁹

News of the Xanadu software came in the summer of 1992. On 20 August, Autodesk, the maker of the AutoCAD drafting software, announced that it was divesting two companies under its information systems division, one of them the Xanadu Operating Company, on the grounds that they “do not fit into Autodesk’s core business,” with a transition period of about thirty days to find other funding. The news traveled slowly. The page on CERN’s site introducing Nelson still said, in its November version, “The Xanadu Operating Company is now owned by Autodesk, but the project continues. Code is due out ‘any time now’,” the quotation marks around any time now being in the original; the later version was changed to “was owned for a while by Autodesk, but later dropped. (Not clear in 1993 what is going to happen there).” This page sat on his own site, and he edited it himself. A man maintaining a page about a rival’s project, waiting for the rival’s code through one version after another: I don’t think it was an anxious kind of waiting, more a glance back every so often to see if there was news, and then back to his own work. In January 1993, in a discussion on the list about charging for documents and how to count copies in a cache, he dropped in a line: Have you read “Literary Machines”? That was Nelson’s self-published book on Xanadu. That goes into this in a lot of detail, “or at least Xanadu should have done.” Half a sentence of needling, in passing, and then back to his caches. By this point his attitude to Xanadu had more or less set: a good book, a product that had not appeared, and no intention of waiting.¹⁰⁹

Autodesk’s press release of 20 August 1992, from the headline to the Xanadu paragraph. Text from the online edition of John Walker’s The Autodesk File, fifth edition, preserved at fourmilab.ch, re-rendered in a monospaced face, omitting the employee letter and the AMIX section that follow; the two shaded areas are ours. The release comes from the divesting party, and its description of what Xanadu does is a public-relations account. © Autodesk; reproduced here as a historical excerpt.

“You Can’t Set Up a System of Bidirectional Links”#

Back to the list in 1992. Xanadu’s product had not appeared, but the standard it stood for was being raised on www-talk one item at a time. In May and June, a user named Robert Raisch sent two long letters, saying he had heard Nelson speak in 1977 and had been thinking about hypertext for the fifteen years since. He listed the link types the Web ought to support. One was “inclusion,” pulling elements from elsewhere straight into the current document, which is exactly Xanadu’s embedded quotation; another was Nelson’s stretch text, a passage that folds up into a summary and unfolds into the full text.¹¹

On 25 June, Kappe sent a “rather long” letter explaining that their own hypertext system, Hyper-G, kept links in a link database outside the documents, rather than writing them into the page as HTML did. The advantage was tracing backwards, knowing who pointed at this page, which “is essential for maintaining the integrity of the hypertext.” He also drew a diagram to show that two overlapping anchors could not be expressed in SGML’s paired tags at all: one anchor marks paragraphs one and two, another marks two and three, the ranges overlap, and the opening and closing tags cannot nest. That same evening, in another thread, Berners-Lee touched on the same point: in principle there was no reason not to have overlapping or nested anchors, “But not now.”¹¹

He sat on Kappe’s letter for twenty days. The reply went out at 00:33 on 15 July, and it opened by owning up: “I am replying very late.” Nearly every question on the list came to him for an answer in those days, so twenty days was not strange; what is striking is that he did reply, and at length. First the overlapping anchors. This was of course counter to the SGML philosophy of strict nesting, he said, and “SGML people can get quite religeous about this, but I can’t.” It was not that he knew nothing about typesetting; in 1978 he had written typesetting software for intelligent printers at a small firm in Dorset. What he did not believe in was the doctrine of strict nesting. He thought being able to refer to two overlapping passages was useful, and the only trouble was that SGML tools would treat such a structure as “BAD.” Only then did he get to link databases. The kind Kappe proposed was the model of Intermedia, the system Norman Meyrowitz and others had built at Brown, where links live together in one database, which Intermedia called a “web,” the same word as the later Web, shared by the readers and centrally coordinated. “If you expand that system to more than one web server, and scale it to global, then you find that the same problems of ensuring consistency occur between the multiple link databases when before they occured between documents. You can’t set up a system of bidirectional links for example, which you could in the non-distributed case.” That should have been the end of it. It was not. He went on: “Link databases have their advantages, though,” and the model he was “rather keen on at the moment” was another one, servers that were both link databases and version control systems, where a link points not at a marker in the document but at a character position in a particular version, and when the document changes, the server translates the old position into the new one. Then he picked holes in it himself: diff cannot cope with someone swapping the two halves of a file, so let the editor explain each change to the server as it goes; and once you are doing that, “you can go a small extra step” and have the server coordinate simultaneous edits by many people, which gives you a group editor. A letter answering “why can’t you do bidirectional links,” written into the small hours, turned into the next thing he wanted to build. The evening before, in another thread, he had proposed a project “up for grabs”: a server that hoards mail, indexes it and fills in all the back-links between messages, which he said “would be a cool excercise in the management of unruly information,” and “Any takers?” Nobody in the archive took it. The idea that a machine could compute reverse links after the fact was already there.¹²

Berners-Lee’s reply to Kappe in the early hours of 15 July 1992, the two paragraphs on link databases. Text from www-talk 1992, message 149, re-rendered in a monospaced face, omitting the Received lines and empty headers; the thirty preceding lines on DTDs and the ending are marked as omitted in square brackets, and the two shaded areas are ours. This is not how it looked on a terminal in 1992. The author of the message is Tim Berners-Lee; reproduced here as a historical excerpt.

The phone call in Chapel Hill was right: Web links go one way. Berners-Lee’s answer was that once there is more than one server, run by different people, bidirectional links cannot be kept, and better to admit it than pretend. A reader who has written programs will ask: the target server can record who points at it, Hyper-G did exactly that, why doesn’t that count? His letter did not say it was impossible in principle. It said the consistency problem moves from between documents to between link databases: when a page on one server changes, the records about it on other servers have to change too, and who guarantees that? My reading is that as long as nobody can guarantee it on behalf of someone else’s server, bidirectional is a fiction. He knew this kind of problem. His trade at CERN was distributed systems, and the remote procedure call work he did from 1984 on was about letting machines from different makers call each other’s programs. I think he had seen, in those years, what it looks like when consistency cannot be kept, and this time he did not intend to pretend. The way out he had in mind was different: reverse links not written by authors but computed by machines after the fact. The cost turned up quickly. A link was a string of address written into the page, with no system to move it along when the document moved. At the end of October, Edward Vielmetti of a network company in Michigan saved a Web page to his own disk, opened it again, and “all of the cross reference links were just gone,” because the links were written relative to the original server and stopped working once they left it. When a link fails today and a page tells you it cannot be found, this is where it comes from: no system guarantees that both ends of a link are still there.¹²

“Explicit Linking Is Impossible”#

In 1993 the objections came from a new set of people, and in a new form: not whether links were bidirectional, but who would maintain them. On 4 May, Eliot Kimber of IBM posted to the SGML newsgroup under the title “Explicit Linking is Impossible.” He did the arithmetic with IBM’s own document libraries: within a book of a hundred pages, one author can maintain a thousand links by hand; across a library of ten such books, a hundred links from each book to the others, but just barely; across libraries, anything above about fifty links cannot be maintained, and cross-enterprise linking “is so clearly impossible as to be not worth considering.” The next day, Marc Andreessen of the National Center for Supercomputing Applications (NCSA) at the University of Illinois, who had just released the graphical browser Mosaic, forwarded the post to www-talk, underlined “not worth considering” with a row of carets, and added a jab: if he didn’t know better he’d say the man worked for IBM, “Didn’t you lose $5 billion last year?” The person forwarding it was 21.¹³

Above: Kimber’s post as forwarded by Andreessen on 5 May 1993, the arithmetic passage and Andreessen’s note, with two omissions marked in square brackets; below: Andreessen’s reply to a Swedish user the same day, in full. Text from www-talk 1993 second quarter, messages 225 and 228, re-rendered in a monospaced face; the three shaded areas are ours. This is not how it looked on a terminal in 1993. The authors are Eliot Kimber and Marc Andreessen; reproduced here as a historical excerpt.

Not one of the replies said Kimber had got the arithmetic wrong. A user in Sweden said his own experience writing HTML supported the conclusion, that maintaining links into other people’s documents was “next to impossible,” and that if Berners-Lee ever renamed a directory at CERN it would be “like pulling the rug away from under my feet”; he proposed a numbering scheme for hypertext like the ISBN, and then said that was too complicated. Vielmetti said deep explicit links were unnecessary, that you could “pre-load a search” to get people to the neighborhood; he imagined a lookup service called spidey that searched by file name, keyword or book number, with probabilistic results that would not fail the moment a document moved, “(you hope).” Tony Sanders of BSDI said reverse references could manage change locally but not globally: “I agree that massive explicit linking is impossible, but it isn’t desirable either.” Andreessen’s answer was the shortest. He did not think maintaining links was easy or cheap now, and there were no magic tools, “but limitations of current technology should not be extrapolated to judge possibility or impossibility; I would have thought the Wright brothers taught us that principle.” The Wright brothers did not have to maintain links.¹³

Nobody in this argument mentioned Xanadu. Kimber prescribed nothing; he only said people could not do it. And what Xanadu promised was precisely to take the job out of people’s hands and have the system maintain the links. The people on the list admitted they could not provide that; they disagreed only about whether it mattered. Berners-Lee did not speak in this thread. He did not need to. He had answered the question a year earlier, and the answer was to let it go: if links could be maintained, that was a matter for people and machines, not something the system guaranteed. Now others were arguing on his behalf, and what they were arguing about was already the problem that came after he let it go. Xanadu had another face on the list as well. In June, when someone imagined librarians on screen annotating documents and drawing lines between them, Raisch replied, “Ahhhh… Dreams of XANADU,” and told people to read Nelson’s books, which he said had changed his life.¹³

“That Doesn’t Mean That Anybody’s Using Them”#

In September 1993, someone pointed out that HTML could express two-way links: the anchor tag had two attributes, REL and REV, one saying what the relationship was from this end and the other from that end. The person pointing it out added half a sentence of his own: “Of course, that doesn’t mean that anybody’s using them. :)” A user in the Netherlands followed up: could someone explain how to use them and what they would do for him? The CERN documentation was vague; did Xmosaic, then the most popular graphical browser, support them? Nobody in the archive answered the second question. The same month, in a discussion of document naming and version numbers, Berners-Lee added an aside in parentheses: multi-level version numbers, “(I think these are basically Ted Nelson’s ‘tumblers’),” which was Xanadu’s scheme for numbering every document and version. He had read Nelson, remembered Nelson’s words and reached for them without thinking; he just had not built to them.¹⁴

On 6 December, Roy Fielding of the University of California, Irvine, put out a request for comments on a program he wanted to write, MOMspider, a maintenance spider for multiple owners. On behalf of a group of site owners it would periodically traverse each owner’s pages, from a top document down to every leaf, and find three things: “referenced objects which cannot be accessed (broken links),” objects recently changed, and objects past their expiry date, then build an index document for each owner, itself hypertext. Berners-Lee replied that evening, “I’ll play devil’s advocate here,” and the only thing he argued about was which tag a page should use to name its owner, so a machine would know whom to tell about a broken link. Fielding had said that using the existing LINK tag for the owner would be a contrived relationship; Berners-Lee answered that LINK was designed for any two-ended relationship, and listed some off the cuff: Jack loves Jill, Jill loves Jack, Jack likes pie, Jill makes pie, Jack eats pie, Jack addicted to pie, Jack needs pie. A man who writes out a list like that while arguing over a tag was, I think, in a good mood that night. As for the spider itself, he raised no objection.¹⁵

The “Problem” and “Partial Solution” sections of Fielding’s request for comments, 6 December 1993. Text from www-talk 1993 fourth quarter, message 710, re-rendered in a monospaced face, omitting the opening remarks and the traversal rules at the end; the two shaded areas are ours. This is not how it looked on a terminal in 1993. The author of the message is Roy T. Fielding; reproduced here as a historical excerpt.

The program was built, and presented six months later at the first WWW conference in Geneva. In the material I have read, it is the first program on the Web side written specifically for broken links. Xanadu’s design promised that links would not break. The Web’s practice was to have a machine go round periodically, see which ones had, and tell the owner. The responsibility moved from the system to the site owner, and from before the fact to after it. The dead-link checkers that websites run today do the same job. The day Fielding sent out his request, two years had passed since the phone call in Chapel Hill.¹⁶

The two years this piece covers. Dates are taken from the letters, the proceedings, the press release and the CERN pages; the Chapel Hill stop is known only as “late fall of 1991,” from Jones’s 2013 recollection. Drawn by us from the sources above; not a contemporary chart.

What the reviewers in San Antonio thought afterwards, and what the voice on the phone in Chapel Hill thought afterwards, is not in the material from 1993. What survives is a parenthesis. On CERN’s page about Nelson, both versions carry a line signed TBL: “I have a copy of LM 90.1.” Literary Machines, the book Nelson published himself and revised edition after edition, 90.1 being the edition, thinks a system of unbreakable links through from beginning to end. Berners-Lee read it, and then built a system that breaks. It was not that he could not build the other kind: ENQUIRE in 1980, the program only he used, had paired relationships, and from any node you could see who pointed here. His reason, in the material I have read, he wrote down once, in that letter at half past midnight on 15 July: in the non-distributed case you can, in the distributed case you can’t. He wanted the distributed one. I don’t think he was untempted. A man who buys a rival’s book and writes “I have a copy” on his own web page is a man who has thought the design through in earnest; thought it through, and then set it down, and the setting down probably had little drama in it, much like the tone of that letter, where he says it can’t be done and turns to the next model he is keen on. The half that breaks has been the reader’s to carry ever since.

Xanadu guaranteed that links would not break; at the end of 1992 the code was still “due out any time now,” and by 1993 even the CERN page could only say it was not clear what would happen there. The Web admitted that links break, and Fielding wrote it into his request for comments as a case history: maintenance “is rarely or inconsistently performed and the infobase eventually becomes corrupted.” When he wrote that, the Web had been public for a little over two years. The Web grew up carrying that case history. Spoiler: this thing that the experts saw as born crippled went on to become the place the world’s information passes through. The links still break. Nobody has gone back to cure it, and nobody has waited for the cure.¹⁵

Where the protagonist’s thoughts and feelings in this piece are not sourced, they are the author’s inference from contemporary material, marked with “I think,” “I suspect,” or “what I read”; the facts and quotations are all in the footnotes.