Over a fortnight in July I travelled from Basel to Vienna, taking in two of the venues where the shape of the web is argued over in detail: the HTTP Workshop and IETF 126. The trip had three purposes: present measurement work drawn from the Common Crawl archive, attend the working groups whose output will determine how crawlers like ours are treated in the coming years, and move an Internet-Draft forward.

A view of the Rhine in Basel, Switzerland

The HTTP Workshop is an invitation-only gathering held every two years.  It puts roughly forty people in a room, among whom are the people who build and operate browsers, clients, servers, proxies, content delivery networks and measurement projects.  This was the seventh edition.  The format is deliberately an intimate one involving short talks and long discussions, and it runs under Chatham House Rule so that participants can speak candidly about what they see in production.

I presented a set of HTTP version adoption statistics^† and used them to make a somewhat broader argument about what I’d argue Common Crawl is for.

The usual framing of a web crawl is as a supply of data for downstream consumers.  That’s true, but that undersells it.  A broad, openly available crawl is also a way to let the community observe protocol deployment at scale, as it actually is rather than as one assumes it to be.  Which HTTP versions are being served, which TLS versions, which IP versions, how they’re changing over numerous crawls and where the dustier parts of the web behave differently from the few hundred big places that most measurements are looking at.

We think that last part is very interesting.  Measurement based on top-site lists tells you what the largest and best-resourced operators are doing but it tells you relatively little about the small business sites, the parish councils, the personal blogs, and the other stuff which is where a giant chunk of the web still lives, and where protocol adoption tends to be behind by years.  A crawl of the whole tail is one of the few ways to see those.

In transit

As it turns out, long train journeys seem to be an excellent format for the sort of discussions that a conference schedule never quite makes room for.

Most of them came back to responsible crawling.  Rate limiting, respecting the preferences expressed by site operators, and being transparent about who you are.  As automated traffic grows and site operators grow less patient with it, the practical question facing every crawler is whether it can be told apart from the badly behaved ones.  A crawler that identifies itself honestly publishes its address ranges, honours robots.txt and explains its purpose.  One that doesn’t is asking to be blocked, and increasingly will be.

The plenary meeting at IETF 126, ViennaVienna: IETF 126

IETF 126 ran from 18 to 24 July at the Hilton Vienna Park. I attended seven working group and research group sessions. Briefly, and in no particular order:

webbotauth (Web Bot Authorisation)

This working group is developing mechanisms by which well-behaved automated clients can cryptographically authenticate themselves to servers, so that site operators can tell legitimate, identifiable bots apart from those spoofing their user agents.

aipref (AI Preferences)

This is the working group developing standard, machine-readable ways for content owners to express preferences about how their content may be used, including by AI systems and crawlers. If you’re not familiar with this one yet, you probably ought to be.

maprg (Measurement and Analysis for Protocols Research Group)

I presented recent IPv6 adoption measurements and spoke to new additions to cc-crawl-statistics.  The measurement community is a great audience, since they can validate, extend, and cite the work, and their questions are a very good guide for what we should be measuring next.

httpbis (HTTP Working Group)

This is the group that maintains and evolves the HTTP specifications, and therefore the source of the protocol whose adoption we report on.  Many of the folks who were at the HTTP Workshop were also here, and it was great to follow up on discussions had the week before.

6man (IPv6 Maintenance)

This group is responsible for core IPv6 specifications.  Keeping up to date on this one keeps our IPv6 measurements aligned with where the protocol is heading.

happy (Happy Eyeballs)

This group is concerned with the algorithms clients use to establish connections quickly across IPv4 and IPv6.  This is relevant to us both as the operator of a large client fleet and as a measurer of connectivity in the wild.  See our recent analysis on IPv6 adoption.

AUDIT side meeting (Agent Use of Delegation and Interaction Traceability)

This was a session on how autonomous agents delegate authority and how their interactions can be made traceable.  As agents proliferate, the questions of who is acting, on whose behalf, and with what accountability basically determine how any automated client is treated.  See this draft from Mirja Kühlewind on the topic.

Machine-readability draft

During IETF 126 a side meeting on my Internet-Draft was held. The draft’s title is “Defining Machine Readability for Usage Preferences and Policy Expression”.  You can read the draft on the Datatracker.

The frontispiece of Ted Nelson's Computer Lib/Dream Machines (1974). Original image by John R. Neill for L. Frank Baum's Tik-tok of Oz (1914).

The draft exists because "machine readable" gets invoked constantly in standards work and in policy and in legislation, and it’s almost never defined precisely enough to tell you whether a machine can actually act on the thing in question.  A terms-of-service document wrapped in a JSON string is parseable, but what a crawler recovers from it is a string of legalese, and that’s not something with which it can make any decisions.  The draft separates this idea into five distinct properties.  Discoverability, parseability, interpretability, actionability, and verifiability.  It then measures existing mechanisms against them, including robots.txt, the AIPREF drafts, ODRL, TDMRep, and C2PA.  The draft doesn’t propose any new mechanisms of its own.  It’s a set of definitions intended to make all of these conversations more precise.

The room was full and the discussion was encouraging.  There seems to be substantial interest in applying the framework in other standards efforts beyond the IETF.

If any of this is relevant to what you're working on, I'd be very glad to hear from you.