Вход на сайт

Просмотр новости

Найдите то, что Вас интересует

Host- and Domain-Level Web Graphs March, April, and May 2026

Дата публикации: 29-05-2026 00:00:00

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, April, and May 2026. The graphs consist of 262.4 million nodes and 8.1 billion edges at the host level, and 118.8 million nodes and 4.3 billion edges at the domain level.

Основное содержимое страницы с новостью.

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, April, and May 2026. The crawls used to generate the graphs were CC-MAIN-2026-12, CC-MAIN-2026-17, and CC-MAIN-2026-21. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior Web Graph Releases. You may also visit the projects cc-webgraph and cc-pyspark which include all scripts and tools required to construct the graphs. Instructions to explore the graphs in the webgraph format are given in our collection of Web Graph Notebooks. You may also wish to explore our Web Graph Statistics page for more information on ranking.

Host-level Graph

The host-level graph consists of 262.4 million nodes and 8.1 billion edges.

There are 198.2 million dangling nodes (75.56%) and the largest strongly connected component contains 36.6 million (13.96%) nodes. Dangling nodes stem from:

  • Hosts that have not been crawled, yet are pointed to from a link on a crawled page
  • Hosts without any links pointing to a different host name
  • Hosts which did only return an error page (eg. HTTP 404).

Host names in the graph are in reverse domain name notation such that: www.example.com becomes com.example.www.

You can download the graph and the ranks of all 262.4 million hosts from AWS S3 on the path s3://commoncrawl/projects/hyperlinkgraph/cc-main-2026-mar-apr-may/host/ (this requires an account on AWS). Alternatively, you can use https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-mar-apr-may/host/ as prefix to access the files from everywhere.

Please note that the text representation of the host-level graph is shipped in 240 gzip-compressed files listed in two path listings - one for the nodes (vertices), one for the edges (arcs). First, download the paths listing and decompress it using gzip -d or gunzip. By adding the prefix s3://commoncrawl/ or https://data.commoncrawl.org/ to each line in the path listing you get the list of URLs to download the entire graph.

Download files of the Common Crawl March, April, and May 2026 host-level Web GraphDomain-level Graph

The domain graph is built by aggregating the host graph on the level of pay-level domains (PLDs) based on the public suffix list maintained on publicsuffix.org. Version (commit) 8b4345f of the public suffix list was used (commit date 2026-05-01T02:51:16Z).

The domain-level graph has 118.8 million nodes and 4.3 billion edges. 63.0% or 74.8 million nodes are dangling nodes, the largest strongly connected component covers 30.2 million or 25.46% of the nodes.

All files related to the domain graph are available on AWS S3 under s3://commoncrawl/projects/hyperlinkgraph/cc-main-2026-mar-apr-may/domain/ or on https://data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-mar-apr-may/domain/.

Download files of the Common Crawl March, April, and May 2026 domain-level Web GraphCredits

Thanks to the authors of the WebGraph framework, whose software made the computation of graph properties and ranks possible. We hope the data will be useful for you to do any kind of research on ranking, graph analysis, link spam detection, etc.

Let us know about your results via Common Crawl's Google Group, or on our Discord Server.

Схожие новости

#Наименование новостиТональностьИнформативностьДата публикации
1Host- and Domain-Level Web Graphs February, March, and April 2026012.1930-04-2026
2Host- and Domain-Level Web Graphs April, May, and June 2026012.1925-06-2026
3Host- and Domain-Level Web Graphs January, February, and March 2026012.1924-03-2026
4Host- and Domain-Level Web Graphs May, June, and July 2026013.0328-07-2026
5Host- and Domain-Level Web Graphs December 2025 and January/February 2026013.8424-02-2026
6February 2026 Crawl Archive Now Available017.5122-02-2026
7May 2026 Crawl Archive Now Available015.9825-05-2026
8April 2026 Crawl Archive Now Available 016.7628-04-2026
9June 2026 Crawl Archive Now Available015.9822-06-2026
10March 2026 Crawl Archive Now Available014.419-03-2026

Классификация: Пресс-релизы. Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 12.19. Источник: commoncrawl.org.