Milestone 1 - Data Crawling - Introduction to Data Science

This milestone builds the raw corpus: every version of every assigned arXiv paper, with its full LaTeX source (figures removed), its metadata, and the metadata of its references. All numbers below are computed from the crawl databases when the book is built.

import json
from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd
from IPython.display import Markdown, display

# ../data when run from m1/ (book build), data/ when the kernel starts at the repo root (Binder)
DATA = next(p for p in (Path("../data/m1"), Path("data/m1")) if p.exists())
members = sorted(p.name for p in DATA.iterdir() if p.is_dir() and p.name != "all")
summaries = {m: json.loads((DATA / m / "summary.json").read_text()) for m in members}
summary = json.loads((DATA / "all" / "summary.json").read_text())
papers = pd.read_parquet(DATA / "all" / "papers.parquet")
resources = {m: pd.read_parquet(DATA / m / "resources.parquet") for m in members}
tables = json.loads((DATA / "tables.json").read_text())


def gnu_time(member):
    values = {}
    for f in (DATA / member).glob("time_*.txt"):
        for line in f.read_text().splitlines():
            key, _, value = line.strip().rpartition(": ")
            values[key] = value
    return values


gnu = {m: gnu_time(m) for m in members}
plt.rcParams.update({"figure.figsize": (7, 3.6), "figure.dpi": 110, "axes.spines.top": False,
                     "axes.spines.right": False, "axes.grid": True, "grid.alpha": 0.3})


def mb(x):
    return f"{x / 2**20:,.1f} MB"


def hms(seconds):
    seconds = int(round(seconds))
    return f"{seconds // 3600}h {seconds % 3600 // 60:02d}m {seconds % 60:02d}s"


def table(rows, columns):
    return pd.DataFrame(rows, columns=columns).style.hide(axis="index")

1. Scope¶

The Lab Instructors assigned each member of Group 15 a range of about 5,000 arXiv IDs. A range that crosses a month boundary (m2) is split into two spans; the open span ends at the first batch of IDs for which arXiv returns no paper.

Member	Name	Range
m1	Nguyễn Cát Huy	2610.02471 - 2610.07470
m2	Tiết Trọng Khiêm	2608.29007 - end of 2608, 2609.00001 - 2609.02833
m3	Nguyễn Trực Phúc	2607.18694 - 2607.23693
m4	Lê Kiến Quốc	2607.08694 - 2607.13693
m5	Trịnh Tấn Phát	2601.08156 - 2601.13155

display(Markdown(
    f"All five ranges were crawled ({', '.join(members)}): {summary['ids_assigned']:,} IDs queried, "
    f"{summary['papers_found']:,} papers on arXiv, {summary['versions_total']:,} versions."
))

2. Implementation¶

The crawler is a Python package (src/introds) driven by one resumable command, python -m introds crawl --member <m> [--s3]. It records the state of every paper, version and reference request in a SQLite database, so an interrupted run continues where it stopped.

assigned IDs ──► 1. discovery (arXiv API) ──► papers + versions
                                         ├──► 2a. S3 bulk archive (latest versions, optional)
                                         │    2b. export.arxiv.org (every version still missing) ──► .tex / .bib
                                         └──► 3. references (Semantic Scholar) ──► reference metadata

2.1 Entry discovery¶

We query the arXiv API through arxiv.py (Client.results(Search(id_list=...))) with batches of 100 IDs. The short ID of each result carries the latest version (2601.08157v3), which gives the number of versions, the v1 date (published) and the date of the latest version (updated). IDs missing from the response are stored as not_found. A second pass queries versioned IDs (2601.08157v2) to obtain the dates of intermediate versions, needed for revised_dates. Batching resolves 5,000 IDs in about 100 requests instead of 5,000.

Methods considered. The specification lists four discovery tools. OAI-PMH (ListRecords with arXivRaw) returns every version with its date, but a from/until sweep covers every paper modified in the period, far more records than our 25,000, and its resumption tokens expire daily. The Kaggle snapshot is a 4+ GB file refreshed weekly that may miss the newest IDs (m1 is from the current month). S3 holds sources, not metadata. The batched arXiv API gives exactly the assigned IDs, their version count and the dates of every version in minutes, so it is the discovery method; S3 is used later for sources (section 2.3).

2.2 Full-text sources from export.arxiv.org¶

Each version is downloaded from https://export.arxiv.org/src/<id>v<k> (never from arxiv.org), with a descriptive User-Agent, one connection at a time and at least 3 seconds between requests. The server cuts long transfers of large archives, so a dropped transfer is continued with an HTTP Range request from the last byte received. The status code of every attempt is logged; HTTP 429 and 5xx responses are retried with exponential backoff, and HTTP 404 marks the version missing. The response is spooled in memory and inspected:

a tar archive (gzipped or not): only regular .tex and .bib members are extracted, keeping their relative paths so that \input / \include still resolve in Milestone 2; figures and every other file are never written to disk. figure environments and \includegraphics commands are left untouched inside the TeX files;
a single gzipped file or a plain file: kept as main.tex if it looks like LaTeX;
a PDF or any other format: the version is kept as an empty record (no_tex), not dropped.

Many submissions carry their bibliography only as a compiled .bbl. The crawl keeps .tex and .bib as the specification asks and only counts .bbl files. For Milestone 2, the .bbl of the 2,913 versions that have one but no .bib was fetched afterwards (python -m introds fetch-bbl), so that their references can still be converted to BibTeX.

2.3 Latest versions from the S3 bulk archive¶

With 3 seconds per request, export.arxiv.org needs at least 6 hours for a range of ~7,000 versions. arXiv also publishes its sources in a requester-pays S3 bucket (s3://arxiv/src/) as monthly tars of ~500 MB, listed in arXiv_src_manifest.xml with the first and last ID of each tar. Each tar holds one gzipped source per paper (2608/2608.28971.gz), as it was when the tar was packed, without a version number. With --s3, the crawler streams the tars that contain the assigned IDs (nothing is stored), infers the version as the latest one submitted before the file’s modification time, and extracts it exactly like a download. Every other version (older versions, revisions after packing) stays pending for export.arxiv.org. verify-s3 re-downloads a random sample from export.arxiv.org and compares every kept file by SHA-256: 15 of 15 matched. The newest month (m1) is not in S3 yet, because tars are added about once a month.

2.4 References¶

Semantic Scholar’s batch endpoint (POST /graph/v1/paper/batch) returns the references of up to 100 papers per request, with flat field names (references.title, references.authors, references.year, references.publicationDate, references.externalIds). Every reference is kept locally; only those with an externalIds.ArXiv value enter the references table.

2.5 Concurrency, robustness and storage¶

arXiv and Semantic Scholar are different hosts with separate rate limits, so sources and references run in two threads, each with its own rate limiter. Failed items are retried up to three times, and every 15-20 minutes an incremental backup (SQLite snapshot plus a tar shard of the newly fetched versions) is written so the crawl survives the loss of the Colab VM.

The corpus follows the suggested organisation: one folder per paper (2601-08157) holding one sub-folder per version with its .tex/.bib files, plus metadata.json and references.json.

Tool / library	Used for
`arxiv` (arxiv.py 4)	entry discovery through the arXiv API
`requests`	source downloads, Semantic Scholar batch API
`boto3`	S3 bulk archive (requester pays)
`tarfile`, `gzip`	in-memory extraction of `.tex` / `.bib`
`sqlite3`	crawl state, metadata, timings, resource samples
`psutil`, GNU `time`	RAM / disk measurement
`pandas`, `pyarrow`	statistics and Parquet export

3. Corpus statistics¶

def corpus_rows(s):
    return [
        s["ids_assigned"], s["papers_found"], s["papers_success"], f"{s['paper_success_rate']:.2%}",
        s["versions_total"], f"{s['avg_versions_per_paper']:.2f}",
        mb(s["avg_paper_bytes_before_figure_removal"]), mb(s["avg_paper_bytes_after_figure_removal"]),
        f"{s['avg_references_per_paper']:.1f}", f"{s['avg_arxiv_references_per_paper']:.1f}",
        f"{s['reference_metadata_success_rate']:.2%}",
    ]


labels = ["IDs queried", "Papers found on arXiv", "Papers scraped successfully", "Success rate",
          "Versions", "Average versions per paper", "Average paper size before figure removal",
          "Average paper size after figure removal", "Average references per paper",
          "Average references with an arXiv ID", "Reference metadata success rate"]
frame = pd.DataFrame({"Statistic": labels} | {m: corpus_rows(summaries[m]) for m in members}
                     | {"All": corpus_rows(summary)})
frame.style.hide(axis="index")

Definitions. IDs queried includes the IDs probed after the end of a month for a range that crosses a month boundary (m2): discovery stops at the first batch of 100 IDs without any paper. A paper is scraped successfully when every one of its versions was resolved: fetched with TeX, kept as an empty record because arXiv has no TeX source for it, or answered with HTTP 404 by arXiv (missing). Sizes are summed over all versions of a paper: before is the total uncompressed size of every file in the source archives, after is the size of the kept .tex/.bib files. The reference metadata success rate is the share of papers for which Semantic Scholar returned a record with its reference list.

rows = []
for m in members + ["all"]:
    s = summary if m == "all" else summaries[m]
    origin = {f"from {k}": v for k, v in s["versions_by_origin"].items()}
    rows.append({"member": m} | s["versions_by_status"] | origin)
counts = pd.DataFrame(rows).set_index("member").fillna(0).astype(int)
counts.style.set_caption("Versions by status and by origin")

fig, ax = plt.subplots()
counts = papers.n_versions.value_counts().sort_index()
ax.bar(counts.index.astype(str), counts.values, color="#4C72B0")
ax.set(xlabel="versions per paper", ylabel="papers", title="Figure 1. Number of versions per paper")
plt.show()

ok = papers[papers.success & (papers.bytes_after > 0)]
fig, ax = plt.subplots()
bins = [10**x for x in [i / 4 for i in range(12, 36)]]
ax.hist(ok.bytes_before, bins=bins, alpha=0.6, label="before figure removal", color="#C44E52")
ax.hist(ok.bytes_after, bins=bins, alpha=0.6, label="after figure removal", color="#4C72B0")
ax.set_xscale("log")
ax.set(xlabel="paper size, all versions (bytes, log scale)", ylabel="papers",
       title="Figure 2. Paper size before and after figure removal")
ax.legend()
plt.show()
reduction = 1 - ok.bytes_after.sum() / ok.bytes_before.sum()
display(Markdown(f"Removing figures and non-TeX files reduces the corpus by **{reduction:.1%}**."))

s2 = papers[papers.s2_status == "ok"]
fig, ax = plt.subplots()
top = int(s2.n_refs_total.quantile(0.99)) + 1
ax.hist(s2.n_refs_total.clip(upper=top), bins=40, alpha=0.6, label="all references", color="#55A868")
ax.hist(s2.n_refs_arxiv.clip(upper=top), bins=40, alpha=0.6, label="with arXiv ID", color="#8172B2")
ax.set(xlabel="references per paper (clipped at the 99th percentile)", ylabel="papers",
       title="Figure 3. References per paper")
ax.legend()
plt.show()

4. Running time¶

Times are measured inside the crawler: every unit of work (a discovery batch, an S3 tar, a version download, a Semantic Scholar batch) is timed with time.perf_counter, including rate-limiter waits, and stored in the timings table. The time per paper is the sum of its own downloads plus its share of the batched stages. Wall time covers the crawl command end to end; GNU time -v measures the last (uninterrupted) run of each member independently.

m2, m3, m4 and m5 were crawled with the S3 bulk archive (--s3); m1 only through export.arxiv.org, because its month is not in the bulk archive yet. Wall time is the sum of all runs of a member: a run interrupted by the loss of its Colab VM resumed from its backup, and the time between two runs is not counted. GNU time covers the last run only. During some runs export.arxiv.org throttled requests (HTTP 429), which shows up as longer waits.

rows = []
for m in members:
    s, g = summaries[m], gnu[m]
    rows.append([m, hms(s["time_wall_total_s"]),
                 g.get("Elapsed (wall clock) time (h:mm:ss or m:ss)", "-"),
                 hms(s["time_discovery_total_s"]), hms(s.get("time_s3_total_s", 0)),
                 hms(s["time_src_total_s"]), hms(s["time_refs_total_s"]),
                 f"{s['time_per_paper_avg_s']:.2f} s",
                 f"{s['time_wall_total_s'] / max(s['papers_found'], 1):.2f} s"])
table(rows, ["Member", "Wall time (all runs)", "GNU time (last run)", "Discovery", "S3 bulk",
             "export.arxiv.org", "References (concurrent)", "Work per paper", "Wall time per paper"])

4.1 Whole team¶

Each range was crawled independently (one Colab runtime per range). The table adds up the five ranges; elapsed if run in parallel is the longest range, the wall time the team would see with one Colab runtime per member started together.

wall = {m: summaries[m]["time_wall_total_s"] for m in members}
gnu_s = {}
for m in members:
    e = gnu[m].get("Elapsed (wall clock) time (h:mm:ss or m:ss)", "")
    parts = [float(x) for x in e.split(":")] if e else []
    gnu_s[m] = sum(v * 60 ** (len(parts) - 1 - i) for i, v in enumerate(parts)) if parts else 0.0
papers_n = sum(summaries[m]["papers_found"] for m in members)
versions_n = sum(summaries[m]["versions_total"] for m in members)
work = sum(summaries[m]["time_per_paper_avg_s"] * summaries[m]["papers_found"] for m in members)
origin = {}
for m in members:
    for k, v in summaries[m]["versions_by_origin"].items():
        origin[k] = origin.get(k, 0) + v
rows = [
    ("Papers / versions crawled", f"{papers_n:,} / {versions_n:,}"),
    ("Versions from S3 / from export.arxiv.org", f"{origin.get('s3', 0):,} / {origin.get('export', 0):,}"),
    ("Total wall time (sum of the five ranges)", hms(sum(wall.values()))),
    ("Elapsed if run in parallel (longest range)", f"{hms(max(wall.values()))} ({max(wall, key=wall.get)})"),
    ("Total GNU time of the last runs", hms(sum(gnu_s.values()))),
    ("Entry discovery, all ranges", hms(sum(summaries[m]["time_discovery_total_s"] for m in members))),
    ("Average wall time per paper (team)", f"{sum(wall.values()) / papers_n:.2f} s"),
    ("Average work per paper (team)", f"{work / papers_n:.2f} s"),
    ("Average wall time per version (team)", f"{sum(wall.values()) / versions_n:.2f} s"),
]
table(rows, ["Whole team", "Value"])

fig, ax = plt.subplots()
for m, color in zip(members, ["#55A868", "#C44E52", "#4C72B0", "#8172B2", "#CCB974"]):
    t = papers[papers.member == m].time_total_s
    ax.hist(t.clip(upper=t.quantile(0.99)), bins=50, alpha=0.6, label=m, color=color)
ax.set(xlabel="seconds per paper (clipped at the 99th percentile)", ylabel="papers",
       title="Figure 4. Crawling time per paper")
ax.legend()
plt.show()
per_version = {m: summaries[m]["time_wall_total_s"] / summaries[m]["versions_total"] for m in members}
display(Markdown(
    "With export.arxiv.org only, a paper with *k* versions needs at least 3*k* seconds. "
    + " ".join(f"**{m}** needs {v:.2f} s of wall time per version." for m, v in per_version.items())
    + " Filling the latest versions from S3 (m2-m5) removes most export requests; the "
    "export-only range (m1) also lost time to periods of throttling by export.arxiv.org."
))

5. Memory footprint¶

A background thread samples the resident set size (RSS) of the crawler process and its children every 2 seconds with psutil, and the size of the data directory every 60 seconds (the equivalent of du -sb). Samples are stored in the resource_samples table. Benchmark environment: Google Colab, CPU-only runtime (2 vCPU Intel Xeon @ 2.20 GHz, 13.6 GB RAM, Python 3.13).

rows = []
for m in members:
    s, g = summaries[m], gnu[m]
    peak_gnu = g.get("Maximum resident set size (kbytes)")
    rows.append([m, mb(s["ram_peak_bytes"]), mb(s["ram_avg_bytes"]),
                 mb(int(peak_gnu) * 1024) if peak_gnu else "-", mb(s["disk_max_bytes"]),
                 mb(s["output_raw_bytes"]), mb(s["output_db_bytes"])])
table(rows, ["Member", "Peak RAM (sampled)", "Average RAM", "Peak RAM (GNU time)",
             "Max disk usage", "Output: TeX corpus", "Output: SQLite"])

rows = [
    ("Peak RAM of any range (sampled)", mb(max(summaries[m]["ram_peak_bytes"] for m in members))),
    ("Largest disk usage of any range", mb(max(summaries[m]["disk_max_bytes"] for m in members))),
    ("Final output: TeX corpus, all ranges", mb(sum(summaries[m]["output_raw_bytes"] for m in members))),
    ("Final output: SQLite, all ranges", mb(sum(summaries[m]["output_db_bytes"] for m in members))),
]
table(rows, ["Whole team", "Value"])

fig, axes = plt.subplots(len(members), 1, figsize=(7, 2.6 * len(members)), squeeze=False)
for ax1, m in zip(axes[:, 0], members):
    # Runs are placed back to back: the time between an interrupted run and its resume is dropped
    parts, offset = [], 0.0
    for _, run in resources[m].sort_values("ts").groupby("run_id", sort=False):
        hours = (run.ts - run.ts.min()) / 3600 + offset
        parts.append(run.assign(hours=hours))
        offset = hours.max()
        ax1.axvline(offset, color="grey", ls="--", lw=0.8)
    res = pd.concat(parts)
    hours = res.hours
    ax1.plot(hours, res.rss_bytes / 2**20, color="#C44E52", lw=1, label="RAM (RSS)")
    ax1.set(ylabel="RAM (MB)", title=f"{m}")
    ax2 = ax1.twinx()
    ax2.plot(hours, res.disk_bytes / 2**20, color="#4C72B0", lw=1, label="disk")
    ax2.set_ylabel("disk (MB)")
    ax2.grid(False)
axes[-1, 0].set_xlabel("crawl hours (dashed line: end of a run)")
fig.suptitle("Figure 5. RAM (red) and disk usage (blue) over time")
fig.tight_layout()
plt.show()
display(Markdown(
    "Peak RAM is higher with S3 because a whole member of the bulk tar is read in memory before "
    "extraction; the export path spools large archives to disk past 64 MB."
))

6. Output data¶

The corpus follows the suggested organisation: one folder per paper, one sub-folder per version with its .tex / .bib files (original relative paths), plus the paper’s metadata and the Semantic Scholar references. The papers and references tables are exported as sharded Parquet for the final dataset (published with Milestone 3).

layout = tables["layout"]
display(Markdown(f"Example, arXiv:{layout['paper']}:\n\n```text\n" + "\n".join(layout["files"]) + "\n```"))
for name in ("papers", "references"):
    t = tables[name]
    display(Markdown(f"**{name}**: {t['rows']:,} rows in {t['shards']} Parquet shard(s)"))
    display(table([(c["name"], c["type"], "yes" if c["nullable"] else "no", str(t["sample"].get(c["name"]))[:80])
                   for c in t["columns"]], ["column", "type", "nullable", "example"]))

7. Limitations¶

Semantic Scholar may lag behind new arXiv uploads; papers it does not know yet are stored as not_found and can be retried with --retry-not-found.
Sources whose bibliography exists only as .bbl have no .bib in the Milestone 1 corpus, as the specification asks; their .bbl was fetched separately for Milestone 2.
The S3 bulk archive is requester-pays (about USD 1.5 per range) and lags about a month behind arXiv, so it cannot help with the newest month (m1).
Versions that still fail after three attempts are kept as failed and counted against the success rate.

8. Requirements checklist¶

Requirement (Milestone 1 statement)	Where
Crawl every version of every assigned paper	Sections 1, 3 (versions by status)
Full-text sources from export.arxiv.org / S3, figures removed by extraction (`.tex`, `.bib` only)	2.2, 2.3, Figure 2
Versions without TeX kept as empty records	2.2, 3 (`no_tex`, `missing`)
Metadata (title, authors, submission and revision dates, venue) and reference metadata	2.1, 2.4, 6
Only references with an arXiv ID in `references`; all references kept locally	2.4, 6
Tools and libraries	2 (table)
Papers scraped successfully and success rate	3
Average paper size before / after figure removal	3, Figure 2
Average references per paper; reference metadata success rate	3, Figure 3
Total time per paper, average time per paper, entry discovery time, end-to-end wall time	4, 4.1, Figure 4
Peak RAM, average RAM, maximum disk usage, final output size	5, Figure 5
Auditable measurements on Colab CPU-only	4, 5 (`perf_counter`, `psutil`, GNU `time -v`)
Figures and tables generated during book compilation	every figure and table on this page