Dashboard

Curator publishes a live dataset analytics databoard:

https://swe-databoard.pages.dev/

Open the Databoard

What it shows

The databoard compares four datasets along one consistent axis — difficulty and semantic tags produced by the same pipeline for every task, so numbers are directly comparable across sources.

id	Display	Source
`self_made`	SWE-Lego-Live-Instances	Curator `swegen-selfmade` (non-top5k + top5k)
`swe_rebench`	SWE-rebench	`nebius/SWE-rebench`
`openswe_filtered`	OpenSWE-filtered	`SWE-Lego/openswe_filtered_for_rl`
`scale_swe`	Scale-SWE	`AweAI-Team/Scale-SWE`

Each dataset panel reports total/tagged counts, a difficulty distribution (label breakdown, percentiles, score bins), and tag clouds for language, area, topic, and bug class.

Difficulty and tag methodology

Difficulty scoring and tagging follow the harbor scripts/task_analysis method, applied uniformly to every dataset.

Difficulty score

A weighted, log-scaled model over five dimensions of the solution patch, tests, and instruction. Each dimension is scaled to 1.0–5.0, combined by weight, then mapped to a final 1.0–10.0 score:

Dimension	Weight	Signal
`patch_scope`	0.30	changed lines + files + hunks
`logic_complexity`	0.25	new defs/classes + control-flow additions
`context_breadth`	0.20	distinct directories touched
`test_complexity`	0.15	test lines + test files
`instruction_complexity`	0.10	instruction length

The score is bucketed into a label:

Label	Score
`easy`	≤ 4.0
`medium`	4.1 – 7.0
`hard`	> 7.0

Scoring is static (no LLM) so it is deterministic and reproducible.

Tags

An LLM assigns exactly four tags per task, in order:

[language, area, topic, bug_class]

language — primary programming language (python, go, typescript, …).
area — one of backend, frontend, fullstack, cli, library, framework.
topic — framework/library or focused technical topic (fastapi, react, async, …).
bug_class — a domain-independent label for the defect mechanism (missing-fallback, incomplete-validation, wrong-default, …), describing the logical failure mode rather than the affected feature.

The tagger disables the model's thinking mode so it returns a strict JSON object, validates the 4-tuple (including the closed area vocabulary), and retries on malformed output. Runs are resumable and idempotent.

Very large or awkward tasks can make the endpoint hang on a full-size prompt. The tagger handles these with progressive truncation: if a request fails, it retries with a shorter patch/instruction/test excerpt (and fewer changed files), stepping down through several levels until the request completes. This guarantees every task receives tags without discarding context for normal-size tasks.

Regenerating the databoard

From subblock/curator/dashboard/:

# 1. Difficulty + tags for every dataset (resumable, 64-way concurrent).
#    The canonical tagger lives in the swegen submodule; --datasets-dir
#    points it at this dashboard's datasets/ directory.
python3 ../repos/swegen/tools/tag_task_metadata.py \
  --datasets-dir datasets --dataset all --jobs 64 --retries 3

# 2. Render the multi-dataset HTML
python3 progress_monitor_multi.py --output-html site/index.html

# 3. Deploy
cd site && CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ACCOUNT_ID=... \
  npx wrangler@latest pages deploy . --project-name=swe-databoard --branch=main

See subblock/curator/dashboard/README.md for dataset export and endpoint configuration.

Docs versus databoard

There are two Cloudflare Pages projects:

Project	URL	Purpose
`swe-swegen-docs`	`https://swe-swegen-docs.pages.dev`	This documentation site
`swe-databoard`	`https://swe-databoard.pages.dev/`	Live dataset analytics dashboard

Use the docs when onboarding or reviewing the pipeline. Use the databoard to compare dataset difficulty and tag composition.