Public status site

Notice Watcher

Generated 2026-07-21 13:40:51 KST

GitHub repository

Overview

1735registered configstracked source definitions
1700watched sitesactive runtime states
269needs reviewfailed or bug markers
6650total jobsfailed: 2361, rejected: 2333, done: 1956

Figures

Watched boards · 1700Content pages · 1544Blocked · 324Dead / bug · 1641
  • static HTML1126
  • JSON API228
  • headless browser203
  • custom adapter139
  • other method4
  • content page1544
  • blocked324
  • dead URL1640
  • system bug1
Figure 1. Every URL we evaluated, arranged from the centre outward by outcome. The inner disc is the set of boards we actively watch — each dot's colour shows how we read it (static HTML, JSON API, headless browser, or a custom adapter; see legend), and the spiral layout has no meaning beyond even packing. The outer rings are URLs we evaluated but did not subscribe to: single content pages, anti-bot blocks, and dead or broken URLs. Hover a dot to highlight the same fetch method and see the domain; click to open the URL in a new tab.
411 cases bolted onto the engine — click a block to open the noteEach square = one case file. Columns = day. Colour = fix layer; above the baseline = improvements that generalised, below = one-off / no-change.Project startSelf-improvement v3Permanent-gate rule5-batch parallel runWorktree isolationAgentic register05-1105-1305-1505-1705-1905-2105-2305-2505-27improved ↑other ↓
Figure 2. Every time the auto-solver fails on a new site, a human reads through what happened and writes a short note in docs/cases/. One block per note, stacked on the day it landed. The thick baseline is "the engine that runs today"; each block above it is a piece of evidence that the engine wasn't enough on its own that day. Tall columns are batch runs (05-21 = 5 site batches in parallel = 118 notes). Colour marks the layer that needed touching — F recognizer / platform code, C probe heuristic, A prompt / agentic, B/D/E engine / writer / validate, or no-change (we wrote a note but no code moved). On top of that colour, the grid splits in two around the baseline: blocks above are the cases that became improvements — patterns the solver now handles on its own next time — while blocks below are everything else (one-off hand patches, no-change notes, rejections) bolted on without generalising. Dashed verticals are infra milestones (hover for what shipped). Click any block to read its note inline, or open the full markdown on GitHub from the modal footer.

Public Source Domains

3268 domains tracked. Type to check whether one is already watched.

Recent Activity

FinishedHostStatusResult
2026-07-20T23:30:50.239952+00:00unknown hostdoneregistered
2026-07-20T23:30:07.006736+00:00unknown hostdoneregistered
2026-07-20T23:30:03.314728+00:00unknown hostdoneregistered
2026-07-20T23:23:29.982060+00:00arca.livedoneregistered
2026-07-20T23:22:49.553319+00:00www.humblebundle.comdoneregistered
2026-07-19T23:31:30.572658+00:00unknown hostdoneregistered
2026-07-19T23:31:11.099385+00:00unknown hostdoneregistered
2026-07-19T23:31:00.467170+00:00unknown hostdoneregistered
2026-07-19T23:23:33.860437+00:00arca.livedoneregistered
2026-07-19T23:22:50.023647+00:00www.humblebundle.comdoneregistered
2026-07-18T23:31:29.437606+00:00unknown hostdoneregistered
2026-07-18T23:31:07.995590+00:00unknown hostdoneregistered
2026-07-18T23:30:57.387388+00:00unknown hostdoneregistered
2026-07-18T23:21:32.748553+00:00arca.livedoneregistered
2026-07-18T23:20:49.563204+00:00www.humblebundle.comdoneregistered
2026-07-17T23:31:35.508753+00:00unknown hostdoneregistered
2026-07-17T23:31:05.202715+00:00unknown hostdoneregistered
2026-07-17T23:30:54.749272+00:00unknown hostdoneregistered
2026-07-17T23:22:31.600547+00:00arca.livedoneregistered
2026-07-17T23:21:49.037277+00:00www.humblebundle.comdoneregistered

What runs when you /watch a new site (HAR)

A first-time /watch <url> cascades top → bottom through bot → worker → register subprocess. Each box is a real call — hover to read the role, click to open the source on GitHub. Dashed boxes are conditional branches (skip / fast-path / early return). Figure 4 below drills into the generate step with a live probe artifact.

bot asyncioworker asyncioregister subprocessbot/main.pywatch()bot/url_gate.pycheck()url_gate…_check_st…url_gate…_check_po…url_gate…_check_ip…url_gate…_check_sa…main.pywatch()skip if alread…bot/db.pyenqueue_j…bot/worker.py_process_job_inner()site_ops…blocking_…bot/site_ops.pyblocking_register()register…_try_know…fast-path: rec…probe.pymain()register…_prefligh…digest.pybuild_dig…codex_ag…run_codex…register…_register…→ configs/<slug>.jso…
Figure 3. /watch call icicle — lane color = which process the call runs in ( bot asyncio,  worker asyncio,  register subprocess). Box width = how many sub-steps live inside — not wall-clock time. Lane transitions mark async hand-off (bot → worker) and OS subprocess spawn (worker → register). On mobile, swipe horizontally if boxes get tight.
Step 1Probe fetchesstatic + browser
Figure 3b (temporary). Probe + register-decide pipeline — the 5-stage funnel that the icicle has not yet absorbed. Click any stage to expand the file-flow detail below. This figure will be folded into the icicle once it expresses the per-stage probe/register order inline.
Step 1

Probe fetches

Static GET first; Playwright captures the rendered page only when static fetch is insufficient.

  1. 1
    scripts/probe.py→main(argv) → _run(args, url, slug)

    CLI entry and output/probe/<slug>/ setup.

  2. 2
    probe/fetch_static.py→fetch()

    First static httpx request with preset headers.

  3. 3
    probe/fetch_headless.py→fetch_with_capture()

    Browser fallback with HAR capture.

Figure 4. From probe artifacts to the agentic config packet

Pick a URL example to read the full path in one place: probe/HAR signals first, then the staged config-generation packet below. Use view buttons for a full-screen explanation of each field plus the raw text that would be shown or derived for the model.

open URL1122 registered probe examples

Probe and HAR signals

entries
206
JSON-ish
15
xhr/fetch
0
HTTP 4xx/5xx
0
sitewww.atptour.com/en/news
HAR mtime2026-05-25T14:25:09+09:00
verdict정적 HTTP로 충분
config strategyplaywright_html
more
probe hosthttps://www.atptour.com/en/news
first article hosthttps://www.atptour.com/en/news/meet-clement-tabur-roland-garros-2026
content-type distribution
content-typecount
x-unknown136
application/javascript18
application/json15
text/javascript14
text/html7
text/plain7
image/gif3
text/css2
keytypecountpreview
List JSON API candidates probe.extract.traffic_api_candidates(har, page_url)
traffic_api_candidates[0]List JSON API3score=7 · GET 200 · - · application/json; charset=utf-8 · list_hits=3 · best_count=5 · keys=PlayerId, PlayerName, PlayerCountryCode, PlayerHeadshot, ScRelativeUrlPlayerProfile, Sc…
traffic_api_candidates[1]List JSON API2score=4 · GET 200 · - · application/json; charset=utf-8 · list_hits=2 · best_count=6 · keys=Type, TeamTieResults, MatchId, Org, IsDoubles, RoundName
Article body JSON candidates probe.extract.traffic_article_body_candidates(har, article_url)
traffic_article_body_candidatesArticle body JSON candidates0Not detected for this probe.
RSS / Atom candidates probe.extract.rss_feed_urls(html, base_url, har)
rss_feed_urlsRSS / Atom candidates0Not detected for this probe.
Pagination candidates probe.extract.pagination_hints(html, base_url, har)
pagination_hintsPagination candidates0Not detected for this probe.
Audio share / player signal probe.extract.audio_share_signal(...)
audio_share_signalAudio share / player signal0Not detected for this probe.
Digest allow-list summary engine.digest.build_digest(...)
list_html/sourceRegister digest1list_html/source · /home/aaaa/notice-watcher/output/probe/host_atptour-com_en_3dfed836/list.html
article_sample/sourceRegister digest1article_sample/source · /home/aaaa/notice-watcher/output/probe/host_atptour-com_en_3dfed836/article.html
signal_countsRegister digest11api=0 rss=11 pag=0 · signal counts
notes[0]Register digest2recommendation · ⚠ 정적 응답 vs Playwright DOM 비교: headless 에만 mosaic/tile 류 반복 패턴 45개 추가됨 (정적 83). 정적 HTML 의 <script id=*-json-data> 같은 JSON island 에서 클라이언트 JS 가 tile 렌더 가능성 — 그렇다…
Stored probe summary (list_candidates.json) list_candidates.jsontype 컬럼 = 저장 필드 값의 자료형 (str / list / dict / null) — 위 signal 종류와 의미 다름. 각 필드 역할은 행 hover.
article_meta_signalsnull—article_meta_signals · null · null
audio_share_host_detectednull—audio_share_host_detected · null · null
body_empty_likelybool—body_empty_likely · bool · false
common_platformnull—common_platform · null · null
discourse_platformnull—discourse_platform · null · null
first_article_urlstr69first_article_url · str (n=69) · "https://www.atptour.com/en/news/meet-clement-tabur-roland-garros-2026"
html_repeating_patternslist15html_repeating_patterns · list (n=15) · [{"selector": "head > meta", "child_count": 34, "first_text": "", "href_common_prefix": null, "href_pattern_guess": null, "sample_url": nul…
hydration_list_candidateslist0hydration_list_candidates · list (n=0) · []
inline_js_data_candidateslist0inline_js_data_candidates · list (n=0) · []
lemmy_platformnull—lemmy_platform · null · null
mastodon_platformnull—mastodon_platform · null · null
mbin_platformnull—mbin_platform · null · null
medium_custom_domainnull—medium_custom_domain · null · null
misskey_platformnull—misskey_platform · null · null
nav_only_same_hostdict6nav_only_same_host · dict (n=6) · {"base_host": "www.atptour.com", "total_same_host": 6, "in_nav": 0, "outside_nav": 6, "nav_only_same_host": false, "sample_nav_ancestors": []}
pagination_hintslist0pagination_hints · list (n=0) · []
peertube_platformnull—peertube_platform · null · null
pixelfed_platformnull—pixelfed_platform · null · null
root_marketing_homepagenull—root_marketing_homepage · null · null
row_external_hostdict8row_external_host · dict (n=8) · {"base_host": "www.atptour.com", "base_registered_domain": "atptour.com", "total_count": 7, "external_count": 3, "external_ratio": 0.429, "sample_…
row_interactive_actionnull—row_interactive_action · null · null
rss_feed_urlslist0rss_feed_urls · list (n=0) · []
runtime_id_candidateslist0runtime_id_candidates · list (n=0) · []
traffic_json_api_candidateslist0traffic_json_api_candidates · list (n=0) · []
wordpress_platformnull—wordpress_platform · null · null
xenforo_platformnull—xenforo_platform · null · null

Agentic config packet

  1. 01Probe artifacts
  2. 02Digest build
  3. 03Tmpdir staging
  4. 04Agent loop
  5. 05Parent publish

Model input packet

에이전트 tmpdir 에 실제로 깔리는 파일. ①직접 입력=매 run 먼저 읽음, ②필요할 때 조회=조건부, ③도구·산출=모델이 콘텐츠로 안 읽음. digest.json 의 HTML 은 모델이 받는 그대로 (compress + 60K). 행 hover=필드 의미, key 클릭=raw 전체.

keytypesizepreview
프로브 산출물 — digest 로 접힘 (모델 직접 입력 아님)
probe608885 bytes608885 bytes
probe259701 bytes259701 bytes
probe6937 bytes{ "html_repeating_patterns": [ { "selector": "head > meta", "child_count": 34, "first_text": "", "href_common_prefix": null, "href_pattern_gues…
probe29179 bytes{ "slug": "host_atptour-com_en_3dfed836", "url": "https://www.atptour.com/en/news", "verdict": "정적 HTTP로 충분", "recommended_strategy": "httpx (S1.H2)", "recommended_heade…
probe3014 bytes{ "page_url": "https://www.atptour.com/en/news", "candidates": [ { "source": "page-feed-link", "url": "https://www.atptour.com/en/feedback", "validated":…
probe229 bytes{ "platform": "Linux-6.18.2-arch2-1-x86_64-with-glibc2.42", "python": "3.13.11", "arch": "x86_64", "outbound_ip_local": "172.30.1.99", "goodbyedpi_running": false, "go…
probe599 bytes{ "url": "https://www.atptour.com/robots.txt", "status": 200, "crawl_delay": null, "disallow": [ "/", "/", "/", "/", "/", "/", "/", "/", …
probe252 bytes{ "page_url": "https://www.atptour.com/en/news", "sitemap_urls_tried": [ "http://www.atptour.com/sitemap" ], "candidates": [], "stats": { "sitemap_count": 1, …
① 직접 입력 — 항상 읽음 (every run)
Prompt11001 bytes# notice-watcher register agent You are an automated configuration agent. ## GOAL Produce a working notice-watcher config JSON. Parent re-validates and publishes. ## STRICT SC…
Prompt1020 bytesTASK: register a notice-watcher config Read AGENTS.md FIRST (in your cwd). Then: - digest.json, slug.txt, url.txt - failure_packet.json if present (previous api_loop_once failed;…
Evidence80069 bytes{ "slug": "host_atptour-com_en_3dfed836", "url": "https://www.atptour.com/en/news", "signal_counts": { "har_entries": 206, "json_entries": 15, "xhr_fetch": 0, …
Examples358 bytes{ "selection_rule": "top 2 scored configs excluding the current slug", "current_strategy": "playwright_html", "manifest": [ { "slug": "arca_GENERATED", "scor…
② 필요할 때 조회 — 조건부 (on-demand)
Examples7137 bytes{ "arca_GENERATED.json": { "version": 1, "site": "arca.live", "board": "akendfield", "strategy": "playwright_html", "headers": { "user-agent": "Mozilla…
Rules31496 bytes너는 게시판/공지 사이트 크롤링 엔진의 "config 작성기"다. 입력으로 한 게시판의 정찰(probe) 요약 digest 를 받고, 그 게시판을 크롤링하는 **선언적 config(JSON)** 를 출력한다. 출력은 **오직 JSON 객체 한 개** — 설명·주석·코드펜스 없이 JSON 객체 그 자체만. 가능한 출력은…
Evidence208 bytes{ "source": "api_loop_once", "attempt": 1, "candidate_config": "previous failed config, if any", "validation_feedback": "short validator failure text", "error": "generat…
③ 도구·산출 — 모델이 콘텐츠로 안 읽음
Evidence296 bytes{ "same_keys_as": "digest.json", "html": "uncompressed in the real tmpdir", "signal_counts": { "har_entries": 206, "json_entries": 15, "xhr_fetch": 0, "traff…
Validator229 bytes{ "launcher": "run_validator.bat on Windows, run_validator.sh elsewhere", "python_path": "/home/aaaa/notice-watcher/.venv/bin/python", "input": "./candidate.json", "timing…
Output343 bytes{ "last_json": { "ok": true, "candidate_path": "./candidate.json", "config": {}, "attempts": [ { "i": 1, "validate_ok": true, "erro…

Raw text view

One full-screen bundle: command shape, stdin prompt, JSON evidence, rules, validator handoff, and expected output contract.

Result path

The agent writes candidate.json; the parent re-validates it and publishes configs/host_atptour-com_en_3dfed836.json. Current published strategy: playwright_html.

Auto-selected each cycle (score-based; sticky to the previous slug when it still qualifies) from output/probe/<slug>/. Generated from the selected probe run plus current prompt/source files. If probe output, config writer rules, agent prompt, validator wrapper, or generate/codex_agentic.py changes, the site manifest changes and this section is rebuilt on the next static-site generation cycle.

Agent guardrail

config 자동 생성은 codex 에이전트가 한다. 에이전트는 repo 밖 임시폴더에 후보 config 만 쓰고, 부모는 ok=true 자기보고를 그대로 받아들이지 않고 사이트를 다시 받아 재검증한 뒤 configs/ 에 발행한다. 임시폴더 격리는 OS 가 강제하지 않는 관례이며, 운영 환경(Windows 단일 머신)에 OS 강제 샌드박스가 없어 임시폴더 밖 변경은 해시 무결성 감사로 실행 후 검출한다.

탐지·대응 (실행 후 검출)

지원 (동시성·원자성)

L4

부모 독립 재검증

에이전트: ok = true(부모는 안 믿음)ok 무시부모가 직접 다시 검증• 사이트 새로 fetch• hard 검사 (글 ≥1건 잡히나)• selector grounding→ 최종 판정

에이전트가 ok=true 를 반환해도 부모는 그대로 받아들이지 않고 validate_built_config 를 다시 실행한다. 사이트를 새로 fetch 해 hard 검사(글 ≥1건 추출 여부 등)와 selector grounding 을 거쳐 부모가 발행 여부를 정한다.

generate/validate.py:380 validate_built_config

모델 codex:gpt-5.4-mini (reasoning effort low), auto 모드 — 싼 1-shot 을 먼저 시도하고, 실패할 때만 멀티턴 에이전트로 올린다.