SingleFile → Tana Pipeline

SingleFile → Tana Pipeline

The clipper Ed uses as of 2026-07-18. Replaces the browser-extension clippers (⌘⇧U Tana, ⌘⇧Y Notebooks), which are retired.

Why

The old clippers stripped page chrome in the browser, before saving, using a class/id name heuristic (“delete anything named like an ad”). That heuristic is what silently deleted whole articles — thespruce.com (sc-ad-container on the article wrapper), Squarespace (overlay in the 2,293-char <body> class), Substack (newsletter-post). A name-match cannot reliably tell chaff from content, and when it guesses wrong it takes the article with it.

This pipeline is Ed’s design and inverts the order: capture the whole page first, clean it afterward with a purpose-built article extractor working on the complete file. It also makes the login/paywall problem vanish — the SingleFile extension saves from the live, logged-in Helium tab, so public and gated pages both work with no cookies or debug port.

How it works

SingleFile extension (your live Helium tab, logged in)
        │  saves a self-contained .html
        ▼
~/Downloads/singlefile/           ← drop folder (extensions may only write under Downloads)
        │  launchd me.edmd.singlefile-tana polls every 5s, waits for the file to stop growing
        ▼
singlefile_to_tana.py (:8789)
   1. reads original URL / title / saved-date from the SingleFile comment in the file
   2. Trafilatura (reader-mode) extracts the article → markdown   [DETERMINISTIC, no LLM]
   3. tidy pass: drop infobox tables, citation + edit-section links; reflow fragments
   4. markdown → Tana Paste (shared tana_paste.py) → #clip node in the Inbox
   5. move the .html to the fidelity archive; node links it via GET /view
        ▼
Tana Inbox: #clip node (URL, Clipped, Site, "View original") + clean article outline
~/Sync/ED/singlefile-archive/     ← perfect self-contained copy, served at localhost:8789/view

Extraction is a reader-mode engine (Trafilatura), not an LLM — it never invents or drops text. Images are inlined by SingleFile as huge data: URIs; Trafilatura leaves them out of the text extraction, so Tana nodes stay light and the pictures live in the fidelity copy.

#clip fields

Deterministic, extracted per clip; a field is only added when it has a value:

Field Source
URL, Clipped, Site the SingleFile comment header
Saved page link to the fidelity copy (/view) — was “PDF” pre-2026-07-18
Author, Published, Excerpt Trafilatura metadata (author deduped; date; publisher description)
Reading word count + estimate (N words · M min, 220 wpm)
Cover the page’s hero/share image (og:image, fallback twitter:image)
Summary, Topics left empty — the pipeline does not use an LLM. These exist so a Tana AI command node can fill them on demand if Ed ever wants (he declined an always-on LM Studio dependency).

Behaviours

  • Dedup. A local URL ledger (clipped-urls.json, synced via ~/Sync/ED) records every clipped URL. Re-clipping a URL already in Tana is skipped (DUP — archived only, no duplicate node). A ledger is used rather than a Tana query because the URL field stores an <a> anchor that Tana’s field-match won’t match against the plain URL.
  • Link-only clips. When the extracted article is under 600 chars (a listing/product page, a login wall) the clip is filed with fields + View original and no body, rather than a fragmented mess.
  • Inline images. SingleFile inlines images as base64 data: URIs (unrenderable in Tana), but the original URL survives in lazy-load attrs (data-src/data-srcset/srcset) or, if the SingleFile “Save original URLs” option is on, data-sf-original-src. _restore_images() rewrites each <img> src to that URL before Trafilatura, dropping images with no URL and small chrome (avatars, logos, book-cover thumbs, banners) by size (IMG_MIN_DIM=500) + a name blocklist. Real article photos render in the note; the fidelity copy keeps every image. Enable SingleFile’s “Save original URLs” to make this work on every site, not just lazy-loading ones.
  • Gzipped archive. archive_file() stores captures as *.html.gz; /view serves with Content-Encoding: gzip. Only ~1.4× smaller in practice (the files are mostly already- compressed JPEG data), but it compounds as the archive grows and syncs to both Macs.
  • Empty Summary/Topics fields are filled by a Tana AI command on demand (Ed’s design) — the pipeline uses no LLM. The MCP can’t configure that command; it’s a UI setup.
  • Hierarchical body. The article’s outline is preserved: each heading is a parent node and its section nests underneath, sub-headings nesting deeper (md_to_tana_tree). A whole-line bold question adopts a short (1-2 paragraph) answer; longer runs stay flat so a question can’t swallow a whole section. Not a Tana Paste limitation — Tana honours 2-space indentation as hierarchy; the old flat output was just how the pipeline emitted nodes. (Dragonfly example: 160 flat nodes → 14 top-level with proper nesting.) Body is imported as per-section pastes (tree_to_pastes) so a subtree is never split and parent links survive.

Setup (one-time)

Nothing to configure. The watcher monitors ~/Downloads directly, and SingleFile saves there by default — so just clip with SingleFile’s button or hotkey and it lands in Tana within a few seconds. (Ed chose Downloads-direct on 2026-07-18 to avoid the SingleFile Companion, a separate helper program SingleFile would otherwise need to save outside the download folder.)

The watcher only ever ACTS on files that carry SingleFile’s signature — every other download is read once, remembered, and left exactly where it is. It never deletes anything. A processed SingleFile clip is moved out of Downloads into the fidelity archive (preserved and linked from its Tana node), so clips don’t pile up in Downloads.

With the old clippers retired, SingleFile can reclaim ⌘⇧Y / ⌘⇧U (they had collided).

Paths & services

Thing Path
Service ~/Sync/ED/scripts/singlefile-tana/singlefile_to_tana.py (:8789)
Shared Tana Paste lib ~/Sync/ED/scripts/tana-clipper/tana_paste.py
launchd me.edmd.singlefile-tana (RunAtLoad, KeepAlive) → runs run.sh
launch wrapper singlefile-tana/run.sh — self-builds the venv on first launch (parity-safe)
venv ~/venvs/tana-clipper (trafilatura, bs4, lxml, markdownify)
URL ledger (dedup) singlefile-tana/clipped-urls.json
Watched folder ~/Downloads (direct; only SingleFile-signed files are acted on)
Fidelity archive (kept) ~/Sync/ED/singlefile-archive/
Log ~/Sync/ED/scripts/singlefile-tana/singlefile-tana.log
Processed-state ~/Sync/ED/scripts/singlefile-tana/processed.json
Tana tag #clip gYJaHULPWCaI (shared with the retired Tana clipper)

The old clippers (me.edmd.clipper :8787, me.edmd.tana-clipper :8788) are retired but left in place (dormant, harmless). Their code, the pdf-manifest/clip-reap protection, and their invariants remain valid for the PDFs they already produced.

Gotchas

  1. This depends on the SingleFile extension being installed and set to save to the drop folder. No file in the folder → nothing happens. That’s the one manual dependency.
  2. Trafilatura fragments citation-heavy pages (Wikipedia leads): each inline link/bold/ citation can become its own block. The tidy_markdown() reflow pass in tana_paste.py rejoins them (Shiitake test: 128 → 73 nodes, 17 → 4 fragments). Wikipedia is the worst case; ordinary articles extract cleanly.
  3. Images don’t appear in the Tana outline — they’re data: URIs in the SingleFile file and Trafilatura drops them. The fidelity copy (View original) has every image. This is by design; recovering original image URLs from SingleFile’s data-sf-original-src attributes is possible later if inline images in Tana are wanted.
  4. Infobox / sidebar tables are dropped (include_tables=False) — they’re chaff for reading and Trafilatura renders them poorly. A genuinely tabular page loses that table from the outline but keeps it in the fidelity copy.
  5. file:// links don’t open from Tana (webview blocks them) — that’s why the fidelity copy is served over http://localhost:8789/view instead of linked as a file path.
  6. Watches ~/Downloads directly and MOVES (never deletes) processed clips to the archive. Non-SingleFile downloads are never touched — only read once and remembered in processed.json (so they’re not re-read every poll; that skip is why the if name in done check in scan_once is load-bearing). If a clip should instead STAY in Downloads, change the shutil.move to shutil.copy2 — but then Downloads accumulates every clip.

Troubleshooting

curl -s http://localhost:8789/health                 # ok tana=up watch=...
tail -20 ~/Sync/ED/scripts/singlefile-tana/singlefile-tana.log
launchctl kickstart -k gui/$(id -u)/me.edmd.singlefile-tana
python3 ~/scripts/check-invariants.py --only singlefile_tana_up
ls ~/Downloads/singlefile/                            # anything stuck here = not processed

Log lines: OK (clean), PARTIAL (some node batch failed — logged), THIN-OK (little text extracted; fidelity copy kept — check the source), TANA-DOWN (Tana app not running; the file stays in the drop folder and retries next poll — nothing lost).

Edit Markdown files in /content/ · Auto-index regenerated every deploy · / or ⌘K to search