
Most read-it-later workflows are write-only graveyards. You save an interesting article, drop it into a folder, and never look at it again—even when your own notes already hold a book you read two years ago or a thread you liked that connects directly to it.
If you have spent years accumulating a personal knowledge base in Obsidian—book catalogues, exported social archives, project drafts, and technical notes—you don’t need another cloud bookmarking SaaS. You need a Serendipity Miner that links every new URL to the knowledge you already own.
Here is how we built ~/scripts/serendipity_miner.py and wired it into a <1s-feeling mine <url> command inside our Zero-Trust Signal Gateway.
Architecture: Extract, Score, Resonate, File

The pipeline runs completely locally in four stages without requiring an LLM round-trip just to file a link:
- Zero-Friction Capture (
mine <url>): Send any URL to Signal “Note to Self” (or runserendipity_miner.pyfrom the CLI). - Clean Extraction & Concept Scoring: Strips navigation, scripts, and boilerplate into clean Markdown, then ranks the top salient keywords.
- Parallel Cross-Corpus Search (
ThreadPoolExecutor): Queries your local vault index across four distinct silos simultaneously. - Obsidian Filing + Blog Seed: Writes a structured note under
03-Resources/Clippings/with real Obsidian wikilinks to matching books, tweets, and projects.
1. Clean Markdown Extraction with Graceful Fallback
When defuddle is available on PATH, the miner delegates parsing to defuddle parse <url> --md. When running on headless nodes without Node tooling on PATH, it falls back transparently to BeautifulSoup4 + html2text:
for tag in soup(["script", "style", "nav", "footer", "aside", "noscript", "svg", "form"]):
tag.decompose()
main_el = soup.find("article") or soup.find("main") or soup.body or soup
converter = html2text.HTML2Text()
converter.body_width = 0
converter.ignore_images = True
md_text = converter.handle(str(main_el)).strip()
This keeps extraction fast, deterministic, and resilient across arbitrary blogs and documentation sites.
2. Deterministic Concept Extraction (Zero-LLM Latency)
To search your vault without waiting 10 seconds for an LLM to summarize the page first, extract_keywords() weights tokens by structural position:
- Title tokens:
+8 - Meta description:
+4 - H1–H3 headings:
+3 - Body sample (first 6,000 chars):
+1
After filtering English and technical boilerplate stopwords, the top 4–6 keywords form a high-signal search query in under 5 milliseconds.
3. Parallel Resonance Across Books, Twitter/X, Knowledge, and Projects
Instead of running one global vault search—which gets dominated by recent logs—we fan out four scoped searches in parallel using ~/scripts/search_vault.py --folder:
CORPORA = [
("Books", "03-Resources/Books"),
("Twitter/X", "03-Resources/Twitter"),
("Knowledge", "Knowledge"),
("Projects", "01-Projects"),
]
Running these four queries concurrently via ThreadPoolExecutor(max_workers=4) takes ~100–250ms total and guarantees that an old book in your library or a tweet you liked in 2021 isn’t crowded out by yesterday’s session notes.
4. Filing with Vault Resonance and Blog Seed Callouts
The resulting note in 03-Resources/Clippings/<YYYY-MM-DD> - <Title>.md opens with an Obsidian > [!INFO] Vault Resonance callout containing clickable wikilinks grouped by corpus. Whenever 2 or more corpora match, it automatically appends a > [!TIP] Blog Seed callout highlighting the cross-domain overlap.
In Signal, the reply stays mobile-compact:
💎 Serendipity Mined: Self-Compacting & Bi-Temporal Agent Memory
• Keywords: memory, context, phase, engineering, bi-temporal (bs4+html2text)
• Saved: 03-Resources/Clippings/2026-11-18 - Self-Compacting & Bi-Temporal Agent Memory.md
• Vault Resonance (12 match(es)):
↳ Books: Political Trust..., Wristwatch Annual...
↳ Twitter/X: Liked Tweets 01, Liked Tweets 07
↳ Knowledge: Regression-gated nightly memory...
↳ Projects: Self-Compacting and Bi-Temporal Agent Memory
Paired with Self-Compacting & Bi-Temporal Agent Memory, every article you read immediately weaves itself into your existing graph instead of sitting isolated in a bookmarks folder.
Key Takeaways
- Scoped parallel searches beat global search: Querying Books, Twitter/X, Knowledge, and Projects separately surfaces unexpected cross-domain connections that global ranking hides.
- TF-weighted structural keywords eliminate LLM cold-start: Title + heading boosting extracts accurate search terms in milliseconds.
- Mobile capture closes the loop: Wiring
mine <url>into Signal turns passive scrolling into structured, backlinked Obsidian notes.