Architecture

Fetches each active source’s RSS feed, or scrapes its page, and stores new posts. A SHA-256 hash of title and link rejects duplicates at insert. Runs at startup and then every scanner.default_interval minutes (180 in production); a scan still running when the next is due makes that one skip.

Code

internal/scanner, cmd/milesahead/scan.go

Reads / writes

Reads sources; writes raw_articles.

Config

scanner.default_interval, scanner.user_agent

Metrics

milesahead_scans_total

Groups the last 48 hours of scanned posts by title similarity (TF-IDF cosine). Only a cluster with posts from at least two different sites goes on to generation, so every article has more than one source to be checked against.

Code

internal/cluster

Reads / writes

Reads raw_articles.

Config

scanner.cluster_threshold

Two checks before anything is written. Raw-ID check: a cluster that reuses a post some article was written from (article_sources) or already covers (covered_raws) is skipped, and its other posts are recorded in covered_raws, so late coverage cannot come back as a second article once the originals leave the 48-hour window. Story check: the cluster is compared with every article from the last 7 days, in any status: a TF-IDF prefilter keeps up to 5 scoring at least 0.25, then one short AI call on the enrichment chain decides “same story, and if so materially new?”. If that call fails, a lexical score of at least 0.55 decides.

Code

internal/storydedup, internal/db/coverage.go, cmd/milesahead/scan.go

Reads / writes

Reads generated_articles, article_sources, covered_raws; writes covered_raws.

Config

scanner.story_dedup (shadow in production; block or off)

Metrics

milesahead_story_dedup_decisions_total{decision,mode}

Read more: Deduplication

Turns a cluster into a checked draft. The generator writes (with a fallback chain); editor notes are split out; citation markers are removed and any numbered source reference or news-site mention is flagged; the critic chain scores it with thinking off; remarks that only confirm a passed check are dropped; if any issue is left, the generator revises once and the higher-scoring version is kept. Key details and links come from the raw reply, with a separate extraction call only as a fallback.

Code

internal/ai, internal/editornote

Reads / writes

None directly; returns a result to the article manager.

Config

ai.generator_model, ai.fallback_generator_model, ai.critic_model, ai.critic_disable_thinking, ai.critic_threshold, ai.request_timeout

Metrics

milesahead_critic_score, milesahead_articles_generated_total, milesahead_ai_api_duration_seconds, milesahead_ai_api_errors_total

Read more: AI pipeline

A background pass that fills what generation leaves empty: Scout’s Take, key details, tags and the image. A miss is simply retried next pass. Each pass first resizes stored oversized images and serves any Find a new image request; the new picture replaces the old one only once it is found.

Code

internal/enrich, internal/images/scorer.go

Reads / writes

Reads and writes generated_articles; article_image_requests, article_rejected_images.

Config

ai.enrichment_model, ai.enrichment_interval, ai.enrichment_max_attempts, ai.enrichment_max_duration, ai.vision_model

Metrics

milesahead_articles_degraded, milesahead_enrichment_passes_total, milesahead_enrichment_last_success_timestamp_seconds, milesahead_image_resize_total, milesahead_article_images_unsized

Read more: Enrichment · Find a new image

Owns an article’s status. A new article is pending at or above critic_threshold (75) and draft below. Approve and reject post the revision the admin saw and are refused if a rewrite landed since; an approval can be undone for 30 s. Rewrites start at once and a revision reconciler retries them every 5 min with backoff (5, 10, 20, 40 min), giving up after 5 failures.

Code

internal/article, internal/cardupdate, internal/linker

Reads / writes

generated_articles, article_revisions, admin_feedback, verification_results, approval_undo, article_sources.

Metrics

milesahead_articles_approved_total, milesahead_revisions_total, milesahead_revision_reconcile_total{outcome}, milesahead_revisions_waiting{state}

Read more: Article workflow

One SQLite file in WAL mode (modernc.org/sqlite, pure Go) at /opt/milesahead/data/milesahead.db. Migrations run at every start. Search uses two FTS5 indexes (articles_fts, cards_fts; porter stemming, accents folded), rebuilt once when their version in search_index_versions changes.

Code

internal/db, deploy/backup.sh, .github/workflows/backup.yml

Config

db.path

Read more: Database tables · Backups

The Go net/http server on port 9090: the public site (home, articles, cards, compare, search, feed, sign-up), the admin dashboard (session login, CSRF, noindex), POST /api/vote, /healthz and /metrics. HTML is no-cache; content-versioned static files are cached for a year, others for an hour. It also sends sign-up confirmation emails. Admin decisions (approve, reject, rewrite, undo) go through the article manager.

Code

internal/web, templates/, static/

Reads / writes

Reads nearly every table; writes admin decisions, votes, subscribers, sessions and image requests.

Config

server.port, server.domain, admin.password_hash, newsletter.signup_enabled

Metrics

milesahead_http_requests_total, milesahead_http_duration_seconds, milesahead_http_response_bytes, milesahead_votes_total

Read more: Reader site · Article workflow

The VM’s shared reverse proxy, in Docker, reading one file per site from /opt/fynans/caddy-sites/. milesahead.caddy serves milesahead.kumaranik.com over HTTPS, compresses responses with encode zstd gzip, sets security headers and proxies to the app on port 9090. The deploy validates the file inside the container before installing it, and fails if the reload fails.

Code

deploy/milesahead.caddy

Read more: Deployment

Builds the weekly issue the evening before the send (AI-ranked candidates, Scout’s Pick, Card of the Week) and sends it on newsletter.send_day at send_time (Saturday 08:00 New York time) to confirmed subscribers. The issue is re-rendered at send time and each copy gets its reader’s own unsubscribe link, in the body and in List-Unsubscribe with one-click List-Unsubscribe-Post.

Code

internal/newsletter, internal/mail/outbound.go, internal/web/subscribe.go

Reads / writes

Reads generated_articles, reference_pages, subscribers; writes newsletters, newsletter_candidates.

Config

newsletter.send_day, newsletter.send_time, newsletter.timezone, newsletter.signup_enabled, newsletter.unsubscribe_mailbox

Metrics

milesahead_emails_sent_total{kind,outcome}, milesahead_subscribers_total

Read more: Newsletter

robfig/cron with panic recovery, wired in cmd/milesahead/main.go. It starts every background job in the process.

Code

internal/scheduler, cmd/milesahead/main.go

Config

scanner.default_interval, ai.enrichment_interval, mail.imap.poll_interval

Runs on the VM under systemd. Scrapes localhost:9090/metrics every 30 s and remote-writes to Grafana Cloud’s Prometheus, and ships the milesahead.service journal to Grafana Cloud’s Loki.

Code

deploy/alloy-config.alloy, deploy/alloy.service

Read more: Observability