Farpoint · client project · 2026
BDC Market Watch
A daily pipeline that turns news into strategic signals for the Business Development Bank of Canada.
- LLM pipelines
- Embeddings
- Production
Overview
An end-to-end pipeline that reads the day’s news from many sources, groups related stories, and has an LLM judge only the strongest clusters — producing decision-ready signal cards for BDC’s leadership.
Contributions and results
- Shipped to BDC end-to-end in about three weeks.
- In-process ONNX sentence embeddings and cosine clustering before the LLM judge — an estimated ~5–10× fewer LLM calls than judging every article, with no per-call embedding cost.
- Multi-source polling (RSS, Atom, sitemaps, JSON feeds) with deduplication by hash, SimHash and string similarity.
- Idempotent at every stage, with crash recovery from durable per-run state in PostgreSQL.