Tiered course-to-role-skill matching engine: tag crosswalk → embedding + keyword scoring → suspected-pair detection → budget-guided LLM arbitration → null-safe ranking → top 5 per (role, skill).

Each candidate course gets a null-safe weighted average of five signals. Any signal that doesn't apply to a given course is dropped, not zeroed — the remaining weights renormalize to sum back to 1 (ranking/scoring.py::composite_score):

Default weights (config.py, sum to 1.0):

signal weight meaning relevance 0.35 Tier 1/2/3 match confidence between the course and the role/skill node authority 0.25 is this source_type (official vs. aggregator) the trusted tier for this specific role/skill, per authority.csv provider_authority 0.15 new — prestige of the provider/instructor behind an aggregator listing engagement 0.15 normalized rating (rating / 5) recency 0.10 not currently populated by any source (always None today → excluded, weight redistributed)

The ask: a Coursera course built by Google or IBM, or taught by a recognized instructor, should outrank an otherwise-similar but unbranded Coursera listing on the same topic. authority_score can't express this — it scores source_type (official vs. aggregator) per role/skill node, not who built a specific aggregator listing.

ranking/provider_authority.py::provider_authority_for(source_type, provider, instructors):

source_type == "official" → returns None (excluded, not zero). An official row (AWS docs published by AWS, Google's own course on Google Cloud) is the authoritative party — there's no separate third-party "who backs this" question to ask, and these rows usually don't even carry provider/instructors data. Scoring them low would wrongly punish them for lacking metadata that doesn't apply to them. Excluding the signal lets composite_score renormalize across relevance / authority / engagement for that course instead.

source_type == "aggregator" → looks up provider in a curated prestige table (mega-cap tech / top universities ≈ 0.9–1.0, recognized ed-tech brands ≈ 0.55–0.8) and instructors in a smaller known-names table, blends them 0.65 * provider + 0.35 * instructor when both are present, and falls back to whichever one is present if only one is. A real-but-unrecognized provider/instructor gets a neutral 0.5 (not punished for being unranked). An aggregator course with neither field populated also returns None — there's nothing to score, so it's excluded rather than guessed at.

Net effect: official-source rows are completely unaffected by branding (as they should be — they can't have a "provider" other than themselves). Among aggregator rows, courses backed by a recognized company/university or taught by a known instructor get a real ranking boost; unbranded listings aren't penalized below the neutral baseline, they just don't get the bonus.

Fixed a coverage-gap bug where a candidate scoring below mid_confidence_low got no mapping row at all — not confirmed by Tier 2, not sent to Tier 3, just silently dropped. That's why niche/specific skills (Kotlin, Apex, Istio, FastAPI, Hugging Face Datasets, ...) were showing 0 courses: the score never crossed either threshold, so the pair vanished before it was ever verified. See settings.coverage_rescue_enabled in config.py and the "Coverage rescue" block in pipeline/run.py for the fix — every taxonomy node with zero resolved candidates now gets its single best remaining candidate escalated to Tier 3 for a real LLM check before it's reported as a gap.

Switched Tier 2 to sentence-transformers (BAAI/bge-base-en-v1.5, local, no API key) from TF-IDF. TF-IDF was the bigger contributor to the bug above: it's pure lexical overlap, so a course that never uses a skill's exact wording scored near-zero against it regardless of actual relevance. (This briefly ran through Gemini's free embedding API, then all-MiniLM-L6-v2, before landing on bge-base-en-v1.5 for stronger semantic separation — see "Recent changes" history if you're diffing against an older version of this repo.) bge-base-en-v1.5 is an asymmetric retrieval model: it wants an instruction prefix on the "query" side that the "passage" side doesn't get. embedder.py and candidate_gen.py apply this automatically (taxonomy/node text is treated as the query, course text as the passage) — a no-op if you swap back to a plain symmetric model like all-MiniLM-L6-v2 or all-mpnet-base-v2 via COURSE_MATCHER_SENTENCE_TRANSFORMER_MODEL. Bigger download than MiniLM (~440MB vs ~80MB) and somewhat slower CPU encoding, in exchange for fewer real matches landing in the ambiguous mid-confidence band that gets escalated to Tier 3 — if Tier 3 volume is still too high after this, the more direct lever is narrowing mid_confidence_low/mid_confidence_high in config.py (currently a wide 0.45–0.75 band) and/or collision_margin, rather than reaching for an even bigger embedding model.

Switched Tier 3 to Llama 4 Scout via Groq (meta-llama/llama-4-scout-17b-16e-instruct, free tier), previously Gemini 2.5 Flash. Same reason as before for not using a batch endpoint: Groq's free tier doesn't have one for chat completions either, so Tier 3 calls are synchronous and rate-limited — but Groq's free tier also enforces a hard daily request cap (1,000 RPD for this model) on top of the per-minute one, which Gemini's free tier didn't, so llm_request_hard_ceiling now defaults to 950 instead of being off by default. See matching/budget_governor.py and check current limits at https://console.groq.com/docs/rate-limits before relying on the numbers baked into config.py — they change without notice.

Added persisted, cross-run daily usage tracking for Tier 3. A new llm_usage_daily table counts real Groq requests per UTC day, checked before every call — not just within one run, but across every run and process that day. Once llm_request_hard_ceiling is reached, submission stops immediately (mid-batch if needed) instead of erroring; the rest of the pipeline (ranking, export, coverage-gap detection) proceeds normally on whatever Tier 3 resolved before stopping. Monitor current usage any time with course-matcher llm-usage, without making an API call.

With both Tier 2/3 provider switches, the pipeline now needs only one API key (COURSE_MATCHER_GROQ_API_KEY) instead of two, and has no dependency on Google's API at all.

Or, if you'd rather use requirements.txt (e.g. your deployment tooling expects one, or you just prefer it): pip install -r requirements.txt -r requirements-dev.txt installs the identical pinned versions. requirements.txt is kept in sync with pyproject.toml by hand; if you bump one, bump the other. requirements-embeddings.txt is the optional sentence-transformers swap-in, kept separate since it pulls in torch.

That's it. It initializes the database, ingests the included sample data (data/raw/, data/crosswalk.csv, data/authority.csv -- data/crosswalk.csv and data/authority.csv are optional and not included; the pipeline runs correctly without them), runs the full matching + ranking pipeline, and prints the top 5 per (role, skill). Everything below is either how main.py works internally, or the granular step-by-step commands if you want to run one piece at a time instead of the whole thing.

To point it at your real data, replace those same paths with your own files (data/raw/ folder, data/crosswalk.csv, data/authority.csv), or override the locations via environment variables — see .env.example. Run python main.py again any time after that; it's always safe, and a run where nothing changed reprocesses zero courses and costs nothing.

This is the actual implementation of the "drop a file in a source folder" half of the architecture:

Column headers don't need to match exactly — RoleID, Role ID, and role_id all work (same for course fields: CourseID, Course Title, Provider, etc.). See ingestion/column_aliases.py for the full alias list; unrecognized columns are just ignored, not an error.

Taxonomy files also don't need a pre-computed "other skills" column. A 163-role, 15-skills-each taxonomy is naturally 2445 rows — one per (role, skill) pair — and other_skills is derived automatically by grouping every row sharing the same role_id. Optional Proficiency / Reason columns, if present, get folded into the embedding context text for extra disambiguating signal.

The subfolder name becomes source automatically. source_type falls back to DEFAULT_SOURCE_TYPES in ingestion/raw_folder.py (coursera/udemy/edx → aggregator, aws/google/microsoft → official) if the file doesn't already have one — extend that map as you onboard new sources, or just put a source_type column in the file itself, which always wins.

If a source isn't in DEFAULT_SOURCE_TYPES and the file has no source_type column, those rows are reported as skipped (not silently guessed) — check the rows_skipped count in the command's output.

The single-file ingest-courses <path> / ingest-taxonomy <path> commands still exist too, for a one-off file or a scripted pipeline that already knows exactly what it's ingesting.

Every ingest-* command accepts .csv, .json, .txt, or .xlsx — pick whatever your source data actually arrives in. Mixed formats across commands are fine (taxonomy from JSON, courses from an Excel export, crosswalk from CSV — all in the same run).

Run course-matcher run again immediately after — it should report courses_reprocessed: 0. That's the idempotency guarantee working: nothing gets re-matched, and nothing gets billed to the LLM tier, unless a course's content or the taxonomy version actually changed.

Every real Tier-3 (Groq) call increments a persisted counter — a row in the llm_usage_daily table, keyed by (UTC date, provider, model) — the moment the call returns, success or failure. That's true across runs and across processes: it's not reset just because you started a new python main.py or course-matcher run, and two runs on the same day share the same count.

Monitor it: course-matcher llm-usage prints today's usage, the configured COURSE_MATCHER_LLM_REQUEST_HARD_CEILING, and how many requests are left — read-only, doesn't call the API. The run summary also reports llm_calls_completed and llm_daily_limit_reached after every pipeline run.

It stops automatically: once today's count reaches LLM_REQUEST_HARD_CEILING (950 by default — just under Groq's 1,000 RPD free-tier cap), submit_and_collect() stops submitting new requests immediately, mid-batch if needed. It doesn't raise or crash the run — whatever Tier 3 already resolved gets ranked and exported normally; the pairs it didn't get to just don't have an LLM-verified mapping this run and either surface as a coverage gap or get picked up automatically on the next run once the daily count resets (at UTC midnight, since the counter is keyed by UTC date).

Set COURSE_MATCHER_LLM_REQUEST_HARD_CEILING to your actual Groq daily limit (check console.groq.com/settings/limits — free-tier limits vary by model and can change) or to blank/None to remove the cap entirely on a paid tier.

tests/test_bugfixes.py — 7 tests covering two real bugs found and fixed during a review of this project (a TF-IDF embedder that densified its matrix and OOM-killed the pipeline on the real catalog size, and a Unicode case-folding mismatch between Python's str.lower() and SQLite's ASCII-only lower() that broke re-running authority-file ingestion on any accented provider/instructor name). This is the only test coverage that currently exists for this project — there is no CI workflow configured either. The rest of the pipeline (candidate generation, scoring, crosswalk, budget governor, ingestion format parity) has been exercised manually against the real sample data but has no automated regression coverage yet.

Concern How it's addressed Persistence Real tables (db/models.py) via SQLAlchemy — SQLite for dev/CI, same code against Postgres via DATABASE_URL Idempotency content_hash + taxonomy_version per course (db/models.py: Course) — re-ingesting or re-running is a no-op unless something actually changed. Not covered by an automated test yet — verified manually by re-running the pipeline against the same DB Incremental processing get_courses_needing_match() only returns the delta — a monthly run touches new/changed courses, not the whole catalog Config config.py, every threshold is an env var with a sensible default — no more hardcoded module constants Validation Pydantic models (domain/models.py) reject bad rows at the boundary with a specific error, not a stack trace three layers deep Retries tenacity-backed exponential backoff on every Groq/Llama 4 Scout call (matching/budget_governor.py) — Tier 2 embeddings run locally via sentence-transformers, so there's no network call there to retry Rate limiting Groq's free tier has no batch endpoint for chat completions, so Tier 3 calls are synchronous, paced to stay under its requests-per-minute ceiling, and also stopped by a hard daily-request ceiling once Groq's 1,000 RPD cap is approached. A failed call after retries is logged and skipped rather than aborting the run Logging Structured JSON (logging_config.py) — pipeline_completed, llm_budget_target_exceeded, coverage_gaps_found, etc. are machine-parseable events, not console prints Method precedence save_mapping_rows() enforces crosswalk > llm > embedding at the database level — a lower-confidence tier can never silently overwrite a higher-confidence one, even across separate runs Observability pipeline_runs table — one row per execution: courses considered/reprocessed, LLM calls, cost, budget-target status, gap count. Queryable history, not just the last run's log lines Multi-format ingestion .csv, .json, .txt, .xlsx all dispatch through ingestion/readers.py into the same validated shape. Not covered by an automated test yet — verified manually against each sample file format Tests 7 tests covering two real bugs found in review (TF-IDF memory blowup, Unicode case-folding in authority-file upsert). No coverage yet for candidate generation, scoring, crosswalk, or budget-governor logic CI Not set up. tests/ runs locally via pytest; no .github/workflows/ yet Packaging Installable (pyproject.toml), course-matcher console entrypoint, Dockerfile

Being direct about the boundary: "production ready" code still needs a production environment to run in. Specifically:

A provisioned Postgres instance. DATABASE_URL swaps in cleanly, but nobody's stood one up for you — RDS/Cloud SQL/wherever your team runs databases.

An actual Airflow deployment. dags/course_matcher_dag.py is structurally correct and will run once deployed to a real scheduler + metadata DB — it does nothing sitting in this repo.

Secrets management. .env is fine for local dev. Production wants GROQ_API_KEY and DATABASE_URL coming from your actual secrets manager (Airflow Connections, AWS Secrets Manager, Vault, whatever your org uses) — not a committed file. This repo had a live, unrevoked Groq API key committed in .env as of this review; it's been removed from the project (only the placeholder .env.example ships now), but the key itself was never rotated here — rotate it at console.groq.com if this project has ever been pushed anywhere.

Real alerting. alert_on_coverage_gaps() in the DAG is a labeled seam, not a Slack/PagerDuty integration — it prints. Wiring it to something that actually pages someone is org-specific and left for you.

Load-tested scale. Proven correct and idempotent on a handful of sample courses. The brute-force embedding matrix is architecturally right for ~2,400 taxonomy nodes, but real catalog-size performance (tens of thousands of courses) hasn't been measured against real hardware.

One API key. Real embeddings (Tier 2) run locally via sentence-transformers — no key, no network dependency, just pip install "course-matcher[embeddings]" (untested in this sandbox, it couldn't install here, but it's a standard PyPI package). Without it installed, the embedder logs a warning and quietly falls back to TF-IDF (weaker matching, but keeps CI/zero-setup runs working). Real LLM arbitration (Tier 3) needs a free Groq key — no credit card, ~1 minute at console.groq.com/keys; without one, Tier 3 falls back to a labeled mock.

None of these are code problems this repo could have solved by writing more Python — they're deployment decisions that belong to whoever owns your infrastructure.