Tech Jobs Radar

Overview

Tech Jobs Radar is a live job search site for tech, software, data and IT vacancies in the Netherlands, collected directly from employers' own career sites. Each posting is classified as tech or non-tech and annotated with structured fields: role family, skills, seniority, experience, degree, salary range, posting language, whether Dutch is required, visa sponsorship, remote policy and, for internships, whether university enrolment is required. Every listing links back to the employer's original page, and optional accounts keep a profile and saved jobs across devices.

Architecture

sources -> queue -> workers + adapters -> normalise, filter, dedup -> classify, extract -> Postgres -> API -> site

• Ingestion. Source adapters for the public job board APIs of common applicant tracking systems (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor, Personio, Workday, SmartRecruiters and others), plus schema.org JobPosting data read through careers site sitemaps.
• Crawling. A Redis queue (RQ) with worker processes, one job per board, with per-host rate limits shared across all workers and robots.txt respected.
• Processing. A Netherlands-only filter, duplicate linking, posting lifecycle tracking and periodic link checks, followed by a rule-based classifier and extractor that fill a typed schema.
• Serving. A FastAPI JSON API with a response cache, and a plain JavaScript front end with Chart.js and D3.

Every adapter maps a board's response to the same raw posting record, so the rest of the pipeline is independent of the source. The crawl steps are plain functions, so the same code runs in a queue worker, in a single local process and in the tests.

Data pipeline

Source discovery

New boards are discovered through the Internet Archive's URL index and the IND register of recognised sponsors, with a weekly discovery run. Many large Dutch employers run their own career site rather than a public tracking board, but publish JobPosting structured data because Google for Jobs requires it. For these, discovery follows a domain's sitemaps to pages that look like job details and registers the sitemap only if sampled pages carry that data.

Crawl scheduling

Crawl frequency adapts to each source's recent yield of Dutch postings, and sources that keep failing are disabled. Rate limits are counted per host in Redis, so all workers draw on one budget, and boards on a shared multi-tenant platform share one limit because such platforms rate-limit by IP. robots.txt is matched with longest-rule precedence, as major search engines apply it, rather than the first-match behaviour of Python's standard parser, which would read "disallow everything, allow /careers" as a ban on the careers pages.

Periodic jobs are queued by a scheduler thread inside the worker rather than as Kubernetes CronJobs, which start a fresh process for every run and would strain the memory budget. They take priority over crawls and run one at a time in a forked child process that frees its memory on exit. Each time slot is claimed in Redis with an atomic set-if-absent, so several workers never queue the same run twice.

Lifecycle

Each posting is normalised into canonical fields, including a city, a duplicate key from employer, title and city, and a content hash. Classification and extraction run only when a posting is new or its hash changes, which keeps re-crawls cheap. Contact details are removed from posting text before storage. Postings that disappear from a board are closed, with two guards: if a board that had at least ten postings suddenly returns fewer than half as many, the response is treated as partial and nothing is closed, and an employer's expiry date is ignored for a source when more than 80% of its dated postings appear expired, since the field is then a template value.

Deduplication

Duplicates are linked to one canonical posting rather than deleted, by six rules: the same vacancy on several sources, reposts under a new URL, the same original URL, per-city copies recognised by identical text once city names and numbers are removed, the same page under another language segment of its URL, and Dutch and English versions of one vacancy. The employer's own copy is preferred over an agency's.

Translated copies cannot be matched by key, because their titles differ. Within a source, a Dutch and an English posting are candidates when they share a day and city and either their titles are near-identical after removing language-specific prefixes or they share most of their numbers (salary, hours, reference numbers). Pairing is greedy and one-to-one on the combined score, so two Dutch jobs cannot collapse onto one English job.

Classification

The tech classifier scores job titles by rules, with an exclusion list for non-technical functions and phrases that keep a title technical despite an excluded word ("Sales Engineer"). Generic titles such as "Engineer" or "Analyst" count as tech only when the role text describes hands-on software, IT or data work. Titles are also mapped to position types by an ordered list in which the most specific pattern is tried first, so "Test Automation Engineer" is QA rather than software engineering.

Information extraction

The extractor fills a typed Pydantic schema. It first segments a posting into the job description and the company description using Dutch and English section headers, and reads skills only from the job part, since company introductions often name the employer's own products. Skills are matched against a taxonomy of canonical skills with alias patterns, which also disambiguates skill names that are common Dutch words. Posting language is decided from function words that occur in only one language.

Requirements are parsed clause by clause, so that "Dutch is a plus" is not a Dutch requirement, office days in the same sentence turn "remote" into hybrid, and negation applies only within its clause. The rules are versioned, so stored postings can be re-extracted after a change. The rule-based extractor serves as a cheap, deterministic baseline that any LLM extractor would have to beat on the golden set.

Serving

API processes keep the live rows in memory. When the pipeline changes the data it increments a version key in Redis. Each API process reloads on its next request, and heavy responses are cached under the current version, so a new version starts a fresh cache namespace without explicit invalidation.

Evaluation

The extractor is scored field by field against a golden set of 46 postings, built by correcting the rule outputs: exact match for scalar fields, and set precision, recall and F1 for skill lists. This evaluation runs in CI on every push as a quality gate, and the build fails if the mean of nine field accuracies and the skills F1 falls below 0.70, so a rule change cannot silently degrade extraction.

A second gate measures the site's filters on 500 live postings sampled across sources, languages and levels, split into 300 for tuning the rules and 200 held out for scoring only. Each filter is measured as it is used, with 95% confidence intervals, and for filters that hide or show postings the costly error is reported separately: for "English, no Dutch required", the share of shown jobs that need no Dutch. Since tech role classification is heavily imbalanced, with some rare categories, scores are tracked per field rather than as a single number.

Deployment

Production runs on k3s on a single AWS EC2 instance within a 2 GB memory budget, packaged as a Helm chart, with the AWS resources defined in Terraform. GitHub Actions runs the tests and evaluation gates on every push and builds an image tagged with the commit. Flux then deploys the release from Git in a GitOps workflow and rolls it back if the new pods do not become ready. Prometheus collects metrics, an alert fires when crawl volume drops, and the database is backed up to S3 nightly.

Data and terms

Only public job board APIs and careers pages are read. LinkedIn, Indeed and Glassdoor are excluded, as their terms prohibit automated access, as are aggregators and sites that block the crawler.