Build a job market tracker with Python and Google Jobs API

Job boards answer just one question, which is what's open today. Everything else stays invisible, including whether Austin posted 40% fewer backend roles this month than last, whether the "remote" tag on half those listings survives contact with the job description, and which companies are hiring at volume while everyone watches the layoff headlines. Every board resets its view each day, and the history disappears with it.

You can build that history yourself. By the end of this tutorial you'll have a Python script that snapshots Google Jobs postings for one role across 5 cities every day, normalizes the messy fields into something you can compute on, stores each snapshot in SQLite, and charts what the current market looks like while the history builds toward trend analysis. The whole thing runs to roughly 200 lines on cron or GitHub Actions, and it consumes about 3% of an entry-level API plan.

The data comes from the SearchApi Google Jobs API, which returns Google Jobs results as structured JSON. That saves you from parsing Google's HTML yourself, and it means the fields you need arrive already labeled.

TL;DR

Google Jobs shows you today's listings and forgets them tomorrow, so you build the history yourself by snapshotting daily and keeping every snapshot instead of overwriting.

• What you build. A Python script that pulls postings for one role across 5 cities, normalizes the messy fields, and writes one row per job per day into SQLite. Roughly 200 lines, running on cron or GitHub Actions.
• What it costs. A 30-day run uses 300 searches, which is 3% of the $40 Developer plan. Building and debugging fits inside the 100-search free tier.
• The 3 fields that fight you. posted_at is relative text and 17% of listings omit it, salaries arrive as unsigned strings like 125K a year, and descriptions come through as HTML that eats your scanning window.
• The response omits job_id. Deduplicate on the htidocid value inside sharing_link, or every run counts the same job as new.
• What the first run showed. A single run returned 104 postings across 5 cities, with genuine remote between 20% and 35% by city once descriptions were checked against the tags, on samples of 4 to 7 listings each.
• What to watch for. Counts that pin to your max_pages ceiling measure your fetch settings rather than the market, and a parser that fails closed will hand you a confident, false finding about the API.

Prerequisites

You'll need Python 3.11 or later, since the code uses datetime.UTC and modern type hints, plus a SearchApi account for the API key. Pin the 3 libraries in requirements.txt so your runs stay reproducible.

requests==2.32.3
pandas==2.2.3
matplotlib==3.9.2

The free plan includes 100 searches, which covers building and debugging the whole pipeline before you pay anything. A production run of 5 cities at 2 pages each uses 10 searches per day, so a 30-day window costs 300 searches.

SearchApi bills by monthly plan rather than pay-as-you-go, which makes that 300 a slice of quota rather than a $1.20 invoice. It represents 3% of the $40 Developer plan's 10,000 monthly searches, meaning this project fits inside the smallest paid tier with room for roughly 30 more like it. Confirm current rates on the SearchApi pricing page before you commit to a schedule.

Export your key as an environment variable so it stays out of the repo.

export SEARCHAPI_KEY="your_key_here"

Understanding the Google Jobs response

Before writing anything, make one live request and read what comes back. This call queries backend engineer roles in Austin and pipes the response through Python's JSON formatter.

curl -G "https://www.searchapi.io/api/v1/search" \
  --data-urlencode "engine=google_jobs" \
  --data-urlencode "q=backend engineer" \
  --data-urlencode "location=Austin,Texas,United States" \
  --data-urlencode "gl=us" \
  --data-urlencode "api_key=$SEARCHAPI_KEY" | python -m json.tool | head -60

The response has 4 top-level keys worth knowing. search_metadata carries the request ID and timings, search_parameters includes location_used (the canonical location Google resolved your input to), jobs holds the listings themselves, and pagination carries the token for the next page.

A single job object contains the following fields, trimmed here to the ones that matter.

{
  "position": 4,
  "title": "Java Backend Engineer",
  "company_name": "Aurum Groups",
  "location": "Fort Worth, TX",
  "via": "via Indeed",
  "description": "<p>Position :- Java Backend Engineer</p>\n<p>Location :- Fort Worth, TX (Onsite) )Only Local)</p>\n<p>Mandatory Technical Skills:</p>\n<p>Core Java</p>\n<p>Java 11/…",
  "extensions": [
    "3 days ago",
    "125K a year",
    "Full-time",
    "No degree mentioned",
    "Health insurance"
  ],
  "detected_extensions": {
    "posted_at": "3 days ago",
    "salary": "125K a year",
    "schedule": "Full-time",
    "no_degree_mentioned": true,
    "health_insurance": true
  },
  "apply_link": "https://www.indeed.com/viewjob?jk=eed5ed5afc3d407f",
  "sharing_link": "https://www.google.com/search?ibp=htl;jobs&…&htidocid=OmZ9vxiPLgLgQpOSAAAAAA%3D%3D…"
}

That object has 3 properties which drive the rest of the build. The location reads "Fort Worth, TX" even though the request asked for Austin, because Google resolved the search to location_used: "Austin,Texas,United States" and returns metro-area results rather than city-limit results. The description arrives as HTML rather than plain text, so anything scanning it needs the tags stripped first.

Most importantly, detected_extensions carries no work_from_home key at all, which is the normal case rather than the exception, and it's why the remote classifier later leans on the description instead.

Field availability shifts between listings inside one response, and again between runs on consecutive days. A later request for the same query and location returned position 1 with no posted_at and no salary, while position 4 carried a posting date and still no salary, so neither of the 2 fields the object above shows together is guaranteed to appear.

The sparsest listing sits at the top of the page, which is worth absorbing before you write any parser. Ranking and completeness are unrelated, so code that reads jobs[0] to decide which fields exist will build the wrong schema.

The trend analysis rests on 5 fields, which are title, company_name, location, detected_extensions, and sharing_link. Everything else is display sugar or the full description text, which you'll want later for skill extraction but leave alone while counting.

A further 2 details will break your pipeline if you miss them, and both hide in fields that look harmless at first read. Catch them now and the rest of the build stays mechanical.

posted_at is relative text rather than a date, and 17% of listings omit it. When present you'll get "3 days ago", "22 hours ago", "Just posted", or the notorious "30+ days ago", and each string means something only relative to the moment you fetched it, so store the snapshot timestamp alongside every row. That omission rate, 18 rows of 104 in the run behind this article, is why the parser returns a missing precision flag rather than raising. Your snapshot date becomes the only date you can trust for those rows.

The response omits job_id entirely. SerpApi returns one and SearchApi leaves it out, so your only stable per-listing identifier is the htidocid value embedded in sharing_link, which is Google's internal document ID for that posting. Extract it and it becomes your deduplication key across snapshots, while skipping it means every run counts the same job as new and turns the whole dataset into noise.

Pagination works by token rather than offset. The response ends with pagination.next_page_token, and you pass that value back as the next_page_token parameter on your next request, so there's no page number to increment.

Fetching postings for one city

The city fetch takes 2 functions, one supplying the key and one walking the pages, surviving the failures, and staying inside the rate limit.

import os
import time
import requests

SEARCHAPI_URL = "https://www.searchapi.io/api/v1/search"

def api_key() -> str:
    """Read the key at request time so imports stay side-effect free."""
    key = os.environ.get("SEARCHAPI_KEY")
    if not key:
        raise SystemExit("Set SEARCHAPI_KEY before fetching.")
    return key

def fetch_city(query: str, location: str, max_pages: int = 2, pause: float = 1.5,
               max_retries: int = 2) -> tuple[list[dict], str | None]:
    """Fetch job listings for one query/location pair. Returns (jobs, location_used)."""
    jobs: list[dict] = []
    token: str | None = None
    location_used: str | None = None
    page, retries = 0, 0

    while page < max_pages:
        params = {
            "engine": "google_jobs",
            "q": query,
            "location": location,
            "gl": "us",
            "hl": "en",
            "api_key": api_key(),
        }
        if token:
            params["next_page_token"] = token

        try:
            resp = requests.get(SEARCHAPI_URL, params=params, timeout=95)
        except requests.RequestException as exc:
            print(f"  [{location}] page {page + 1} network error: {exc}")
            break

        if resp.status_code == 429:
            if retries >= max_retries:
                print(f"  [{location}] still limited after {max_retries} retries, stopping")
                break
            retries += 1
            print(f"  [{location}] rate limited, retry {retries} in 60s")
            time.sleep(60)
            continue                      # same page, token unchanged

        if resp.status_code != 200:
            print(f"  [{location}] page {page + 1} HTTP {resp.status_code}: {resp.text[:200]}")
            break

        retries = 0
        data = resp.json()
        location_used = location_used or data.get("search_parameters", {}).get("location_used")
        page_jobs = data.get("jobs", [])
        jobs.extend(page_jobs)
        print(f"  [{location}] page {page + 1}: {len(page_jobs)} jobs")

        page += 1
        token = data.get("pagination", {}).get("next_page_token")
        if not token or not page_jobs:
            break
        time.sleep(pause)

    return jobs, location_used

Both details in that loop matter more than they look. Reading the key inside api_key() rather than at module scope keeps the import side-effect free, so your analysis script and your tests run without a key in the environment. The retry path uses while with an explicit page += 1 because a for page in range(max_pages) loop combined with continue silently spends a page slot on the rate-limited attempt, which costs you the second page of results without fetching it.

The 95-second timeout matches SearchApi's own ceiling, which returns a 503 when it can't retrieve results within 90 seconds, so a shorter client timeout means you pay for searches you never receive. Only successful requests count against your quota, which makes a 429 cost nothing but time. The hourly cap sits at 20% of your monthly allocation, and a daily job this size stays comfortably under it, but the backoff belongs in the code anyway for the day you scale to 50 cities.

Log location_used on every run, because passing "Austin" and passing "Austin,Texas,United States" can resolve differently, and a silent resolution change mid-series will look like a demand shift in your charts. The SearchApi Locations API returns canonical strings when you want to pin them precisely.

Normalizing the messy parts

This section separates a working tracker from a script that produces confident nonsense. Three problems need solving, listed in ascending order of how much they'll annoy you.

Relative dates into absolute dates

The parser below converts Google's relative strings into ISO dates and returns a precision flag beside each one, since some strings are far vaguer than others.

import re
from datetime import date, datetime, timedelta

RELATIVE_RE = re.compile(r"(\d+)\s*(\+?)\s*(minute|hour|day|week|month)s?\s+ago", re.I)
FLOOR_HINT = re.compile(r"\b(over|more than|at least)\b", re.I)
IMMEDIATE = {"just posted", "just now", "today", "posted today"}

UNIT_DELTA = {
    "minute": lambda n: timedelta(minutes=n),
    "hour": lambda n: timedelta(hours=n),
    "day": lambda n: timedelta(days=n),
    "week": lambda n: timedelta(weeks=n),
    "month": lambda n: timedelta(days=30 * n),
}

def parse_posted_at(raw: str | None, snapshot: date) -> tuple[str | None, str]:
    """Convert '3 days ago' into an ISO date plus a precision flag."""
    if not raw:
        return None, "missing"

    text = raw.strip().lower()
    if text in IMMEDIATE:
        return snapshot.isoformat(), "day"

    match = RELATIVE_RE.search(text)
    if not match:
        return None, "unparsed"

    n, plus, unit = int(match.group(1)), match.group(2), match.group(3).lower()
    posted = snapshot - UNIT_DELTAunit

    if plus == "+" or FLOOR_HINT.search(text):
        precision = "floor"          # true date is this old or older
    elif unit in ("minute", "hour", "day"):
        precision = "day"
    else:
        precision = "approx"         # weeks and months round hard

    return posted.isoformat(), precision

Keep the precision flag, because "30+ days ago" is a floor rather than a date, and Google applies it to anything older than 30 days, with no upper bound you can see. Filtering on precision != "floor" while you compute freshness metrics stops a pile of stale listings from masquerading as a specific Tuesday.

The missing flag earns its place too. In the live run behind this article, 18 of 104 rows had no posted_at, so 17% of listings carry no age at all and any freshness metric describes the other 83%. Count your missing rows before you build anything on posting age.

Detecting genuinely remote roles

The location field lies constantly. A fully in-office role at a company headquartered in San Francisco reads "San Francisco, CA", and so does a remote-first role that lists SF for tax reasons, while detected_extensions.work_from_home goes missing from most listings, and was absent from every listing in the run behind this article.

The classifier below layers 3 signals, starting with the cheapest, and returns one of 4 labels.

TAG_RE = re.compile(r"<[^>]+>")          # descriptions arrive as HTML
REMOTE_LOC = re.compile(r"\b(remote|anywhere|work from home|telecommute)\b", re.I)
HYBRID_HINT = re.compile(
    r"\b(hybrid|"
    r"\d+\s*days?\s*(?:per|a)\s*week\s*(?:in|at|from)|"
    r"return[- ]to[- ]office)\b", re.I
)
ONSITE_HINT = re.compile(r"\b(on-?site|in[- ]office|in[- ]person)\b", re.I)

def classify_work_mode(job: dict) -> str:
    """Return one of: remote, hybrid, onsite, unknown."""
    flagged = job.get("detected_extensions", {}).get("work_from_home")
    location = job.get("location") or ""
    head = TAG_RE.sub(" ", job.get("description") or "")[:2000]

    location_says_remote = bool(REMOTE_LOC.search(location))
    hybrid = bool(HYBRID_HINT.search(head))
    onsite = bool(ONSITE_HINT.search(head))

    if flagged is True or location_says_remote:
        return "hybrid" if (hybrid or onsite) else "remote"
    if hybrid:
        return "hybrid"
    if onsite or flagged is False:
        return "onsite"
    return "unknown"

Keep ONSITE_HINT separate from HYBRID_HINT. Folding "onsite" into the hybrid pattern, which is the obvious first version, means a description reading "you will work onsite with the platform team" classifies as hybrid, and that quietly inflates the exact number this pipeline exists to measure. Remote-flagged listings are the one place both patterns behave alike, since a remote role mentioning either term is describing an exception, so either one downgrades it to hybrid.

Strip the HTML before you measure that window. Descriptions arrive as markup, and tags burn roughly 7% of a 2,000-character budget, which is enough to push a work-mode sentence sitting near the boundary out of range and return unknown for a listing that stated its arrangement plainly.

Reading only the first 2,000 characters is otherwise deliberate, because boilerplate about flexible working arrangements lives in the benefits section near the bottom of almost every listing and will flip half your dataset to remote once you scan the whole thing. That window is the single knob most worth tuning if your unknown bucket comes back large.

Salary out of free text

The parser runs 2 passes with different strictness. The structured detected_extensions.salary field is known to hold a salary, so a currency symbol stays optional there, while free description text keeps the $ anchor to prevent version numbers and dates matching as money. Both passes strip HTML, annualize whatever period they find, and apply a sanity range before returning anything.

AMOUNT = r"(\d{1,3}(?:,\d{3})*(?:\.\d+)?)\s*([KkMm])?"
SEPARATOR = r"\s*(?:-|to|\u2013|\u2014)\s*"
PERIOD = r"(?:\s*(?:per|a|an|/)\s*(hour|hr|year|yr|month|mo|week|wk))?"

# The structured field is known to be a salary, so a currency symbol is optional there.
# Free description text needs the "$" anchor to avoid matching version numbers and dates.
FIELD_RANGE = re.compile(r"\$?\s?" + AMOUNT + SEPARATOR + r"\$?\s?" + AMOUNT + PERIOD, re.I)
FIELD_SINGLE = re.compile(r"\$?\s?" + AMOUNT + PERIOD, re.I)
DESC_RANGE = re.compile(r"\$\s?" + AMOUNT + SEPARATOR + r"\$?\s?" + AMOUNT + PERIOD, re.I)
DESC_SINGLE = re.compile(r"\$\s?" + AMOUNT + PERIOD, re.I)

# Dropping the "$" requirement on the field pass lets non-USD amounts through,
# so reject them outright rather than storing foreign currency as dollars.
NON_USD = re.compile(
    r"[\u00a3\u20ac\u00a5\u20b9]|\b(?:EUR|GBP|CAD|AUD|INR)\b|\b(?:C|A|CA|AU|NZ)\$", re.I)

ANNUALIZE = {"hour": 2080, "hr": 2080, "week": 52, "wk": 52,
             "month": 12, "mo": 12, "year": 1, "yr": 1}

def _to_number(amount: str, suffix: str | None) -> float:
    value = float(amount.replace(",", ""))
    if suffix and suffix.lower() == "k":
        value *= 1_000
    elif suffix and suffix.lower() == "m":
        value *= 1_000_000
    return value

def _extract(text: str, rng: re.Pattern, single: re.Pattern) -> tuple[float, float] | None:
    match = rng.search(text)
    if match:
        low = _to_number(match.group(1), match.group(2))
        high = _to_number(match.group(3), match.group(4))
        period = (match.group(5) or "year").lower()
    else:
        match = single.search(text)
        if not match:
            return None
        low = high = _to_number(match.group(1), match.group(2))
        period = (match.group(3) or "year").lower()

    factor = ANNUALIZE.get(period, 1)
    low, high = low * factor, high * factor
    if low > high:
        low, high = high, low
    if not (10_000 <= low <= 2_000_000):
        return None
    return low, high

def parse_salary(job: dict) -> dict:
    """Annualized salary range from the structured field, falling back to the description."""
    field = job.get("detected_extensions", {}).get("salary")
    if field and not NON_USD.search(field):
        found = _extract(field, FIELD_RANGE, FIELD_SINGLE)
        if found:
            return {"salary_min": found[0], "salary_max": found[1],
                    "salary_source": "detected_extensions"}

    description = TAG_RE.sub(" ", job.get("description") or "")[:4000]
    if description:
        found = _extract(description, DESC_RANGE, DESC_SINGLE)
        if found:
            return {"salary_min": found[0], "salary_max": found[1],
                    "salary_source": "description"}

    return {"salary_min": None, "salary_max": None, "salary_source": None}

What this parser gets wrong comes from running it against real strings rather than from assumption, including strings this project's own snapshot returned.

The equity row is the instructive one. That 10_000 <= low check stops a garbage number reaching your database, but it does so by throwing away a listing which genuinely disclosed, so the parser fails safe rather than failing accurate and you should know which of those you're getting.

The pound row shows why NON_USD exists. Dropping the currency requirement on the field pass lets foreign amounts through, and the failure is worse than a plain mismatch, because the range pattern breaks on the second symbol while the single-amount pattern still matches the first number. A £50,000 to £65,000 range lands in your database as a firm 50000, in dollars, with no signal that anything went wrong. Rejecting non-USD strings outright is the safer trade, and widening this to real multi-currency support means storing the symbol and a rate, not loosening the regex.

Coverage is the bigger caveat, and the first measurement of it was wrong in an instructive way. Across 104 listings from 5 cities, an earlier version of this parser reported 9% disclosing any salary, with 0% of those credited to detected_extensions.salary. Reading that as "Google rarely populates the structured field" is the obvious conclusion and the wrong one, because the field was populated on listings the parser then threw away. Google writes salaries there as 125K a year with no currency symbol, and the original $-anchored regex rejected every such string in silence.

That mistake carries 2 lessons. A parser that fails closed produces clean-looking data and a confident, false finding about your source, so treat any 0% attribution as a bug report on your own code before you treat it as a fact about the API. Verify field-level coverage directly against the raw response rather than inferring it from parser output, because those 2 numbers answer different questions.

Measure both rates on your own database, since coverage shifts with query, city, and time. Compare the second number against a direct count of how many raw responses carried the field at all, because a gap between them points at your regex rather than at Google.

has_salary = df.salary_min.notna().mean() * 100
from_field = df[df.salary_min.notna()].salary_source.eq("detected_extensions").mean() * 100
print(f"{has_salary:.0f}% disclose; {from_field:.0f}% of those via detected_extensions")

Whatever number you get, salary analysis on Google Jobs describes the subset of employers who disclose, which skews toward jurisdictions with pay transparency laws. Treat whatever rate you measure as a signal about disclosure norms rather than about market compensation.

Storing snapshots over time

The instinct is to keep one row per job and update it, but that instinct destroys the product. Comparing yesterday's set of job keys against today's tells you what appeared, what disappeared, and how long a listing survives, which is the one thing job boards withhold.

The schema stores one row per job per snapshot date, with job_key derived from Google's own document ID and a content hash standing in whenever the sharing link goes missing. A small migrate step alongside it keeps older databases usable as the column list grows.

import hashlib
import sqlite3
from pathlib import Path
from urllib.parse import unquote

DB_PATH = Path(__file__).with_name("jobs.db")   # independent of cwd
HTIDOCID_RE = re.compile(r"htidocid=([^&#]+)")

SCHEMA = """
CREATE TABLE IF NOT EXISTS snapshots (
    snapshot_date   TEXT NOT NULL,
    job_key         TEXT NOT NULL,
    query           TEXT NOT NULL,
    city            TEXT NOT NULL,
    title           TEXT,
    company         TEXT,
    location        TEXT,
    via             TEXT,
    posted_at_raw   TEXT,
    posted_date     TEXT,
    posted_precision TEXT,
    work_mode       TEXT,
    schedule        TEXT,
    salary_raw      TEXT,
    salary_min      REAL,
    salary_max      REAL,
    salary_source   TEXT,
    position        INTEGER,
    PRIMARY KEY (snapshot_date, city, query, job_key)
);
CREATE INDEX IF NOT EXISTS idx_key_date ON snapshots(job_key, snapshot_date);
CREATE INDEX IF NOT EXISTS idx_city_date ON snapshots(city, snapshot_date);
"""

def job_key(job: dict) -> str:
    """Google's htidocid, or a content hash when the sharing link is missing."""
    match = HTIDOCID_RE.search(job.get("sharing_link") or "")
    if match:
        return unquote(match.group(1))
    seed = "|".join([
        (job.get("title") or "").strip().lower(),
        (job.get("company_name") or "").strip().lower(),
        (job.get("location") or "").strip().lower(),
    ])
    return "h:" + hashlib.sha1(seed.encode()).hexdigest()[:16]

def migrate(conn) -> None:
    """CREATE TABLE IF NOT EXISTS skips existing tables, so add columns explicitly."""
    existing = {row[1] for row in conn.execute("PRAGMA table_info(snapshots)")}
    for column, decl in [("salary_raw", "TEXT")]:
        if column not in existing:
            conn.execute(f"ALTER TABLE snapshots ADD COLUMN {column} {decl}")
    conn.commit()

def save_snapshot(conn, rows: list[dict]) -> int:
    if not rows:
        return 0
    columns = list(rows[0].keys())
    placeholders = ", ".join("?" * len(columns))
    sql = (f"INSERT OR REPLACE INTO snapshots ({', '.join(columns)}) "
           f"VALUES ({placeholders})")
    conn.executemany(sql, [tuple(r[c] for c in columns) for r in rows])
    conn.commit()
    return len(rows)

INSERT OR REPLACE against that composite primary key makes the whole run idempotent, so re-running after a partial failure repairs the day instead of duplicating it. Partial failures happen often enough on scheduled runners that this property earns its keep quickly.

migrate exists because CREATE TABLE IF NOT EXISTS skips a table that already exists, columns included. Adding salary_raw to SCHEMA therefore does nothing to a database created before that column, and the next insert dies with "table snapshots has no column named salary_raw" partway through a run. Checking PRAGMA table_info and issuing ALTER TABLE keeps existing history intact, which matters once your series is weeks long and deleting the file costs you the whole dataset.

Schedule the script for early morning so each snapshot samples a comparable point in the posting cycle. This cron line runs it at 06:15 daily and appends everything to a log file.

15 6 * * * cd /home/you/jobtracker && /usr/bin/env SEARCHAPI_KEY=xxx ./venv/bin/python tracker.py >> run.log 2>&1

GitHub Actions works just as well and gives you free hosting plus a version history of every snapshot, because the workflow commits the database back to the repo after each run.

name: job-snapshot
on:
  schedule:
    - cron: "15 6 * * *"
  workflow_dispatch:

jobs:
  snapshot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: python tracker.py
        env:
          SEARCHAPI_KEY: ${{ secrets.SEARCHAPI_KEY }}
      - run: |
          git config user.name "job-bot"
          git config user.email "bot@users.noreply.github.com"
          git add jobs.db
          git commit -m "snapshot $(date -u +%F)" || exit 0
          git push

GitHub's scheduled runners drift by up to an hour under load and skip entirely during outages, so expect occasional gaps. Plot against actual snapshot dates rather than assuming a continuous series, because a missed day looks identical to a quiet market once the dates are implied rather than recorded.

Running across multiple cities

The runner loops the fetcher over your city list, normalizes each result through the 3 parsers, and writes one batch per city.

from datetime import UTC, datetime   # UTC requires Python 3.11+

CITIES = [
    "Austin,Texas,United States",
    "Denver,Colorado,United States",
    "Seattle,Washington,United States",
    "Atlanta,Georgia,United States",
    "New York,New York,United States",
]
QUERY = "backend engineer"

def run_snapshot(cities: list[str] = CITIES, max_pages: int = 2) -> None:
    snapshot = datetime.now(tz=UTC).date()
    conn = sqlite3.connect(DB_PATH)
    conn.executescript(SCHEMA)
    migrate(conn)

    for city in cities:
        jobs, location_used = fetch_city(QUERY, city, max_pages=max_pages)
        posted = [parse_posted_at(
            j.get("detected_extensions", {}).get("posted_at"), snapshot) for j in jobs]

        rows = [{
            "snapshot_date": snapshot.isoformat(),
            "job_key": job_key(j),
            "query": QUERY,
            "city": city.split(",")[0],
            "title": j.get("title"),
            "company": j.get("company_name"),
            "location": j.get("location"),
            "via": (j.get("via") or "").replace("via ", ""),
            "posted_at_raw": j.get("detected_extensions", {}).get("posted_at"),
            "posted_date": p[0],
            "posted_precision": p[1],
            "work_mode": classify_work_mode(j),
            "schedule": j.get("detected_extensions", {}).get("schedule"),
            "salary_raw": j.get("detected_extensions", {}).get("salary"),
            "position": j.get("position"),
            **parse_salary(j),
        } for j, p in zip(jobs, posted)]

        if rows:
            print(f"{city.split(',')[0]}: saved {save_snapshot(conn, rows)} rows "
                  f"(resolved as {location_used})")
        time.sleep(2)

    conn.close()from datetime import UTC, datetime   # UTC requires Python 3.11+

CITIES = [
    "Austin,Texas,United States",
    "Denver,Colorado,United States",
    "Seattle,Washington,United States",
    "Atlanta,Georgia,United States",
    "New York,New York,United States",
]
QUERY = "backend engineer"

def run_snapshot(cities: list[str] = CITIES, max_pages: int = 2) -> None:
    snapshot = datetime.now(tz=UTC).date()
    conn = sqlite3.connect(DB_PATH)
    conn.executescript(SCHEMA)
    migrate(conn)

    for city in cities:
        jobs, location_used = fetch_city(QUERY, city, max_pages=max_pages)
        posted = [parse_posted_at(
            j.get("detected_extensions", {}).get("posted_at"), snapshot) for j in jobs]

        rows = [{
            "snapshot_date": snapshot.isoformat(),
            "job_key": job_key(j),
            "query": QUERY,
            "city": city.split(",")[0],
            "title": j.get("title"),
            "company": j.get("company_name"),
            "location": j.get("location"),
            "via": (j.get("via") or "").replace("via ", ""),
            "posted_at_raw": j.get("detected_extensions", {}).get("posted_at"),
            "posted_date": p[0],
            "posted_precision": p[1],
            "work_mode": classify_work_mode(j),
            "schedule": j.get("detected_extensions", {}).get("schedule"),
            "salary_raw": j.get("detected_extensions", {}).get("salary"),
            "position": j.get("position"),
            **parse_salary(j),
        } for j, p in zip(jobs, posted)]

        if rows:
            print(f"{city.split(',')[0]}: saved {save_snapshot(conn, rows)} rows "
                  f"(resolved as {location_used})")
        time.sleep(2)

    conn.close()

Your monthly quota works out to cities × queries × pages × days searches. The 5-city, single-query, 2-page daily run above comes to 300 searches, or 3% of the Developer plan, while 3 job titles takes it to 900 and 9%. Scaling to 10 cities across 5 titles at 3 pages hits 4,500 searches, which is 45% and still inside the same $40 plan, so the number that matters is whether you stay under your monthly allocation rather than what any individual run "costs."

The variable that hurts is pages, because it multiplies against everything else. Before committing to 2 pages daily, this snippet checks whether page 2 returns listings that page 1 missed.

jobs, _ = fetch_city(QUERY, CITIES[0], max_pages=2)
keys = [job_key(j) for j in jobs]
print(f"{len(keys)} fetched, {len(set(keys))} distinct")

Close numbers mean page 2 earns its quota, while a second page that mostly repeats the first means you should drop max_pages to 1 and halve your consumption. On the backend engineer query across all 5 cities, every city returned listings on page 2 that page 1 missed, so 2 pages stayed in the schedule. Re-run this check whenever you change the query, because a narrower title can fit entirely on page 1.

Putting it together

The snippets above assemble into 2 files, plus the workflow and a database that appears on first run.

jobtracker/
├── tracker.py           # fetch, normalize, store (everything through run_snapshot)
├── analyze.py           # remote-share chart, plus trend queries later
├── tests/test_tracker.py  # parser unit tests, no API key needed
├── requirements.txt
├── jobs.db              # created on first run
└── .github/workflows/snapshot.yml

All imports for tracker.py go at the top, in this order, and cover every function in the piece.

import hashlib
import os
import re
import sqlite3
import time
from datetime import UTC, date, datetime, timedelta
from pathlib import Path
from urllib.parse import unquote

import requests

Then come the constants (SEARCHAPI_URL, DB_PATH, SCHEMA, CITIES, QUERY), the regex patterns, and the functions in the order they appear above, ending with an entry point that accepts 2 flags.

if __name__ == "__main__":
    import argparse

    ap = argparse.ArgumentParser(description="Snapshot Google Jobs postings.")
    ap.add_argument("--one-city", action="store_true",
                    help="fetch only the first city in CITIES")
    ap.add_argument("--max-pages", type=int, default=2,
                    help="pages per city (1 halves your quota use)")
    args = ap.parse_args()

    run_snapshot(cities=CITIES[:1] if args.one_city else CITIES,
                 max_pages=args.max_pages)

Running python tracker.py --one-city --max-pages 1 costs a single search, which is the cheapest way to confirm your key works and rows land in the database before you turn the scheduler on.

analyze.py needs sqlite3, pandas, matplotlib.pyplot, and its own copies of DB_PATH and QUERY. Because api_key() reads the environment lazily, test_tracker.py can import every parser and exercise it against fixture strings without touching the network or holding a key, which keeps the test suite free to run.

Analyzing the data

Only the remote-share chart works from a single snapshot, so you can sanity-check the classifier on day 1. Counting postings per city over time needs both a lifted pagination ceiling and a couple of weeks of history, so that chart belongs in "What arrives with history" rather than in your first publishable run.

This script loads every snapshot into pandas and draws the remote-share chart, labeling each bar with the underlying counts so every percentage arrives with the sample size attached. It selects the Agg backend so the same file runs headless under cron and GitHub Actions, creates images/ rather than assuming it exists, and exits cleanly against an empty database.

import sqlite3
from pathlib import Path

import matplotlib
matplotlib.use("Agg")          # headless, so this runs under cron and Actions
import matplotlib.pyplot as plt
import pandas as pd

DB_PATH = Path(__file__).with_name("jobs.db")   # same constants as tracker.py
QUERY = "backend engineer"

IMAGES = Path(__file__).with_name("images")     # same cwd independence as DB_PATH
IMAGES.mkdir(exist_ok=True)

conn = sqlite3.connect(DB_PATH)
df = pd.read_sql_query("SELECT * FROM snapshots", conn, parse_dates=["snapshot_date"])
if df.empty:
    raise SystemExit("No rows yet. Run tracker.py first.")

latest = df[df.snapshot_date == df.snapshot_date.max()]
totals = latest.groupby("city")["job_key"].nunique()
remote = (latest[latest.work_mode == "remote"]
          .groupby("city")["job_key"].nunique()
          .reindex(totals.index, fill_value=0))
share = (remote / totals * 100).sort_values()

fig, ax = plt.subplots(figsize=(8, 4.5))
share.plot.barh(ax=ax, color="#3b6ea5")
ax.bar_label(
    ax.containers[0],
    labels=[f"{share[c]:.0f}%  (n={remote[c]}/{totals[c]})" for c in share.index],
    padding=4, fontsize=9,
)
ax.set_xlim(0, max(share.max() * 1.5, 10))
ax.set_title(f'Genuinely remote "{QUERY}" postings, {latest.snapshot_date.max():%d %b %Y}')
ax.set_xlabel("% of listings in that city")
ax.set_ylabel("")
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
fig.savefig(IMAGES / "remote_share.png", dpi=150)

Putting n=7/20 on the bar alongside 35% costs one line and keeps the chart honest about its own weight, because a reader who sees only the percentage will assume a sample large enough to compare cities. Those counts belong in the chart itself rather than a caption, since charts travel and captions get stripped.

What the first snapshot showed

A single run across 5 cities returned 104 rows, which is the baseline every later comparison measures against. A single snapshot supports claims about composition, so the numbers below are all cross-sectional. Trend claims need days, and those arrive on their own.

Salary disclosure came in at 9% under a parser that rejected every unsigned amount, which makes that figure a floor rather than a measurement. Recomputing it from this snapshot is impossible, because the schema at the time stored parser output and discarded the raw string, so the corrected rate arrives with the first snapshot written under the salary_raw column.

Posting dates were absent from 17% of listings, or 18 rows of 104, leaving the snapshot date as the only timestamp for those. Both results shape what this dataset can answer, since a freshness metric built on posted_at would describe a filtered subset rather than the market.

Live posting counts came to 25 in Austin, 20 each in Denver, New York, and Seattle, and 19 in Atlanta, totaling 104. That clustering is a finding about the collector rather than the market, because each page returns about 10 results, so 2 pages cap a city near 20. Every Austin row carries a position between 1 and 10, so its 25 distinct keys came from a smoke test and a full 2-page pull landing on the same date rather than from deeper coverage, which makes it an artifact to exclude rather than a signal.

Atlanta's 19 came from a 10-plus-9 pull and is the only count that genuinely fell below the cap. The other cities held listings this run never fetched, so these counts describe sample size and stay unusable as a demand signal until max_pages rises high enough that cities return fewer results than the ceiling allows.

Work mode splits by city, and unknown is the largest bucket everywhere, running from 53% in Atlanta to 68% in Austin. Genuine remote covered 35% of New York listings, 30% of Denver, 25% of Seattle, 21% of Atlanta, and 20% of Austin, while onsite classification fired only in Atlanta at 11% and Austin at 8% and stayed at zero across the other 3 cities.

Both figures carry a health warning worth printing next to them. Those remote percentages rest on 4 to 7 listings per city, where a single posting moves the number by 4 to 5 percentage points, so the apparent gap between New York and Austin is 7 listings against 5 and carries no weight.

The dominant unknown share also means most listings describe their work arrangement nowhere in the first 2,000 characters, which splits the metric in 2, since remote runs at 20% to 35% of all listings but 44% to 88% of the listings the classifier could actually place. Near-total absence of onsite classifications points the same way, because employers who list a city and expect you to infer attendance rarely write the word "onsite" at all, so ONSITE_HINT needs widening before that bucket means anything.

Trend claims need 2 conditions this run lacks, which are a lifted pagination ceiling and several weeks of history. Both arrive on their own once the scheduler runs, and the queries waiting for them are below.

What arrives with history

The delta query is where the snapshot design pays off, because new postings on any given day are simply the keys present today and absent yesterday. It needs 2 snapshots minimum and a couple of weeks to be worth reading, so run it once collection has been going a while.

by_day = df.groupby("snapshot_date")["job_key"].apply(set)
churn = pd.DataFrame({
    "new": [len(by_day.iloc[i] - by_day.iloc[i - 1]) for i in range(1, len(by_day))],
    "gone": [len(by_day.iloc[i - 1] - by_day.iloc[i]) for i in range(1, len(by_day))],
}, index=by_day.index[1:])
print(churn.describe())

The postings-per-city line chart belongs here too, once your counts stop hitting the pagination ceiling described in "Limitations".

daily = (df.groupby(["snapshot_date", "city"])["job_key"]
           .nunique().unstack(fill_value=0).sort_index())

fig, ax = plt.subplots(figsize=(11, 5))
daily.plot(ax=ax, marker="o", linewidth=1.8, markersize=4)
ax.set_title(f'Live "{QUERY}" postings by city')
ax.set_ylabel("distinct postings")
ax.set_xlabel("")
ax.grid(alpha=0.25)
ax.legend(title=None, frameon=False, ncol=5)
fig.tight_layout()
fig.savefig(IMAGES / "postings_by_city.png", dpi=150)

Median listing lifespan falls out of the same data with a group-by on job_key, giving you first-seen and last-seen dates per posting. Nothing in the collector changes to unlock any of this, because the snapshots are already accumulating.

Wrapping up

You now have a collector that snapshots Google Jobs postings across 5 cities every day, normalizes relative dates, work mode, and salary into columns you can query, and keeps every snapshot instead of overwriting yesterday's. The first run returned 104 rows and answered composition questions straight away, putting genuine remote between 20% and 35% by city once the descriptions were checked against the tags.

The parts that took several attempts are the ones worth carrying into your own build. A parser that fails closed produces clean-looking data and a false conclusion, which is how a populated salary field read as 0% coverage. City counts that pin to your max_pages ceiling measure your fetch settings rather than the market. Both mistakes look like findings until you check them against the raw response, so budget time for that check rather than trusting the first chart your pipeline draws.

Everything else arrives with patience. Leave the scheduler running for a few weeks and the churn query, the line chart, and median listing lifespan all become readable from data you're already collecting, with no change to the collector itself.

The complete tracker, schema, analysis script, and a suite of 39 unit tests covering the parsers live on GitHub. Grab a free SearchApi key and you can have your first snapshot inside 15 minutes.