Job source dashboard for RapidAPI scraping pipeline.

This page renders from a generated DB snapshot (`dashboard-data.js`) sourced from strict URL-policy views and `vw_latest_source_runs`.

Catalog APIs --
Sources With Runs --
Normalized Listings --
Latest Run State --
Salary Coverage --
Dead URLs --

API List

Intended pipeline APIs currently used by the site. Job-board-focused APIs are separated below as blacklisted.

Snapshot source: dashboard-data.js (generated from Postgres)
API Family Endpoint Base Template Params Pagination Request Limit Requests Left Jobs Limit Jobs Left
Run ./db/publish_dashboard_data.sh to populate live rows.
Visible: --
Page -- / --

Blacklisted APIs (Grouped by Status)

Only blacklisted providers are shown below, grouped by status with policy reason for each API.

Loading blacklisted API groups...

API Family Endpoint Base Pagination Request Limit Jobs Limit Jobs Left Policy Reason
Run ./db/publish_dashboard_data.sh to populate blacklisted API rows.

Listings by API

Period-based ingestion performance view for active APIs (weekly, monthly, 90-day, or custom date range).

Snapshot source: `source_performance_daily` + `source_quality.jobs_total` from dashboard snapshot.
Range: -- Days: --
Run ./db/publish_dashboard_data.sh to populate performance bars.
Source API Metric Value Metric Share Runs Fetched Mapped Mapping Success % Current Active Listings Share of Total
Run ./db/publish_dashboard_data.sh to populate listing statistics.
Visible: --
Page -- / --

Job Listings

Live listing rows from strict policy views that only allow direct company-career URLs or ATS-hosted portals.

Snapshot source: dashboard-data.js (generated from Postgres)
Visible: --
Page -- / --
Source API Missing Description Missing Job Title Missing Apply Link Missing Location
Run ./db/publish_dashboard_data.sh to populate quality rows.
Visible: --
Page -- / --
Job Title Company Location Work Mode Type Level Category Salary Apply Method Apply Link Link Type Link Host Policy Source API Posted Description Raw JSON URL Health Sync
Run ./db/publish_dashboard_data.sh to populate live rows.

Normalized Job Schema (Core Fields)

job_id job_title employer_name job_location work_mode employment_type experience_level job_category job_posted_at_datetime_utc job_apply_link job_min_salary job_max_salary salary_currency job_salary_period salary_display job_description job_publisher apply_options[] job_highlights source_api scraped_at

Scraper Blueprint (Static)

1) Source Selection Pick API + time window (`24h`, `7d`, `6m`) and load URL template params.
2) Request Assembly Build query string, append pagination state (`page`, `offset`, or `cursor`).
3) Response Mapping Map provider fields into normalized schema and tag `source_api`.
4) Validation + Persistence De-duplicate by external id/apply link and persist listing snapshot.

JSearch Response Keys Used For Mapping

top_level: status, request_id, parameters, data[]

record keys:
job_id, job_title, employer_name, job_location, job_city, job_country,
job_employment_type, job_is_remote, job_posted_at_datetime_utc,
job_apply_link, job_salary, job_min_salary, job_max_salary,
job_salary_period, job_description, job_publisher, apply_options[]

Scraper Run Contract (Proposed)

{
  "session_id": "scrape_20260303_001",
  "source_api": "jsearch",
  "request_mode": "search",
  "query": "software engineer",
  "pagination": {"page": 1, "num_pages": 1},
  "records_fetched": 20,
  "records_mapped": 20,
  "records_failed": 0,
  "ended_state": "completed"
}

Blacklisted Job Posts (Debug)

Rows blocked by strict URL policy because they resolve to job-board/aggregator links instead of direct company or ATS portals.

Snapshot source: `blocked_jobs` from `rapidapi_scraper.vw_job_listing_table_policy_all` where `policy_is_allowed = false`.
Visible: --
Page -- / --
Job Title Company Source API Link Type Link Host Blocked URL Policy Posted Raw JSON
Run ./db/publish_dashboard_data.sh to populate blocked rows.

Scheduler Debug (Canary)

Debug view for canary automation (dry-run or live). Reads scheduler artifacts from the local `db/` directory to verify cadence, lock behavior, quota guards, and source execution decisions.

Expected files: `db/scheduler-state.json`, `db/scheduler.log`, `db/cron.log`.
Loading Last refresh: --
Mode --
Last Tick --
Next Tick --
Due Sources --
Skipped Sources --
Host --
Source Cadence Decision Reason Next Due Last Attempt Failure Streak Cooldown Until
Loading scheduler debug state...
Visible: --
Page -- / --

Scheduler Log Tail

Loading scheduler.log...

Cron/Loop Log Tail

Loading cron.log...

External Test Connectors

Test-only connection probes for non-RapidAPI sources. This does not write jobs into the production ingest pipeline.

Source: `external-connectors-debug.json` refreshed by `render_dashboard_data.sh`.
Generated At --
Accepted Fit --
Review Needed --
Not Configured --
Visible: --
Page -- / --
Connector Type Policy Fit Status HTTP Records Sample Apply Hosts Notes Tested At
Run ./db/render_dashboard_data.sh to populate external connector probe rows.

Randomizer Analytics

Measures how keyword rotation affects ingestion performance by comparing randomizer runs versus static runs per source.

Computed from `scrape_sessions.request_params.randomizer`, `records_*`, persisted jobs (`jobs.source_session_id`), and URL policy outcomes where available.
Lookback Days --
Randomizer Runs --
Static Runs --
Mapped Uplift --
Persisted Uplift --
Combo Coverage --
Run ./db/publish_dashboard_data.sh to populate randomizer uplift bars.
Visible: --
Page -- / --
Source API Mode Runs Mapped Total Avg Mapped / Run Persisted Total Avg Persisted / Run Policy Pass Combos Tested Coverage
Run ./db/publish_dashboard_data.sh to populate randomizer source summary.
Visible: --
Page -- / --
Source API Combo Key Runs Mapped Persisted Policy Pass Failures
Run ./db/publish_dashboard_data.sh to populate combo-level stats.
Visible: --
Page -- / --
Date (UTC) Source API Mode Runs Mapped Persisted
Run ./db/publish_dashboard_data.sh to populate daily trend rows.

Ingestion Runs

Latest per-source ingestion sessions from `rapidapi_scraper.vw_latest_source_runs` (backed by `scrape_sessions`). This section does not update on `publish_dashboard_data.sh` alone.

Updates when `ingest_*.sh` / `run_ingest_cycle.sh` runs; snapshot publish only refreshes `generated_at`.
Session UUID Started Source API Status Fetched Mapped Failed Retries
Run ./db/publish_dashboard_data.sh to populate live rows.
Visible: --
Page -- / --

Snapshot History

Recent dashboard publish events recorded in `rapidapi_scraper.vw_recent_dashboard_snapshots`.

Tracks `publish_dashboard_data.sh` executions, output path, and snapshot row counts.
Visible: --
Page -- / --
Published At Snapshot Generated At Jobs Runs Sources Quality Rows Output Path
Run ./db/publish_dashboard_data.sh to populate snapshot history rows.