Built a full-stack job portal aggregating government, private, IT, and other sector jobs. Develop...Built a full-stack job portal aggregating government, private, IT, and other sector jobs. Develop...
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started
Back to feedPost
Vyankatesh's avatar
Vyankatesh Gadge
• Aug 30
Built a full-stack job portal aggregating government, private, IT, and other sector jobs. Developed the platform with Next.js 14 and Supabase PostgreSQL, including Google OAuth authentication. Created a Python web scraper using BeautifulSoup and Requests to collect job listings automatically, with daily execution through GitHub Actions. Deployed on Vercel with GitHub CI/CD and implemented responsive UI and SEO optimization.
Completed work
Web DevelopmentPostgreSQLPythonFull Stack DevelopmentNext.js
Post image
Eric's avatar
Eric Moore
pro
• Aug 30
I'd show when each listing was last checked and hide it if the source takes it down. That makes it easier to tell whether the job is still live.
Vyankatesh's avatar
Vyankatesh Gadge
• Aug 31
Each job listing will show a "last checked" timestamp, so users can see exactly when we last confirmed it's still live on the source site. my scraper runs daily, so this will usually show as checked within the last 24 hours.
If a listing gets taken down from the source, it'll...
Eric's avatar
Eric Moore
pro
• Aug 31
I'd tie the timestamp to the last successful scrape, so a failed run can't make an old listing look freshly checked.
Vyankatesh's avatar
Vyankatesh Gadge
• Aug 31
You're right the timestamp should only update on a confirmed successful check of that specific listing, not just because the scraper job ran. So if the scraper run fails partway through, times out, or hits an error before reaching a listing, that listing's "last checked" stays...
Jaden's avatar
Jaden Green
• Aug 30
The scraper is the part most people underestimate on a job board. Listings move, sites change markup, and half your value is freshness. Honestly just curious how you're handling rescrapes -- full sweep on a schedule, or diffing against what's already in Postgres? I've built...
Vyankatesh's avatar
Vyankatesh Gadge
• Aug 31
Mine did break in smaller ways — early on it occasionally grabbed the wrong table off a job's page. One time it pulled an answer-key table (numbered rows, A/B/C/D options) instead of the actual job-details table. Another time it stitched together two unrelated tables into one...
Jaden's avatar
Jaden Green
• Aug 31
Keying off what the table says instead of where it sits in the DOM is the move -- field labels survive a redesign, selectors don't. And the answer-key trap is the kind of tell you only learn by getting burned once. Numbered rows plus single-letter values is a great...
Vyankatesh's avatar
Vyankatesh Gadge
• Aug 31
that's exactly the right approach. Keying off field labels instead of DOM position makes total sense for durability, and yeah, I've been burned by structural assumptions before, so that's a lesson well learned.
For the update-detection piece I already diff titles on insert, so...
Dusko's avatar
Dusko Licanin
pro
• Sep 5
The "last checked" timestamp idea in the thread is a good fix for stale listings. On the scraping side, how is it holding up against structure changes on the government job boards specifically? Those sites tend to be the least stable and the hardest to get consistent access to,...
Vyankatesh's avatar
Vyankatesh Gadge
• Sep 6
Right now the scraper pulls from a specific site site and I've built it to key off field labels rather than DOM position or CSS selectors so it looks for things like "Post Name," "Last Date," "Qualification" by their label text, not by where they sit in the page structure....
Back to feed
The network for creativity
Join 1.25M professional creatives like you
Connect with clients, get discovered, and run your business 100% commission-free
Creatives on Contra have earned over $150M and we are just getting started