Linkedin job scraping using Python | Web Scraper

An ed-tech startup sought to accelerate growth by expanding its offerings from course instruction to job search support, aiming to broaden its customer base. They envisioned a tool that would consolidate LinkedIn data and highlight top key skills for job titles relevant to their customers.

Extract recent job listings from LinkedIn based on specified job titles.

Filter jobs based on criteria like location, date posted, etc.

Organize these listings into a clear, accessible format.

Identify key skills from job descriptions through data mining.

Web Scraping Script: Developed a Python-based web crawler to retrieve LinkedIn job descriptions for specific roles, with filters applied (region, posted within the last 24 hours, etc.).

Data Cleaning and Organization: Processed and cleaned the scraped data, removing duplicates, irrelevant entries, formatting inconsistencies and organized the data into an Excel file.

Skills Extraction: Developed a Python script to mine the top required skills from job descriptions, helping users identify key competencies for each role.

Automation: Automated the crawler to run daily and populate new data into separate tabs of the Excel file.

Error Handling: Built-in error handling for scenarios like changes to LinkedIn’s page structure, CAPTCHA issues, or blocked requests, ensuring the scraper remained functional over time.

User-friendly application: Converted the solution into a client-ready application, providing knowledge transfer.

Documentation: Documented the code, approach, and methodology in a Google Doc.

Post-launch Support: Offered support to extend the crawler's capabilities, allowing it to fetch job descriptions for any new titles the startup targeted for its customers (students).

Boosted customer base by fostering trust and offering valuable career support.

Broadened the startup's reach, expanding from individual digital users to educational institutions seeking job search tools for their enrolled students.