Craigslist Data Scraper (MySQL)

I don’t normally talk about personal life, however I have been in the market for a new Ski Boat as my family and I are avid Skiers and Wake boarders. In this pursuit I have found that while Craigslist is an awesome tool for finding a boat, spending hours manually searching the listings is not my cup of tea. So with that said, I found myself looking for a way to automate the process and Python seemed to fit the bill. While I am certainly not an expert in Python, I was able to find many examples online that helped me quickly put together a script that would automate my entire search process. What I needed from the script was the ability to crawl Craigslist to search for listings that matched my criteria and then store the listings in a database and email the results. Below is what I came up with, hopefully this will be as useful for others as it has been for me. In order for this script to run on Debian or Ubuntu, you will need to ensure you have the appropriate python modules installed as seen in the command below.

Now that we have the required per-requisites installed, lets move on and get our script configured. First, you will need to edit theĀ ### Database Initialization ### section with your MySQL or MariaDB Server info. Next, you will need edit the ### Search Parameters ### section with the information you are searching for. Lastly, you will need to edit the ### Email Parameters ### section so you are not sending me your search results. Note, that after running the script the first time, you can safely remove the database creation area annotated by ### Database Creation, remove after first run ###.

Note: Following the success of the Craigslist Data Scraper (MySQL), I was asked to re-write the script for use with SQLite. If you would prefer the SQLite version please see Craigslist Data Scraper (SQLite).

Once you have edited the script to your liking, you can schedule the script to run at intervals using your crontab.

download-icon Craigslist Data Scraper (MySQL) Download

Did you find this article useful? Why not share it with your friends?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.