{"id":4446,"date":"2023-06-15T14:50:14","date_gmt":"2023-06-15T21:50:14","guid":{"rendered":"https:\/\/parkingreform.org\/?p=4446"},"modified":"2023-06-26T07:44:29","modified_gmt":"2023-06-26T14:44:29","slug":"building-a-u-s-parking-minimums-database","status":"publish","type":"post","link":"https:\/\/parkingreform.org\/2023\/06\/15\/building-a-u-s-parking-minimums-database\/","title":{"rendered":"Building a U.S. Parking Minimums Database"},"content":{"rendered":"\n

Using data to inform and reform policies isn\u2019t new. However, in order to analyze data, you need to be able to access the data \u2014 even better if it\u2019s already processed and formatted. There\u2019s no unified database for parking minimums in the United States, so we were tasked to build one. <\/p>\n\n\n\n

What are parking minimums?<\/h5>\n\n\n\n

Parking minimums are a set of parking requirements for new developments, ranging from residential to commercial buildings. Developers have to, at a minimum, build a certain number of parking spaces, which can be based on characteristics from square footage to number of employees.<\/p>\n\n\n\n

\"\"
Los Angeles County: Required parking spaces<\/figcaption><\/figure>\n\n\n\n

<\/p>\n\n\n\n

Parking minimums are also known as parking requirements, parking ratios, or parking schedules. These tables are scattered across municipalities\u2019 Code of Ordinances or Unified Development Ordinance (UDO).\u00a0<\/p>\n\n\n\n

Our ten-week project\u2019s main goal was to build and populate a database to hold all parking minimum requirements across the United States, using automation to speed up the data parsing.<\/p>\n\n\n\n

Developed workflow<\/h5>\n\n\n\n

We mainly web-scraped the requirements from Municode<\/a>, which is a digital library of Code of Ordinances across the United States. A parking section URL<\/a> is inputted into our program for web-scraping and data processing. The data entries are then inserted into the database.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
The database<\/h5>\n\n\n\n

The data lives in a PostgreSQL database through Supabase. Each entry states the state, region, and use case, which maps to the raw requirement.<\/p>\n\n\n\n

\"Figure
A note about the interaction with Supabase: Our pipeline interacts with the database through SQLAlchemy, instead of Supabase\u2019s Python Client. This decision accounts for future plans involving switching hosting providers.<\/figcaption><\/figure>\n\n\n\n
Web-scraping parking requirements<\/h5>\n\n\n\n

In order to efficiently extract data from the parking tables, we developed a web scraper refined for our purpose and for a website like Municode.<\/p>\n\n\n\n

We used Selenium Webdriver to obtain the HTML of a webpage. We can then parse the HTML according to the structure of data representation. In our case, most of the information is presented in tables and bullet points, so we have functions that extract data from both structures. <\/p>\n\n\n\n

We encountered one difficulty when using our current table parsing function. When scraping tabular data, the function will acquire all of the tables in a webpage, including the ones we don\u2019t need. So, we tried to write a function that would automatically select the tables needed. We attempted to resolve this problem by using natural language processing (NLP) machine learning approach. Each table will be assessed by their column names, and they\u2019ll be categorized as \u201cuseful\u201d or \u201cuseless.\u201d <\/p>\n\n\n\n

Ideally, the model will reach above 95% in accuracy to be implemented. However, the accuracy of our trained model is only 80%. We still have to double-check the results and manually select some tables from the prediction. Having this model didn\u2019t increase our efficiency when adding data into the database, so we ended up abandoning it. <\/p>\n\n\n\n

Web crawling<\/h5>\n\n\n\n

For the majority of our project, the parking code URLs are found manually, which is tedious. There are almost 3,500 municipalities in the United States. Using a generous estimate of five minutes to find a parking code on Municode, it would take about 290 hours. We\u2019re not expecting to crawl for every single parking code, but if we could find one out of three parking minimums, we\u2019d save 100 hours of manual work.<\/p>\n\n\n\n

What if we could write a script that can search keywords and find the parking code for us? We experimented with Python\u2019s scrapy library (which also has scraping functions) and focused on its \u201ccrawling\u201d ability to find the pages we need.<\/p>\n\n\n\n

However, Municode has fancy elements, like a search bar and drop-down tabs, which can only be loaded by waiting for a browser to interpret the code. We landed on using a joint library called scrapy-playwright, which implements a browser.<\/p>\n\n\n\n

Our crawler is able to search for keywords in the search bar, wait for its results, and extract the first link.<\/p>\n\n\n\n

This feature isn\u2019t fully implemented, as the spider can only search for one keyword and only scrapes the first link, which isn\u2019t always the right one with the parking code.<\/p>\n\n\n\n

What we would have done differently<\/h5>\n\n\n\n

We wanted to automate as much of the process as possible, from trying to parse every single format of parking minimums to using machine learning to find the right table. It\u2019s difficult to get computers to work 100% correctly, and it\u2019s easy to speed up a human\u2019s workflow. Rather than approaching this project as an end-to-end system with no humans involved, we realized our time is better spent building tools to make the manual work easier.<\/p>\n\n\n\n

We were also torn between inserting as many entries as possible and building a viable system to onboard other volunteers. As a result, we developed our personal scraping and parsing pipeline without fully considering how other people can use it.<\/p>\n\n\n\n

What comes next<\/h5>\n\n\n\n

There are a few more core and auxiliary features that can be implemented:<\/p>\n\n\n\n