MapnStreets Panel Web App

Have you ever walked down a street and thought, “huh…another street named ‘Main?’ How creative…I wonder how many more ‘Main Streets’ there are across the US? 🤔”

If, like me, you have, wonder no longer! MapnStreets allows you to explore the prevalence and locations of street names in the United States. Try it on HuggingFace, or run it locally by cloning from GitHub!

Curious about how MapnStreets works? Keep reading!

Built on the Shoulders of Giants: OSS Packages 🚀

MapnStreets is a combination of various open-source packages that make the magic happen:

• Fugue: This nifty Python library handles scalable data processing and analytics. It’s like the brains behind the operation, allowing for efficient querying and processing of large datasets.
• Panel: This is the package responsible for creating the interactive web interface of MapnStreets! With Panel, you don’t need to know JavaScript; simply define the widgets, layout, and interactivity in Python—then serve!
• GeoPandas: GeoPandas takes pandas to the next level by adding geospatial capabilities. It’s all about handling geospatial data structures and operations. In other words, it’s a lifesaver when dealing with geographical datasets, like shapefiles.
• GeoViews: GeoViews provides interactive visualizations of geospatial data. With GeoViews, you can create dynamic maps and plots that you can zoom in and out of. You can also pan left and right.
• Parquet: Ever heard of a file format that’s both efficient and scalable? That’s Parquet! In MapnStreets, Parquet ensures fast and efficient storage and retrieval of geospatial data.
• DuckDB: DuckDB is a lightning-fast in-memory analytical database. In MapnStreets, DuckDB (alongside Fugue) takes care of executing your queries.
• Ray: Ray is a superstar when it comes to distributed execution. It’s a flexible and high-performance framework that allows you to parallelize and distribute computations across a cluster. With Ray, MapnStreets is able to download and process over 3,000 files in a flash!

These amazing packages, along with their supporting dependencies, form the powerhouse behind MapnStreets to provide efficient data processing and interesting visualizations.

Pre-Processing: The Magic of Data Preparation

Before we discuss the front-end aspect of MapnStreets, let’s take a peek behind the curtain at the enchanting world of data pre-processing. Here’s how the magic happens:

1. Fetching the data:
   We begin by sending a GET request to the list of available TIGER/Line® Edges partitions provided by the US Census Bureau. Then, we parse the response’s content with the help of BeautifulSoup to gather all the file URLs.

2. Downloading and processing shapefiles:
   With the file URLs in hand, we embark on a downloading spree to download all the compressed shapefiles! To ensure idempotent downloads and avoid redundant processing, we utilize a clever trick. Before downloading each shapefile, we check for the existence of a corresponding “.done” file using the done_path variable. If the file already exists, we gracefully skip to the next one, saving valuable time and resources. Once the file is downloaded, we load into GeoPandas, but because geometry type is not natively supported in Parquet, we use another trick—we transform the geometry column into well-known text (WKT) format, so that it can be exported as Parquet!

3. Embracing parallelization with Fugue and Ray:
   With Fugue, you can use any distributed computing backend you’d like, whether it’s Dask, Ray, or Spark! Here, Ray was chosen for its automatic progress bars.

With these steps complete, we’ve prepared the stage for MapnStreets to shine. The world of street names is just a click away, waiting to be explored with MapnStreets!

The Web App: The Map of All the Streets

Now that you understand the tech behind it, let’s see how MapnStreets works in the frontend.

1. Feed the curiosity 🧐
   When you open MapnStreets, you’re greeted with an input box. That’s where you can enter the name of the street you’re curious about. Let’s say you type in “Main St.” That’s when the fun begins!

2. Magic happens ✨
   MapnStreets jumps into action! It uses Fugue to parallelize the looking up of street data based on the name you entered, and it scales amazingly, whether you have two CPUs (HuggingFace free tier), four CPUs, or n CPUs! It filters using a parameterized SQL query, and voila! It outputs a DataFrame that contains all the juicy details about those “Main Streets” across the United States. The DataFrame’s geometry column is de-serialized using `wkt.loads` and then converted into a GeoPandas GeoDataFrame!

3. Visualize, explore, repeat 🗺️
   Now it’s time to visualize those Main Streets! MapnStreets employs GeoViews to create an interactive map. The map shows you the locations of all the Main Streets in the country. You can zoom in, zoom out, and explore different areas. It’s like taking a virtual street-name adventure! MapnStreets also provides a table that highlights the exact locations of each Main Street. It’s like a treasure map, guiding you to each Main Street’s whereabouts. How cool is that?

And the best part? Everything is done in Python, thanks to the magic of Panel. The user interface is smooth, responsive, and super engaging. It’s like having your own personal street-name explorer at your fingertips.

Feed Your Curiosity with MapnStreets!

MapnStreets is the ultimate app for street-name enthusiasts. It combines Fugue, Panel, GeoPandas, GeoViews, Parquet, DuckDB, and Ray to bring you an interactive and visually stunning experience. So, the next time you encounter a street with an interesting name, satisfy your curiosity with MapnStreets. It’s the perfect way to explore the prevalence and locations of street names across the United States. Happy street-name hunting! 🚗🗺️