The Indie Cini

🎬 The Indie Cini

A data-driven planner for Chicago's independent cinemas.

Chicago is home to fantastic independent cinemas and many eager moviegoers. But these cinemas and moviegoers are often separated by a digital gulf, one largely unnoticed despite its expanse.

• Fandango and other major showtime aggregators tend not to list screenings from these venues.
• Critical reviews and aggregated scores are strewn across innumerable websites, making movie discovery needlessly difficult.

To address these gaps, The Indie Cini consolidates showtimes and film metadata from Chicago’s leading independent cinemas and enriches them with aggregated critic reviews. It currently covers the programming of the Gene Siskel Film Center, the Music Box Theatre, and FACETS.

Built on an end-to-end pipeline of scrapers, data modeling, and a tailored UI, The Indie Cini delivers an interactive calendar, a critic-score dashboard, flexible filters, and a responsive layout, all designed to simplify the discovery of Chicago's indie film culture.

🧭 Overview

The Indie Cini automates the collection, normalization, and transformation of cinema slate data from Chicago’s top independent venues, alongside film review data from a major aggregator. The result is a unified and consistently updated view of the city’s indie screening landscape.

Core outcomes include:

• Unified dataset of all screening events across Chicago's major indie venues

• Aggregated critic reviews powering a sortable, detail-rich dashboard

• Interactive calendar interface, with filters that refine both the calendar and the review dashboard

• Analytics-ready data foundation for analyzing score dynamics and theatrical longevity, with initial support for capturing user interaction events.

🏗️ Architecture

The system follows a modular, AWS-hosted architecture that separates data collection, observability, ingestion, transformation, application serving, and cloud operations into distinct layers.

Data Pipeline

High-level data pipeline architecture for The Indie Cini (click to enlarge).

The core data flow remains ETL-style: scrapers generate structured artifacts, observability checks validate them before loading, ingestion refreshes the production warehouse, and dbt models produce application-facing marts.

Production Platform

Architecture of The Indie Cini’s AWS-hosted production platform (click to enlarge).

In production, the public frontend and scheduled pipeline run as containerized workloads on ECS/Fargate, with artifacts in S3, PostgreSQL on RDS, and HTTPS traffic routed through Route 53, ACM, and an Application Load Balancer.

Layer Breakdown

1. Venue Scraping Layer (Python + Selenium + BeautifulSoup)

   • Separate scrapers for Siskel, Music Box, and FACETS, with seasonal expansion support for CIFF
   • Custom RotatingDriver manages browser restarts, stalled sessions, and JS-heavy pages
   • ScrapeResult bundles parsed DataFrames, source HTML, and run metadata

2. Review Scraping Layer (Python + Selenium + BeautifulSoup)

   • Metacritic pages identified via automated search with empirically tuned relevance ranking
   • Dynamically loaded content retrieved through simulated browser interaction, including scrolling, “Load more" handling, cookie dismissal, and targeted JavaScript execution through Selenium
   • Master files (mc_<data_type>_accum) reduce redundant scraping
   • (Planned): Letterboxd score integration

3. Observability Layer (Python + pandas)

   • Artifact inventory reports verify that each expected scrape artifact exists, loads successfully, and is fresh enough to trust
   • Artifact quality reports inspect the latest artifacts for structural validity, empty datasets, duplicate identifiers, missing values, invalid runtimes or review scores, unmatched Metacritic searches, and source-specific volume anomalies
   • Findings are classified by operational severity, distinguishing routine observations from warnings and pipeline-blocking issues
   • The main observability runner acts as a pre-load quality gate, preserving diagnostic reports while preventing invalid scrape artifacts from reaching production tables
   • Post-load database freshness reports verify core PostgreSQL tables after ingestion, checking row counts, latest scrape timestamps, and table-level freshness status

4. Ingestion Layer (Python + pandas + SQLAlchemy):

   • Standardized loading routines prepare validated scrape artifacts for relational storage by normalizing column names and enforcing expected dtypes
   • Loads only the latest artifacts that have passed the observability gate
   • Core scrape-derived tables follow a deterministic full-refresh pattern, keeping the warehouse synchronized with the latest validated scrape
   • Data is routed to either a local MySQL dev database or the production AWS RDS PostgreSQL warehouse

5. Storage Layer (AWS S3 + AWS RDS PostgreSQL)

   • Unified StorageBackend abstraction for consistent local/S3 file I/O
   • pathing.py defines centralized naming conventions and directory structure
   • ScrapeResult objects serialize directly into versioned S3 archives
   <!-- - Postgres tables (`showtimes`, `show_info`, `reviews`) feed unified analytical views (`v_screenings_enriched`, `metascore_report`) -->
   • Raw scrape artifacts persist in S3/local storage, while AWS RDS PostgreSQL houses the application's raw, transformed, and analytics tables

6. Transformation Layer (dbt)

   • dbt transforms raw PostgreSQL tables into layered models spanning staging, intermediate, and mart tiers:
     • Staging: Standardizes venue slate and review data
     • Intermediate: Constructs stable join keys, deduplicates Metacritic records, and aggregates film reviews by publication and film
     • Marts: Two feature-specific marts power the frontend: mart_show_calendar and mart_review_dashboard
   • dbt tests enforce core data integrity assumptions, including null, uniqueness, and composite-key checks.

7. Frontend (Next.js + React Big Calendar)

   • Public Next.js application runs as a containerized ECS/Fargate service behind an Application Load Balancer
   • Interactive calendar with daily/weekly/agenda views
   • Event blocks sized by runtime and color-coded by venue
   • Tooltips preview event details and link directly to booking pages
   • Critic-score dashboard with sorting, hovercards, and click-to-highlight behavior
   • Filter controls for venue, release type, and runtime
   • Fully responsive layout, plus a desktop resizable split panel between calendar and dashboard

8. Usage Analytics (Next.js API routes + PostgreSQL)

   • Lightweight frontend instrumentation captures user interaction events, including page views and outbound clicks
   • Events are sent via a server-side API route and persisted to an append-only analytics_events table in AWS RDS PostgreSQL
   • A dbt staging model (stg_analytics_events) standardizes this data for potential downstream analysis
   • This layer establishes a foundation for future behavioral insights, without being a core feature of the current application

9. Cloud Deployment & Operations (AWS)

   • Frontend and pipeline workloads run as separate Docker images stored in Amazon ECR and executed on ECS/Fargate
   • EventBridge Scheduler launches the recurring production pipeline task, replacing the former Render cron workflow
   • Route 53, ACM, and an Application Load Balancer provide public DNS, HTTPS, and canonical-domain routing for https://indiecini.org
   • Secrets Manager injects runtime database and API configuration into ECS tasks without storing sensitive values in the codebase
   • CloudWatch Logs and pipeline alerting provide operational visibility into scheduled runs and production services

🧩 Key Features

🗂️ Data Sources

🧮 Technologies Used

🧰 Development Notes

• Environment: Windows-based local development with Dockerized Linux production workloads and ChromeDriver-backed Selenium scraping
• Storage Structure: data/ and test/ directories mirror production vs. sandbox runs
• Output Format: CSV and PKL scrape outputs archived under /data/pkl/{venue or review source}/{category}/
• Observability Outputs: Timestamped inventory, quality, and database freshness reports are archived under /data/pkl/observability/
• Containerization: Separate Docker images support the Next.js frontend service and the Python/dbt ingestion pipeline
• Cloud Runtime: Production workloads run on AWS ECS/Fargate, with runtime configuration injected through AWS Secrets Manager
• Logging: Timestamped, structured application logs are supplemented by CloudWatch logs for AWS-hosted services and scheduled pipeline runs (e.g., facets_scrape | +0.01s | https://facets.org/...)

📸 Showcase

Live Demo

• Direct: indiecini.org

Visual Highlights

• 📅 Main Calendar: Weekly and agenda views with color-coded venues
• 🔍 Filters Panel: Filter by release type, runtime, and venue
• 🗞️ ReviewSpew Dashboard: Sortable critic-review table with hovercards
• 🎯 Click-to-Highlight: Selecting a film or director isolates its screenings in the calendar
• 🖥️ Resizable Split-Panel Layout: Drag to adjust the space between the ReviewSpew dashboard and the calendar.
• 📱 Responsive Mobile Layout
• 🧩 Architecture Diagrams:: indiecini.org/architecture

🚀 Future Work

• Integrate Letterboxd audience review scores
• Add a scrollable film-poster ribbon beneath the ReviewSpew dashboard for quick visual browsing.
• Implement predictive models for audience–critic divergence and for the duration of a film's theatrical run.
• Expand coverage to additional Chicago-area independent venues.
• Codify the AWS infrastructure with Terraform, AWS CDK, or CloudFormation to improve repeatability and reduce configuration drift.

Frontend-specific

• Add filters for review scores, specified date ranges, and polarizing genres.
• Surface subtle features (like click-to-highlight) through small UX cues.

👤 Author

Max Ruther
M.S. Computer Science (Data Science concentration) — DePaul University
Solo developer and data engineer behind The Indie Cini.
💌 Portfolio

This repository remains private to protect proprietary logic and sensitive API configurations.
Public assets, screenshots, and diagrams are available on the case-study page.