The Indie Cini is a full-stack data application for exploring Chicago’s independent cinema landscape. To ease the discovery and logistics of moviegoing for our city’s busy film fans, it consolidates showtimes, film details, and critic reviews from multiple sources into a unified, interactive interface.
This case study traces both the product motivation behind this project and the engineering system built to support it. It follows The Indie Cini from its underlying user problem through its scraping pipeline, storage and observability layers, PostgreSQL warehouse, dbt transformations, frontend experience, usage analytics, and AWS-hosted production platform.
Problem
Chicago is home to fantastic independent cinemas and many eager moviegoers. But these beloved venues and their audiences can still be kept apart by several digital gaps:
- Major showtime aggregators like Fandango rarely list Chicago’s independent cinemas, so many moviegoers begin browsing or planning without their screenings in view
- Showtimes and critic reviews live on separate sites, splitting discovery across multiple tabs and slowing it
- Deciding between films, venues, or dates requires hopping between different lists of showtimes, typically without consistent filtering, sorting, or structure
- Traditional showtime listings reduce screenings to flat text entries, obscuring their duration and overlap
Taken together, these frictions make Chicago’s indie film landscape harder to explore than it should be. Moviegoers deserve a clearer, more unified way through it. And the cinemas at its center deserve digital representation that honors their contribution.
Goals
My vision of a unified indie moviegoing platform didn’t arrive as a sudden insight, nor as a neatly defined specification. It developed gradually out of organizing countless cinema outings, both with friends and occasionally through Meetup. Over time, the friction points faced by modern moviegoers became unmistakable, clarifying what would be needed in a digital tool.
From those accumulated experiences, the project’s goals solidified along two dimensions: the moviegoing support I wanted to provide, and the engineering foundation required for reliability, extensibility, and automation.
Product Goals
- Create a single, authoritative view of Chicago’s indie film landscape by consolidating showtimes from multiple venues.
- Reduce friction in discovery by pairing showtimes with critic reviews, both individual and aggregated, in one environment.
- Improve navigability with intuitive, interactive views and consistent filtering across calendar and critic-review components.
- Provide decision-making clarity through consolidated critic reviews, rich event context, and visual cues that reveal how screenings relate in time.
Engineering Goals
- Automate the nightly collection and preservation of showtime, film, and review data through robust scrapers and durable artifact storage.
- Validate scrape outputs before ingestion, preventing stale, missing, empty, or malformed artifacts from reaching production tables.
- Normalize and transform source data into stable, application-facing datasets with clear contracts across pipeline stages.
- Design for extensibility, allowing additional cinemas and review sources to slot naturally into the system.
- Operate the application as a cloud-hosted production system, with managed PostgreSQL, scheduled pipeline execution, secure configuration, and operational logging.
- Deliver a dense-but-legible frontend, built to present substantial information while preserving clarity, intuitive control, and responsiveness across devices.
- Prepare for advanced analysis, including predictive modeling of theatrical run lengths, longitudinal studies of review scores, and analysis of user-interaction events.
Aiming for these goals helps ensure that The Indie Cini’s screening data is accurate, continuously updated, richly contextualized, and intuitively accessible. These priorities continue to guide the app’s evolution.
They anchor future work such as incorporating additional venues and metadata sources, refining the frontend, and expanding the system’s analytic capabilities. As the interface and its datasets continue to mature, so too does the project’s ability to help moviegoers discover screenings and help independent cinemas share their programming.
Architecture
Overview
The Indie Cini is a full-stack, AWS-hosted data application that automates the collection of cinema showtimes and critic reviews, validates scrape artifacts before ingestion, transforms warehouse tables into application-facing marts, and serves those modeled datasets through a responsive browser interface. The system separates scraping, storage, observability, ingestion, transformation, presentation, and cloud operations into distinct conceptual layers. This allows each part to evolve independently without disrupting data flows from the external websites to the application interface.
At the data-pipeline level, nightly Selenium scrapers collect showtimes, film-level details, and critic reviews, applying normalization to reconcile source-specific formatting inconsistencies. Their structured outputs are written to Amazon S3 as timestamped artifacts, inspected by a pre-load observability layer, and ingested into AWS RDS PostgreSQL, the system’s canonical production warehouse. Once loading completes, dbt reshapes the raw tables into staging, intermediate, and mart models. These transformations standardize source-specific records, construct stable join keys, deduplicate film and review data, aggregate critic reviews, and produce frontend-ready datasets that power key features, like the showtime calendar and review dashboard.
At the application-platform level, The Indie Cini runs as containerized AWS workloads. The public Next.js frontend is deployed from a dedicated Docker image on ECS/Fargate behind an Application Load Balancer, with DNS handled through Route 53 and HTTPS managed through AWS Certificate Manager. The backend data pipeline runs separately as a scheduled ECS/Fargate task launched by EventBridge Scheduler. Both workloads use Amazon ECR for image storage, AWS Secrets Manager for runtime configuration, and CloudWatch for production logging and operational visibility.
This architecture keeps the system’s responsibilities deliberately separated. Scraping produces durable, traceable artifacts; observability evaluates those artifacts before warehouse loading; ingestion refreshes the production database; dbt isolates semantic transformation in explicit modeling layers; and the frontend consumes prepared datasets rather than embedding data-shaping logic in the interface. Around those application layers, AWS services provide container execution, managed PostgreSQL, secure configuration, public routing, HTTPS, and operational logging. The result is a platform that remains inspectable at each stage, reliable in production, and extensible across new venues, review sources, analytics features, and monitoring needs.
Scraping
To collect data from a range of online sources, The Indie Cini uses an array of custom scrapers. Each scraper adapts to a site’s unique structure, quirks, and inconsistencies, which vary across sources and shift over time.
Despite these source-specific differences, the scrapers share a robust core: a fault-tolerant, memory-aware Selenium driver that restarts itself when stalled. With this RotatingDriver, the scrapers can reliably navigate JavaScript-heavy pages and execute targeted page-interaction scripts to reveal content yet unrendered. These interactions allow the scrapers to access data that would otherwise remain inaccessible.
Scraping starts the scheduled pipeline each night, proceeding in two phases: cinema venues first, reviews second. The venue scraping phase retrieves showtimes, film metadata, and selected source HTML. These outputs are normalized and validated during scraping, formed into structured DataFrames, and saved to S3 as pickled DataFrames and HTML snapshots.
This freshly written venue data then seeds the review scraping phase. For each film, the review scraper searches for the corresponding Metacritic page by iterating over a subset of identifying fields and applying an empirically tuned relevance ranking. To reduce the expense of this operation, the scraper maintains persistent master files of previous searches and their outcomes. Finally, as with the venue data, the scraped reviews are normalized, validated, and saved to S3 as pickled DataFrames.
Observability
The observability layer inspects the pipeline at two critical boundaries: the scrape-artifact boundary before warehouse loading, and the database boundary after ingestion. It produces timestamped reports that make pipeline health visible across artifact storage, pre-load validation, and PostgreSQL freshness. This context matters most when scrapes fail unevenly: the reports can show which artifacts were freshly produced, which were carried forward from earlier runs, and which are missing or unusable.
Artifact Inventory & Quality Gating
Before ingestion, the artifact observability runner produces two reports: an inventory report and a quality report. The inventory report checks whether each expected scrape artifact exists in S3, and whether the latest available version is current enough to trust. For each artifact, it records the file path, artifact date, row count, column count, and load status, flagging whether the artifact is stale, missing, or empty. This prevents artifacts from being treated as healthy merely because a file exists.
The artifact quality report goes deeper by loading each latest artifact and evaluating its structure and contents. It produces both summary-level metrics and entity-level findings, classifying each finding by operational severity as a routine observation, warning, or pipeline-blocking issue. Its checks assess dataset emptiness, identifier duplication, missing values, invalid film runtimes or review scores, Metacritic search matching, and artifact-specific volume patterns such as reviews or showtimes per film.
Together, these artifact reports function as a pre-load quality gate. If a saved quality summary flags a blocking issue, the observability runner raises an exception rather than passing control to database ingestion. By saving its integrity assessments and blocking on their basis, this layer preserves diagnostic evidence while preventing failed artifacts from reaching production tables.
Database Freshness
Once artifacts are loaded to the PostgreSQL warehouse in AWS RDS, a database freshness report checks the core tables that feed the downstream transformation layer. For each table, it records row count, latest scrape timestamp, and a freshness status such as “ok,” “stale,” “empty,” “missing scrape time,” or “query failed.” Those table-level results are then rolled up into a database-level summary, including an overall freshness verdict and counts of tables by status. These outputs are saved for later inspection, and the database-level summary is also printed in a compact key-value format suited to CloudWatch logs and alerting workflows.
By separating diagnostics from ingestion and transformation, the observability layer gives the pipeline a clearer operational feedback loop. Instead of allowing artifact or warehouse issues to surface later as failures in loading, transformation, or frontend behavior, it checks for them at dedicated boundaries in the pipeline. This structure supports future refinements in cloud-based alerting and orchestration, while making maintenance of the nightly pipeline less reactive and stressful.
Ingestion
After each nightly scrape, the ingestion layer loads the latest validated showtime, metadata, and review artifacts into the structured tables that drive the rest of the system. It reads the scraped DataFrames through the same StorageBackend interface used by the scrapers and observability reports, preserving a consistent contract across pipeline stages. Once artifacts have passed the pre-load observability gate, ingestion focuses on preparing them for relational storage: standardizing field names, casting values where needed, and aligning each dataset with the database’s expected column types.
The loaders then use SQLAlchemy to replace the core, scrape-derived tables in AWS RDS PostgreSQL using data from the latest validated artifacts. This full-refresh pattern keeps the production warehouse aligned to the most recent successful scrape, rather than reconciling incremental changes across runs. The same ingestion logic remains portable across environments: in production, it reads from S3 and writes to RDS PostgreSQL, while local development can use local artifact storage and a local database target. Those refreshed tables then provide stable inputs for dbt models and the frontend, keeping the application’s datasets current and consistent.
Storage
Scraping, observability, and ingestion do not exchange data directly; instead, they meet at the storage layer. The storage layer ensures that scrape artifacts are durably saved and organized into clear, run-based directories.
This design gives the system consistency and reproducibility— qualities that make debugging easier, support retrospective analysis, and help preserve data integrity. These goals are supported by two core mechanisms: the unified StorageBackend interface and a centralized pathing.py module that defines directory and filename conventions.
The StorageBackend Abstraction
At the center of the storage layer is the StorageBackend abstraction, which provides a unified interface for reading and writing scrape artifacts, regardless of their physical location. Whether the system is using local disk or Amazon S3, the pipeline layers interact with artifacts through the same set of methods.
This keeps file-I/O concerns separate from the scraping, observability, and ingestion logic: those components issue simple load and save calls, and the backend resolves the underlying mechanics. Because both backend implementations mirror the same interface, switching between local and S3-backed storage only requires configuration changes rather than code rewrites.
Pathing and Directory Structure
pathing.py centralizes all naming rules and directory conventions for the pipeline, ensuring that every scrape stores its outputs in a predictable, reproducible layout. Each run’s DataFrames are written as .pkl files under a structured hierarchy:
data/pkl/<venue-or-review-source>/<scrape-type>/<YYYY-MM-DD>.pkl
This pattern makes the storage location itself encode essential metadata— where the data came from, what type it represents, and when it was produced. The same conventions apply to HTML snapshots and other scrape artifacts, giving the storage layer a uniform shape across both local and S3 backends.
In combination, the StorageBackend abstraction and the pathing.py module form the core of the storage layer. StorageBackend abstracts file I/O across local disk and S3, while pathing.py standardizes naming and organization. In tandem, these mechanisms make reads and writes predictable and reproducible, providing a stable intermediary between scraping, observability, and ingestion. By establishing a shared, programmatic contract for how data is written and later retrieved, the storage layer ensures continuity across pipeline stages, which is essential to the system’s overall integrity.
Database
The production database, hosted on AWS RDS as a PostgreSQL instance, serves as the system’s canonical, query-optimized representation of the latest successful scrape. It is designed to hold a consistent, queryable snapshot of currently slated screenings and their associated critic reviews, rather than function as a long-term archive.
To support this role, the ingestion pipeline follows a full-refresh pattern: on each successful run, core tables are cleared and repopulated from the latest scraped datasets. This approach simplifies ingestion logic and avoids reconciliation issues across runs, favoring consistency and correctness over incremental updates. The result is a stable foundation for downstream transformation. This full-refresh pattern applies to the scrape-derived tables, but a table for usage analytics is instead persistent and append-only, so that a history of frontend interaction can accumulate over time.
In this architecture, the database does not directly shape application-facing data. Instead, it acts as the boundary between ingestion and transformation, storing normalized yet largely source-aligned tables that are then consumed by the dedicated dbt modeling layer. That layer creates staging, intermediate, and mart relations in the same database environment, so PostgreSQL stores these transformed datasets alongside the inputs used to build them.
This design gives the database a central but bounded role. PostgreSQL provides the shared relational environment for both loaded scrape data and transformed dbt relations, while the modeling layer remains responsible for defining how loaded scrape data becomes application-facing datasets. By keeping those responsibilities separate, the system avoids burying business logic in ingestion and preserves a transformation process that is modular, inspectable, and easier to extend.
Transformation
After ingestion refreshes the scrape-derived tables in PostgreSQL, data transformation in The Indie Cini is handled by a dedicated modeling layer built with dbt. This layer reshapes source-aligned tables into structured, application-facing datasets.
The modeling layer organizes transformations across staging, intermediate, and mart tiers. Staging models standardize tables aligned with individual scrape artifacts — showtimes, film-level details, and critic reviews — bringing them into a consistent format while preserving their original structure. Intermediate models then perform the core relational work of the system, combining the data of multiple cinema venues into unified representations of showtimes and films, then linking those screenings with their corresponding reviews. The final mart models produce feature-specific datasets that directly support the application’s interfaces, which include the screening calendar and the review dashboard.
Within this structure, the modeling layer expresses transformation logic declaratively and in a modular form. It centralizes the operations required to turn heterogeneous scrape outputs into tailored datasets, while keeping each step explicit and inspectable. By formalizing those steps into sequences of well-scoped models, this layer makes the logic of its data-shaping easier to reason about, revise, and extend.
This design simplifies the frontend’s queries, establishes clear data contracts across components, and keeps semantic shaping close to the data itself. As the application evolves — whether through new venues, richer metadata, or expanded review logic — the modeling layer provides a flexible but disciplined surface for expressing how loaded scrape data is transformed into meaningful structures.
Review Aggregation
One transformation plays a distinct semantic role in the system: the model that prepares film reviews from multiple publications for comparative browsing, producing the dataset that powers the application’s review dashboard (mart_review_dashboard). By grouping reviews at the publication level rather than leaving them as individual critic entries, this transformation presents them in a form that is more recognizable to users and more intuitive for their comparison of critical reception across films.
In the scraped source data, critic reviews appear as individual entries, sometimes associated with the same publication. Reconciling these many-to-one relationships requires aggregation logic that consolidates scores by publication while resolving occasional duplication. The resulting review roundups summarize scores for comparison while preserving constituent reviews as context. This allows the frontend to present both concise publication-level metrics and the richer commentary behind them.
In this way, review aggregation defines how a film’s reviews are represented to users. It balances comparative clarity with preserved critical context, and its logic remains explicit in the modeling layer rather than buried in the frontend.
Frontend
The frontend is what makes The Indie Cini an usher, rather than a marquee. It turns a dense set of screenings and reviews into an accessible, interactive guide. This layer deliberately constrains itself to interpreting upstream data, rather than additionally transforming it. Its focus is to present that data clearly, responsively, and in forms that support moviegoers’ decision-making.
The interface first offers broad overviews, helping users orient themselves easily. As their interest narrows, additional detail appears through hovercards, tooltips, and focused interaction. In this way, the frontend manages informational density as users move between exploration and decision-making. Its role as a guide depends on that continuity, sustaining users’ curiosity rather than disrupting it with mode switches or navigation breaks.
That guiding role is further realized through several coordinated surfaces. The calendar views help users orient their plans, visualizing available screenings by start time, runtime, and venue. The review dashboard then interprets that same slate evaluatively, using side-by-side critic scores to place those screenings in the context of critical reception.
When review comparison prompts narrowing, highlighting allows selected films or directors to be tracked across views, isolating their associated screenings in the calendar without breaking the frame of reference. Global filters operate at a broader level, constraining the overall scope by venue, runtime, or release type.
Across these surfaces, focus can narrow progressively without splintering across the interface, supporting exploration and decision-making as a single continuous process.
Usage Analytics
To better understand how users interact with the application, I implemented a lightweight, event-based analytics system that captures key frontend events and stores them in a dedicated table in AWS RDS PostgreSQL.
The system is designed around a simple API route ( /api/analytics ) that receives structured event payloads from the frontend and writes them to an append-only analytics_events table. Each event includes a timestamp, event type, and an extensible metadata structure, enabling the system to capture evolving interaction patterns without requiring frequent schema changes.
Two core event types are currently tracked:
page_view— logs visits to application pages, recording the URL pathoutbound_click— captures clicks from the calendar to official venue pages for screened films, recording the destination URL and associated screening metadata
The outbound_click events are particularly meaningful, as they bridge application interaction with real-world intent. These events reflect moments when users move from browsing The Indie Cini to engaging with a film at its source, providing a direct signal of interest.
Importantly, this analytics system operates independently of the ingestion and transformation pipeline. Unlike the application’s core datasets, which are fully refreshed through scheduled pipeline runs, the analytics event table persists and grows incrementally over time. This makes it possible to analyze user behavior longitudinally.
Future analyses could include:
- identifying high-interest films or venues
- comparing engagement across time ranges
- measuring the effectiveness of UI changes or feature additions
By logging and preserving user-interaction events, the analytics layer serves as a strong foundation for continued product insight and iteration.
Cloud Platform & Production Environment
The Indie Cini’s production platform is organized around two AWS-hosted workloads: an always-available frontend service for users, and a scheduled backend pipeline that refreshes the datasets behind that interface. The frontend workload remains lean and public-facing, while the backend pipeline carries the heavier dependencies required for the data-refresh process, which involves scraping, observability, ingestion, and dbt transformation.
Public requests to the application resolve through Route 53 DNS. They then reach an Application Load Balancer, which terminates HTTPS using an AWS Certificate Manager certificate before forwarding requests to the frontend ECS/Fargate service. This service runs as a containerized Next.js application built from a dedicated Docker image, with Amazon ECR serving as the image repository. The full request path is therefore handled by a dedicated web-service layer, which allows the application code to remain focused on serving pages, API routes, and prepared datasets rather than managing public infrastructure concerns.
The data pipeline runs separately as a scheduled ECS/Fargate task launched by EventBridge Scheduler. Its container image, also stored in Amazon ECR, packages the backend pipeline code, Python dependencies, and the dbt project needed to execute scheduled data refreshes. Each scheduled run reads and writes scrape artifacts in Amazon S3; loads validated data into AWS RDS PostgreSQL; and runs dbt transformations that produce the marts consumed by the frontend.
Sensitive configuration and access control are handled through AWS-managed security mechanisms, rather than by hardcoding sensitive values or granting broad application permissions. Runtime configuration is stored in AWS Secrets Manager, and at startup the required values are injected into the frontend and pipeline containers by ECS task definitions. Access to AWS resources and network paths is then limited to each workload’s needs through IAM roles and security groups. Together, these mechanisms structure the platform as distinct, least-privilege components rather than as a single, broad-permission application environment.
Operational visibility is provided through CloudWatch logs and alerts. Frontend logs support debugging of server startup, API route behavior, and database connectivity, while backend logs surface pipeline failures across scraping, observability, loading, and transformation. These logs help separate web-serving issues from data-refresh issues, making production failures easier to locate and investigate.