Show Calendar model lineage
Standardizes venue-specific showtimes and film metadata, merges both branches across cinemas, and joins them into mart_calendar_shows for the application calendar.
01 — Where the system runs
This deployment view shows the public web request path, the scheduled runtime that keeps the application’s data current, and the AWS services shared by both these workloads.
02 — How data reaches the application
This data-flow view opens the scheduled pipeline itself: collection writes artifacts to S3, a pre-load observability gate validates them, and approved data proceeds through database loading and dbt modeling before being served by the frontend.
03 — Inside the Modeling Layer
dbt Model Lineage
The dbt project separates source normalization, reusable intermediate logic, and frontend-facing marts. The lineage diagrams below show how venue, film, and critic-review data converge to power the application’s calendar and review dashboard.
Review Dashboard model lineage
Transforms critic reviews from multiple publications into a comparative-browsing dataset, producing mart_review_dashboard for the application's review dashboard.
Transformation Highlights
- Canonical review-key construction
- Deterministic review deduplication
- Publication-level aggregation
- Film-level review structures
- Scoping to currently slated films
Beyond the lineage shown here, schema tests and custom data tests enforce key integrity assumptions across the dbt project.
04 — Architecture Decisions
Design choices behind the system
These decisions define how the pipeline passes data between stages, protects production tables from unhealthy inputs, maintains a consistent current-state warehouse, and transforms source-aligned records into application-facing datasets.
01
Pass data between pipeline stages through durable artifacts
Scraping, observability, and ingestion exchange data through timestamped artifacts behind a shared storage interface. This preserves diagnostic evidence, makes each run inspectable, and allows stages to be rerun independently.
- Alternative Approach
- Pass results directly between processes in memory, or write scraped data immediately to PostgreSQL.
- Tradeoff
- The pipeline must manage additional storage organization, pathing conventions, and stale-artifact handling.
02
Gate database loads on artifact quality
Inventory and quality reports evaluate the latest artifacts before ingestion. Critical gate violations stop the refresh so that unhealthy scrape outputs do not replace trusted production data.
- Alternative Approach
- Load artifacts immediately and shift some validation burden downstream, relying on database tests, dbt failures, or frontend symptoms to expose defects.
- Tradeoff
- A strict quality check may delay a data refresh by stopping a run that could otherwise have partially succeeded.
03
Maintain production tables as a current-state snapshot
Each successful run of the pipeline replaces the scrape-derived core tables with the latest validated datasets, matching the application’s need for a consistent view of the current cinema slate.
- Alternative Approach
- Incrementally reconcile additions, changes, and removals across successive scrape runs.
- Tradeoff
- Full refreshes repeat more work and do not preserve longitudinal history inside the core application tables.
04
Shape application data in a dedicated modeling layer
Ingestion keeps tables largely source-aligned, while dbt owns standardization, joins, deduplication, aggregation, and frontend-facing marts. Semantic logic therefore remains explicit, modular, and testable.
- Alternative Approach
- Maintain transformations as PostgreSQL views outside a version-controlled modeling layer, or embed them in Python loaders or application queries.
- Tradeoff
- The architecture gains another tool, modeling layer, and orchestration step that must be maintained.
05 — Implementation Stack
Core Technologies
The application combines a TypeScript frontend, a Python data pipeline, PostgreSQL and dbt for relational modeling, and managed AWS services for execution, storage, routing, security, and observability.
Application
Presents screening and review data through a responsive, interactive browser interface.
- Next.js
- React
- TypeScript
- Tailwind CSS
- react-big-calendar
Collection & Ingestion
Collects source data, normalizes scrape outputs, and loads validated artifacts into relational tables.
- Python
- Selenium
- BeautifulSoup
- Requests
- pandas
- SQLAlchemy
Storage & Modeling
Preserves pipeline artifacts and transforms source-aligned tables into application-facing datasets.
- Amazon S3
- PostgreSQL on Amazon RDS
- dbt
- SQL
Cloud Platform
Runs, schedules, secures, routes, and monitors the application's production workloads.
- ECS Fargate
- Amazon ECR
- EventBridge Scheduler
- Route 53
- AWS Certificate Manager
- Application Load Balancer
- Secrets Manager
- CloudWatch