Track housing development from permit application to certificate of occupancy. Built for California HCD Annual Progress Report compliance.
Interactive dashboard with sortable tables, timeline visualization, APR comparison, and height analysis.
Explore DataInteractive map of all Berkeley housing projects with status, units, and timeline data.
View MapExecute the full pipeline in Google Colab. No local setup required.
Open in ColabSource code, data files, and documentation for the complete analysis pipeline.
View CodeClick any notebook to open it directly in Google Colab:
| Stage | Notebook | Description | Run |
|---|---|---|---|
| 00 | 00A_tour_of_the_pipeline | Non-technical tour of housing pipeline concepts | Colab |
| 00 | 00B_first_notebook_in_colab | Your first hands-on notebook - create data and charts | Colab |
| A | A1_data_sources_setup | Connect to Berkeley Open Data API, handle WAF blocks | Colab |
| A | A2_address_standardization | Normalize addresses (FIFTH ↔ 5TH, Ave ↔ AV) | Colab |
| A | A3_geocoding_pipeline | Match to lat/lon using 563K address lookup | Colab |
| A | A4_apn_enrichment | Match projects to Assessor Parcel Numbers | Colab |
| A | A5_buildingeye_import | Import permit dates from BuildingEye planning portal | Colab |
| A | A6_community_map_import | Parse Gellerman KML, extract news links from 203 projects | Colab |
| A | A7_comprehensive_integration | Fuzzy match and merge official + community data | Colab |
| A | A9_city_profile_builder | Generate city profile and setup checklist for adaptation | Colab |
| B | B1_lifecycle_tracking | Track permit stages: Zoning → Building → CO | Colab |
| B | B2_status_classification | Classify status into pipeline categories | Colab |
| B | B3_progress_indicators | Identify stalled projects (>180 days) | Colab |
| C | C0_methods_overview | Core methods: joins, aggregation, plotting | Colab |
| C | C1_pipeline_analysis | Projects by status, conversion rates | Colab |
| C | C2_timeline_analysis | Processing times, bottleneck identification | Colab |
| C | C3_proposal_vs_reality | RHNA tracking, "Making It Pencil" analysis | Colab |
| C | C4_quality_checks | Data validation: soft asserts, null checks, bounds | Colab |
| D | D1_monthly_report_generator | HTML/PDF/JSON report generation | Colab |
| D | D2_dashboard_data_export | SQLite + Datasette deployment | Colab |
| D | D3_alerts_monitoring | Status change alerts, milestone tracking | Colab |
| D | D4_hcd_apr_tables | Map pipeline data to HCD APR Table A2 fields | Colab |
| F | F1_development_math | Basic feasibility: TDC, NOI, Return on Cost | Colab |
| F | F2_pro_forma_transparent | Full pro forma with IRR, draw schedule, policy scenarios | Colab |
Progress toward California HCD Annual Progress Report requirements:
| Category | Coverage | Status |
|---|---|---|
| Direct mappings (year, address, permits) | 5 fields | Ready |
| Derivable (unit_category, SB35 flag) | 7 fields | Ready |
| APN (Assessor Parcel Number) | 96.5% | Ready |
| Income breakdown (VLI, LI, MOD) | 0% | Needs Data |
| Permit dates (entitlement, BP, CO) | 0% | In Progress |
Three ways to explore the data:
git clone https://github.com/blockXblock/berkeley-housing-analysis && jupyter lab MASTER_ANALYSIS.ipynbPlanned additions to expand the course:
| Module | Description | Status |
|---|---|---|
| E1: Environmental Overlays | CEQA, flood zones, historic districts | Planned |
| E2: Zoning Analysis | Parcel-level zoning constraints and allowed density | Planned |
| G1: Comparative Analysis | Cross-city benchmarking for Bay Area jurisdictions | Planned |
| Multi-City Network | Adapt pipeline for Oakland, SF, San Jose using city_template.yaml | Seeking Contributors |