Snowflake-Labs/horizontal-federated-learning

Snowflake-Labs/horizontal-federated-learning

Description: RITM0611822

Language: PLpgSQL

License: Apache-2.0

Stars: 0

Forks: 0

Open issues: 0

Created: 2026-06-08T09:02:22Z

Pushed: 2026-06-25T07:46:22Z

Default branch: main

Fork: no

Archived: no

README:

Snowflake Horizontal Federated Learning (HFL)

A reference implementation of Horizontal Federated Learning (HFL) using only Snowflake native features. Model parameters are exchanged across 3 Snowflake accounts via Private Listings to collaboratively train a binary classification model for diabetes prediction on CDC BRFSS 2024 data.

---

Use Case Overview

Horizontal Federated Learning (HFL) enables multiple organizations to collaboratively train machine learning models without sharing raw data with each other. This implementation focuses on the healthcare domain. Patient health and lifestyle data is extremely valuable for improving early disease detection and prediction accuracy, but direct data sharing between healthcare institutions and organizations faces significant regulatory and ethical barriers due to patient privacy concerns. With federated learning, each institution retains its data locally and exchanges only model parameters (weights), enabling collaborative training. This achieves generalization performance unattainable from any single institution's data alone, without compromising privacy.

In this implementation, we use CDC BRFSS 2024 (Behavioral Risk Factor Surveillance System) data, geographically splitting 53 U.S. states across 10 clients to train a diabetes prediction model. Each client holds data from only a limited set of states, but through federated learning, a global model reflecting patterns across all states is constructed, outperforming every single client's local model. Even with differential privacy (DP) noise applied, performance remains above all local baselines.

This approach is not limited to healthcare. Applicable use cases include: fraud detection across financial institutions (banks collaboratively learn fraud patterns without sharing transaction data), network anomaly detection among telecommunications carriers, cross-supply-chain quality prediction in manufacturing, and consumer behavior analysis across retail companies. Any domain where data sensitivity is high yet integrating data from multiple organizations creates significant value is a candidate for federated learning.

---

Table of Contents

1. [Use Case Overview](#use-case-overview) 2. [Architecture Overview](#architecture-overview) 3. [Experiment Results](#experiment-results) 4. [Directory Structure](#directory-structure) 5. [Prerequisites](#prerequisites) 6. [Build & Deploy](#build--deploy) 7. [Running the HFL Workflow](#running-the-hfl-workflow) 8. [Stored Procedures Reference](#stored-procedures-reference) 9. [Technical Details](#technical-details) 10. [Alternative Approaches Considered](#alternative-approaches-considered)

---

Architecture Overview

┌─────────────────────────────────────────────────┐
│ ORCHESTRATOR_ACCOUNT (Orchestrator) │
│ │
│ HFL_APP_DB.HFL_META │
│ - HFL_ROUNDS │
│ - HFL_GLOBAL_WEIGHTS │
│ - HFL_EVAL_RESULTS │
│ - EVAL_DATA_SAMPLE (10K) │
│ - Model Registry │
│ │
│ ┌──────────────────┐ ┌─────────────────┐ │
│ │AWS2_HFL_CLIENT_DB│ │AZ1_HFL_CLIENT_DB│ │
│ │ (Shared DB) │ │ (Shared DB) │ │
│ └────────┬─────────┘ └────────┬────────┘ │
└───────────┼─────────────────────┼───────────────┘
│ Private Listing │ Private Listing
│ (param receive) │ (param receive)
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ CLIENT_ACCOUNT_AWS │ │ CLIENT_ACCOUNT_AZURE │
│ (Clients C1–C5) │ │ (Clients C6–C10) │
│ │ │ │
│ HFL_APP_DB.HFL_LOCAL │ │ HFL_APP_DB.HFL_LOCAL │
│ TRAIN_DATA_SAMPLE_C1│ │ TRAIN_DATA_SAMPLE_C6│
│ TRAIN_DATA_SAMPLE_C2│ │ TRAIN_DATA_SAMPLE_C7│
│ TRAIN_DATA_SAMPLE_C3│ │ TRAIN_DATA_SAMPLE_C8│
│ TRAIN_DATA_SAMPLE_C4│ │ TRAIN_DATA_SAMPLE_C9│
│ TRAIN_DATA_SAMPLE_C5│ │ TRAIN_DATA_SAMPLE_C10│
│ HFL_CLIENT_UPDATES │ │ HFL_CLIENT_UPDATES │
│ │ │ │
│ GLOBAL_HFL_DB │ │ GLOBAL_HFL_DB │
│ (Shared from Orch.) │ │ (Shared from Orch.) │
└──────────────────────┘ └──────────────────────┘

Account Roles

| Account | Role | Region | |---|---|---| | ORCHESTRATOR_ACCOUNT | Orchestrator (aggregation, evaluation, Model Registry) | AWS (your region) | | CLIENT_ACCOUNT_AWS | Clients AWS2_C1–C5 | AWS (your region) | | CLIENT_ACCOUNT_AZURE | Clients AZ1_C6–C10 | Azure (your region) |

Client State Assignments

All 53 U.S. states/territories are distributed across 10 clients using greedy bin packing, balancing each client's training pool to ~28K–30K rows. Each client subsamples 1,000 rows from its pool.

| Client | States | Pool Size | Samples | |---|---|---|---| | AWS2_C1 | 1, 8, 32, 34, 49, 51 | 27,757 | 1,000 | | AWS2_C2 | 2, 13, 27, 37, 55 | 26,771 | 1,000 | | AWS2_C3 | 4, 6, 17, 50, 56, 66 | 27,968 | 1,000 | | AWS2_C4 | 5, 15, 26, 28, 40, 48 | 28,274 | 1,000 | | AWS2_C5 | 9, 16, 20, 30, 31, 35 | 28,647 | 1,000 | | AZ1_C6 | 10, 36 | 27,311 | 1,000 | | AZ1_C7 | 11, 21, 24, 38, 41, 45 | 28,267 | 1,000 | | AZ1_C8 | 12, 18, 29, 33, 42, 44 | 28,282 | 1,000 | | AZ1_C9 | 19, 22, 53, 54 | 26,866 | 1,000 | | AZ1_C10 | 23, 25, 39, 46, 72, 78 | 27,830 | 1,000 |

HFL Round Flow

Round 0 (Initialization)
Orchestrator: Insert R00_INIT (zero vector) → distribute to clients via Private Listing

Round N (N = 1, 2, 3)
1. CLIENT_AWS/CLIENT_AZURE: Fetch previous round's global weights from GLOBAL_HFL_DB
2. 10 clients: Train with warm_start on local data (1K each)
→ Store NONE + DP (noise_scale=0.05) updates in HFL_CLIENT_UPDATES
3. Orchestrator: Poll client weights via Private Listing shared DBs
4. Orchestrator: Aggregate via FedAvg → store in HFL_GLOBAL_WEIGHTS
5. Orchestrator: Register in Model Registry

Final
Orchestrator: SP_EVAL_MODELS evaluates all 12 models via ROC-AUC
- FEDERATED_NONE, FEDERATED_DP
- LOCAL_ONLY_{AWS2_C1..C5, AZ1_C6..C10}

---

Experiment Results

Logistic Regression — FedAvg

| Model | ROC-AUC | |---|---| | FEDERATED_NONE | 0.8118 | | FEDERATED_DP | 0.8106 | | LOCAL_ONLY_AWS2_C5 | 0.8059 | | LOCAL_ONLY_AZ1_C7 | 0.8045 | | LOCAL_ONLY_AZ1_C6 | 0.8039 | | LOCAL_ONLY_AZ1_C10 | 0.8019 | | LOCAL_ONLY_AZ1_C8 | 0.8018 | | LOCAL_ONLY_AWS2_C4 | 0.8010 | | LOCAL_ONLY_AWS2_C3 | 0.7994 | | LOCAL_ONLY_AZ1_C9 | 0.7973 | | LOCAL_ONLY_AWS2_C2 | 0.7910 | | LOCAL_ONLY_AWS2_C1 | 0.7890 |

• Aggregation: FedAvg (weighted average of coefficients + scaler params)

-...