Snowflake-Labs/nifihub

Snowflake-Labs/nifihub

Language: Python

License: Apache-2.0

Stars: 3

Forks: 211

Open issues: 5

Created: 2026-05-22T07:35:44Z

Pushed: 2026-06-05T21:14:08Z

Default branch: main

Fork: no

Archived: no

README:

NiFi Hub

A community-driven repository of independently-versioned Apache NiFi extension bundles and versioned flow definitions.

> Disclaimer: The contents of this repository are community-driven and provided on an "AS IS" basis, without warranties of any kind, express or implied. They are NOT supported by Snowflake and do not constitute a Snowflake product or service. Snowflake makes no guarantees regarding functionality, compatibility, availability, or fitness for any particular purpose, and assumes no liability arising from use of this repository. Community support is available through GitHub Issues and Pull Requests.

Table of Contents

• [Overview](#overview)
• [Repository Structure](#repository-structure)
• [Getting Started](#getting-started)
• [Creating a New Bundle](#creating-a-new-bundle)
• [Creating a New Flow](#creating-a-new-flow)
• [Quality Gates](#quality-gates)
• [Releases](#releases)
• [Environments (CD)](#environments-cd)
• [CI/CD Workflows](#cicd-workflows)
• [Technology Stack](#technology-stack)
• [License](#license)

Overview

NiFi Hub provides:

• Extension Bundles: Reusable NiFi processors, controller services, and reporting tasks packaged as NARs
• Flow Definitions: Versioned NiFi flow definitions with documentation and automated validation
• Strict Quality Gates: Checkstyle, PMD, license header checks, and 80% code coverage enforced on every PR
• Independent Versioning: Each bundle has its own version and release lifecycle
• Agent-Friendly: Every bundle and flow includes machine-readable documentation for AI coding assistants

Repository Structure

nifihub/
├── extensions/ # NiFi extension bundles
│ └── /
│ └── /
│ └── nifi--bundle/
│ ├── pom.xml # Bundle aggregator POM
│ ├── SKILL.md # Agent-readable documentation
│ ├── nifi--processors/
│ └── nifi--nar/
├── environments/ # Openflow-as-Code environment configs
│ ├── schema.json # JSON Schema for validation
│ └── /
│ └── config.yaml # Deployments, runtimes, and flows
├── flows/ # NiFi flow definitions
│ └── /
│ ├── .json # Flow definition
│ ├── .md # Flow documentation
│ └── tests/
│ ├── test_.py # Validation + runtime tests
│ └── test_.yaml # CI runtime config
├── scripts/cd/ # CD orchestration scripts
├── scripts/ci/ # CI ephemeral runtime scripts
├── pom.xml # Root parent POM
├── checkstyle.xml # Checkstyle rules (NiFi conventions)
├── pmd-ruleset.xml # PMD rules (NiFi conventions)
└── codecov.yml # Coverage configuration

Getting Started

Recommended Approach

The recommended way to use NiFi Hub is to fork this repository into your organization (public or private). Your fork is where you make environment-specific changes (account configs, custom flows, proprietary extensions), while contributions to improve the framework itself, submit reusable flows, or add extensions are welcome back to this upstream repository via Pull Requests.

GitHub Environment Setup

The CD pipeline requires a GitHub Environment with the following secrets and variables:

| Name | Type | Purpose | |------|------|---------| | SNOWFLAKE_PAT | Secret | Programmatic Access Token for Snowflake SQL operations (CREATE/ALTER/DROP deployment, runtime, connectors, etc.) | | NIFI_RUNTIME_PAT | Secret | Programmatic Access Token for the NiFi runtime REST API (flow checkout, parameters, controller services) | | NIFIHUB_REGISTRY_PAT | Secret | GitHub Personal Access Token with repo read access, injected into Flow Registry Clients so NiFi can pull flow definitions | | SNOWFLAKE_ACCOUNT_URL | Variable | Snowflake account URL (e.g. https://myorg-myaccount.snowflakecomputing.com) | | SNOWFLAKE_USER | Variable | Snowflake user for PAT-based authentication | | SNOWFLAKE_ROLE | Variable | Role used for SQL operations (e.g. OPENFLOW_ADMIN) |

Additional secrets/variables referenced via ${{ secrets.* }} or ${{ vars.* }} in config.yaml are resolved from the same GitHub Environment at deploy time.

Prerequisites

• Java 21+
• Python 3.12+ (for flow tests)
• Git

Maven is provided via the included wrapper (./mvnw).

Building an Extension Bundle

Each bundle builds independently from its own directory:

# Build and test
./mvnw clean verify -f extensions/examples/getting-started/nifi-example-bundle/pom.xml

# Build with all quality checks (checkstyle + PMD + license headers)
./mvnw clean verify -Pcontrib-check -f extensions/examples/getting-started/nifi-example-bundle/pom.xml

# Build with code coverage report
./mvnw clean verify -Preport-code-coverage -f extensions/examples/getting-started/nifi-example-bundle/pom.xml

Running Flow Tests

pip install -r flows/requirements.txt
pytest flows/examples/tests/ -v

Creating a New Bundle

Use the provided scaffold script:

./create-bundle.sh
# Example: ./create-bundle.sh data snowflake nifi-snowflake

Or manually create the directory structure following the [example bundle](extensions/examples/getting-started/nifi-example-bundle/) as a template.

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed instructions.

Creating a New Flow

1. Export your flow definition as JSON from NiFi 2. Place it at flows//.json 3. Create a companion flows//.md with documentation 4. Add validation tests at flows//tests/test_.py

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed instructions.

Quality Gates

All pull requests must pass these checks before merging:

Extension Bundles

• Checkstyle: NiFi coding conventions (200 char lines, no star imports, etc.)
• PMD: Static analysis rules aligned with Apache NiFi
• Apache RAT: License header verification on all source files
• Unit Tests: All tests pass via Maven Surefire
• Code Coverage: Minimum 80% line coverage (enforced via Codecov)

Flow Definitions

• Flow Diff: Human-readable diff of flow changes (via Snowflake Flow Diff)
• Flow Checkstyle: Best practice violations (concurrent tasks, self-loops, backpressure)
• Validation Tests: Structural validation via pytest + nipyapi

###…