RepoClarifaiClarifaipublished Oct 16, 2023seen 5d

Clarifai/clarifai-pyspark

Python

Open original ↗

Captured source

source ↗
GH
GitHub/github.com/Clarifai/clarifai-pyspark
Clarifai/clarifai-pyspark repository metadata
Source ↗
published Oct 16, 2023seen 5dcaptured 13hhttp 200method plain

Clarifai/clarifai-pyspark

Description: Interfaces for Unstructured data and ML pipelines with Databricks and Clarifai

Language: Python

License: Apache-2.0

Stars: 7

Forks: 2

Open issues: 2

Created: 2023-10-16T13:00:01Z

Pushed: 2024-11-11T13:19:56Z

Default branch: main

Fork: no

Archived: no

README:

ClarifaiPySpark

Introduction

This readme provides overview of the Software Development Kit (SDK) under development for integrating Clarifai with Databricks. The primary use case for this SDK is to facilitate the interaction between Databricks and Clarifai for tasks related to uploading client datasets, annotating data, and exporting and storing annotations in Spark DataFrames or Delta tables.

!Screenshot 2023-11-17 at 5 21 04 PM

The initial use case for this SDK revolves around three main objectives:

Uploading Client Datasets into Clarifai App:

The SDK should enable the seamless upload of datasets into the Clarifai application, simplifying the process of data transfer from Databricks to Clarifai.

Annotate the Data:

It should provide features for data annotation, making it easier for users to add labels and metadata to their datasets within the Clarifai platform.

Export Annotations to Spark DataFrames/Delta Tables:

The SDK should offer functionality to export annotations and store them in Spark DataFrames or Delta tables, facilitating further data analysis within Databricks.

Requirements:

  • Databricks : Runtime 13.3 or later
  • Clarifai : `` pip install clarifai
  • Create your Clarifai account
  • Follow the instructions to get your own Clarifai PAT
  • Protocol Buffers : version 4.24.2 pip install protobuf==4.24.2

Setup:

Install the package and initialize the clarifaipyspark class to begin.

pip install clarifai-pyspark

Getting Started:

from clarifaipyspark.client import ClarifaiPySpark

Create a Clarifai-PySpark client object to connect to your app on Clarifai. You can also choose the dataset or create one in your clarifai app to upload the data.

claps_obj = ClarifaiPySpark(user_id=USER_ID, app_id=APP_ID, pat=CLARIFAI_PAT)
dataset_obj = claps_obj.dataset(dataset_id=DATASET_ID)

Examples:

Checkout these notebooks for various operations you can perform using clarifai-pyspark SDK. | Notebook | Description | GitHub | |----------|--------|---------------- |

## If you want to enhance your AI journey with workflows and leveraging custom models (programmatically) our Clarifai SDK might be good place to start with. Please refer below resources for further references.