cohere-ai/llama-recipes

cohere-ai/llama-recipes

Description: Examples and recipes for Llama 2 model

Language: Python

License: NOASSERTION

Stars: 1

Forks: 1

Open issues: 1

Created: 2023-08-04T14:40:54Z

Pushed: 2024-07-03T11:54:54Z

Default branch: main

Fork: yes

Parent repository: meta-llama/llama-cookbook

Archived: no

README:

Llama 2 Fine-tuning / Inference Recipes and Examples

The 'llama-recipes' repository is a companion to the Llama 2 model. The goal of this repository is to provide examples to quickly get started with fine-tuning for domain adaptation and how to run inference for the fine-tuned models. For ease of use, the examples use Hugging Face converted versions of the models. See steps for conversion of the model [here](#model-conversion-to-hugging-face).

Llama 2 is a new technology that carries potential risks with use. Testing conducted to date has not — and could not — cover all scenarios. In order to help developers address these risks, we have created the Responsible Use Guide. More details can be found in our research paper as well. For downloading the models, follow the instructions on Llama 2 repo.

Table of Contents

1. [Quick start](#quick-start) 2. [Model Conversion](#model-conversion-to-hugging-face) 3. [Fine-tuning](#fine-tuning)

• [Single GPU](#single-gpu)
• [Multi GPU One Node](#multiple-gpus-one-node)
• [Multi GPU Multi Node](#multi-gpu-multi-node)

4. [Inference](./docs/inference.md) 5. [Repository Organization](#repository-organization) 6. [License and Acceptable Use Policy](#license)

Quick Start

[Llama 2 Jupyter Notebook](./examples/quickstart.ipynb): This jupyter notebook steps you through how to finetune a Llama 2 model on the text summarization task using the samsum. The notebook uses parameter efficient finetuning (PEFT) and int8 quantization to finetune a 7B on a single GPU like an A10 with 24GB gpu memory.

Installation

Llama-recipes provides a pip distribution for easy install and usage in other projects. Alternatively, it can be installed from source.

Install with pip

pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes

Install from source

To install from source e.g. for development use this command. We're using hatchling as our build backend which requires an up-to-date pip as well as setuptools package.

pip install -U pip setuptools
pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 -e .

For development and contributing to llama-recipes please install all optional dependencies:

pip install -U pip setuptools
pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 -e .[tests,auditnlg,vllm]

Install with optional dependencies

Llama-recipes offers the installation of optional packages. There are three optional dependency groups. To run the unit tests we can install the required dependencies with:

pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes[tests]

For the vLLM example we need additional requirements that can be installed with:

pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes[vllm]

To use the sensitive topics safety checker install with:

pip install --extra-index-url https://download.pytorch.org/whl/test/cu118 llama-recipes[auditnlg]

Optional dependencies can also be combines with [option1,option2].

⚠️ Note ⚠️ Some features (especially fine-tuning with FSDP + PEFT) currently require PyTorch nightlies to be installed. Please make sure to install the nightlies if you're using these features following this guide.

Note All the setting defined in [config files](src/llama_recipes/configs/) can be passed as args through CLI when running the script, there is no need to change from config files directly.

Note In case need to run PEFT model with FSDP, please make sure to use the PyTorch Nightlies.

For more in depth information checkout the following:

• [Single GPU Fine-tuning](./docs/single_gpu.md)
• [Multi-GPU Fine-tuning](./docs/multi_gpu.md)
• [LLM Fine-tuning](./docs/LLM_finetuning.md)
• [Adding custom datasets](./docs/Dataset.md)
• [Inference](./docs/inference.md)
• [FAQs](./docs/FAQ.md)

Where to find the models?

You can find llama v2 models on HuggingFace hub here, where models with hf in the name are already converted to HuggingFace checkpoints so no further conversion is needed. The conversion step below is only for original model weights from Meta that are hosted on HuggingFace model hub as well.

Model conversion to Hugging Face

The recipes and notebooks in this folder are using the Llama 2 model definition provided by Hugging Face's transformers library.

Given that the original checkpoint resides under models/7B you can install all requirements and convert the checkpoint with:

## Install HuggingFace Transformers from source
pip freeze | grep transformers ## verify it is version 4.31.0 or higher

git clone git@github.com:huggingface/transformers.git
cd transformers
pip install protobuf
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path

Fine-tuning

For fine-tuning Llama 2 models for your domain-specific use cases recipes for PEFT, FSDP, PEFT+FSDP have been included along with a few test datasets. For details see [LLM Fine-tuning](./docs/LLM_finetuning.md).

Single and Multi GPU Finetune

If you want to dive right into single or multi GPU fine-tuning, run the examples below on a single GPU like A10, T4, V100, A100 etc. All the parameters in the examples and recipes below need to be further tuned to have desired results based on the model, method, data and task at hand.

Note:

• To change the dataset in the commands below pass the dataset arg. Current options for integrated dataset are grammar_dataset, alpaca_datasetand samsum_dataset. A description of how to use your own dataset and how to add custom datasets can be found in [Dataset.md](./docs/Dataset.md#using-custom-datasets). For grammar_dataset, alpaca_dataset please make sure you use the suggested instructions from…