UpstageAI/evalverse-EQBench

UpstageAI/evalverse-EQBench

Description: Submodule of evalverse forked from EQ-Bench/EQ-Bench

Language: Python

License: MIT

Stars: 0

Forks: 0

Open issues: 0

Created: 2024-05-09T08:22:25Z

Pushed: 2024-05-09T08:33:34Z

Default branch: main

Fork: no

Archived: yes

README: 

EQ-Bench

EQ-Bench is a benchmark for language models designed to assess emotional intelligence. You can read more about it in our paper.

The latest leaderboard can be viewed at EQ-Bench Leaderboard.

News

Version 2 Released

V2 of EQ-Bench contains 171 questions (compared to 60 in v1) and a new system for scoring. It is better able to discriminate performance differences between models. V2 is less subject to variance caused by perturbations (e.g. temp, sampler, quantisation, prompt format, system message). Also added is the ability to upload results to firebase.

We encourage you to move to v2, and to note which version you are using (EQ-Bench v1 or EQ-Bench v2) when publishing results to avoid confusion.

NOTE: V1 scores are not directly comparable to v2 scores.

More v2 details

Version 2 of the benchmark brings three major changes:

1. Increased the number of test questions from 60 to 171. 2. Changed the scoring system from _normalised_ to _full scale_. 3. Uploading results to firebase.

Known issues:

• When using oobabooga as the inferencing engine, the api plugin stops responding after approx. 30 queries. This is handled by the benchmark pipeline by the query timing out (according to the value set in config.cfg), and then reloading ooba. The cause is unknown at this stage; the benchmark should however still complete.

Score sensitivity to perturbations

Originally 200 dialogues were generated for the test set, of which 60 of the best (most coherent & challenging) were selected for v1 of the benchmark. We had initially established very low variance between runs of the v1 benchmark, when holding all parameters the same. However it has become apparent that minor perturbations to the model or inferencing parameters can cause score variance beyond what is explained by the actual change in performance.

Traditional multiple choice tests are less prone to this kind of variance because these perturbations are unlikely to change an answer from "A" to "B". In contrast, EQ-Bench questions require a subjective prediction of emotional intensity on a range of 0-10. Small perturbations to the model or inferencing params can produce significantly different numerical predictions. This is a source of noise that can be mitigated by increasing the number of questions. So for v2 we opted to expand the test set to 171 out of the originally generated 200.

We tested v1 against v2 for a number of models, while controlling a range of parameters (temp, sampler, quantisation, prompt format, system message). We find v2 scores to be significantly more stable to perturbations to these variables, and so we expect the scores to be more closely representative of the true performance of the model.

Scoring system changes

In v1 of EQ-Bench we elected to normalise the four emotional intensity ratings in each question to sum to 10. The reasoning for this was that different subjects might have different ideas about, for example, what constitutes a _10_ rating. Given the subjectivity here, multiple perspectives can be valid.

A systematic bias in how the subject rates emotional intensity might correlate with a similar systematic bias in the creators of the reference answers, resulting in an artificially inflated score. So to eliminate this issue we normalised both the reference answer and the subject answer so that we are only comparing the _relative_ intensity of each emotion.

This seemed like a good idea at the time, however normalising in this way is far from a perfect solution. It handles certain edge cases poorly, and several models benchmarked with numbers that were significant outliers compared to other major benchmarks (E.g. Mixtral 8x7 produced unusually low scores). In addition, normalising the answers means we are losing the ability to assess the model's ability to make reasonable predictions about the absolute intensity of emotions.

In v2 we opted for a different approach: We still calculate the score by computing the difference from the reference answer, however, we no longer normalise the values. To mitigate the subjective nature of rating emotional intensity, we scale down smaller differences (differences between 1-4 from reference) on a curve. Differences from 5 to 10 are counted 1:1.

The result of these changes is better discriminative ability of the benchmark, and generally slightly higher scores compared to v1. As with v1, the score baseline is calibrated so that a score of 0 corresponds to answering randomly, and a score of 100 matches the reference answers exactly.

Version 1.1 Released

This version adds support for Oobabooga. The benchmark pipeline can automatically download each model, launch the model with ooba using the specified parameters, and close the ooba server after the run completes, optionally deleting the model files.

Requirements

• Linux
• Python 3x
• Working install of Oobabooga (optional)
• Sufficient GPU / System RAM to load the models
• Python libraries listed in install_reqs.sh

Show python libraries

EQ-bench requirements

• tqdm
• sentencepiece
• hf_transfer
• openai
• scipy
• torch
• peft
• bitsandbytes
• transformers (preferably the latest version installed directly from GitHub: huggingface/transformers)
• trl
• accelerate
• tensorboardX
• huggingface_hub

Requirements for QWEN models

• einops
• transformers_stream_generator (version 0.0.4)
• deepspeed
• tiktoken
• flash-attention (the latest version installed directly from GitHub: Dao-AILab/flash-attention)
• auto-gptq
• optimum

Requirements for uploading results

• gspread
• oauth2client
• firebase_admin

Installation

Quick start:

If you are installing EQ-Bench into a fresh linux install (like a runpod or similar), you can run ooba_quick_install.sh. This will install oobabooga into the current user's home directory, install all EQ-Bench dependencies, then run the benchmark pipeline.

Install (not using quick start)

Note: Ooobabooga is optional. If you prefer to use…