cohere-ai/sandbox-multilingual

cohere-ai/sandbox-multilingual

Description: A demonstration of a multilingual semantic search engine you can be quickly built using Cohere's platform.

Language: Python

License: MIT

Stars: 63

Forks: 5

Open issues: 3

Created: 2022-12-05T11:59:50Z

Pushed: 2023-07-25T20:53:09Z

Default branch: main

Fork: no

Archived: yes

README:

################################################################################
# ____ _ ____ _ _ #
# / ___|___ | |__ ___ _ __ ___ / ___| __ _ _ __ __| | |__ _____ __ #
# | | / _ \| '_ \ / _ \ '__/ _ \ \___ \ / _` | '_ \ / _` | '_ \ / _ \ \/ / #
# | |__| (_) | | | | __/ | | __/ ___) | (_| | | | | (_| | |_) | (_) >

(where ` is the key you obtained, without the ` brackets).

Alternatively, you can pass COHERE_TOKEN= as an additional argument to any make command below.

Building an index

Follow these steps to first build a semantic index of article titles. These could steps could be adapted to any dataset.

Step 1: Get the data

First, download the BBC articles by running one of the following commands.

If you want to get started quickly, use

make download-bbc-news

This will download BBC news articles for 45 languages and save it as csv inside ./data/

Step 2: Process the text into a index of embeddings (representations)

Once you have some text, we need to process it into a search index of embeddings and addresses.

This can be done by using the command

make embeddings

assuming your target csv is under the ./data/ directory.

The command will search the ./data/ directory recursively for files with a .csv extension, and build a simple database of the embeddings, file name and line number of each paragraph.

Warning: If you have a lot of text to search, this can take a little while to finish!

Step 3: Build and launch the search engine

Once you have an embeddings.npz file built, you can use the following command to build a docker image which will serve a simple REST app to allow you to query the database you have made:

make build

You can then start the server using

make run

This is slightly overkill for a simple example, but it's designed to reflect the fact that building an index of a large body of text is relatively slow, and ensures that querying the engine is fast.

If you want to use this project as a building block for a real application, it is likely that you will want to maintain your database of text embeddings in a server architecture and query it with a lightweight client. Packaging the server as a docker application means that it is very simple to turn this into a 'real' application by deploying it to a cloud service.

Step 4: Query your search engine

If you open a new terminal window for any of the options below, remember to run

export COHERE_TOKEN=

Via a viewer script

By far the easiest option is to run our helper script:

scripts/search.sh "My query here in my preferred language searching for news"

to query the database. The script takes an optional second argument specifying the number of desired results.

The script pops up a modified vim interface, with the following commands:

• Press q to quit.
• Press the UP or LEFT arrow to page up in the list of results (show in the bottom pane)
• Press the DOWN or RIGHT arrow to page down in the list of results

The top pane will show you the position in the document where the result is found.

Via a REST API

Once the server is running, you can query it using a simple REST api. You can explore the API directly by going to /docs#/default/search_search_post here. It's a simple JSON REST API; here's how you can ask a query using curl:

curl -X POST -H "Content-Type: application/json" -d '{"query": "Artifical Intelligence", "num_results": 3}' http://localhost:8080/search

This will return a JSON list of length num_results, each with the filename and line-number (doc_url and block_url) of the blocks that were the closest semantic match to your query. But you probably want to actually just read the bit of the files that's the best answer.

Via vim

As we are searching through local text files, it's actually a bit easier to parse the output using command line tools; use the provided python script utils/query_server.py to query it on the command line. query_server.py prints out the results in the standard file_name:line_number: format, so we can page through the actual results in a nice way be leveraging vim's quickfix mode.

Assuming you have vim on your machine, you can simply

vim +cw -M -q <(python utils/query_server.py "my_query" --num_results 3)

to get vim to open the indexed text files at the locations returned by the search algorithm. (use :qall to close both the window and the quickfix navigator). You can cycle through the returned results using :cn and :cp. The results aren't perfect; it's semantic search, so you would expect the matching to be a bit fuzzy. Despite this, I often find you can get the answer to your question in the first few results, and using Cohere's API lets you express your question in natural language, and let's you build a suprisingly effective search engine in just a few lines of code.

Example queries

Some good-to-try queries that show the search working well on generic, natural language questions are:

• How improving children's diets can aid development
• كيف تتقن مهارة جديدة في وقت قياسي؟
• هل يفرض سد النهضة الإثيوبي واقعا جديدا على مصر؟
• WhatsApp: comment l'application gagne de l'argent si son service est gratuit pour la plupart des utilisateurs ?
• 米アップル、動画ストリーミングサービスを発表 クレジットカードやゲームサービスも
• グーグル、メインロゴを刷新

Algorithm Details

This repo uses a very simple strategy to index a document, and search for the best match. First, it breaks up every document into paragraphs, or 'blocks'. Then, it calls co.embed on each paragraph, in order to generate a vector embedding using Cohere's language model. It then stores each embedding vector, along with the corresponding document and line number of the paragraph, in a simple array as a 'database'.

In order to actually do the search, we use the FAISS similarity search library. When we get a query, we use the same Cohere API call to embed the query. We then use FAISS to find the top $n$ results closest to the query,…