Deepseek R1 Got Eyes

DeepSeek R1 Just Got Eyes with Fireworks AI Document Inlining

GLM 5.2 is live! Opus-level intelligence at open-source rates. Pay per token on serverless. Try it today.

Blog

Deepseek R1 Got Eyes DeepSeek R1 Just Got Eyes with Fireworks AI Document Inlining

PUBLISHED 2/5/2025

Table of Contents A Quick Recap of DeepSeek R1 Bridging the Gap: Document Inlining with Fireworks AI

How Does Document Inlining Work? Technical Deep Dive: API Integration

1. Imports and Dependencies

2. API Client Configuration

3. Constructing the API Request

4. Processing the Output Experimenting with Different Prompts

Top 5 Findings

Main Contributions

Prompt 2:

Key Benchmark Comparisons:

Distilled Models:

Key Strengths:

Limitations: Why This Matters to AI Engineers Conclusion

Table of Contents

A smart reasoning LLM is good, but a smart reasoning VLM is better! So let’s give DeepSeek R1 eyes. We’re excited to demonstrate how DeepSeek R1 , a state-of-the-art reasoning model from DeepSeek AI, can now process and reason over both text and image inputs using the Fireworks AI Document Inlining feature. This capability extends DeepSeek R1’s powerful reasoning to multimodal analysis, opening new avenues for research and application in AI. A Quick Recap of DeepSeek R1

DeepSeek R1 has been making waves in the AI research community, consistently performing at the top of industry benchmarks and rivaling even some of the most prominent closed-source models. DeepSeek R1, developed by DeepSeek AI, is a state-of-the-art reasoning model with a massive 671 billion parameter (671B) configuration. It has demonstrated top-tier performance across various benchmarks, positioning itself as a leading open-source alternative in the AI space. For this demo, we are running DeepSeek R1 on the Fireworks AI platform, where it operates at an impressive 100 tokens per second (100t/s) output speed. However, until now, the model's API was limited to textual reasoning—leaving a gap in scenarios that demand a visual understanding. Bridging the Gap: Document Inlining with Fireworks AI

While DeepSeek R1 excels in complex reasoning tasks, many practical use cases, such as document analysis, research paper summarization, and multimedia content understanding, require a multimodal approach. This is where Fireworks AI Document Inlining comes into play. With this new feature, you can effortlessly extend DeepSeek R1’s capabilities into the visual domain. How Does Document Inlining Work?

The concept is deceptively simple yet incredibly powerful: by appending #transform=inline to the end of your PDF or image URL, you enable the DeepSeek R1 model to inline the document. This transforms your text-based reasoning model into a fully capable vision-language model (VLM) without any additional overhead. For example, if you have a research paper URL like this: https://arxiv.org/pdf/2501.12948 simply modify it to: https://arxiv.org/pdf/2501.12948#transform=inline This small tweak instructs the model to process and inline the visual content of the document, empowering it to analyze both the textual and visual components seamlessly. Technical Deep Dive: API Integration

Below, we provide a detailed technical breakdown of how to integrate and utilize the DeepSeek R1 model with the Fireworks AI API for multimodal analysis. 1. Imports and Dependencies

Before making the API call, ensure you have the necessary imports: 1 2 3 4 5 import json from base64 import b64encode from fireworks import Fireworks from pydantic import BaseModel

2. API Client Configuration

Set up the Fireworks client by initializing it with your API key: 1 2 fireworks_client = Fireworks(api_key="YOUR_API_KEY")

3. Constructing the API Request

The request is designed to handle multimodal input, combining both text and images: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 response = client.chat.completions.create( model="model-deployment-name", messages=[ { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "https://arxiv.org/pdf/2501.12948#transform=inline" }, }, { "type": "text", "text": "Summarize the research paper with top 5 findings, and the main contributions of the paper" }, ], } ],

response_format={"type": "text"} )

4. Processing the Output

Finally, handle the API response: 1 2 print(response)

The printed output contains the summary generated by DeepSeek R1, which now leverages both its reasoning and newly acquired visual analysis capabilities to deliver a comprehensive understanding of the research paper. Experimenting with Different Prompts

We tested DeepSeek R1 using Fireworks AI Document Inlining with the following two prompts: 1 2 {"text": "Summarize the research paper with top 5 findings, and the main contributions of the paper"}

Result: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Okay, so I need to summarize this research paper. Let me start by reading through the user-provided content carefully. The paper is titled "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" by DeepSeek-AI. The main idea seems to be using reinforcement learning (RL) to enhance the reasoning capabilities of large language models (LLMs) without relying on supervised fine-tuning (SFT) initially.

First, the paper introduces two models: DeepSeek-R1-Zero and DeepSeek-R1. The former is trained using RL directly on the base model, while the latter incorporates some cold-start data and a multi-stage training pipeline. The authors claim that DeepSeek-R1-Zero achieves performance comparable to OpenAI's 01-0912 and that DeepSeek-R1 matches OpenAI-01-1217 on reasoning tasks. They also mention distilling these models into smaller versions, which performed well on benchmarks.

The key contributions mentioned are using RL without SFT, developing a pipeline for DeepSeek-R1, and showing that smaller models can be empowered through distillation. The evaluations cover various reasoning tasks, knowledge benchmarks, and other applications like creative writing.

Top findings probably include the effectiveness of RL in improving reasoning, the self-evolution observed in R1-Zero, the impact of cold-start data in enhancing readability and performance, comparison with existing models like OpenAI's, and the success of distillation. The main contributions would be the RL approach without SFT, the multi-stage training pipeline, distillation results, and releasing open-source models.

Now, I need to structure the summary with five...