databricks/vector

databricks/vector

Description: A high-performance observability data pipeline.

Language: Rust

License: MPL-2.0

Stars: 2

Forks: 3

Open issues: 6

Created: 2024-01-26T23:26:48Z

Pushed: 2024-10-04T00:22:38Z

Default branch: master

Fork: yes

Parent repository: vectordotdev/vector

Archived: yes

README:

Quickstart • Docs • Guides • Integrations • Chat • Download • Rust Crate Docs

What is Vector?

Vector is a high-performance, end-to-end (agent & aggregator) observability data pipeline that puts you in control of your observability data. [Collect][docs.sources], [transform][docs.transforms], and [route][docs.sinks] all your logs and metrics to any vendors you want today and any other vendors you may want tomorrow. Vector enables dramatic cost reduction, novel data enrichment, and data security where you need it, not where it is most convenient for your vendors. Additionally, it is open source and up to 10x faster than every alternative in the space.

To get started, follow our [quickstart guide][docs.quickstart] or [install Vector][docs.installation].

Principles

• Reliable - Built in [Rust][urls.rust], Vector's primary design goal is reliability.
• End-to-end - Deploys as an [agent][docs.roles#agent] or [aggregator][docs.roles#aggregator]. Vector is a complete platform.
• Unified - [Logs][docs.data-model.log], [metrics][docs.data-model.metric] (beta), and traces (coming soon). One tool for all of your data.

Use cases

• Reduce total observability costs.
• Transition vendors without disrupting workflows.
• Enhance data quality and improve insights.
• Consolidate agents and eliminate agent fatigue.
• Improve overall observability performance and reliability.

Community

• Vector is relied on by startups and enterprises like Atlassian, T-Mobile,

Comcast, Zendesk, Discord, Fastly, CVS, Trivago, Tuple, Douban, Visa, Mambu, Blockfi, Claranet, Instacart, Forcepoint, and [many more][urls.production_users].

• Vector is downloaded over 100,000 times per day.
• Vector's largest user processes over 30TB daily.
• Vector has over 100 contributors and growing.

Documentation

About

• [Concepts][docs.about.concepts]
• [Under the hood][docs.about.under-the-hood]
• [Architecture][docs.under-the-hood.architecture] - [data model][docs.architecture.data-model] ([log][docs.data-model.log], [metric][docs.data-model.metric]), [pipeline model][docs.architecture.pipeline-model], [concurrency model][docs.architecture.concurrency-model], [runtime model][docs.architecture.runtime-model]
• [Networking][docs.under-the-hood.networking] - [ARC][docs.networking.adaptive-request-concurrency]
• [Guarantees][docs.under-the-hood.guarantees]

Setup

• [Quickstart][docs.setup.quickstart]
• [Installation][docs.setup.installation] - [operating systems][docs.installation.operating_systems], [package managers][docs.installation.package_managers], [platforms][docs.installation.platforms] ([Kubernetes][docs.platforms.kubernetes]), [manual][docs.installation.manual]
• [Deployment][docs.deployment] - [roles][docs.deployment.roles], [topologies][docs.deployment.topologies]

Reference

• Configuration
• [Sources][docs.configuration.sources] - [docker_logs][docs.sources.docker_logs], [file][docs.sources.file], [http][docs.sources.http], [journald][docs.sources.journald], [kafka][docs.sources.kafka], [socket][docs.sources.socket], and [many more...][docs.sources]
• [Transforms][docs.configuration.transforms] - [dedupe][docs.transforms.dedupe], [filter][docs.transforms.filter], [geoip][docs.transforms.geoip], [log_to_metric][docs.transforms.log_to_metric], [lua][docs.transforms.lua], [remap][docs.transforms.remap], and [many more...][docs.transforms]
• [Sinks][docs.configuration.sinks] - [aws_cloudwatch_logs][docs.sinks.aws_cloudwatch_logs], [aws_s3][docs.sinks.aws_s3], [clickhouse][docs.sinks.clickhouse], [elasticsearch][docs.sinks.elasticsearch], [gcp_cloud_storage][docs.sinks.gcp_cloud_storage], and [many more...][docs.sinks]
• [Unit tests][docs.configuration.tests]
• [Remap Language][docs.reference.vrl]
• [API][docs.reference.api]
• [CLI][docs.reference.cli]

Administration

• [Management][docs.administration.management]
• [Monitoring & observing][docs.administration.monitoring]
• [Upgrading][docs.administration.upgrading]
• [Validating][docs.administration.validating]

Resources

• [Community][urls.vector_community] - [chat][urls.vector_chat], [calendar][urls.vector_calendar], [@vectordotdev][urls.vector_twitter]
• [Releases][urls.vector_releases]
• Policies - [Code of Conduct][urls.vector_code_of_conduct], [Privacy][urls.vector_privacy_policy], [Releases][urls.vector_releases_policy], [Security][urls.vector_security_policy], [Versioning][urls.vector_versioning_policy]

Comparisons

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF | |-------------------------------------------------------------------------------------------------------------------:|:---------------:|:--------:|:---------------:|:---------:|:---------:|:---------------:|:--------:| | TCP to Blackhole | _86mib/s_ | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a | | File to TCP | _76.7mib/s_ | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s | | Regex Parsing | 13.2mib/s | n/a | _20.5mib/s_ | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s | | TCP to HTTP | _26.7mib/s_ | n/a | 19.6mib/s | Developed with ❤️ by Datadog - Security Policy - Privacy Policy

[docs.about.concepts]: https://vector.dev/docs/about/concepts/ [docs.about.under-the-hood]: https://vector.dev/docs/about/under-the-hood/ [docs.administration.monitoring]:…