VAST Event Broker 5.5 adds JSON-to-table expansion, mTLS and 950MB messages

Topics stored in a database, not in log segments

The VAST Event Broker first shipped with VAST 5.4. According to Jason Russler, Technical Director at VAST, it is a Kafka implementation built on the company's Disaggregated Shared-Everything (DASE) architecture. Russler wrote the announcement post.

Traditional Kafka implementations use a shared-nothing design, in which topic data is written to log segments on local storage and partition replicas provide resilience and high availability. VAST takes a different route and commits messages to the VAST DataBase. The company describes this as a flexible and scalable presentation layer for structured and vector data, built on a columnar table format with partitioning, sorting, vector clustering and auto-defragmentation.

VAST says DataBase tables incur very little write amplification, support full high availability and make scaling easy because no rebalancing is needed. Messages committed this way can also be queried directly by the VAST Native Query Engine, Trino, Spark and Python, in addition to being consumed through Kafka protocols. Data can be processed by partition, offsets, timestamps, message headers and the messages themselves.

Blob expansion turns JSON streams into tables

The headline feature of 5.5 is called blob expansion, also referred to as Topics to Tables. It parses JSON messages entering a topic, flattens them if they contain nesting, and writes them as structured data to a DataBase table in parallel with the topic itself.

The process is synchronous. The acknowledgement to the producer is not sent until the message has been committed to both the topic and the table.

According to VAST, this lets users apply arbitrary data optimizations to the expansion table based on the message schema:

• The table can be partitioned, sorted or both.
• The selected fields can be a subset of those in the message.
• The table can also be a superset of the message schema, to be filled in later through merge or update operations.

In practice, VAST says, streaming JSON data becomes immediately queryable once it is produced to the brokers, and the system can handle these expansions at high throughput.

Performance, security and message size

Performance was again a focus of the release. VAST says Event Broker 5.5 is faster than any other broker available and represents a major improvement over VAST 5.4, where the figure was 2.2 million messages per second per broker. The company measures maximum sustained throughput per cluster node for a topic with a read-amplification factor of 1, using 1KiB messages. The result is a single rate assumed to be half reads and half writes, so 2GB/s equals 1GB/s read plus 1GB/s write. The updated figure appears in a chart in the original post and is not given in the text.

VAST argues that higher throughput means more value from hardware, support for more consumer groups on a single topic and more operational headroom. It also says that expanding a cluster only requires adding nodes. Because DASE removes the idea of a node owning data, all partitions are immediately available without rebalancing.

On security, the release adds mTLS authentication so that clients can prove their identity to the broker. VAST notes that mTLS is popular in hardened zero-trust environments, IoT ecosystems and microservice architectures.

The maximum message size has been raised to 950MB to ensure interoperability with existing Apache environments. VAST also lists internal updates that improve latency and monitoring of broker topics.

What it means for users

For teams running event-driven workflows, the main change is that JSON streams can be queried as structured tables without a separate pipeline, according to VAST. Organizations with strict identity requirements gain mTLS, and the larger message limit is meant to ease moves from existing Apache Kafka setups. The performance claims come from VAST itself, and the company invites interested customers to contact it or schedule a demo.