Topic of Kafka

Kafcache: Memcached + Kafka Streams

Introducing Kafcache a Memcached Kafka state store bridge for processing topologies in which low-latency matters. At TokenAnalyst we take low-latency seriously and use Memcached for our machine learning model application, for instance labelling, but also for lookup intensive data transformations. [...]

Tunnel your Kafka Cluster from AWS

Tunneling your AWS Kafka topics to your local developer box for fun and profit: Experimenting with real data is way more fun, plus, it decreases the turn-over if the developer has to cope with complex data. In the latter case, one can easily consume data and build test fixtures for a new stream processor, or test against real data directly (fail fast). [...]

Exposing Remote JMX in your Kafka Setup

This article is missing documentation of how to setup JMX to be used remotely. Proper monitoring and alerting is essential. It allows developers to understand the system and its subsystem by numbers. JMX to Graphite or Prometheus bridges exist, however, one might try to avoid putting these adapters on the same machine to de-couple the monitoring from the actual service. [...]

Kafka Streams in Scala with Schema Registry

In this example, Ill use the new Scala API which was released in Kafka 2.0 with Confluents schema registry and Avro4s to convert a GenericRecord into case class. I couldnt find anything on the net when working on this, so I might as well provide an example here. Further, Circe and Avro4s are great to also create JSON schema files programmatically. [...]

Using Monix with Kafka, Avro and Schema Registry

In this brief tech article, Ill show how to add a serializer, deserializer to monix-kafka to work with Confluents schema registry. This is important to ensure out-of-the-box KSQL and Kafka connect compatibility when integrating Monix microservices. [...]

Monitoring Kafka Streams Metrics via JMX

Ill present a way to access the Kafka Streams metrics using the command-line application jmxterm, and Ill give insights about the semantics of these metrics. At the end, I listed valueable resources related to Monitoring and Kafka Streams. [...]

Released Mocked Streams for Apache Kafka

I wrote a little helper library Mocked Streams in Scala, which allows you to create lightweight parallelizable unit-tests for your topologies without running a full Kafka cluster neither an embedded one. [...]

Stateful Streaming in Spark and Kafka Streams

This article is about aggregates in stateful stream processing in general. I write about the differences between Apache Spark and Apache Kafka Streams along concrete code examples. Further, I list the requirements which we might like to see covered by a stream processing framework. [...]

Testing Topologies in Kafka Streams

I wrote a little helper library Mocked Streams in Scala, which allows you to create lightweight parallelizable unit-tests for your Kafka Streams topologies without running a full Kafka cluster neither an embedded one. [...]

Processing Tweets with Kafka Streams

Ill present an example application for Kafka Streams which computes word counts in tumbling windows of tweets fetched from Twitters Streaming API [...]