I wrote a little helper library Mocked Streams in Scala, which allows you to create lightweight parallelizable unit-tests for your topologies without running a full Kafka cluster neither an embedded one.

Kafka Streams is a deployment-agnostic stream processing library written in Java. Even though Kafka has a great test coverage, there is no easy way to write unit-tests for processing topologies, until now. I released Mocked Streams for Scala, which simply allows you to unit-test multi-input and output stateful stream topologies (since Apache Kafka >= 0.10.1) without Zookeeper and Kafka Brokers in your favorite Scala testing framework e.g. ScalaTest and Specs2.

import com.madewithtea.mockedstreams.MockedStreams

val input = Seq(("x", "v1"), ("y", "v2"))
val expe = Seq(("x", "V1"), ("y", "V2"))
val strings = Serdes.String()

MockedStreams()
  .topology { builder => builder.stream(...) [...] }
  .input("topic-in", strings, strings, input)
  .output("topic-out", strings, strings, expe.size) shouldEqual expe

Mocked Streams

Mocked Streams 1.0 is a library which allows you to do the latter without much boilerplate code and in your favourite Scala testing framework. It wraps the org.apache.kafka.test.ProcessorTopologyTestDriver class, but adds more syntactic sugar to keep your test code simple. The features of the 1.0 release include:

• Multiple input and output streams
• Specify your topologies as a function: (KStreamBuilder => Unit)
• Use .config to pass your Kafka Streams configuration (e.g. Timestamp extractor)
• Use .outputTable to match against the compacted table output as Map[K,V]
• Support for Scala 2.11.8
• Support for Apache Kafka 0.10.1.0

Multiple Inputs and Outputs

val ms = MockedStreams()
  .topology { builder => builder.stream(...) [...] }
  .input("in-a", strings, ints, inputA)
  .input("in-b", strings, ints, inputB)
  .stores(Seq("store-name"))

ms.output("out-a", strings, ints, expA.size) shouldEqual(expectedA)
ms.output("out-b", strings, ints, expB.size) shouldEqual(expectedB)

Getting Started

See the README for using Mocked Streams. For examples check out the test code of Mocked Streams itself. Mocked Streams is also located in the Maven Central Repository, hence you just need to add:

libraryDependencies += "com.madewithtea" %% "mockedstreams" % "1.0.0" % "test"

If you have any questions or issues regarding Mocked Streams, get in touch via Github. Alternatively, you can get in contact via mail.