Apache Flink and DC/OS

Current developments around Apache Flink in the context of DC/OS.

Aug 02, 2016

Michael Hausenblas


Apache Flink has come a long way. Having its roots in a research project back in 2010, it became an Apache Incubator project in March 2014 and later that year was promoted to an Apache Software Foundation top-level project.


The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala. Flink executes arbitrary dataflow programs in a data-parallel and pipelined manner, enabling the execution of batch and stream processing programs. It bundles with libraries for domain-specific use cases such as a complex event processing library, a Machine Learning library, as well as a graph processing API and library.


Increasingly we notice interest in the community to run Flink on DC/OS, in order to benefit from the dynamic partitioning capabilities DC/OS offers. Another reason for the desire to have Flink as a first-class citizen available in the Universe is that Flink typically works alongside other distributed systems such as Kafka and Cassandra, which are already available there.


The Mesos-Flink integration work, known as FLINK-1984 in the community, started more than a year ago with a considerable increase of activity in the recent time. Independently of this work carried out in the Flink community, a member of the DC/OS community recently created a JS-based Mesos framework for Flink.


Apache Flink community growth 2015 (Source: ASF). Apache Flink community growth 2015 (Source: ASF).


Apache Flink has a bright future with a vibrant community and with data Artisans a commercial entity backing it as well as providing solutions and training around Flink. We're looking forward to the upcoming yearly Flink Forward conference in September taking place in Berlin, Germany and will update you on new developments in this space over the coming weeks.

Ready to get started?