Apache Flink and DC/OS
For more than five years, DC/OS has enabled some of the largest, most sophisticated enterprises in the world to achieve unparalleled levels of efficiency, reliability, and scalability from their IT infrastructure. But now it is time to pass the torch to a new generation of technology: the D2iQ Kubernetes Platform (DKP). Why? Kubernetes has now achieved a level of capability that only DC/OS could formerly provide and is now evolving and improving far faster (as is true of its supporting ecosystem). That’s why we have chosen to sunset DC/OS, with an end-of-life date of October 31, 2021. With DKP, our customers get the same benefits provided by DC/OS and more, as well as access to the most impressive pace of innovation the technology world has ever seen. This was not an easy decision to make, but we are dedicated to enabling our customers to accelerate their digital transformations, so they can increase the velocity and responsiveness of their organizations to an ever-more challenging future. And the best way to do that right now is with DKP.
Apache Flink has come a long way. Having its roots in a research project back in 2010, it became an Apache Incubator project in March 2014 and later that year was promoted to an Apache Software Foundation top-level project.
The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala. Flink executes arbitrary dataflow programs in a data-parallel and pipelined manner, enabling the execution of batch and stream processing programs. It bundles with libraries for domain-specific use cases such as a complex event processing library, a Machine Learning library, as well as a graph processing API and library.
Increasingly we notice interest in the community to run Flink on DC/OS, in order to benefit from the dynamic partitioning capabilities DC/OS offers. Another reason for the desire to have Flink as a first-class citizen available in the Universe is that Flink typically works alongside other distributed systems such as Kafka and Cassandra, which are already available there.
The Mesos-Flink integration work, known as FLINK-1984 in the community, started more than a year ago with a considerable increase of activity in the recent time. Independently of this work carried out in the Flink community, a member of the DC/OS community recently created a JS-based Mesos framework for Flink.
Apache Flink community growth 2015 (Source: ASF).
Apache Flink has a bright future with a vibrant community and with data Artisans a commercial entity backing it as well as providing solutions and training around Flink. We're looking forward to the upcoming yearly Flink Forward conference in September taking place in Berlin, Germany and will update you on new developments in this space over the coming weeks.