Introduction
At the beginning, we used to cluster our Spark applications using Apache Yarn as our main Resource Manager. At that time we considered an RM that was more like a “Spark extension,” which was basically used to optimize Spark processes and nothing more. Our usage of Yarn then never went beyond deploying those applications, such as monitoring them via web browser by typing something like http://yarn.url:4040/<spark_app>.
Problems in Yarn came with the need to deploy applications using different Spark versions, which were thus submitted to the cluster as Docker containers. This requirement with the relatively first deploy attempt raised a set of structural issues and unplanned limitations using Yarn. One example: deployed containers used a private ipv4 IP (eg: 10.0.0.20) in the slave, which of course was preventing us from connecting to the Spark-monitoring interface from outside.
After some brainstorming, we worked around this situation by setting up an IPv6 stack for each Yarn slave, thus assigning an IPv6 address to each container and finally reaching the exposed Spark web interface. It was surely working, but then we moved to Mesos.
Continue reading