Impact Analysis of system.exit() usage in Spark Jobs
Is system.exit() a show stopper in Spark Jobs running through YARN in Cluster Mode. What is its impact? Motivation The first question needs to be addressed - Is it good to use the system.exit() in the spark jobs? . Ideally, not only in Spark jobs but in general for any JVM-based applications, it's a big NaaaaaH ! It kills the Spark Job prematurely by shutting down the whole JVM sequence abruptly. Although its usage is the least encouraged way even for lowest business priority applications. Please do some extensive research on it. This post clearly articulates only the impacts of the system.exit usage on the Spark applications. I faced a real-time issue that motivated me and it's entirely my experience that I'm expressing with you which I put forward into this article. Background Precisely speaking, I recently started working on a project for one of my financial client where they have already migrated traditional...