SparkPi: an introduction to applications using Apache Spark with radanalytics.io

The SparkPi microservice that you will build during this tutorial is quite simple in its design and functionality. SparkPi is deployed as a single server pod and several Apache Spark pods. The microservice provides an HTTP server that accepts GET requests and responds with an estimation of Pi. It does this by using Apache Spark to calculate the value by using a Monte Carlo method.

Note that usually you would not run extended calculations during an HTTP request unless the calculations can be completed quickly. However, this simplifies the implementation details. As you build further applications from the material provided here, keep this simplification of design in mind.

Regardless of the implementation language or the HTTP framework, all of the tutorial applications linked from this page have the same general architecture. The following diagram shows the major components involved with deployment and usage of the microservice:

Component key

The SparkPi component is the main component that you will be concerned with throughout the tutorial. You should already have a computer with the necessary tools installed for making Pi requests, editing your source files and commanding OpenShift to deploy your microservice. The Apache Spark cluster is created automatically for you by the Oshinko source-to-image tools.

Source-to-image technology is fundamental to the operation of this tutorial. Source-to-image provides a convenient way for you to produce container images directly from your source repositories. This work is handled by language-specific builder images which can ingest your source code, assemble it and then produce an application image which is ready to be deployed. In the case of the Oshinko source-to-image builders, an Apache Spark cluster is also created dynamically for your application when it runs.

To use the source-to-image workflow you will need to have a Git repository to store your work. This repository is then used by the source-to-image builders to create the final application image which will be run on OpenShift. For the purposes of this tutorial your repository must be accessible by the OpenShift cluster. The easiest way to do this is by using an external service like GitHub, BitBucket, or GitLab.

Create a new repository on your preferred platform and make a local clone of it. For more information see BitBucket’s documentation and GitHub’s documentation.

With your freshly minted repository you are ready to start coding!

This tutorial was developed for multiple languages and HTTP frameworks. Functionally, all of these microservices work in the same manner with similar results to an end user. In this way they represent a black box style microservice which can be replaced with another component that satisfies the same input and output requirements.

Each of these language specific tutorials provide instructions on how to structure your repository and what files to create. You will also learn how each microservice works and the commands necessary to build and launch them in OpenShift. Because the source-to-image templates that you will use are language specific, you must choose a single implementation to exist in your repository. When you are finished, return here to learn about interacting with your microservice and exposing it outside of OpenShift. Choose one of the following language and framework options:

At this point you should have created a code repository for your microservice, populated the repository with source files, built an application image, and launched that image on OpenShift. The final stage in this tutorial is to expose your microservice outside of OpenShift and begin interacting with it as a user.

The first step in this process is to expose a route to your microservice. OpenShift contains an edge router which associates domain name URIs with services. By default, applications you create through source-to-image have services that can expose routes which will contain their name, the project name and a hostname for the OpenShift server. You can create these routes by using the following command:

To see the routes available in your project, run oc get routes. This command should return output similar to the following example:

Notice that in addition to the sparkpi route you created earlier that there is also a route prefixed with cluster-. This is the route created automatically by the Oshinko source-to-image tooling and it provides access to the Apache Spark web interface. You can visit either one of these routes in your web browser to inspect the results.

In addition to accessing the SparkPi microservice through the web browser, you can use the curl tool and command scripting to achieve the results. The following commands substitute in the URI for your service and the results should look similar for you:

You can see that this rough approximation is close, but not quite close enough. Try adding the scale parameter to your URL to see how it affects the outcome.

You have created and deployed your first radanalytics.io application onto OpenShift. At this point you are beginning to understand the core concepts behind the Oshinko source-to-image tooling. Investigate the other applications and examples in the Tutorials section and also revisit the Get Started page to learn how you can use the Oshinko WebUI to control the Apache Spark clusters in your projects.