CloverDX in a Kubernetes Pod using Google Kubernetes Engine (GKE)

Containerized deployments have become increasingly popular amongst our clients. We here at CloverDX maintain a Docker image for developers to use throughout their organizations. By leveraging our Docker repo along with a container management system (like Kubernetes) you gain the ability to easily spin up new instances of CloverDX Server and scale them as needed.

In this tutorial, we're going deploy several Docker containers within a Kubernetes pod on Google Cloud Platform using Google Kubernetes Engine (GKE). 

What we need:

Getting License and software files

If you haven’t already done so, create a trial account for CloverDX. Log in using your CloverCARE credentials sent to your email. The log-in page can be found here: https://support.cloverdx.com

Once you're logged in, go to the "Downloads and Licenses" section using the button on the top bar.

image-png-Sep-20-2021-07-23-04-03-PM

  • Download the “Tomcat Server Download” link. You should see a "clover.war" file pop up in your downloads
  • On the right side, you should see a "CloverDX Server License".
    1. Click to open a text file showing license information.
    2. Click the button that says “Copy to Clipboard”.
    3. Go to any text editor (like Notepad++), make a new file.
    4. Paste the license information over from your clipboard.
    5. Finally, save this new file as "license.dat".

Now we have everything we need to start working GCP!

Google Cloud Project creation

After logging into your Google Cloud Project create new Google Cloud Project with a name of your choice.

  1. Click on the project tab in the top left to open up the projects menu
  2. Select “New Project” in the top right of the menu
  3. Name your project as you'd like. We went with "cloverdx" for our example project
  4. Click "Create" to finish making your new project

Navigate to your new project in the Google Cloud Console.

  1. Click on the Activate Cloud Shell button in the top right corner of the Cloud Console
  2. A Cloud Shell session opens inside a new frame at the bottom of the Cloud Console and displays a command-line prompt.

Create Docker Repository in GCP

  1. Set the PROJECT_ID environment variable to your Google Cloud project ID (PROJECT_ID). You'll use this environment variable when you build the container image and push it to your repository.
    export PROJECT_ID=<created project id> // e.g. cloverdx-123456
    export REGION=<target region> // e.g. europe-west3 echo $PROJECT_ID $REGION
  2. Set your project ID for the gcloud command-line tool:
    gcloud config set project $PROJECT_ID

    Output:
    Updated property [core/project].

    Note: Your gcloud tool might already be set to your $PROJECT_ID—you will still see the Updated property output.
  3. Create the hello-repo repository with the following command:
    gcloud artifacts repositories create hello-repo \
      --repository-format=docker \
      --location=$REGION \
      --description="Docker repository"
    Replace REGION with the a region for the repository, such as us-west1. To see a list of available locations, run the command:

Setting up and Building our Container Image

  1. We need to pull our base Docker image into our GCP Machine and make some modifications to get up and running. From Google Cloud Shell:
    git clone https://github.com/cloverdx/cloverdx-server-docker.git
  2. Now we need to upload the clover.war and license.dat files we got in previous steps into our cloverdx-server-docker directory
    1. In the google console, go into your cloverdx-server-docker directory:
      cd cloverdx-server-docker
    2. Now we need to upload our .war file using Google’s cloud console. On the right side, click the "…" button to open up more options and choose "Upload a File"
    3. Navigate to local directory where you downloaded clover.war and upload it.
    4. Once you get a successful upload message, run the list files command: ls and confirm that you see clover.war in the current directory.
    5. Now let’s do the same and upload our saved license.dat file. Move it into the examples/Kubernetes directory:
      cd examples/kubernetes
    6. Now upload the license.dat file from your local computer to the shell cloud console using the same process as before.
    7. Once you get a successful upload message, run the list files command:
      ls
    8. Confirm that you see license.dat in the current directory.

Making a gcloud Kubernetes cluster

  1. Running the command:
    gcloud container clusters create hello-cluster --num-nodes=1 --region=$REGION
    Will make a gcloud Kubernetes cluster called hello-cluster in $REGION (we set this variable in very first step in gcloud console) availability zone. It should take a few minutes of provisioning and then you should get a confirmation of it running that looks like this:
  2. Now, when infrastructure is ready, we need to deploy our Kubernetes example on this cluster.
    1. We need to grab credentials (switch console context to environment we want to deploy) from our Kubernetes cluster by running:
      gcloud container clusters get-credentials --region $REGION hello-cluster
    2. Next, we need to run a command to initiate the launch of our Kubernetes instance.
      ./run.sh eu.gcr.io/$PROJECT

      This application will now be forwarding traffic through port 8082. Press ctrl+c to exit out of the forwarding script running.
  3. All services (pods) created using CloverDX public example register themselves in a namespace cloverdx.

Adding a Load Balancer

To be able to access instance from the internet, you need to create load balancer which will proxy incoming traffic into your CloverDX GKS cluster.

Example project published on Github uses Gravitee Gateway to distribute HTTP calls to multiple applications and is set up to route traffic coming to port 8082. To route external calls coming from HTTP port (80) to exposed one (8082), run:
kubectl expose deployment gravitee-gateway --name=internet-lb --type=LoadBalancer --port 80 --target-port 8082 --namespace=cloverdx

Accessing our CloverDX Server Instance

  1. Now we need to look for the IP address so we can publicly access our cluster from a browser.
    kubectl get service --namespace=cloverdx | grep internet-lb

    Note: it may take a while till load balancer receives an IP address. When IP address is still pending assignment, you'll see <pending> instead of an IP address. It usually takes just couple of seconds.
  2. In the fourth column, you should see an IP address. Copy and paste this into your browser. You should be able to access CloverDX Server console on http://<IP address>/clover.
  3. To login, use the default credentials that come with every evaluation instance
    Username: clover
    Password: clover

We have now successfully provisioned and accessed our CloverDX Server instance running in a Kubernetes Pod on Google Cloud Platform. Be mindful about the fact, this is in no way production-grade deployment but rather an example of how to deploy CloverDX Server (or Cluster) in K8s.

To Shut down your Kubernetes Pod:
kubectl delete --all pods --namespace=cloverdx. To completely remove all resources created by this tutorial. You also need to remove K8s cluster and container repository!

More from Tech Blog

  • CloverDX as Kafka event consumer

    In previous article, we covered how to establish a Kafka connection and how to publish messages into a Kafka topic. This article will cover the other side... Connectors
  • How to connect and publish messages to Kafka

    Kafka is a distributed event streaming platform capable of handle massive volumes of events. It is designed and operates similar to a messaging queue.... Data Transformation
  • How to verify if you are affected by Log4Shell vulnerability

    Overview Log4Shell is a remote code execution vulnerability discovered in December 2021 in log4j Java library. The vulnerability allows attacker to abuse...
  • Using Python in CloverDX

    Python has long been one of the premier choices for data engineers looking to organize, clean, and analyze data from various sources. It is relatively easy... CloverDX How-To
  • Understanding the CloverDX Project Lifecycle

    Data Integration pipelines, like all software systems, have a lifecycle that includes transitions between development, test and production environments.... CloverDX How-To

Visit CloverDX Blog

Read On