hub /  hasura /  hdfs
3 min(s)
20 Pull(s)
Icon

Sample project to deploy HDFS on a Hasura cluster

apache
hadoop
hdfs
big-data
Detailed description

hdfs

Follow the below steps to get the setup working on your cluster.

Prerequisites

  • Ensure that you have the hasura cli tool installed on your system.
$ hasura version

Once you have installed the hasura cli tool, login to your Hasura account

$ # Login if you haven't already
$ hasura login
  • Ensure that you have kubectl installed on your system.
$ kubectl version
  • You should also have git installed.
$ git --version

Getting started

$ # Run the quickstart command to get the project
$ hasura quickstart hasura/hdfs

Note the name of the cluster printed in the output.

$ # Navigate into the Project
$ cd hdfs

Deploy app

Deployment is a 3-step process.

Step-1: Deploy Persistent Volumes and Persistent Volume Claims (https://kubernetes.io/docs/concepts/storage/persistent-volumes/)

$ # Ensure that you are in the hdfs directory
$ cd custom_k8s/pv
$ kubectl create -f k8s.yaml --context=<cluster_name>

Step-2: Deploy Namenode

$ # Ensure that you are in the hdfs directory
$ cd custom_k8s/namenode
$ kubectl create -f k8s.yaml --context=<cluster_name>

Step-3: Deploy Datanodes

$ # Ensure that you are in the hdfs directory
$ cd custom_k8s/datanode
$ kubectl create -f k8s.yaml --context=<cluster_name>

Optional Step: Deploy Namenode web UI

To get a HTTPS endpoint to view your Namenode web UI. Append the following snippet to your routes.yaml

namenode:
  /:
    corsPolicy: allow_all
    upstreamService:
      name: namenode-web
      namespace: '{{ cluster.metadata.namespaces.user }}'
    upstreamServicePath: /
    upstreamServicePort: 80

The upstream service namenode-web was already deployed as part of Step-2 above.

$ # Ensure that you are in the hdfs directory
$ cd conf
$ # Add above snippet to the end of routes.yaml
$ vim routes.yaml
$ # Commit changes and push
$ git add . && git commit -m "add namenode ui route"
$ git push hasura master

Goto namenode.<cluster_name>.hasura-app.io to access the Namenode web UI.

Explore HDFS

Exec into namenode-0 pod to run commands using hdfs client.

$ kubectl exec -it namenode-0  --context=<cluster_name> -- /bin/bash

root> hdfs dfs -put  /etc/hosts /
root> hdfs dfs -ls /

And thats it!


Author & License

Source: https://github.com/k8s-platform-hub/hub-hdfs
OSS license: MIT License

Comments