Lighter is an opensource application for interacting with Apache Spark on Kubernetes or Apache Hadoop YARN. It is heavily inspired by Apache Livy and has some overlapping features.
Lighter support:
- Interactive Python Sessions through Sparkmagic kernel
- Batch application submissions through the REST API
You can read a description on how Lighter works here.
- Quickstart with Minikube
- Installation on Kubernetes
- Installation on Docker
- Configuration Properties
- Securing a deployment
- Configuring Sparkmagic
- Using REST API
Important
Lighter has no built-in authentication: anyone who can reach its REST API can run code on your Spark cluster. Read Securing a deployment before exposing it.
To build Docker image, containing application, you need to run
docker build -t lighter .
Lighter will always try to support the latest patch version for all officially recommended Apache Spark releases i.e.: 3.5.9 and 4.2.0.
Lighter is MIT licensed.