Skip to content
This repository was archived by the owner on Sep 3, 2022. It is now read-only.

Development Environment

Jump to bottom
nikhilk edited this page Dec 5, 2014 · 23 revisions

This page outlines setting up a working development environment for working with the DataLab repository.

Environment Setup

Once you've cloned the repository locally, you'll need to setup your environment to include the tools and packages required to build and run the various components.

# Run once to setup tools
./tools/initonce.sh

# Run once per command prompt to setup environment vars etc.
source ./tools/initenv.sh

Dependencies

  • Build system - Gradle 2.0
  • Java - Java 1.7
  • JavaScript - NodeJS 0.10.x, TypeScript 1.0
  • Docker
  • SDKs - gcloud
  • Tools - ipython (and associated python dependencies - see below)

Python Environment

On the mac, the easiest way to perhaps get IPython and associated dependencies is to use a full python distribution ... like the Anaconda one. It includes dependencies like pandas and various python libraries. You'll want to install it and then update the packages to latest version (using the conda update <name of package> command).

Running Locally

DataLab is designed to be run within the cloud, specifically on Google Compute Engine and using Google Cloud Platform services. It relies on the metadata service available within the cloud to resolve contextual information such as current project id, and retrieve tokens to authorize cloud API calls. A tool is included in the repository to simulate this metadata service. Run the following commands:

# Setup gcloud (once usually suffices, unless you need to change projects)
gcloud auth login
gcloud config set project <your cloud project>
gcloud config set compute/zone <zone name - eg. us-central1-a>

# Start local emulation of metadata service
node ./tools/metadata/server.js

Running IPython quickly

IPython notebooks can be used to work against BigQuery APIs provided by DataLab.

# Run ipython with (transient) in-memory notebooks
ipym.sh

# Run ipython with notebooks in a cloud storage bucket
ipyc.sh

# Run ipython with notebooks in a local directory
ipy.sh <path to local notebooks dir>

Check out the sample notebooks in the sample directory to get started.

Clone this wiki locally