Skip to content
This repository was archived by the owner on Sep 3, 2022. It is now read-only.

Development Environment

Jump to bottom
Drew Bryant edited this page Mar 18, 2015 · 23 revisions

This page outlines setting up a working development environment for working with the DataLab repository.

Environment Setup

Once you've cloned the repository locally, you'll need to setup your environment to include the tools and packages required to build and run the various components.

# Run once to setup tools
./tools/initonce.sh

# Run once per command prompt to setup environment vars etc.
source ./tools/initenv.sh

Dependencies

  • Java - Java 1.7
  • JavaScript - NodeJS 0.10.x, TypeScript 1.0
  • Docker
  • SDKs - gcloud
  • Tools - ipython (and associated python dependencies - see below)
  • Pkg-config (for building DataLab server - see below)

Python Environment

On the mac, the easiest way to perhaps get IPython and associated dependencies is to use a full python distribution ... like the Anaconda one. It includes dependencies like pandas and various python libraries. You'll want to install it and then update the packages to latest version (using the conda update <name of package> command).

Running Locally

DataLab is designed to be run within the cloud, specifically on Google Compute Engine and using Google Cloud Platform services. It relies on the metadata service available within the cloud to resolve contextual information such as current project id, and retrieve tokens to authorize cloud API calls. A tool is included in the repository to simulate this metadata service. Run the following commands:

# Setup gcloud (once usually suffices, unless you need to change projects)
gcloud auth login
gcloud config set project <your cloud project>
gcloud config set compute/zone <zone name - eg. us-central1-a>

# Start local emulation of metadata service
node ./tools/metadata/server.js

Running IPython quickly

IPython notebooks can be used to work against BigQuery APIs provided by DataLab.

# Run ipython with (transient) in-memory notebooks
ipym.sh

# Run ipython with notebooks in a cloud storage bucket
ipyc.sh

# Run ipython with notebooks in a local directory
ipy.sh <path to local notebooks dir>

Check out the sample notebooks in the sample directory to get started.

Installing 0mq on OS X

DataLab Server has a dependency on the 0mq messaging library used for communicating with language kernels via the IPython protocol. However, installing the bindings for 0mq available from NPM requires that the 0mq library already be installed and that pkg-config be available. Steps to setup:

  • First setup prereqs
    • Install Homebrew (http://brew.sh/)
    • Install zeromq library and pkg-config: brew install zeromq
  • Next setup the environment
    • Make sure pkg-config is available by adding Homebrew's binary path to $PATH: PATH=$PATH:/usr/local/bin (add this to your ~/.bashrc configuration file)
    • Verify that pkg-config is available: which pkg-config
Clone this wiki locally