Skip to content

Installing Istio with a Custom Envoy Proxy

Jump to bottom
Samuel Waggoner edited this page Dec 9, 2019 · 3 revisions

Version of Istio

At the time of writing, we are targeting Istio 1.4.0 as the version to build our custom proxy sidecar docker image against.

Create a VM to build the custom Envoy/Proxy image

We created an N1 VM on GCP in the oak project with 96 cpus (high cpu type) and with 120 GB SSD drive and selected Ubuntu 18.04 LTS as the operating system.

Connect to the VM in preparation of acquiring the necessary build tools. We used an ssh command we found in the Google Cloud Console:

gcloud beta compute --project "cf-identity-service-oak" ssh --zone "us-central1-b" "istio-proxy-build-tools-2019-11-18"

Install the required build tools

# Install Bazel
sudo wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/download/v0.0.8/bazelisk-linux-amd64
sudo chmod +x /usr/local/bin/bazel

# Install some basic build tools
sudo apt-get update
sudo apt-get install -y libtool cmake automake autoconf make ninja-build \
    curl wget unzip virtualenv build-essential llvm clang libc++-dev libc++abi-dev lld

# Install golang
sudo add-apt-repository -y ppa:longsleep/golang-backports
sudo apt-get update
sudo apt-get install -y golang-go

# Install docker (see https://docs.docker.com/install/linux/docker-ce/ubuntu)
sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
# Allow non-root users to run docker commands (see https://docs.docker.com/install/linux/linux-postinstall)
sudo usermod -aG docker $USER
sudo systemctl enable docker
# Log out and log back in so that your group membership is re-evaluated
exit

After you have logged out, log back into the VM and continue installing build tools.

# Verify that docker is installed correctly:
docker run hello-world

# Uninstall gcloud, because this version does not allow installing plugins
sudo snap remove google-cloud-sdk
# Then reinstall gcloud (see https://cloud.google.com/sdk/docs/#linux)
wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-271.0.0-linux-x86_64.tar.gz # or latest
tar xvf google-cloud-sdk-271.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/install.sh \
  --quiet \
  --command-completion true \
  --path-update true \
  --additional-components docker-credential-gcr
# Log out and log back in so that your path is updated
exit

Log back in to be ready to continue.

Clone the repositories needed to build the proxy image

On the build server:

mkdir -p ~/workspace
cd ~/workspace
git clone https://github.com/istio/envoy.git envoy
git clone https://github.com/istio/proxy.git proxy
git clone https://github.com/istio/istio.git istio
  • The envoy project is the Istio community's fork of Envoy used to provide the envoy proxy server. We need this project because we will make our custom changes in this project.
  • The proxy project is used to build Istio's version of Envoy which includes Istio's custom extensions. We will use this to build the binary that includes our custom changes.
  • The istio project is the top-level repo for Istio. We will use it to build the docker image for the sidecar, which will include the proxy binary built from the proxy repo.

Use istio/proxy and istio/envoy together to build the envoy binary

  1. cd ~/workspace/envoy
  2. git checkout release-1.4
  3. Apply these changes (also shown in the diff below):
    1. Manually apply this PR, because someone deleted the GitHub repo and this PR points the code to a mirror of the original repo: https://github.com/envoyproxy/envoy/pull/9072/commits/05f9c66748bba3ef60fb7ffc3511ecb3e42e59b4
    2. Manually apply this PR, as it contains our bug fix: https://github.com/envoyproxy/envoy/pull/8668/commits/d4db053425274b8d7778868c6dc29122fc9002bf 
diff --git a/bazel/repositories.bzl b/bazel/repositories.bzl
index 46978aaa4..f6b3466a5 100644
--- a/bazel/repositories.bzl
+++ b/bazel/repositories.bzl
@@ -128,7 +128,7 @@ def envoy_dependencies(skip_targets = []):
     _com_github_circonus_labs_libcircllhist()
     _com_github_cyan4973_xxhash()
     _com_github_datadog_dd_opentracing_cpp()
-    _com_github_eile_tclap()
+    _com_github_mirror_tclap()
     _com_github_envoyproxy_sqlparser()
     _com_github_fmtlib_fmt()
     _com_github_gabime_spdlog()
@@ -243,14 +243,14 @@ def _com_github_envoyproxy_sqlparser():
         actual = "@com_github_envoyproxy_sqlparser//:sqlparser",
     )

-def _com_github_eile_tclap():
+def _com_github_mirror_tclap():
     _repository_impl(
-        name = "com_github_eile_tclap",
+        name = "com_github_mirror_tclap",
         build_file = "@envoy//bazel/external:tclap.BUILD",
     )
     native.bind(
         name = "tclap",
-        actual = "@com_github_eile_tclap//:tclap",
+        actual = "@com_github_mirror_tclap//:tclap",
     )

 def _com_github_fmtlib_fmt():
diff --git a/bazel/repository_locations.bzl b/bazel/repository_locations.bzl
index 0d298f3bc..e664a4396 100644
--- a/bazel/repository_locations.bzl
+++ b/bazel/repository_locations.bzl
@@ -73,10 +73,10 @@ REPOSITORY_LOCATIONS = dict(
         strip_prefix = "sql-parser-5f50c68bdf5f107692bb027d1c568f67597f4d7f",
         urls = ["https://github.com/envoyproxy/sql-parser/archive/5f50c68bdf5f107692bb027d1c568f67597f4d7f.tar.gz"],
     ),
-    com_github_eile_tclap = dict(
+    com_github_mirror_tclap = dict(
         sha256 = "f0ede0721dddbb5eba3a47385a6e8681b14f155e1129dd39d1a959411935098f",
         strip_prefix = "tclap-tclap-1-2-1-release-final",
-        urls = ["https://github.com/eile/tclap/archive/tclap-1-2-1-release-final.tar.gz"],
+        urls = ["https://github.com/mirror/tclap/archive/tclap-1-2-1-release-final.tar.gz"],
     ),
     com_github_fmtlib_fmt = dict(
         sha256 = "4c0741e10183f75d7d6f730b8708a99b329b2f942dad5a9da3385ab92bb4a15c",
diff --git a/source/extensions/filters/http/ext_authz/ext_authz.cc b/source/extensions/filters/http/ext_authz/ext_authz.cc
index 3878c2ba9..a6ea4a43d 100644
--- a/source/extensions/filters/http/ext_authz/ext_authz.cc
+++ b/source/extensions/filters/http/ext_authz/ext_authz.cc
@@ -216,10 +216,14 @@ void Filter::onComplete(Filters::Common::ExtAuthz::ResponsePtr&& response) {
           ENVOY_STREAM_LOG(trace,
                            "ext_authz filter added header(s) to the local response:", callbacks);
           for (const auto& header : headers) {
-            ENVOY_STREAM_LOG(trace, " '{}':'{}'", callbacks, header.first.get(), header.second);
+            // ENVOY_STREAM_LOG(trace, " '{}':'{}'", callbacks, header.first.get(), header.second);
             response_headers.remove(header.first);
-            response_headers.addCopy(header.first, header.second);
+            // response_headers.addCopy(header.first, header.second);
           }
+         for (const auto& header : headers) {
+            ENVOY_STREAM_LOG(trace, " '{}':'{}'", callbacks, header.first.get(), header.second);
+           response_headers.addCopy(header.first, header.second);
+         }
         },
         absl::nullopt, RcDetails::get().AuthzDenied);
     callbacks_->streamInfo().setResponseFlag(StreamInfo::ResponseFlag::UnauthorizedExternalService);
  1. cd ~/workspace/proxy
  2. git checkout 1.4.0
  3. Apply these changes (also shown in the diff below):
    1. Mount our local clone of istio/envoy into the build container, and do not remove the container when the build finishes
    2. Edit WORKSPACE to use the local clone of istio/envoy as the envoy source code instead of downloading it from github
    3. In Makefile.overrides.mk we configure a different container image for use with BUILD_WITH_CONTAINER 
diff --git a/Makefile b/Makefile
index 027b86f..da09cce 100644
--- a/Makefile
+++ b/Makefile
@@ -67,7 +67,7 @@ $(info Building with the build container: $(IMG).)
 # the path of the file.
 TIMEZONE=`readlink $(READLINK_FLAGS) /etc/localtime | sed -e 's/^.*zoneinfo\///'`

-RUN = $(CONTAINER_CLI) run -t -i --sig-proxy=true -u $(UID):docker --rm \
+RUN = $(CONTAINER_CLI) run -t -i --sig-proxy=true -u $(UID):docker \
        -e IN_BUILD_CONTAINER="$(BUILD_WITH_CONTAINER)" \
        -e TZ="$(TIMEZONE)" \
        -e TARGET_ARCH="$(TARGET_ARCH)" \
@@ -76,6 +76,7 @@ RUN = $(CONTAINER_CLI) run -t -i --sig-proxy=true -u $(UID):docker --rm \
        -e HUB="$(HUB)" \
        -e TAG="$(TAG)" \
        -v /etc/passwd:/etc/passwd:ro \
+       -v /home/pivotal/workspace/envoy:/sw-rr-envoy \
        $(DOCKER_SOCKET_MOUNT) \
        $(CONTAINER_OPTIONS) \
        --mount type=bind,source="$(PWD)",destination="/work" \
diff --git a/Makefile.overrides.mk b/Makefile.overrides.mk
index a29486e..c9beff8 100644
--- a/Makefile.overrides.mk
+++ b/Makefile.overrides.mk
@@ -14,3 +14,4 @@

 # this repo is not on the container plan by default
 BUILD_WITH_CONTAINER ?= 0
+IMG = gcr.io/istio-testing/build-tools-proxy:master-2019-11-08T17-21-18
diff --git a/WORKSPACE b/WORKSPACE
index d8d64db..21cab53 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -38,25 +38,25 @@ bind(
 # 2. Update .bazelrc and .bazelversion files.
 #
 # envoy commit date: 11/11/2019
-ENVOY_SHA = "429ee52b1f45b57420db15f434a36ad9efd81828"
+#ENVOY_SHA = "429ee52b1f45b57420db15f434a36ad9efd81828"

-ENVOY_SHA256 = "144aac08d1c5491e948546f2780ba938d92ba59f079bbef038e8174a5a877588"
+#ENVOY_SHA256 = "144aac08d1c5491e948546f2780ba938d92ba59f079bbef038e8174a5a877588"

-LOCAL_ENVOY_PROJECT = "/PATH/TO/ENVOY"
+LOCAL_ENVOY_PROJECT = "/sw-rr-envoy"

-http_archive(
-    name = "envoy",
-    sha256 = ENVOY_SHA256,
-    strip_prefix = "envoy-" + ENVOY_SHA,
-    url = "https://github.com/istio/envoy/archive/" + ENVOY_SHA + ".tar.gz",
-)
+#http_archive(
+#    name = "envoy",
+#    sha256 = ENVOY_SHA256,
+#    strip_prefix = "envoy-" + ENVOY_SHA,
+#    url = "https://github.com/istio/envoy/archive/" + ENVOY_SHA + ".tar.gz",
+#)

 # TODO(silentdai) Use bazel args to select envoy between local or http
 # Uncomment below and comment above http_archive to depends on local envoy.
-#local_repository(
-#     name = "envoy",
-#     path = LOCAL_ENVOY_PROJECT,
-#)
+local_repository(
+     name = "envoy",
+     path = LOCAL_ENVOY_PROJECT,
+)

 load("@envoy//bazel:api_binding.bzl", "envoy_api_binding")
  1. cd ~/workspace/proxy
  2. bazel clean --expunge --async && BUILD_WITH_CONTAINER=1 make build to build
  3. docker container ls to find the container that was just used to build
  4. docker cp $CONTAINER_ID:/work/bazel-out/k8-fastbuild/bin/src/envoy/envoy . to copy the built binary out of the container to your current directory

Use istio/istio to build the container image

  1. cd workspace/istio
  2. git checkout 1.4.0
  3. export GOPATH=$HOME/go The following steps will use this directory as a cache. It does not need to exist before running the following steps. Feel free to rm -rf $HOME/go if you want to start clean.
  4. This will copy your envoy binary from your istio/proxy repo into the appropriate place for this repo to use: 
    rm -f /home/pivotal/go/out/linux_amd64/release/istio_is_init
USE_LOCAL_PROXY=1 ISTIO_ENVOY_LOCAL_PATH=/home/pivotal/workspace/proxy/envoy make init
  5. Build the docker container which will include your custom built envoy binary: USE_LOCAL_PROXY=1 ISTIO_ENVOY_LOCAL=/home/pivotal/workspace/proxy/envoy make docker.proxyv2
  6. Confirm that the envoy executable can be started and has the right git sha, run "envoy --version" inside the container: docker run --entrypoint envoy <the-built-image-id> --version
  7. tag and push the container image that you just built: 
    docker tag <the-built-image-id> gcr.io/cf-identity-service-oak/patched-sidecar:istio-envoy-1.4.0-plus-patch
gcloud auth login
gcloud auth configure-docker
docker push gcr.io/cf-identity-service-oak/patched-sidecar:istio-envoy-1.4.0-plus-patch
  8. Once you have pushed the container image, you are most likely finished with the build server, so you can log out of glcoud using gcloud auth revoke and then stop the VM to save money.

Download the Official Istio Release

We are targeting Istio 1.4.0 as the version to build our custom proxy sidecar docker image against.

  1. Download the official Istio release on your iMac: 
    cd workspace
wget https://github.com/istio/istio/releases/download/1.4.0/istio-1.4.0-osx.tar.gz
tar xvf istio-1.4.0-osx.tar.gz
  2. Edit /install/kubernetes/istio-demo.yaml to specify that the injected sidecars should use the custom built container:
diff --git a/install/kubernetes/istio-demo.yaml b/install/kubernetes/istio-demo.yaml
index a4c97a6..38fcc95 100644
--- a/install/kubernetes/istio-demo.yaml
+++ b/install/kubernetes/istio-demo.yaml
@@ -20561,11 +20561,7 @@ data:
       {{- end }}
       containers:
       - name: istio-proxy
-      {{- if contains "/" (annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image) }}
-        image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.proxy.image }}"
-      {{- else }}
-        image: "{{ annotation .ObjectMeta `sidecar.istio.io/proxyImage` .Values.global.hub }}/{{ .Values.global.proxy.image }}:{{ .Values.global.tag }}"
-      {{- end }}
+        image: "gcr.io/cf-identity-service-oak/patched-sidecar:istio-envoy-1.4.0-plus-patch"
         ports:
         - containerPort: 15090
           protocol: TCP
  1. Add the istioctl to the path 
    cd istio-1.4.0
echo 'export PATH=$PWD/bin:$PATH' > .envrc
direnv allow

Create a cluster and install Istio

  1. Create a GKE cluster:
    1. Go to the GCP Console and select the project in which you would like to create a k8s cluster, e.g. the OAK project
    2. If you don’t already have them, install the gcloud CLI and the kubectl CLI
    3. Click “Create Cluster”, choose “Standard Cluster”, and fill in a name for your cluster, e.g. “ryan-cluster”
    4. Note the k8s version option. See the Istio install docs for the supported versions of k8s for the version of Istio that you would like to use. “Istio 1.2 has been tested with these Kubernetes releases: 1.12, 1.13, 1.14”.
    5. Otherwise, you can accept the defaults.
    6. Click “Create”.
  2. When it is finished creating, next we will install Istio into the cluster
    1. Prepare your k8s cluster
      1. For testing purposes, we do not need to enable the Istio CNI feature, so we can skip that step
      2. Login: gcloud auth login your_email@pivotal.io
      3. Retrieve your credentials for kubectl: gcloud container clusters get-credentials <your_cluster_name> --zone <your_cluster_zone> --project cf-identity-service-oak
      4. If kubectl commands do not work, grant cluster administrator permissions to the current user: kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value core/account)
    2. Follow the setup instructions for Istio
      1. Install Istio 
        for i in install/kubernetes/helm/istio-init/files/crd*yaml; do
  kubectl apply -f ${i};
done
kubectl apply -f install/kubernetes/istio-demo.yaml
      2. Verify the installation by ensuring the pods are running. Then verify that the Istio Kubernetes services are deployed and verify they all have an appropriate CLUSTER-IP except the jaeger-agent service. Also wait for the external IP of the istio-ingressgateway to appear in the watch command below. 
        kubectl get pods -n istio-system
watch kubectl get svc -n istio-system
      3. Enable automatic sidecar injection in the default namespace: kubectl label namespace default istio-injection=enabled

Install the authcode demo app and protect it using the authservice

  1. cd ~/workspace/istio-authn-authz-demo
  2. Check the image name for the authservice container in the authcode-sample-app deployment in ~/workspace/istio-authn-authz-demo/sample-istio-policies/authcode/authcode-sample-app.yaml. Make sure it is set to the version of the authservice which you would like to test.
  3. Get the ingress hostname: source ./scripts/sample-apps/print-gateway-info-and-sample-app-url.sh
  4. Update the callback.hostname and landing_page IP addresses in this file: ~/workspace/istio-authn-authz-demo/sample-istio-policies/authcode/authcode-sample-app-authservice-configmap.yaml Use the value of $INGRESS_HOSTNAME as the new IP address.
  5. Generate and update certs for the authcode-app-gateway: ./scripts/istio/generate-self-signed-certs-for-gateway.sh
  6. Deploy the authcode and resource server app to your k8s cluster, and set up the gateway for them: ./scripts/sample-apps/deploy-sample-apps.sh
  7. Wait for the deployment to finish: watch kubectl get pods
Clone this wiki locally