1.1.0

This release adds the ability for operators to configure their deployment to enable self-service app to app policy creation for all space developers.

When enabled, network.write does not need to be explicitly granted to individual space developers in order for them to be able to create policies between apps in spaces for which they have the SpaceDeveloper role.

Space developers now have a configurable quota for the maximum number of policies they can create for any given app as a source. The quota defaults to 50 but does not apply to users with network.admin.

Give us your feedback in the #container-networking channel on cloudfoundry.slack.com. Take a look at known issues for current limitations and known issues.

Verified with the following:

• CF deployment

Manifest Changes

New Properties

• An optional parameter has been added to allow all space developers to create policies (default false).
  If this property is not set, a space developer must have network.write to create policies.
  • cf_networking.enable_space_developer_self_service
• An optional parameter has benn added to configure the maximum number of policies a space
  developer can write for a given source app. Defaults to 50 if it is not set. Does not apply to
  users with network.admin:
  • cf_networking.max_policies_per_app_source

Significant Changes

Space Developer Self-Service

• An operator can configure the max policies/app at deploy time
• As an operator I can enable self service for all space developers

Logging

• As an operator, I can grep through my logs to find all the CF Networking stuff

Windows Compatibility

• garden-external-networker works on Windows
• cloudfoundry-incubator/cf-networking-release #9: Allow garden-external-networker to be used on windows

1.0.0

CF networking features are now generally available! This release doesn't include any major changes, just some metrics and logging enhancements and miscellaneous chores.

Try it out and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Manifest Changes

New Properties

• The following optional parameters have been added to override the timeout values for
  database connections and DNS health checks for the silk controller and policy server:

  • cf_networking.silk_controller.connect_timeout_seconds
  • cf_networking.policy_server.connect_timeout_seconds

• This optional property has been added to override the metron port on the silk controller:

  • cf_networking.silk_controller.metron_port

• This optional property has been added to override the health check port on the silk controller:

  • cf_networking.silk_controller.health_check_port

Removed Properties

• The following properties have been removed from the silk-controller job,
  but still must be set on the silk-daemon job.
  • cf_networking.silk_daemon.ca_cert
  • cf_networking.silk_daemon.client_cert
  • cf_networking.silk_daemon.client_key

Significant Changes

Metrics

• Silk controller emits metrics for subnet usage and HTTP health
• Silk daemon emits metrics

Logging

• Standardize logging across CF networking
• As an operator, I can grep through my logs to find all the CF Networking stuff

Stability

• vxlan-policy-agent fails when policy server is not reachable on start-up
• Requests to policy server and silk controller on their internal endpoints do not hang forever when there is a network partition to their respective databases
• Expose the lease poll interval as a BOSH property

Chores

• mysql ci fails with deadlock
• Remove remaining flannel code and dependencies from cf-networking-release
• Add an integration concurrency test for the silk controller
• Policy server database reads should check for errors at the end of the loop.
• Silk controller database reads should check for errors at the end of the loop.
• An acceptance test covers that ASGs work when an app has a policy

0.25.0

Primary changes include stability related fixes and changes to policy enforcement to make container networking policy independent of ASG configuration.

We also tested and documented how to detect problems with overlapping overlay network and underlay network ranges.

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com. Take a look at known issues for current limitations and known issues.

Verified with the following:

• CF deployment

Manifest Changes

New Properties

The optional parameter cf_networking.lease_poll_interval_seconds has been added to allow operators to override the default polling interval between silk-daemon and silk-controller.

Changed Properties

The value for cf_networking.garden_external_networker.cni_config_dir now defaults to /var/vcap/jobs/silk-cni/config/cni We recommend that you remove any overrides for this property, unless you are intending to use a 3rd party CNI plugin.

Other Changes

Since silk is now deployed by default, there is no more silk.yml ops file. Deploying with flannel is no longer supported.

Significant Changes

iptables

• cf-networking smoke tests and acceptance tests pass on systems configured with a wide open ASG

Stability

• vxlan-policy-agent fails when policy server is not reachable on start-up
• Validate behavior when the overlay network configuration overlaps with the underlay IP range

BOSH Links

• Use bosh links for databases in ops files

Miscellaneous

• Get toque passing with 100 proxy apps using GCP concourse workers
• Standardize naming convention for repos and stuff
• CF networking ops file should install silk

0.24.0

This release allows operators to configure the UDP port used for the VXLAN devices and makes it possible to change the IP address range allocated for the container network.

In addition, it adds further improvements to the control scripts for all BOSH jobs and introduces basic uptime monitoring for the silk-daemon.

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Manifest Changes

New Properties

• The host port for receiving VXLAN packets is now configurable as cf_networking.vtep_port for flannel and silk. Overriding this value is optional.

Significant Changes

BOSH configuration

• As an operator I can change the overlay network mask to enlarge the network
• CF networking jobs use links for database configuration
• An operator can override the UDP port used for VXLAN traffic

BOSH job control scripts

• All control scripts are using a common pid_utils
• Fix monit issues with silk-daemon

Silk metrics

• silk-daemon emits an uptime metric

Other

• Explore using context in policy server
• Smoke tests continue to pass during deploys

0.23.0

This release improves the interaction between monit and silk-daemon. It also includes improvements to iptables configuration and logging.

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Manifest Changes

None

Significant Changes

iptables

• iptables logs leave a space between the container handle and the next field in the log
• Our iptables rules are agnostic to system defaults and lower-priority chains
• Default deny is agnostic to changes in the overlay network range

Policy Server

• VPA and silk-daemon connecting to controllers should have reasonable timeouts

Bug fixes

• When the cf_networking.mtu is not set, it is incorrectly passed through as -50.

0.22.0

This release introduces a beta version of a new container networking fabric called "silk" and contains significant changes to job and property names. In order to deploy silk you must upgrade to Diego release v1.15.0 or higher.

Silk is a replacement for flannel, which uses a central controller node backed by a SQL database. Etcd is no longer required by CF Networking Release when running Silk.

There are several manifest changes required to enable Silk, and we highly recommend reading the manifest changelog to understand the changes.

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Manifest Changes

Too many to list here - please take a look at the manifest changelog

Significant Changes

Silk Controller and daemon

• Containers use sensible defaults for MTU
• Subnets are reclaimed once they expire
• I can ping from one container to the other on the overlay network
• Routing rules are cleaned up when Diego cells are removed
• silkd crashes if there is a subnet theft
• silkd is deployed by BOSH
• Error handling in silkd when it cannot get routes from silk controller
• Container networking with silk is reliable at 20K app instances across 100 Diego cells
• Operators can override the pre-encap MTU
• Silk controller does not return expired leases
• Manifest changelog has instructions to deploy silkd and silk controller
• diego manifest generation scripts work with silk
• Client config LoadConfig is covered by unit tests
• silk-teardown deletes VTEP even when release fails
• Standardize names across BOSH user surface area

Acceptance Tests

• Acceptance tests support separate system and app domains

Policy Server

• Policy server is using golang context

0.21.0

Fixes compatibility issues with BOSH stemcell version 3363.19. If you are deploying cf-networking-release you must upgrade to this release or a subsequent release to use this and future stemcells.

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Significant Changes

Manifest Changes

Changed Properties

• The value for cf_networking.garden_external_networker.cni_plugin_dir now defaults to /var/vcap/packages/silk-cni/bin
  We recommend that you remove any overrides for this property, unless you are intending to use a 3rd party CNI plugin.

Stemcell Compatibility

• Silk plays nice with latest stemcell (which disables IPv6)

Silk Controller and daemon

• I can ping from one host to the other on the overlay network
• Silk controller has an endpoint to renew a lease
• silkd is deployed by BOSH
• silk-controller is deployed by BOSH

0.20.0

No major changes in this release. Most commits are feature work to support a new daemon and controller to set up container networking.

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Significant Changes

CNI

• Make path to CNI plugin a default

Application Security Groups

• As an operator I can configure an ASG for ICMP traffic

Chores

• garden external networker logs have higher signal:noise ratio

Documentation

• Document smoke test usage

0.19.0

The first release to include a new layer-3 only CNI plugin. Highlights include:

• Silk CNI plugin to replace Flannel CNI plugin
• NetIn and NetOut rules are configured through CNI
• Networking features to enable BOSH DNS for CF apps

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues. Verified with the following:

• CF deployment

Manifest Changes

Changed Properties

• The value for cf_networking.garden_external_networker.cni_plugin_dir must be updated to /var/vcap/packages/silk/bin if you are not swapping out CNI with your own plugin. (There is no default currently, but we plan to add one in the next release)
• The property for global ASG logging has changed from cf_networking.garden_external_networker.iptables_asg_logging
  to cf_networking.iptables_asg_logging.

Removed Properties

• cf_networking.flannel_watchdog.no_bridge is now removed.

New Properties

A new property has been added to support an upcoming feature. Users can specify DNS servers and access will be automatically allowed for link-local DNS servers:

• cf_networking.dns_servers

The new feature will require garden-runc-release versions >=1.4.0.

Significant Changes

New CNI plugin

• CF Wrapper plugin fails if there is a subnet theft
• CF Networking Release can use the Silk CNI plugin instead of the flannel + bridge plugins
• Flannel watchdog has a bridgeless mode where it inspects the the container metadata store
• An acceptance environment is running a BOSH deployed silkd

NetIn/NetOut Changes

• Wrapper CNI plugin can configure NetIn and NetOut
• The external networker defers to the CNI plugin to write NetIn/NetOut rules

BOSH DNS support

• An iptables input rule is written for every local DNS server
• DNS servers are returned from the external networker to garden - Requires garden-runc-release versions >1.3.0

Logging enhancements

• Logging for denied outbound non-c2c packets
• As an operator I know how to find the source app using a packet capture
• ASG deny logging is rate limited to a hardcoded interval
• Troubleshooting docs include information about ASG logging through BOSH property

Chores

• Switch all dev machines to only use go bosh cli
• cloudfoundry-incubator/cf-networking-release #5: bosh-lite ops file

0.18.0

Lots of good stuff in this release. Highlights include:

• Logging for c2c iptables can be enabled through a BOSH property
• Container networking scales to 20K application instances with 3 policies per application.
• Initial support for logging ASG iptables through a BOSH property. ASG logs will be prefixed with OK_ or DENY_.
• If you are running Diego release v1.10.1 you must upgrade to this release

We do not recommend using cf-networking-release in production yet, but give it a try and give us your feedback in the #container-networking channel on cloudfoundry.slack.com.

Take a look at known issues for current limitations and known issues.

Verified with the following:

• CF deployment

New Manifest Properties

• cf_networking.rep_listen_addr_admin enables our drain scripts to wait for the Diego rep to exit.
  It should always be the same value as diego.rep.listen_addr_admin. It defaults to 127.0.0.1:1800.
• cf_networking.garden_external_networker.iptables_asg_logging globally enables iptables logging for
  all ASGs, including logging of denied packets. Defaults to false.
• cf_networking.vxlan_policy_agent.iptables_c2c_logging enables iptables logging for
  container-to-container traffic. It defaults to false. Note: this is already
  configurable at runtime.
• cf_networking.plugin.health_check_port allows BOSH to better health-check the flanneld process
  required for connectivity.

Removed Manifest Properties

• cf_networking.policy_server.database.connection_string was deprecated in v0.10.0 and is now removed.

Significant Changes

Scalability

• container networking is reliable with 20k app instances across 100 diego cells
• Scalability test for popular server
• Our docs include recommendations on scaling policy server instances and DB
• The policy server can handle our scalability target of 20K AIs

Upgrades

• As an operator I can upgrade from kawasaki to cf-networking

Manifest Changes

• Remove deprecated BOSH property policy_server.database.connection_string

Security

• Policy server can communicate with UAA over TLS

Chores

• Investigate and fix "Ginkgo timed out waiting for parallel nodes to report back"
• Improve stop behavior of monit ctl scripts

Stability

• Flannel has a healthcheck endpoint for monit
• A cell with a subnet mismatch can be recovered by a BOSH restart of the cell
• Policy server monit script checks a healthcheck endpoint

Logging

• Logging for c2c iptables is configurable through a BOSH property
• Logging for denied outbound non-c2c packets

Internal integration

• Garden External Networker handles NetIn/NetOut fields during Up