Skip to content

Dev #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 103 commits into
base: main
Choose a base branch
from
Open

Dev #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
103 commits
Select commit Hold shift + click to select a range
e8bd55b
Copy in gerbil wg config
oschwartz10612 Feb 21, 2025
f69a7f6
Move wg into more of a class
oschwartz10612 Feb 21, 2025
66edae4
Clean up implementation
oschwartz10612 Feb 21, 2025
fb199cc
Tidy
oschwartz10612 Feb 21, 2025
45a1ab9
Dont always do wg
oschwartz10612 Feb 21, 2025
56e7590
Adjust ws types
oschwartz10612 Feb 21, 2025
95eab50
Get wg working
oschwartz10612 Feb 21, 2025
bff6707
Basic create wg seems to be working
oschwartz10612 Feb 21, 2025
18d99de
Handle messages correctly
oschwartz10612 Feb 21, 2025
0affef4
Properly handle key
oschwartz10612 Feb 21, 2025
270ee9c
Fix panic
oschwartz10612 Feb 22, 2025
afa93d8
Add static port and udp hole punch
oschwartz10612 Feb 22, 2025
4aa718d
Initial hp working but need to fix port issue
oschwartz10612 Feb 22, 2025
8795c57
HP works!
oschwartz10612 Feb 22, 2025
f6429b6
Basic holepunch working
oschwartz10612 Feb 23, 2025
b68502d
Basic relay working!
oschwartz10612 Feb 23, 2025
cd3ec0b
Support relay switch
oschwartz10612 Feb 24, 2025
5e673c8
Clean up when wg is used
oschwartz10612 Feb 24, 2025
067e079
Handle / better
oschwartz10612 Mar 13, 2025
14eff8e
Merge branch 'holepunch' of github.com:fosrl/newt_dg into holepunch
oschwartz10612 Mar 13, 2025
f9e52c4
Working on encryption
oschwartz10612 Mar 14, 2025
2a19856
Merge branch 'main' into holepunch
oschwartz10612 Mar 15, 2025
1501de6
Handle encrypted messages
oschwartz10612 Mar 16, 2025
f08378b
Fix segfault if no wgService created
oschwartz10612 Mar 26, 2025
7f9a31a
Remove listen port - unused
oschwartz10612 Mar 26, 2025
c5978d9
Handle port correctly and delete interface
oschwartz10612 Mar 28, 2025
f677376
Merge branch 'dev' into hp-multi-client
oschwartz10612 Mar 31, 2025
09d6829
Add update message
oschwartz10612 Mar 31, 2025
65dc81c
Add wgtester
oschwartz10612 Mar 31, 2025
e47ddaa
Merge branch 'holepunch' into hp-multi-client
oschwartz10612 Apr 1, 2025
6b0ca9c
Adjust wgtester to work with bpf
oschwartz10612 Apr 4, 2025
0ced66e
Relaying working
oschwartz10612 Apr 12, 2025
027d9a0
Remove redundant log message
oschwartz10612 Apr 12, 2025
6a146ed
Add more sensible controls
oschwartz10612 Apr 14, 2025
175718a
Handle order of opertions of hole punch better
oschwartz10612 Apr 23, 2025
494e307
Set to 127
oschwartz10612 May 13, 2025
a4d4976
Update to use docker network checking against newt networking
JonnyBooker Jun 9, 2025
a52260b
Add an enforce network validation flag for docker to not break previo…
JonnyBooker Jun 9, 2025
eaf812a
Merge branch 'dev' into clients-pops
oschwartz10612 Jun 9, 2025
ce6d340
Update go
oschwartz10612 Jun 10, 2025
a5f4d5f
Resolve merge issues
oschwartz10612 Jun 10, 2025
d4b88c3
Merge branch 'fosrl:main' into docker-network-checking
JonnyBooker Jun 10, 2025
e335bb8
Rename added functions for docker client
JonnyBooker Jun 10, 2025
cbbd5b0
Add extra pre-condition check for enforcing docker network
JonnyBooker Jun 10, 2025
126ced6
Merge branch 'docker-network-checking' of https://github.com/JonnyBoo…
JonnyBooker Jun 10, 2025
5476a69
Log the container name and id
JonnyBooker Jun 10, 2025
22f44c8
Merge branch 'dev' into clients-pops
oschwartz10612 Jun 10, 2025
e26552a
Small refinement to how the docker enforcement setting is applied
JonnyBooker Jun 10, 2025
5cb86f3
Update to readme with new configuration settings
JonnyBooker Jun 10, 2025
d0e2205
First pass at pinging
oschwartz10612 Jun 11, 2025
8d4d8b9
Working on pining
oschwartz10612 Jun 13, 2025
6d9160a
Simplified based on PR feedback and support checking use of "bridge" …
JonnyBooker Jun 14, 2025
58f7835
Revise README docs
JonnyBooker Jun 14, 2025
48cb0bf
Minor README update for consistentcy
JonnyBooker Jun 14, 2025
7c971d2
Revert the newt version placeholder text
JonnyBooker Jun 16, 2025
bbea9a9
adjust weight calculation
miloschwartz Jun 16, 2025
95d4cb2
add exit node name to logs
miloschwartz Jun 17, 2025
7f9b700
Add mtu to readme
oschwartz10612 Jun 17, 2025
5f9c041
Update README.md
robtec Jun 14, 2025
50df49e
Tweak to call out when docker deployment
oschwartz10612 Jun 15, 2025
c82b841
Add mtu to readme
oschwartz10612 Jun 17, 2025
850c230
Print ping latency better and add connect message
oschwartz10612 Jun 17, 2025
b397016
give preference to previously connected node
miloschwartz Jun 18, 2025
4b64b04
Update ping check
oschwartz10612 Jun 19, 2025
1c75eb3
New tunnel reconnect works
oschwartz10612 Jun 19, 2025
bb13182
Reorg and add timeout
oschwartz10612 Jun 19, 2025
a14f70d
Faster detection on ws side
oschwartz10612 Jun 19, 2025
3719192
Change terminate to reconnect
oschwartz10612 Jun 19, 2025
6d3938e
Send version
oschwartz10612 Jun 19, 2025
ca3ffa0
Failover is working?
oschwartz10612 Jun 19, 2025
2276316
Properly check the http status code now
oschwartz10612 Jun 20, 2025
040f537
Merge branch 'docker-network-checking' of github.com:JonnyBooker/newt…
oschwartz10612 Jun 22, 2025
5f14ff3
Resolve some basic comments
oschwartz10612 Jun 22, 2025
a684905
Merge branch 'JonnyBooker-docker-network-checking' into dev
oschwartz10612 Jun 22, 2025
e2dd965
Merge branch 'dev' into clients-pops
oschwartz10612 Jun 22, 2025
e642983
Move updates
oschwartz10612 Jun 22, 2025
ac67df6
Add way to stop initial ping
oschwartz10612 Jun 22, 2025
c1a2a32
Add log message
oschwartz10612 Jun 22, 2025
63aea70
Change math random
oschwartz10612 Jun 22, 2025
678d82f
added healthy check in main.go
woutervanelten Jun 25, 2025
a76e6c9
added healthy check in main.go
woutervanelten Jun 25, 2025
e357e7b
Update main.go
woutervanelten Jun 30, 2025
7002871
Update main.go
woutervanelten Jun 30, 2025
9db3b78
Update main.go
woutervanelten Jun 30, 2025
071a51a
Update main.go
woutervanelten Jun 30, 2025
0add2ec
Merge branch 'add-healthcheck' of github.com:woutervanelten/newt into…
oschwartz10612 Jun 30, 2025
295cadc
Merge pull request #71 from woutervanelten/add-healthcheck
oschwartz10612 Jun 30, 2025
837f749
Merge branch 'dev' into clients-pops
oschwartz10612 Jun 30, 2025
b7aa5d1
Merge branch 'main' into dev
oschwartz10612 Jun 30, 2025
450fc6c
Merge branch 'dev' into clients-pops
oschwartz10612 Jun 30, 2025
a39d056
Remove binary mistake
oschwartz10612 Jun 30, 2025
ec8fc20
Comment out WIP clients flags
oschwartz10612 Jun 30, 2025
a896291
Update go.mod
woutervanelten Jul 1, 2025
be56550
Merge pull request #75 from woutervanelten/patch-4
oschwartz10612 Jul 3, 2025
c2a326c
Working packages?
oschwartz10612 Jul 3, 2025
a88d25f
Fix missing netstack pack
oschwartz10612 Jul 3, 2025
61a9097
Make linux clients build correctly
oschwartz10612 Jul 3, 2025
d10c5e0
Merge branch 'main' into dev
oschwartz10612 Jul 3, 2025
b7d4ea0
Fix nil issues
oschwartz10612 Jul 4, 2025
07bd283
Remove dup code
oschwartz10612 Jul 6, 2025
e4bdbbe
Adjust logging
oschwartz10612 Jul 8, 2025
221d586
Fix disconnect errors about closed connection
oschwartz10612 Jul 8, 2025
5339766
Adjust logging
oschwartz10612 Jul 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cicd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
run: |
TAG=${{ env.TAG }}
if [ -f main.go ]; then
sed -i 's/Newt version replaceme/Newt version '"$TAG"'/' main.go
sed -i 's/version_replaceme/'"$TAG"'/' main.go
echo "Updated main.go with version $TAG"
else
echo "main.go not found"
Expand Down
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
newt
.DS_Store
bin/
nohup.out
.idea
*.iml
certs/
certs/
newt_arm64
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,13 @@ When Newt receives WireGuard control messages, it will use the information encod
- `endpoint`: The endpoint where both Gerbil and Pangolin reside in order to connect to the websocket.
- `id`: Newt ID generated by Pangolin to identify the client.
- `secret`: A unique secret (not shared and kept private) used to authenticate the client ID with the websocket in order to receive commands.
- `mtu`: MTU for the internal WG interface. Default: 1280
- `dns`: DNS server to use to resolve the endpoint
- `log-level` (optional): The log level to use. Default: INFO
- `updown` (optional): A script to be called when targets are added or removed.
- `tls-client-cert` (optional): Client certificate (p12 or pfx) for mTLS. See [mTLS](#mtls)
- `docker-socket` (optional): Set the Docker socket to use the container discovery integration
- `docker-enforce-network-validation` (optional): Validate the container target is on the same network as the newt process

- Example:

Expand Down Expand Up @@ -99,6 +101,26 @@ services:
- DOCKER_SOCKET=/var/run/docker.sock
```

#### Hostnames vs IPs

When the Docker Socket Integration is used, depending on the network which Newt is run with, either the hostname (generally considered the container name) or the IP address of the container will be sent to Pangolin. Here are some of the scenarios where IPs or hostname of the container will be utilised:
- **Running in Network Mode 'host'**: IP addresses will be used
- **Running in Network Mode 'bridge'**: IP addresses will be used
- **Running in docker-compose without a network specification**: Docker compose creates a network for the compose by default, hostnames will be used
- **Running on docker-compose with defined network**: Hostnames will be used

### Docker Enforce Network Validation

When run as a Docker container, Newt can validate that the target being provided is on the same network as the Newt container and only return containers directly accessible by Newt. Validation will be carried out against either the hostname/IP Address and the Port number to ensure the running container is exposing the ports to Newt.

It is important to note that if the Newt container is run with a network mode of `host` that this feature will not work. Running in `host` mode causes the container to share its resources with the host machine, therefore making it so the specific host container information for Newt cannot be retrieved to be able to carry out network validation.

**Configuration:**

Validation is `false` by default. It can be enabled via setting the `--docker-enforce-network-validation` CLI argument or by setting the `DOCKER_ENFORCE_NETWORK_VALIDATION` environment variable.

If validation is enforced and the Docker socket is available, Newt will **not** add the target as it cannot be verified. A warning will be presented in the Newt logs.

### Updown

You can pass in a updown script for Newt to call when it is adding or removing a target:
Expand Down
162 changes: 130 additions & 32 deletions docker/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@ import (
"context"
"fmt"
"net"
"os"
"strconv"
"strings"
"time"

"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters"
"github.com/docker/docker/client"
"github.com/fosrl/newt/logger"
)
Expand Down Expand Up @@ -67,13 +70,60 @@ func CheckSocket(socketPath string) bool {
return true
}

// IsWithinHostNetwork checks if a provided target is within the host container network
func IsWithinHostNetwork(socketPath string, targetAddress string, targetPort int) (bool, error) {
// Always enforce network validation
containers, err := ListContainers(socketPath, true)
if err != nil {
return false, err
}

// Determine if given an IP address
var parsedTargetAddressIp = net.ParseIP(targetAddress)

// If we can find the passed hostname/IP address in the networks or as the container name, it is valid and can add it
for _, c := range containers {
for _, network := range c.Networks {
// If the target address is not an IP address, use the container name
if parsedTargetAddressIp == nil {
if c.Name == targetAddress {
for _, port := range c.Ports {
if port.PublicPort == targetPort || port.PrivatePort == targetPort {
return true, nil
}
}
}
} else {
//If the IP address matches, check the ports being mapped too
if network.IPAddress == targetAddress {
for _, port := range c.Ports {
if port.PublicPort == targetPort || port.PrivatePort == targetPort {
return true, nil
}
}
}
}
}
}

combinedTargetAddress := targetAddress + ":" + strconv.Itoa(targetPort)
return false, fmt.Errorf("target address not within host container network: %s", combinedTargetAddress)
}

// ListContainers lists all Docker containers with their network information
func ListContainers(socketPath string) ([]Container, error) {
func ListContainers(socketPath string, enforceNetworkValidation bool) ([]Container, error) {
// Use the provided socket path or default to standard location
if socketPath == "" {
socketPath = "/var/run/docker.sock"
}

// Used to filter down containers returned to Pangolin
containerFilters := filters.NewArgs()

// Used to determine if we will send IP addresses or hostnames to Pangolin
useContainerIpAddresses := true
hostContainerId := ""

// Create a new Docker client
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
Expand All @@ -86,16 +136,54 @@ func ListContainers(socketPath string) ([]Container, error) {
if err != nil {
return nil, fmt.Errorf("failed to create Docker client: %v", err)
}

defer cli.Close()

hostContainer, err := getHostContainer(ctx, cli)
if enforceNetworkValidation && err != nil {
return nil, fmt.Errorf("network validation enforced, cannot validate due to: %w", err)
}

// We may not be able to get back host container in scenarios like running the container in network mode 'host'
if hostContainer != nil {
// We can use the host container to filter out the list of returned containers
hostContainerId = hostContainer.ID

for hostContainerNetworkName := range hostContainer.NetworkSettings.Networks {
// If we're enforcing network validation, we'll filter on the host containers networks
if enforceNetworkValidation {
containerFilters.Add("network", hostContainerNetworkName)
}

// If the container is on the docker bridge network, we will use IP addresses over hostnames
if useContainerIpAddresses && hostContainerNetworkName != "bridge" {
useContainerIpAddresses = false
}
}
}

// List containers
containers, err := cli.ContainerList(ctx, container.ListOptions{All: true})
containers, err := cli.ContainerList(ctx, container.ListOptions{All: true, Filters: containerFilters})
if err != nil {
return nil, fmt.Errorf("failed to list containers: %v", err)
}

var dockerContainers []Container
for _, c := range containers {
// Short ID like docker ps
shortId := c.ID[:12]

// Skip host container if set
if hostContainerId != "" && c.ID == hostContainerId {
continue
}

// Get container name (remove leading slash)
name := ""
if len(c.Names) > 0 {
name = strings.TrimPrefix(c.Names[0], "/")
}

// Convert ports
var ports []Port
for _, port := range c.Ports {
Expand All @@ -112,44 +200,36 @@ func ListContainers(socketPath string) ([]Container, error) {
ports = append(ports, dockerPort)
}

// Get container name (remove leading slash)
name := ""
if len(c.Names) > 0 {
name = strings.TrimPrefix(c.Names[0], "/")
}

// Get network information by inspecting the container
networks := make(map[string]Network)

// Inspect container to get detailed network information
containerInfo, err := cli.ContainerInspect(ctx, c.ID)
if err != nil {
logger.Debug("Failed to inspect container %s for network info: %v", c.ID[:12], err)
// Continue without network info if inspection fails
} else {
// Extract network information from inspection
if containerInfo.NetworkSettings != nil && containerInfo.NetworkSettings.Networks != nil {
for networkName, endpoint := range containerInfo.NetworkSettings.Networks {
dockerNetwork := Network{
NetworkID: endpoint.NetworkID,
EndpointID: endpoint.EndpointID,
Gateway: endpoint.Gateway,
IPAddress: endpoint.IPAddress,
IPPrefixLen: endpoint.IPPrefixLen,
IPv6Gateway: endpoint.IPv6Gateway,
GlobalIPv6Address: endpoint.GlobalIPv6Address,
GlobalIPv6PrefixLen: endpoint.GlobalIPv6PrefixLen,
MacAddress: endpoint.MacAddress,
Aliases: endpoint.Aliases,
DNSNames: endpoint.DNSNames,
}
networks[networkName] = dockerNetwork
// Extract network information from inspection
if c.NetworkSettings != nil && c.NetworkSettings.Networks != nil {
for networkName, endpoint := range c.NetworkSettings.Networks {
dockerNetwork := Network{
NetworkID: endpoint.NetworkID,
EndpointID: endpoint.EndpointID,
Gateway: endpoint.Gateway,
IPPrefixLen: endpoint.IPPrefixLen,
IPv6Gateway: endpoint.IPv6Gateway,
GlobalIPv6Address: endpoint.GlobalIPv6Address,
GlobalIPv6PrefixLen: endpoint.GlobalIPv6PrefixLen,
MacAddress: endpoint.MacAddress,
Aliases: endpoint.Aliases,
DNSNames: endpoint.DNSNames,
}

// Use IPs over hostnames/containers as we're on the bridge network
if useContainerIpAddresses {
dockerNetwork.IPAddress = endpoint.IPAddress
}

networks[networkName] = dockerNetwork
}
}

dockerContainer := Container{
ID: c.ID[:12], // Show short ID like docker ps
ID: shortId,
Name: name,
Image: c.Image,
State: c.State,
Expand All @@ -159,8 +239,26 @@ func ListContainers(socketPath string) ([]Container, error) {
Created: c.Created,
Networks: networks,
}

dockerContainers = append(dockerContainers, dockerContainer)
}

return dockerContainers, nil
}

// getHostContainer gets the current container for the current host if possible
func getHostContainer(dockerContext context.Context, dockerClient *client.Client) (*container.InspectResponse, error) {
// Get hostname from the os
hostContainerName, err := os.Hostname()
if err != nil {
return nil, fmt.Errorf("failed to find hostname for container")
}

// Get host container from the docker socket
hostContainer, err := dockerClient.ContainerInspect(dockerContext, hostContainerName)
if err != nil {
return nil, fmt.Errorf("failed to find host container")
}

return &hostContainer, nil
}
23 changes: 16 additions & 7 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,22 @@ toolchain go1.23.2

require (
github.com/docker/docker v28.3.1+incompatible
github.com/google/gopacket v1.1.19
github.com/gorilla/websocket v1.5.3
github.com/vishvananda/netlink v1.3.0
golang.org/x/crypto v0.39.0
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa
golang.org/x/net v0.41.0
golang.zx2c4.com/wireguard v0.0.0-20231211153847-12269c276173
golang.zx2c4.com/wireguard/wgctrl v0.0.0-20230429144221-925a1e7659e6
gvisor.dev/gvisor v0.0.0-20230927004350-cbd86285d259
golang.zx2c4.com/wireguard v0.0.0-20250521234502-f333402bd9cb
golang.zx2c4.com/wireguard/wgctrl v0.0.0-20241231184526-a9ab2273dd10
gvisor.dev/gvisor v0.0.0-20250503011706-39ed1f5ac29c
software.sslmate.com/src/go-pkcs12 v0.5.0
)

require (
github.com/Microsoft/go-winio v0.6.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/docker/go-units v0.4.0 // indirect
Expand All @@ -27,23 +30,29 @@ require (
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/josharian/native v1.1.0 // indirect
github.com/mdlayher/genetlink v1.3.2 // indirect
github.com/mdlayher/netlink v1.7.2 // indirect
github.com/mdlayher/socket v0.5.1 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/term v0.5.2 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/vishvananda/netns v0.0.4 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel v1.36.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0 // indirect
go.opentelemetry.io/otel/metric v1.36.0 // indirect
go.opentelemetry.io/otel/trace v1.36.0 // indirect
golang.org/x/crypto v0.39.0 // indirect
golang.org/x/mod v0.12.0 // indirect
golang.org/x/mod v0.23.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.33.0 // indirect
golang.org/x/time v0.7.0 // indirect
golang.org/x/tools v0.13.0 // indirect
golang.org/x/tools v0.30.0 // indirect
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
)
Loading