Mocking GCS / S3 connections

[AntonyKing7D]

Hi,,

I've got a question around the GCS vfs module - as part of our deploy and test pipeline, we are trying to mock GCS using the fakeGCS package. Everything we are building uses Docker, so the DNS lookup for the google storage is working as expected and redirecting to the fake GCS package. However, sftpgo is failing to connect to it because the certificate is not what it expects (which is what you would normally want as effectively we are performing a man-in-the-middle attack on ourselves)

Is there a way to either ignore the certificate check, or foce sftpgo to use http instead of https ?

Thanks,
Antony.

[drakkan]

Hi,

for both S3 and Azure we can specify a custom endpoint and so use an emulator or an alternate implementation. I'll add the same option for GCS too.

[drakkan]

Hi,,

I've got a question around the GCS vfs module - as part of our deploy and test pipeline, we are trying to mock GCS using the fakeGCS package. Everything we are building uses Docker, so the DNS lookup for the google storage is working as expected and redirecting to the fake GCS package. However, sftpgo is failing to connect to it because the certificate is not what it expects (which is what you would normally want as effectively we are performing a man-in-the-middle attack on ourselves)

Is there a way to either ignore the certificate check, or foce sftpgo to use http instead of https ?

Thanks, Antony.

Hi,

I did a quick test, setting an env var like this STORAGE_EMULATOR_HOST="http://192.168.1.12:4443" seems enough. In this way all the GCS users will use the specified endpoint. If you need a per-user endpoint I'll add an option to the GCS configuration

[drakkan]

I started to write a patch to add the endpoint per-user

---
 go.mod                           |  1 +
 go.sum                           |  2 --
 httpd/httpd_test.go              |  4 ++++
 httpd/webadmin.go                |  1 +
 httpdtest/httpdtest.go           |  3 +++
 openapi/openapi.yaml             |  3 +++
 templates/webadmin/fsconfig.html | 33 ++++++++++++++++++++++----------
 vfs/filesystem.go                |  1 +
 vfs/gcsfs.go                     | 13 ++++++++++---
 vfs/vfs.go                       |  3 +++
 10 files changed, 49 insertions(+), 15 deletions(-)

diff --git a/go.mod b/go.mod
index 3e5a592..88ee7ba 100644
--- a/go.mod
+++ b/go.mod
@@ -138,6 +138,7 @@ replace (
 	github.com/eikenb/pipeat => github.com/drakkan/pipeat v0.0.0-20210805162858-70e57fa8a639
 	github.com/fclairamb/ftpserverlib => github.com/drakkan/ftpserverlib v0.0.0-20220113173527-7442aa777ac0
 	github.com/jlaffaye/ftp => github.com/drakkan/ftp v0.0.0-20201114075148-9b9adce499a9
+	github.com/sftpgo/sdk => /home/nicola/goprojects/sftpgo-sdk
 	golang.org/x/crypto => github.com/drakkan/crypto v0.0.0-20220130095207-a206cf284b7c
 	golang.org/x/net => github.com/drakkan/net v0.0.0-20220130095023-bd85f1236c34
 )
diff --git a/go.sum b/go.sum
index b978f70..efe618b 100644
--- a/go.sum
+++ b/go.sum
@@ -691,8 +691,6 @@ github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdh
 github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
 github.com/secsy/goftp v0.0.0-20200609142545-aa2de14babf4 h1:PT+ElG/UUFMfqy5HrxJxNzj3QBOf7dZwupeVC+mG1Lo=
 github.com/secsy/goftp v0.0.0-20200609142545-aa2de14babf4/go.mod h1:MnkX001NG75g3p8bhFycnyIjeQoOjGL6CEIsdE/nKSY=
-github.com/sftpgo/sdk v0.0.0-20220201111021-563c373f8012 h1:tkzS0kxhatqIVrWZePzsFlp1xQgR9q6Wt0UYKsBiCUU=
-github.com/sftpgo/sdk v0.0.0-20220201111021-563c373f8012/go.mod h1:gcYbk4z578GfwbC9kJOz2rltYoPYUIcGZgV13r74MJw=
 github.com/shirou/gopsutil/v3 v3.21.13-0.20220106132423-a3ae4bc40d26 h1:nkvraEu1xs6D3AimiR9SkIOCG6lVvVZRfwbbQ7fX1DY=
 github.com/shirou/gopsutil/v3 v3.21.13-0.20220106132423-a3ae4bc40d26/go.mod h1:BToYZVTlSVlfazpDDYFnsVZLaoRG+g8ufT6fPQLdJzA=
 github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24/go.mod h1:M+9NzErvs504Cn4c5DxATwIqPbtswREoFCre64PpcG4=
diff --git a/httpd/httpd_test.go b/httpd/httpd_test.go
index e90bcca..f0b588c 100644
--- a/httpd/httpd_test.go
+++ b/httpd/httpd_test.go
@@ -2981,6 +2981,7 @@ func TestUserHiddenFields(t *testing.T) {
 	u2.FsConfig.GCSConfig.Bucket = "test"
 	u2.FsConfig.GCSConfig.Credentials = kms.NewPlainSecret("fake credentials")
 	u2.FsConfig.GCSConfig.ACL = "bucketOwnerRead"
+	u2.FsConfig.GCSConfig.Endpoint = "http://127.0.0.1:4443"
 	user2, _, err := httpdtest.AddUser(u2, http.StatusCreated)
 	assert.NoError(t, err)
 
@@ -14788,6 +14789,7 @@ func TestWebUserGCSMock(t *testing.T) {
 	user.FsConfig.GCSConfig.KeyPrefix = "somedir/subdir/"
 	user.FsConfig.GCSConfig.StorageClass = "standard"
 	user.FsConfig.GCSConfig.ACL = "publicReadWrite"
+	user.FsConfig.GCSConfig.Endpoint = "http://192.168.1.12:4443"
 	form := make(url.Values)
 	form.Set(csrfFormToken, csrfToken)
 	form.Set("username", user.Username)
@@ -14812,6 +14814,7 @@ func TestWebUserGCSMock(t *testing.T) {
 	form.Set("gcs_bucket", user.FsConfig.GCSConfig.Bucket)
 	form.Set("gcs_storage_class", user.FsConfig.GCSConfig.StorageClass)
 	form.Set("gcs_acl", user.FsConfig.GCSConfig.ACL)
+	form.Set("gcs_endpoint", user.FsConfig.GCSConfig.Endpoint)
 	form.Set("gcs_key_prefix", user.FsConfig.GCSConfig.KeyPrefix)
 	form.Set("pattern_path0", "/dir1")
 	form.Set("patterns0", "*.jpg,*.png")
@@ -14849,6 +14852,7 @@ func TestWebUserGCSMock(t *testing.T) {
 	assert.Equal(t, user.FsConfig.GCSConfig.Bucket, updateUser.FsConfig.GCSConfig.Bucket)
 	assert.Equal(t, user.FsConfig.GCSConfig.StorageClass, updateUser.FsConfig.GCSConfig.StorageClass)
 	assert.Equal(t, user.FsConfig.GCSConfig.ACL, updateUser.FsConfig.GCSConfig.ACL)
+	assert.Equal(t, user.FsConfig.GCSConfig.Endpoint, updateUser.FsConfig.GCSConfig.Endpoint)
 	assert.Equal(t, user.FsConfig.GCSConfig.KeyPrefix, updateUser.FsConfig.GCSConfig.KeyPrefix)
 	if assert.Len(t, updateUser.Filters.FilePatterns, 1) {
 		assert.Equal(t, "/dir1", updateUser.Filters.FilePatterns[0].Path)
diff --git a/httpd/webadmin.go b/httpd/webadmin.go
index fef566a..d3fee81 100644
--- a/httpd/webadmin.go
+++ b/httpd/webadmin.go
@@ -998,6 +998,7 @@ func getGCSConfig(r *http.Request) (vfs.GCSFsConfig, error) {
 	config.Bucket = r.Form.Get("gcs_bucket")
 	config.StorageClass = r.Form.Get("gcs_storage_class")
 	config.ACL = r.Form.Get("gcs_acl")
+	config.Endpoint = r.Form.Get("gcs_endpoint")
 	config.KeyPrefix = r.Form.Get("gcs_key_prefix")
 	autoCredentials := r.Form.Get("gcs_auto_credentials")
 	if autoCredentials != "" {
diff --git a/httpdtest/httpdtest.go b/httpdtest/httpdtest.go
index d55dcd6..ce5be99 100644
--- a/httpdtest/httpdtest.go
+++ b/httpdtest/httpdtest.go
@@ -1332,6 +1332,9 @@ func compareGCSConfig(expected *vfs.Filesystem, actual *vfs.Filesystem) error {
 	if expected.GCSConfig.AutomaticCredentials != actual.GCSConfig.AutomaticCredentials {
 		return errors.New("GCS automatic credentials mismatch")
 	}
+	if expected.GCSConfig.Endpoint != actual.GCSConfig.Endpoint {
+		return errors.New("GCS endpoint mismatch")
+	}
 	return nil
 }
 
diff --git a/openapi/openapi.yaml b/openapi/openapi.yaml
index a1a47a4..4c11d6b 100644
--- a/openapi/openapi.yaml
+++ b/openapi/openapi.yaml
@@ -4657,6 +4657,9 @@ components:
           type: string
           description: 'key_prefix is similar to a chroot directory for a local filesystem. If specified the user will only see contents that starts with this prefix and so you can restrict access to a specific virtual folder. The prefix, if not empty, must not start with "/" and must end with "/". If empty the whole bucket contents will be available'
           example: folder/subfolder/
+        endpoint:
+          type: string
+          description: optional endpoint. A custom endpoint is useful to test emulators and/or alternative implementations. Leave blank to use the default Google servers
       description: 'Google Cloud Storage configuration details. The "credentials" field must be populated only when adding/updating a user. It will be always omitted, since there are sensitive data, when you search/get users'
     AzureBlobFsConfig:
       type: object
diff --git a/templates/webadmin/fsconfig.html b/templates/webadmin/fsconfig.html
index 58729bb..368619a 100644
--- a/templates/webadmin/fsconfig.html
+++ b/templates/webadmin/fsconfig.html
@@ -209,17 +209,8 @@
         </div>
 
         <div class="form-group row fsconfig fsconfig-gcsfs">
-            <label for="idGCSKeyPrefix" class="col-sm-2 col-form-label">Key Prefix</label>
-            <div class="col-sm-3">
-                <input type="text" class="form-control" id="idGCSKeyPrefix" name="gcs_key_prefix" placeholder=""
-                    value="{{.GCSConfig.KeyPrefix}}" aria-describedby="GCSKeyPrefixHelpBlock">
-                <small id="GCSKeyPrefixHelpBlock" class="form-text text-muted">
-                    Similar to a chroot for local filesystem. Cannot start with "/". Example: "somedir/subdir/".
-                </small>
-            </div>
-            <div class="col-sm-2"></div>
             <label for="idGCSACL" class="col-sm-2 col-form-label">ACL</label>
-            <div class="col-sm-3">
+            <div class="col-sm-10">
                 <input type="text" class="form-control" id="idGCSACL" name="gcs_acl" placeholder=""
                     value="{{.GCSConfig.ACL}}" maxlength="255" aria-describedby="GCSACLHelpBlock">
                 <small id="GCSACLHelpBlock" class="form-text text-muted">
@@ -228,6 +219,28 @@
             </div>
         </div>
 
+        <div class="form-group row fsconfig fsconfig-gcsfs">
+            <label for="idGCSEndpoint" class="col-sm-2 col-form-label">Endpoint</label>
+            <div class="col-sm-10">
+                <input type="text" class="form-control" id="idGCSEndpoint" name="gcs_endpoint" placeholder=""
+                    value="{{.GCSConfig.Endpoint}}" aria-describedby="GCSEndpointHelpBlock">
+                <small id="GCSEndpointHelpBlock" class="form-text text-muted">
+                    Optional endpoint. Leave blank to use the default Google servers
+                </small>
+            </div>
+        </div>
+
+        <div class="form-group row fsconfig fsconfig-gcsfs">
+            <label for="idGCSKeyPrefix" class="col-sm-2 col-form-label">Key Prefix</label>
+            <div class="col-sm-10">
+                <input type="text" class="form-control" id="idGCSKeyPrefix" name="gcs_key_prefix" placeholder=""
+                    value="{{.GCSConfig.KeyPrefix}}" aria-describedby="GCSKeyPrefixHelpBlock">
+                <small id="GCSKeyPrefixHelpBlock" class="form-text text-muted">
+                    Similar to a chroot for local filesystem. Cannot start with "/". Example: "somedir/subdir/".
+                </small>
+            </div>
+        </div>
+
         <div class="form-group row fsconfig fsconfig-azblobfs">
             <label for="idAzContainer" class="col-sm-2 col-form-label">Container</label>
             <div class="col-sm-3">
diff --git a/vfs/filesystem.go b/vfs/filesystem.go
index 58bb80d..7962818 100644
--- a/vfs/filesystem.go
+++ b/vfs/filesystem.go
@@ -266,6 +266,7 @@ func (f *Filesystem) GetACopy() Filesystem {
 				StorageClass:         f.GCSConfig.StorageClass,
 				ACL:                  f.GCSConfig.ACL,
 				KeyPrefix:            f.GCSConfig.KeyPrefix,
+				Endpoint:             f.GCSConfig.Endpoint,
 			},
 			Credentials: f.GCSConfig.Credentials.Clone(),
 		},
diff --git a/vfs/gcsfs.go b/vfs/gcsfs.go
index 12429bb..8427c46 100644
--- a/vfs/gcsfs.go
+++ b/vfs/gcsfs.go
@@ -74,14 +74,20 @@ func NewGCSFs(connectionID, localTempDir, mountPath string, config GCSFsConfig)
 		return fs, err
 	}
 	ctx := context.Background()
+	var opts []option.ClientOption
+	if fs.config.Endpoint != "" {
+		opts = append(opts, option.WithEndpoint(fs.config.Endpoint))
+		opts = append(opts, option.WithoutAuthentication())
+	}
 	if fs.config.AutomaticCredentials > 0 {
-		fs.svc, err = storage.NewClient(ctx)
+		fs.svc, err = storage.NewClient(ctx, opts...)
 	} else if !fs.config.Credentials.IsEmpty() {
 		err = fs.config.Credentials.TryDecrypt()
 		if err != nil {
 			return fs, err
 		}
-		fs.svc, err = storage.NewClient(ctx, option.WithCredentialsJSON([]byte(fs.config.Credentials.GetPayload())))
+		opts = append(opts, option.WithCredentialsJSON([]byte(fs.config.Credentials.GetPayload())))
+		fs.svc, err = storage.NewClient(ctx, opts...)
 	} else {
 		var creds []byte
 		creds, err = os.ReadFile(fs.config.CredentialFile)
@@ -97,7 +103,8 @@ func NewGCSFs(connectionID, localTempDir, mountPath string, config GCSFsConfig)
 		if err != nil {
 			return fs, err
 		}
-		fs.svc, err = storage.NewClient(ctx, option.WithCredentialsJSON([]byte(secret.GetPayload())))
+		opts = append(opts, option.WithCredentialsJSON([]byte(secret.GetPayload())))
+		fs.svc, err = storage.NewClient(ctx, opts...)
 	}
 	return fs, err
 }
diff --git a/vfs/vfs.go b/vfs/vfs.go
index afce556..a99cd8a 100644
--- a/vfs/vfs.go
+++ b/vfs/vfs.go
@@ -318,6 +318,9 @@ func (c *GCSFsConfig) isEqual(other *GCSFsConfig) bool {
 	if c.ACL != other.ACL {
 		return false
 	}
+	if c.Endpoint != other.Endpoint {
+		return false
+	}
 	if c.Credentials == nil {
 		c.Credentials = kms.NewEmptySecret()
 	}
-- 
2.35.1

as you can see, to make the patch works in any case, I had to add

opts = append(opts, option.WithoutAuthentication())

if an endpoint is specified otherwise if no other credentials are provided the google storage sdk will fail with an error like this:

dialing: google: could not find default credentials

so I also need to add another option to disable the authentication and change the validation for the GCS fs too. This seems quite excessive for a test only scenario and setting an env var you can already get the same result.

Is the env var enough for you use case? Thank you

[AntonyKing7D]

STORAGE_EMULATOR_HOST seems to be enough for testing, thanks. I don't think we would make use of a 'per user' setting for it; this requirement is quite meta - it's a local pre-deployment test for the code that we have in place to run synthetic tests against the sftp endpoint.