Merge pull request #60 from golemcloud/go-multi-rpc

Go example project with RPC and multiple components

Author: noise64

README.md

@@ -24,17 +24,19 @@ The following fields are required:
 The following fields are optional:
 
 - `requiresAdapter` is a boolean, defaults to **true**. If true, the appropriate version of the WASI Preview2 to Preview1 adapter is copied into the generated project (based on the guest language) to an `adapters` directory.
-- `requiresGolemHostWIT` is a boolean, defaults to **false**. If true, the Golem specific WIT interface gets copied into `wit/deps`.
+- `requiresGolemHostWIT` is a boolean, defaults to **false**. If true, the Golem specific WIT interface gets copied into `wit/deps`. 
 - `requiresWASI` is a boolean, defaults to **false**. If true, the WASI Preview2 WIT interfaces which are compatible with Golem Cloud get copied into `wit/deps`.
+- `witDepsPaths` is an array of directory paths, defaults to **null**. When set, overrides the `wit/deps` directory for the above options and allows to use multiple target dirs for supporting multi-component examples.
 - `exclude` is a list of sub-paths and works as a simplified `.gitignore` file. It's primary purpose is to help the development loop of working on examples and in the future it will likely be dropped in favor of just using `.gitignore` files.
+- `instructions` is an optional filename, defaults to **null**. When set, overrides the __INSTRUCTIONS__ file used for the example, the file needs to be placed to same directory as the default instructions file.
 
 ### Template rules
 
 Golem examples are currently simple and not using any known template language, in order to keep the examples **compilable** as they are - this makes it very convenient to work on existing ones and add new examples as you can immediately verify that it can be compiled into a _Golem template_.
 
 When calling `golem-new` the user specifies a **template name**. The provided component name must use either `PascalCase`, `snake_case` or `kebab-case`.
 
-There is an optional parameter for defining a **package name**, which defaults to `golem:component`. It has to be in the `pack:name` format.
+There is an optional parameter for defining a **package name**, which defaults to `golem:component`. It has to be in the `pack:name` format. The first part of the package name is called **package namespace**.
 
 The following occurrences get replaced to the provided component name, applying the casing used in the template:
 - `component-name`
@@ -46,6 +48,8 @@ The following occurrences get replaced to the provided component name, applying
 - `pack-name`
 - `pack/name`
 - `PackName`
+- `pack-ns`
+- `PackNs`
 
 ### Testing the examples
 The example generation and instructions can be tested with a test [cli app](/src/test/main.rs).

examples/go/INSTRUCTIONS-mage

@@ -0,0 +1,14 @@
+See the documentation about installing tooling: https://learn.golem.cloud/docs/go-language-guide/setup
+and see more details in the README.md file about other commands.
+
+Before first time build, generate RPC stubs:
+  go run mage.go updateRpcStubs
+
+Generate bindings, compile and compose components:
+  go run mage.go build
+
+Deploy components to default profile:
+  go run mage.go deploy
+
+Test the deployed components:
+  go run mage.go testIntegration

examples/go/go-multi-rpc/.gitignore

@@ -0,0 +1,2 @@
+binding
+/target/

examples/go/go-multi-rpc/README.md

@@ -0,0 +1,188 @@
+# Golem Go Example with Multiple Components and Worker to Worker RPC Communication
+
+## Building
+The project uses [magefile](https://magefile.org/) for building. Either install the tool binary `mage`,
+or use the __zero install option__: `go run mage.go`. This readme will use the latter.
+
+To see the available commands use:
+
+```shell
+go run mage.go
+Targets:
+  addStubDependency             adds generated and built stub dependency to componentGolemCliAddStubDependency
+  build                         alias for BuildAllComponents
+  buildAllComponents            builds all components
+  buildComponent                builds component by name
+  buildStubComponent            builds RPC stub for component
+  clean                         cleans the projects
+  deploy                        adds or updates all the components with golem-cli\'s default profile
+  generateBinding               generates go binding from WIT
+  generateNewComponent          generates a new component based on the component-template
+  stubCompose                   composes dependencies
+  testIntegration               tests the deployed components
+  tinyGoBuildComponentBinary    build wasm component binary with tiny go
+  updateRpcStubs                builds rpc stub components and adds them as dependency
+  wasmToolsComponentEmbed       embeds type info into wasm component with wasm-tools
+  wasmToolsComponentNew         create golem component with wasm-tools
+```
+
+For building the project for the first time (or after `clean`) use the following commands:
+
+```shell
+go run mage.go updateRpcStubs
+go run mage.go build
+```
+
+After this, using the `build` command is enough, unless there are changes in the RPC dependencies,
+in that case `updateRpcStubs` is needed again.
+
+The final components that are usable by golem are placed in the `target/components` folder.
+
+## Deploying and testing the example
+
+In the example 3 simple counter components are defined, which can be familiar from the smaller examples. To showcase the remote calls, the counters `add` functions are connected, apart from increasing their own counter:
+ - **component one** delegates the add call to **component two** and **three** too,
+ - and **component two** delegates to **component three**.
+
+In both cases the _current worker name_ will be used as _target worker name_ too. 
+
+Apart from _worker name_, remote calls also require the **target components' deployed ID**. For this the example uses environment variables, and uses the `lib/cfg` subpackage (which is shared between the components) to extract it.
+
+The examples assume a configured default `golem-cli` profile, and will use that.
+
+To test, first we have to build the project as seen in the above:
+
+```shell
+go run mage.go updateRpcStubs
+go run mage.go build
+```
+
+Then we can deploy our components with `golem-cli`, for which a wrapper magefile command is provided:
+
+```shell
+go run mage.go deploy
+```
+
+Once the components are deployed, a simple example integration test suite can be used to test the components.
+The tests are in the [/integration/integration_test.go](/integration/integration_test.go) test file, and can be run with:
+
+```shell
+go run mage.go testIntegration
+```
+
+The `TestDeployed` simply tests if our components metadata is available through `golem-cli component get`.
+
+The `TestCallingAddOnComponentOneCallsToOtherComponents` will:
+ - get the _component URNs_ with `golem-cli component get`
+ - generates a _random worker name_, so our tests are starting from a clean state
+ - adds 1 - 1 worker for component one and component two with the required _environment variables_ containing the other workers' _component ids_
+ - then makes various component invocations with `golem-cli worker invoke-and-await` and tests if the counters - after increments -  are holding the right value according to the delegated `add` function calls.
+
+## Adding Components
+
+Use the `generateNewComponent` command to add new components to the project:
+
+```shell
+go run mage.go generateNewComponent component-four
+```
+
+The above will create a new component in the `components/component-four` directory based on the template at [/component-template/component](/component-template/component).
+
+After adding a new component the `build` command will also include it.
+
+## Using Worker to Worker RPC calls
+
+### Under the hood 
+
+Under the hood the _magefile_ commands below (and for build) use generic `golem-cli stubgen` subcommands:
+ - `golem-cli stubgen build` for creating remote call _stub WIT_ definitions and _WASM components_ for the stubs
+ - `golem-cli stubgen add-stub-dependency` for adding the _stub WIT_ definitions to a _component's WIT_ dependencies
+ - `golem-cli stubgen compose` for _composing_ components with the stub components
+
+### Magefile commands and required manual steps
+
+The dependencies between components are defined in  the [/magefiles/magefile.go](/magefiles/magefile.go) build script:
+
+```go
+// componentDeps defines the Worker to Worker RPC dependencies
+var componentDeps = map[string][]string{
+    "component-one": {"component-two", "component-three"},
+    "component-two": {"component-three"},
+}
+```
+
+After changing dependencies the `updateRpcStubs` command can be used to create the necessary stubs:
+
+```shell
+go run mage.go updateRpcStubs
+```
+
+The command will create stubs for the dependency projects in the ``/target/stub`` directory and will also place the required stub _WIT_ interfaces on the dependant component's `wit/deps` directory.
+
+To actually use the dependencies in a project it also has to be manually imported in the component's world.
+
+E.g. with the above definitions the following import has to be __manually__ added to `/components/component-one/wit/component-one.wit`:
+
+```wit
+import pack-ns:component-two-stub;
+import pack-ns:component-three-stub;
+```
+
+So the component definition should like similar to this:
+
+```wit
+package pack-ns:component-one;
+
+// See https://component-model.bytecodealliance.org/design/wit.html for more details about the WIT syntax
+
+interface component-one-api {
+  add: func(value: u64);
+  get: func() -> u64;
+}
+
+world component-one {
+  // Golem dependencies
+  import golem:api/host@0.2.0;
+  import golem:rpc/types@0.1.0;
+
+  // WASI dependencies
+  import wasi:blobstore/blobstore;
+  // .
+  // .
+  // .
+  // other dependencies
+  import wasi:sockets/instance-network@0.2.0;
+
+  // Project Component dependencies
+  import pack-ns:component-two-stub;
+  import pack-ns:component-three-stub;
+
+  export component-one-api;
+}
+```
+
+After this `build` (or the `generateBinding`) command can be used to update bindings, which now should include the
+required functions for calling other components.
+
+Here's an example that delegates the `Add` call to another component and waits for the result:
+
+```go
+import (
+	"github.com/golemcloud/golem-go/std"
+
+	"golem-go-project/components/component-one/binding"
+)
+
+
+func (i *Impl) Add(value uint64) {
+    std.Init(std.Packages{Os: true, NetHttp: true})
+    
+    componentTwo := binding.NewComponentTwoApi(binding.GolemRpc0_1_0_TypesUri{Value: "uri"})
+    defer componentTwo.Drop()
+    componentTwo.BlockingAdd(value)
+
+    i.counter += value
+}
+```
+
+Once a remote call is in place, the `build` command will also compose the stub components into the caller component.

examples/go/go-multi-rpc/component-generator/main.go

@@ -0,0 +1,142 @@
+package main
+
+import (
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+func main() {
+	if len(os.Args) != 3 {
+		exit(0, fmt.Sprintf("Usage: %s <package"+"-org> <component"+"-name>", os.Args[0]))
+	}
+
+	componentTemplateRoot := "component-template/component"
+	pkgOrg := os.Args[1]
+	compName := os.Args[2]
+	componentDir := filepath.Join("components", compName)
+
+	_, err := os.Stat(componentDir)
+	if err == nil {
+		exit(1, fmt.Sprintf("Target component directory already exists: %s", componentDir))
+	}
+	if err != nil && !os.IsNotExist(err) {
+		exit(1, err.Error())
+	}
+
+	err = os.MkdirAll(componentDir, 0755)
+	if err != nil {
+		exit(1, err.Error())
+	}
+
+	err = fs.WalkDir(
+		os.DirFS(componentTemplateRoot),
+		".",
+		func(path string, d fs.DirEntry, err error) error {
+			srcFilePath := filepath.Join(componentTemplateRoot, path)
+			fileInfo, err := os.Stat(srcFilePath)
+			if err != nil {
+				return fmt.Errorf("stat failed for template %s, %w", srcFilePath, err)
+			}
+
+			if fileInfo.IsDir() {
+				return nil
+			}
+
+			switch path {
+			case "main.go":
+				err = generateFile(pkgOrg, compName, srcFilePath, filepath.Join(componentDir, path))
+			case "wit/component.wit":
+				err = generateFile(pkgOrg, compName, srcFilePath, filepath.Join(componentDir, "wit", compName+".wit"))
+			default:
+				err = copyFile(srcFilePath, filepath.Join(componentDir, path))
+			}
+			if err != nil {
+				return fmt.Errorf("template generation failed for %s, %w", srcFilePath, err)
+			}
+
+			return nil
+		})
+	if err != nil {
+		exit(1, err.Error())
+	}
+}
+
+func generateFile(pkgOrg, compName, srcFileName, dstFileName string) error {
+	pascalPkgOrg := dashToPascal(pkgOrg)
+	pascalCompName := dashToPascal(compName)
+
+	fmt.Printf("Generating from %s to %s\n", srcFileName, dstFileName)
+
+	contentsBs, err := os.ReadFile(srcFileName)
+	if err != nil {
+		return fmt.Errorf("generateFile: read file failed for %s, %w", srcFileName, err)
+	}
+
+	contents := string(contentsBs)
+
+	contents = strings.ReplaceAll(contents, "comp-name", compName)
+	contents = strings.ReplaceAll(contents, "pck-ns", pkgOrg)
+	contents = strings.ReplaceAll(contents, "CompName", pascalCompName)
+	contents = strings.ReplaceAll(contents, "PckNs", pascalPkgOrg)
+
+	dstDir := filepath.Dir(dstFileName)
+	err = os.MkdirAll(dstDir, 0755)
+	if err != nil {
+		return fmt.Errorf("generateFile: mkdir failed for %s, %w", dstDir, err)
+	}
+
+	err = os.WriteFile(dstFileName, []byte(contents), 0644)
+	if err != nil {
+		return fmt.Errorf("generateFile: write file failed for %s, %w", dstFileName, err)
+	}
+
+	return nil
+}
+
+func dashToPascal(s string) string {
+	parts := strings.Split(s, "-")
+	for i, part := range parts {
+		if len(part) > 0 {
+			parts[i] = strings.ToUpper(string(part[0])) + part[1:]
+		}
+	}
+	return strings.Join(parts, "")
+}
+
+func copyFile(srcFileName, dstFileName string) error {
+	fmt.Printf("Copy %s to %s\n", srcFileName, dstFileName)
+
+	src, err := os.Open(srcFileName)
+	if err != nil {
+		return fmt.Errorf("copyFile: open failed for %s, %w", srcFileName, err)
+	}
+	defer func() { _ = src.Close() }()
+
+	dstDir := filepath.Dir(dstFileName)
+	err = os.MkdirAll(dstDir, 0755)
+	if err != nil {
+		return fmt.Errorf("copyFile: mkdir failed for %s, %w", dstDir, err)
+	}
+
+	dst, err := os.Create(dstFileName)
+	if err != nil {
+		return fmt.Errorf("copyFile: create file failed for %s, %w", dstFileName, err)
+	}
+	defer func() { _ = dst.Close() }()
+
+	_, err = io.Copy(dst, src)
+	if err != nil {
+		return fmt.Errorf("copyFile: copy failed from %s to %s, %w", srcFileName, dstFileName, err)
+	}
+
+	return nil
+}
+
+func exit(code int, message string) {
+	fmt.Println(message)
+	os.Exit(code)
+}