Skip to content

[Elixir] add AnyType support #14071

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/generators/elixir.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ These options may be applied as additional-properties (cli) or configOptions (pl
|Uuid|✗|
|Array|✓|OAS2,OAS3
|Null|✗|OAS3
|AnyType||OAS2,OAS3
|AnyType||OAS2,OAS3
|Object|✓|OAS2,OAS3
|Maps|✓|ToolingExtension
|CollectionFormat|✓|OAS2
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,32 +17,57 @@

package org.openapitools.codegen.languages;

import com.samskivert.mustache.Mustache;
import com.samskivert.mustache.Template;
import io.swagger.v3.oas.models.OpenAPI;
import io.swagger.v3.oas.models.info.Info;
import io.swagger.v3.oas.models.media.ArraySchema;
import io.swagger.v3.oas.models.media.Schema;
import io.swagger.v3.oas.models.responses.ApiResponse;
import static org.openapitools.codegen.utils.StringUtils.camelize;
import static org.openapitools.codegen.utils.StringUtils.underscore;

import java.io.File;
import java.io.IOException;
import java.io.Writer;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.apache.commons.lang3.StringUtils;
import org.openapitools.codegen.*;
import org.openapitools.codegen.meta.features.*;
import org.openapitools.codegen.CliOption;
import org.openapitools.codegen.CodegenConstants;
import org.openapitools.codegen.CodegenModel;
import org.openapitools.codegen.CodegenOperation;
import org.openapitools.codegen.CodegenParameter;
import org.openapitools.codegen.CodegenProperty;
import org.openapitools.codegen.CodegenResponse;
import org.openapitools.codegen.CodegenType;
import org.openapitools.codegen.DefaultCodegen;
import org.openapitools.codegen.GeneratorLanguage;
import org.openapitools.codegen.SupportingFile;
import org.openapitools.codegen.meta.features.ClientModificationFeature;
import org.openapitools.codegen.meta.features.DataTypeFeature;
import org.openapitools.codegen.meta.features.DocumentationFeature;
import org.openapitools.codegen.meta.features.GlobalFeature;
import org.openapitools.codegen.meta.features.ParameterFeature;
import org.openapitools.codegen.meta.features.SchemaSupportFeature;
import org.openapitools.codegen.meta.features.SecurityFeature;
import org.openapitools.codegen.model.ModelMap;
import org.openapitools.codegen.model.OperationMap;
import org.openapitools.codegen.model.OperationsMap;
import org.openapitools.codegen.utils.ModelUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.IOException;
import java.io.Writer;
import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import com.samskivert.mustache.Mustache;
import com.samskivert.mustache.Template;

import static org.openapitools.codegen.utils.StringUtils.camelize;
import static org.openapitools.codegen.utils.StringUtils.underscore;
import io.swagger.v3.oas.models.OpenAPI;
import io.swagger.v3.oas.models.info.Info;
import io.swagger.v3.oas.models.media.ArraySchema;
import io.swagger.v3.oas.models.media.Schema;
import io.swagger.v3.oas.models.responses.ApiResponse;

public class ElixirClientCodegen extends DefaultCodegen {
private final Logger LOGGER = LoggerFactory.getLogger(ElixirClientCodegen.class);
Expand Down Expand Up @@ -88,6 +113,9 @@ public ElixirClientCodegen() {
.includeClientModificationFeatures(
ClientModificationFeature.BasePath
)
.includeDataTypeFeatures(
DataTypeFeature.AnyType
)
);

// set the output folder here
Expand Down Expand Up @@ -573,6 +601,8 @@ public String getTypeDeclaration(Schema p) {
return this.moduleName + ".Model." + super.getTypeDeclaration(p) + ".t";
} else if (ModelUtils.isFileSchema(p)) {
return "String.t";
} else if (ModelUtils.isAnyType(p)) {
return "any";
} else if (ModelUtils.isStringSchema(p)) {
return "String.t";
}
Expand Down Expand Up @@ -634,6 +664,7 @@ public ExtendedCodegenResponse(CodegenResponse o) {
this.isArray = o.isArray;
this.isBinary = o.isBinary;
this.isFile = o.isFile;
this.isAnyType = o.isAnyType;
this.schema = o.schema;
this.jsonSchema = o.jsonSchema;
this.vendorExtensions = o.vendorExtensions;
Expand Down Expand Up @@ -830,6 +861,8 @@ private void buildTypespec(CodegenParameter param, StringBuilder sb) {
sb.append(param.dataType);
} else if (param.isFile || param.isBinary) {
sb.append("String.t");
} else if (param.isAnyType) {
sb.append("any");
} else if ("String.t".equals(param.dataType)) {
// uuid, password, etc
sb.append(param.dataType);
Expand Down
10 changes: 6 additions & 4 deletions modules/openapi-generator/src/main/resources/elixir/api.mustache
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,18 @@ defmodule {{moduleName}}.Api.{{classname}} do
:body => :body
{{/isBodyParam}}
{{^isBodyParam}}
{{#atom}}{{baseName}}{{/atom}} => {{#isFormParam}}:form{{/isFormParam}}{{#isQueryParam}}:query{{/isQueryParam}}{{#isHeaderParam}}:headers{{/isHeaderParam}}{{^-last}},{{/-last}}
{{#atom}}{{&baseName}}{{/atom}} => {{#isFormParam}}:form{{/isFormParam}}{{#isQueryParam}}:query{{/isQueryParam}}{{#isHeaderParam}}:headers{{/isHeaderParam}}{{^-last}},{{/-last}}
{{/isBodyParam}}
{{#-last}}
}

{{/-last}}
{{/optionalParams}}
request =
%{}
|> method({{#atom}}{{#underscored}}{{httpMethod}}{{/underscored}}{{/atom}})
|> url("{{replacedPathName}}")
%Tesla.Env{
method: {{#atom}}{{#underscored}}{{httpMethod}}{{/underscored}}{{/atom}},
url: "{{replacedPathName}}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m not sure why this change was made. Yes, it saves two function calls, but it also means that it is harder to extract Tesla from the execution path in favour of https://github.com/wojtekmach/req.

(A different set of changes that I’m going to be looking at will be to replace the structs with validated maps, because there’s a lot of atoms generated in a sufficiently complex API.)

Copy link
Author

@paulbalomiri paulbalomiri Nov 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can revert this one. It helped me with debugging the generated code.

It shows from the get-go that the pipe inputs/outputs are a Tesla.Env.
I think a good way forward with Tesla would be to just compose a middleware stack instead of piping, which cannot be altered at runtime.

I don't have experience with req, but I can look into it if you are interested in also having continued support for Tesla.

(A different set of changes that I’m going to be looking at will be to replace the structs with validated maps, because there’s a lot of atoms generated in a sufficiently complex API.)

Do you want to completely do away with structs?
I concur with you on not creating atoms to the extent that they should not be generated from input (e.g. for values supplied in function calls).

Here, on the other hand, we do have a finite number of atoms, as each struct generates only it's fields' descriptors as atoms at compile time.

So IMHO this is not a case of unbounded atom generation. Ex/Beam is good at handling lots of atoms, their number should just not be unbounded or dependent on runtime and unsafe input.

Utility module for assemling middelwares ?

If you want I can look at creating a utility Module to handle a list of middlewares instead of the current approach. The Api functions would then only use this utility class to insert the necessary middlewares to set the url, params e.t.c.

This way the utility Module would allow for runtime transformations, even outside the spec (e.g. caching, delegation to other nodes e.t.c. from client libraries depending on the generated dep).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure about this discussion.

Since it's not related to the implementation of AnyType, it would be great to bring this concern into an issue. This would also allow the discussion a more retractable character.

For the change itself: (by itself, I just mean the line without the changing-possibility and debugability it could bring)

I can't see the additional value it brings for breaking the style of composing/piping, and therefore I can't just let it through.

I would appreciate it if you could extract this change into a different branch and start the discussion with an issue. You could also link the new branch in the form of a PR, so it doesn't get lost.

Copy link
Author

@paulbalomiri paulbalomiri Nov 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will come back with a new pr, on this
EDIT
cned here: #14070 (comment)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comments on removing atoms are completely orthogonal to this discussion, just brought up as a point (I’ve hit a very large OpenAPI spec that makes a lot more than I’m comfortable with, but yes—I want to explore how structs get handled).

}
{{#requiredParams}}
{{^isPathParam}}
|> add_param({{#isBodyParam}}:body{{/isBodyParam}}{{#isFormParam}}{{#isMultipart}}{{#isFile}}:file{{/isFile}}{{^isFile}}:form{{/isFile}}{{/isMultipart}}{{^isMultipart}}:form{{/isMultipart}}{{/isFormParam}}{{#isQueryParam}}:query{{/isQueryParam}}{{#isHeaderParam}}:headers{{/isHeaderParam}}, {{#isBodyParam}}:body{{/isBodyParam}}{{^isBodyParam}}{{#atom}}{{baseName}}{{/atom}}{{/isBodyParam}}, {{#underscored}}{{paramName}}{{/underscored}})
Expand All @@ -72,6 +73,7 @@ defmodule {{moduleName}}.Api.{{classname}} do
{{#requiresHttpcWorkaround}}
|> ensure_body()
{{/requiresHttpcWorkaround}}
|> Map.from_struct()
|> Enum.into([])

connection
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ defmodule {{moduleName}}.Deserializer do
model
|> Map.update!(field, &(Poison.Decode.decode(&1, Keyword.merge(options, [as: [struct(mod)]]))))
end

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra whitespace.

def deserialize(model, field, :struct, mod, options) do
model
|> Map.update!(field, &(Poison.Decode.decode(&1, Keyword.merge(options, [as: struct(mod)]))))
end

# Just keep the value parsed from json for any
def deserialize(model, field, :any, mod, options), do: model
def deserialize(model, field, :map, mod, options) do
maybe_transform_map = fn
nil ->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
]

@type t :: %__MODULE__{
{{#vars}}{{#atom}}{{&baseName}}{{/atom}} => {{{datatype}}}{{#isNullable}} | nil{{/isNullable}}{{^isNullable}}{{^required}} | nil{{/required}}{{/isNullable}}{{^-last}},
{{#vars}}{{#atom}}{{&baseName}}{{/atom}} => {{#isAnyType}}any(){{/isAnyType}}{{^isAnyType}}{{{datatype}}}{{/isAnyType}}{{#isNullable}} | nil{{/isNullable}}{{^isNullable}}{{^required}} | nil{{/required}}{{/isNullable}}{{^-last}},
{{/-last}}{{/vars}}
}
end
Expand All @@ -23,7 +23,7 @@ defimpl Poison.Decoder, for: {{&moduleName}}.Model.{{&classname}} do
value
{{#vars}}
{{^isPrimitiveType}}
{{#baseType}}|> deserialize({{#atom}}{{&baseName}}{{/atom}}, {{#isArray}}:list, {{&moduleName}}.Model.{{{items.baseType}}}{{/isArray}}{{#isMap}}:map, {{&moduleName}}.Model.{{{items.baseType}}}{{/isMap}}{{#isDate}}:date, nil{{/isDate}}{{#isDateTime}}:date, nil{{/isDateTime}}{{^isDate}}{{^isDateTime}}{{^isMap}}{{^isArray}}:struct, {{moduleName}}.Model.{{baseType}}{{/isArray}}{{/isMap}}{{/isDateTime}}{{/isDate}}, options)
{{#baseType}}|> deserialize({{#atom}}{{&baseName}}{{/atom}}, {{#isAnyType}}:any, :any{{/isAnyType}}{{#isArray}}:list, {{&moduleName}}.Model.{{{items.baseType}}}{{/isArray}}{{#isMap}}:map, {{&moduleName}}.Model.{{{items.baseType}}}{{/isMap}}{{#isDate}}:date, nil{{/isDate}}{{#isDateTime}}:date, nil{{/isDateTime}}{{^isDate}}{{^isDateTime}}{{^isMap}}{{^isArray}}{{^isAnyType}}:struct, {{moduleName}}.Model.{{baseType}}{{/isAnyType}}{{/isArray}}{{/isMap}}{{/isDateTime}}{{/isDate}}, options)
{{/baseType}}
{{/isPrimitiveType}}
{{/vars}}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ defmodule {{moduleName}}.RequestBuilder do
end

def add_param(request, :headers, key, value) do
Tesla.put_header(request, key, value)
Tesla.put_header(request, to_string(key), value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that Tesla headers (like Phoenix headers) are supposed to be all lowercase, we may want something a little more robust than just Kernel.to_string/1.

Copy link
Author

@paulbalomiri paulbalomiri Nov 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, also because Http header field names are case insensitive

key
|> to_string()
|> String.downcase()

Would this be sufficient, or do you mean something different by "robust" ?

Copy link
Contributor

@halostatue halostatue Nov 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’d be a start, yes, and a good one.

Copy link
Contributor

@mrmstn mrmstn Nov 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Funny enough, this function call is actually wrong and Tesla.put_header must not be used here anyway.

The function should look something like this:

  def add_param(request, :headers, key, value) do
    headers =
      request
      |> Map.get(:headers, [])
      |> List.keystore(key, 0, {key, value})

    Map.put(request, :headers, headers)
  end

Secondly, I wouldn't change cases as the server should/would do this to respect the case insensitivity

Copy link
Author

@paulbalomiri paulbalomiri Nov 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR

I promised to come back, only to now postpone 2 more weeks. Shall we leave this open, or shall i create new pr's and retire this PR ?
This is still a blocker for my release & i don't want to keep fork around and release [my software], so i'll definitely come back on this.

New PR proposal

I'd like to address these topics in separate or in one PR:

  • Anytype support
  • Header bugfix, using camel_cased :atoms not :Atom for headers
  • Supply some generated usage example in Readme.md using options EDIT: showcasing the supplied generator options for documentation.
Tesla configurability

Separately I'd like to use exposed middlewares in Tesla, because I think that is what Tesla's composability is about, instead of creating and running the client in it's api functions, but we'll have to decide whether that is a custom generator, an alternative generator in this repo or whether we come to the conclusion that req and tesla should be configurable settings of the same generator.

The Api implementation should just compose a list of middlewares, applied to a user supplied (and defaulted) %Tesla.Env{}
deserialisation e.t.c. should be implemented via middleware composition (e.g. one per field).
This way the lib user could also use non-spec ways of modifying the calls (authentication, session headers, caching e.t.c), while OFC the default is to run according to spec

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With respect to the Tesla configurability, I’d need to see more of what you think. We may be thinking similarly, but I have absolutely shifted my thinking on Tesla away from use Tesla with plug macros and instead using runtime client building. In my opinion, this item should absolutely be opened as a discussion / discussion ticket, because there are several modernizations to the Elixir code base that should be done beyond the work that I did earlier this year and may not have time to get to myself.

end

def add_param(request, :file, name, path) do
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,12 @@ defmodule OpenapiPetstore.Api.AnotherFake do
@spec call_123_test_special_tags(Tesla.Env.client, OpenapiPetstore.Model.Client.t, keyword()) :: {:ok, OpenapiPetstore.Model.Client.t} | {:error, Tesla.Env.t}
def call_123_test_special_tags(connection, client, _opts \\ []) do
request =
%{}
|> method(:patch)
|> url("/another-fake/dummy")
%Tesla.Env{
method: :patch,
url: "/another-fake/dummy",
}
|> add_param(:body, :body, client)
|> Map.from_struct()
|> Enum.into([])

connection
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,11 @@ defmodule OpenapiPetstore.Api.Default do
@spec foo_get(Tesla.Env.client, keyword()) :: {:ok, OpenapiPetstore.Model.FooGetDefaultResponse.t} | {:error, Tesla.Env.t}
def foo_get(connection, _opts \\ []) do
request =
%{}
|> method(:get)
|> url("/foo")
%Tesla.Env{
method: :get,
url: "/foo",
}
|> Map.from_struct()
|> Enum.into([])

connection
Expand Down
Loading