BaseMiddleware is nil when using endpoint-level custom Go plugin (but plugin still works)

[fnkayadev]

Branch/Environment/Version

• Branch/Version: release-5.8.2

Describe the bug
I’m using an endpoint-level custom Go plugin in Tyk Gateway, and everything works as expected — but during startup, I see a confusing log error related to m.BaseMiddleware being nil.

Reproduction steps
Steps to reproduce the behavior:

My api definition

tyk_plugins.go

Logs (debug mode or log file):
time="Jul 17 12:16:31" level=error msg="Recovered from panic while loading Go-plugin" error="runtime error: invalid memory address or nil pointer dereference" mwPath=plugins/tyk_plugins.so mwSymbolName=ClearRate

Configuration (tyk config file):
tyk.conf.json
Configs from envs
API_LISTEN_PATH: "/local/"
API_URL: "http://httpbin"
TYK_GW_SECRET: "SECRET"
TYK_GW_APPPATH: "config/apps/"
TYK_GW_MIDDLEWAREPATH: "tyk/middleware/"
TYK_GW_POLICIES_POLICY_PATH: "tyk/middleware/"
TYK_GW_TEMPLATEPATH: "tyk-gw/templates"
TYK_GW_LOGLEVEL: "debug"

in the plugin loader code, I see the following lines in func (m *GoPluginMiddleware) loadPlugin() bool in mw_go_plugin.go:

At this point, m.BaseMiddleware is nil, which causes a log error during Tyk Gateway startup.

Behavior Observed
• The /clear endpoint is correctly handled by the plugin function (ClearRate).
• It does not proxy the request and returns a custom response (like a virtual endpoint), which is the desired behavior.
• However, during bootstrapping, Tyk logs an error related to m.BaseMiddleware.Copy() because m.BaseMiddleware is nil at that time.

Questions
1. Is it expected for BaseMiddleware to be nil for endpoint-level Go plugins (i.e., virtual endpoint-like behavior)?
2. If m.BaseMiddleware is not required in this case, should Tyk avoid calling .Copy() or initializing successHandler altogether?
3. Can this log error be suppressed or handled more gracefully when the plugin is not using BaseMiddleware?

[buger]

/probe - what do you think?

[github-actions]

Analysis of BaseMiddleware nil issue with endpoint-level Go plugins

Issue Description

The user is experiencing a nil pointer dereference error during Tyk Gateway startup when using an endpoint-level custom Go plugin. The error occurs in the loadPlugin() method of EndpointLevelGoPluginMiddleware when trying to access m.BaseMiddleware.Copy(), but m.BaseMiddleware is nil at that point.

Root Cause

After examining the code, I can identify the exact cause of the issue:

1. When a request matches an endpoint-level Go plugin, the CheckSpecMatchesStatus function in api_loader.go creates a new EndpointLevelGoPluginMiddleware instance with only the Path and SymbolName fields set:

return true, &EndpointLevelGoPluginMiddleware{
    Path:       v.GoPluginMeta.Path,
    SymbolName: v.GoPluginMeta.SymbolName,
}

2. This new middleware instance doesn't have its BaseMiddleware field initialized.

3. Later, when loadPlugin() is called on this middleware, it tries to create a SuccessHandler with m.BaseMiddleware.Copy():

// to record 2XX hits in analytics
m.successHandler = &SuccessHandler{BaseMiddleware: m.BaseMiddleware.Copy()}

4. Since m.BaseMiddleware is nil, calling Copy() on it results in a nil pointer dereference.

Why It Still Works

Despite this error, the plugin still works correctly at runtime because:

1. The loadPlugin() method is called again when a request is actually processed
2. The error is caught by the defer recovery block in the method
3. The plugin function is still executed correctly

Solution

The simplest solution would be to add a nil check in the loadPlugin() method before trying to call Copy() on m.BaseMiddleware. This would prevent the panic while still allowing the plugin to function correctly:

func (m *EndpointLevelGoPluginMiddleware) loadPlugin() bool {
	defer func() {
		if e := recover(); e != nil {
			err = fmt.Errorf("%v", e)
			m.Logger.WithError(err).Error("Recovered from panic while loading Go-plugin")
		}
	}()

	if m.handler, err = goplugin.GetHandler(m.Path, m.SymbolName); err != nil {
		m.Logger.WithError(err).Error("Could not load Go-plugin")
		return false
	}

	// to record 2XX hits in analytics
	if m.BaseMiddleware != nil {
		m.successHandler = &SuccessHandler{BaseMiddleware: m.BaseMiddleware.Copy()}
	}
	return true
}

Alternatively, a more comprehensive solution would be to properly initialize the BaseMiddleware field when creating the EndpointLevelGoPluginMiddleware in CheckSpecMatchesStatus. However, this would require more significant changes to the codebase.

Answers to User's Questions

1. Is it expected for BaseMiddleware to be nil for endpoint-level Go plugins?
   No, this appears to be an oversight in the implementation. The BaseMiddleware should be initialized for all middleware types, but for endpoint-level Go plugins, it's not being set when created in CheckSpecMatchesStatus.

2. If m.BaseMiddleware is not required in this case, should Tyk avoid calling .Copy() or initializing successHandler altogether?
   The successHandler is used to record analytics for successful API calls. If analytics are important for endpoint-level Go plugins, then BaseMiddleware should be properly initialized. If not, then the code could be modified to skip creating the successHandler for endpoint-level plugins.

3. Can this log error be suppressed or handled more gracefully when the plugin is not using BaseMiddleware?
   Yes, adding a nil check as suggested in the solution would prevent the panic and the corresponding error log, making the behavior more graceful.

The key difference between API-level Go plugins and endpoint-level Go plugins is that API-level plugins are properly initialized with a BaseMiddleware instance in the processSpec method, while endpoint-level plugins are created on-the-fly during request processing without proper initialization.

---

Tip: Mention me again using /probe <request>.
Powered by Probe AI

[buger]

/probe is it fair to say that such panic will happen ALL the time when endpoint plugin is loaded, or only in some specific cases, like when it use some specific code or similar?

[github-actions]

Yes, this panic will happen ALL the time when an endpoint-level Go plugin is loaded, not just in specific cases. The issue is structural in how the EndpointLevelGoPluginMiddleware is initialized.

Here's why this happens consistently:

1. When an endpoint-level Go plugin is matched in the request flow, a new EndpointLevelGoPluginMiddleware instance is created with only the Path and SymbolName fields set:

// In CheckEndpointMatchesStatus or goPluginFromRequest
mw := &EndpointLevelGoPluginMiddleware{
    Path:       pluginPath,
    SymbolName: pluginSymbol,
}

2. The BaseMiddleware field is never initialized for these endpoint-level plugins.

3. Later, in the loadPlugin() method, it unconditionally tries to access m.BaseMiddleware.Copy():

// to record 2XX hits in analytics
m.successHandler = &SuccessHandler{BaseMiddleware: m.BaseMiddleware.Copy()}

4. Since m.BaseMiddleware is nil, this always causes a nil pointer dereference panic.

The panic is caught by the defer recovery block in the loadPlugin() method:

defer func() {
    if e := recover(); e != nil {
        err = fmt.Errorf("%v", e)
        m.logger.WithError(err).Error("Recovered from panic while loading Go-plugin")
    }
}()

This is why the plugin still works correctly at runtime despite the error during initialization. The panic is caught, logged, and the plugin continues to function.

This is a structural issue in the codebase - it will happen for every endpoint-level Go plugin, not just in specific cases or with specific plugin code. The issue is in how these plugins are initialized, not in the plugin code itself.

---

Tip: Mention me again using /probe <request>.
Powered by Probe AI

[buger]

/probe I want you suggestion, on how fix it properly

Alternatively, a more comprehensive solution would be to properly initialize the BaseMiddleware field when creating the EndpointLevelGoPluginMiddleware in CheckSpecMatchesStatus. However, this would require more significant changes to the codebase.

[github-actions]

Proper Fix for BaseMiddleware nil Issue in Endpoint-Level Go Plugins

After analyzing the code, I can see that the issue occurs because when an endpoint-level Go plugin is created in the CheckSpecMatchesStatus function, it only initializes the Path, SymbolName, and Logger fields, but doesn't initialize the embedded GoPluginMiddleware pointer. This causes a nil pointer dereference when loadPlugin() tries to access m.BaseMiddleware.Copy().

The Problem

In gateway/api_loader.go, when an endpoint-level Go plugin is detected, it creates a new EndpointLevelGoPluginMiddleware with only these fields set:

return true, &EndpointLevelGoPluginMiddleware{
    Path:       check.PluginPath,
    SymbolName: check.Symbol,
    Logger:     logger.WithField("mwPath", check.PluginPath).WithField("mwSymbolName", check.Symbol),
}

Later, in the loadPlugin() method of EndpointLevelGoPluginMiddleware, it tries to create a successHandler with:

m.successHandler = &SuccessHandler{BaseMiddleware: m.BaseMiddleware.Copy()}

Since m.GoPluginMiddleware is nil, accessing m.BaseMiddleware causes a nil pointer dereference.

The Solution

Looking at how other middleware types are initialized in the same function (like TrackEndpointMiddleware and GraphQLMiddleware), we can see that they properly initialize the BaseMiddleware field. The proper fix is to initialize the GoPluginMiddleware pointer when creating the EndpointLevelGoPluginMiddleware:

return true, &EndpointLevelGoPluginMiddleware{
    GoPluginMiddleware: &GoPluginMiddleware{
        BaseMiddleware: BaseMiddleware{
            Spec: spec,
            Gw:   gw,
        },
        Path:       check.PluginPath,
        SymbolName: check.Symbol,
        logger:     logger.WithField("mwPath", check.PluginPath).WithField("mwSymbolName", check.Symbol),
    },
    Path:       check.PluginPath,
    SymbolName: check.Symbol,
    Logger:     logger.WithField("mwPath", check.PluginPath).WithField("mwSymbolName", check.Symbol),
}

This will properly initialize the GoPluginMiddleware pointer with a BaseMiddleware that has the necessary Spec and Gw fields set. This prevents the nil pointer dereference when loadPlugin() is called.

Alternative Approach

If you prefer a more minimal change, you could also add a nil check in the loadPlugin() method:

func (m *EndpointLevelGoPluginMiddleware) loadPlugin() bool {
    var err error
    defer func() {
        if e := recover(); e != nil {
            err = fmt.Errorf("%v", e)
            m.Logger.WithError(err).Error("Recovered from panic while loading Go-plugin")
        }
    }()

    if m.handler, err = goplugin.GetHandler(m.Path, m.SymbolName); err != nil {
        m.Logger.WithError(err).Error("Could not load Go-plugin")
        return false
    }

    // to record 2XX hits in analytics
    if m.GoPluginMiddleware != nil && m.BaseMiddleware != nil {
        m.successHandler = &SuccessHandler{BaseMiddleware: m.BaseMiddleware.Copy()}
    }
    return true
}

However, the first approach is more robust as it ensures that BaseMiddleware is properly initialized, which might be needed for other functionality beyond just the successHandler.

---

Tip: Mention me again using /probe <request>.
Powered by Probe AI