Custom logging for better app monitoring

[Netail]

Goals

1. Customising logging format (e.g. JSON format)
2. Added more relevant data to logging (timestamp, client name, pid, pathname, remote addr, headers?)

Non-Goals

No response

Background

Websites with a lot of traffic have multiple instances/pods running, which all throw logs regarding errors, readiness & info logs. Going through all of the instances/pods one by one to search for certain logs is inefficient, so people use log collectors like Kibana & Grafana. However, as nextjs logs are not formatted in an easy parsable way (in this case for Kibana it would like a JSON format), it makes it hard to search and filter certain information.

For now there's an solution called next-logger, which basically overwrites the current logger import. This solution is not ideal as it requires a bit of hacking, so a build-in solution would be appreciated.

Proposal

Method 1

With this method, you get a few variables (type & message - which are currently the only 2 variables for the current logging in log.ts) and it's up to the user how they want to log messages. This can be through any logging library or something they build themself.

const nextConfig = {
    logger: ({ type, message }) => {
        // Example 1
        console.log(`${Date.now()}: ${type} - {message}`);
                
        // Example 2
        const log = {  
            timestamp: Date.now(),
            type,
            message,
        };
        console.log(log.toString());
    }
};

This logger could also be a seperate file, just like instrumentation.ts

Method 2

Or a kind of templating style like Spring LogBack? With this method the user is only allowed to format the string, the console.log and interpolation is handled by nextjs.

const nextConfig = {
    logger: "%timestamp%: %type% - %message%"
};

Note

• As a fallback, when the user didn't specifies their own logger, it would default to the current logging behaviour.
• Instead of adding a logging function to the next.config.js, a seperate file for this handling (like instrumentation.ts) could be used.

[Gawdfrey]

I understand this proposal is for build-time setup of log formats, but it would also be nice to be able to forward the server-side logs to a third party logging provider, e.g Splunk.

We recently had an error in one of our environments where we got the client-side digest, but we could not access the server-side matching log that actually tells you what is wrong, without accessing the pod in Kubernetes (We are running standalone mode.)

Hopefully not to unrelated :)

[Netail]

No no, this proposal is actually meant for custom server-side logs. The current NextJS setup also logs error on the server-side, but not easily parsable, so for us a different format of the log is the end goal.

We use a setup where the logs from all pods are collected, while your setup sends the logs to a certain endpoint? (If I understand correctly) With method 1 in the proposal you could add a POST to your logging endpoint :)

[Gawdfrey]

Yeah, meant build-time setup of runtime logs 😅
If it is possible to run async code there, great, that would be the perfect solution for my use case 😄

We have a case now where we get a client side digest sent to our logging service, but I am not able to get the server side digest 😓 This is what I hope this will solve, but also looking at other "hacky" solutions

[Gawdfrey]

You could always forward all logs from the pod as you mention 🤔 But a bit too far down the road for that now 😓

[Netail]

I wonder what other people think of the proposed methods, as I could give it a try to build something in myself. But I need other people's opinion on this topic, preferably someone from the NextJS team, else my work would be for nothing. I think my preference goes to a separate file named logger.ts like instrumentation.ts, where it should export a certain function name.

[mostlylikeable]

For SSR apps, server-side logging shouldn't be an afterthought, imho. It's essential to production application observability. It's a little wild that the best solutions for this currently are inarguably hacks to get around next not providing a way to easily do this. Most frameworks like this just provide some sort of middleware feature, to hook into things and do what you want (one of which, could be logging). I'd be in favor of something extensible like that. Next does have a middleware concept via middleware.js, but it's only supported for edge runtime, which makes me 😞.

[maxy9]

What about having a logger config option in NextJS that you could pass in a logger that adhered to a particular interface?

Then all one would need to do is instantiated their logger of choice, e.g. pino, bunyan, winston, and then assuming the interface is suitable, i.e. implements the relevant methods, we can pass an instance of this to NextJS.

[dir]

Been grappling with this all day, for those of us who use alternative hosting outside of Vercel, structured logging would be absolutely fantastic. The only real "solutions" I've seen right now are both incredibly sketchy:

1. next-logger - patches next.js console globals and allows you to replace them with pino or winston, no disrespect to the developers but it seems fairly poorly written.
2. Patching Next itself and overriding the log handler

[Netail]

Your NextJS patcher does not seem to cover console logs/errors/warnings send by NextJS, only incoming requests

[akvadrako]

This is what we do in instrumentation.js, which is essentially the magic that lets next-logger work:

  if (process.env.NEXT_RUNTIME === 'nodejs') {
    console.error = (msg, ...args) => consoleAdapter('error', msg, args);
    console.warn = (msg, ...args) => consoleAdapter('warn', msg, args);
    console.info = (msg, ...args) => consoleAdapter('info', msg, args);
    console.log = (msg, ...args) => consoleAdapter('info', msg, args);
    console.debug = (msg, ...args) => consoleAdapter('debug', msg, args);
  }

It works alright, though there is some weird stuff going on with the edge runtime calling console.log() twice, so it would be better if next supported a custom logger.

[dir]

You've piqued my curiosity... I have a couple questions if you don't mind:

1. I don't see you doing the whole require('next-logger') thing here in your instrumentation file, how are you setting up/initializing next-logger, if at all?

2. What is consoleAdapter? Is it just a custom wrapper class you've made? If so, is there any chance you could share a snippet? This solution you've posted looks like it could solve a couple issues I've been having.

[petejodo]

one issue that I think this doesn't solve for is when Next throws errors. That seems use a different logger or something which is annoying because the most interesting logs to read obviously are uncaught errors but they end up being the only ones not structured

[akvadrako]

I'm not using next-logger, this code replaces next-logger. consoleAdapter() is your custom logging function, which can for example write the logs like console._stderr.write(...).

When next throws errors doesn't it call console.error(error) ?

[theogravity]

If you want DataDog (or other cloud providers) support along with the logging library you use (eg pino), use my logging library, LogLayer.

Full Guide

Here's an example impl:

// instrumentation.ts
import { LogLayer, type ILogLayer } from 'loglayer';
import { DataDogTransport } from "@loglayer/transport-datadog";
import { PinoTransport } from "@loglayer/transport-pino";
import pino from "pino";
import { serializeError } from "serialize-error";

function stripAnsiCodes(str: string): string {
  // This regex matches all ANSI escape sequences that next.js likes to put in the console logs
  return str.replace(
    /[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/g,
    "",
  );
}

function createConsoleMethod(log: ILogLayer, method: "error" | "info" | "warn" | "debug" | "log") {
  let mappedMethod = method;

  if (method === "log") {
    mappedMethod = "info";
  }

  return (...args: unknown[]) => {
    const data: Record<string, unknown> = {};
    let hasData = false;
    let error: Error | null = null;
    const messages: string[] = [];

    for (const arg of args) {
      if (arg instanceof Error) {
        error = arg;
        continue;
      }

      if (typeof arg === "object" && arg !== null) {
        Object.assign(data, arg);
        hasData = true;
        continue;
      }

      if (typeof arg === "string") {
        messages.push(arg);
      }
    }

    let finalMessage = stripAnsiCodes(messages.join(" ")).trim();

    // next.js uses an "x" for the error message when it's an error object
    if (finalMessage === "⨯" && error) {
      finalMessage = error?.message || "";
    }

    if (error && hasData && messages.length > 0) {
      log.withError(error).withMetadata(data)[mappedMethod](finalMessage);
    } else if (error && messages.length > 0) {
      log.withError(error)[mappedMethod](finalMessage);
    } else if (hasData && messages.length > 0) {
      log.withMetadata(data)[mappedMethod](finalMessage);
    } else if (error && hasData && messages.length === 0) {
      log.withError(error).withMetadata(data)[mappedMethod]("");
    } else if (error && messages.length === 0) {
      log.errorOnly(error);
    } else if (hasData && messages.length === 0) {
      log.metadataOnly(data);
    } else {
      log[mappedMethod](finalMessage);
    }
  };
}

export async function register() {
  const logger = new LogLayer({
    errorSerializer: serializeError,
    transport: [
      new PinoTransport({
        logger: pino(),
      }),
      new DataDogTransport(...),
    ]
  })
 
  if (process.env.NEXT_RUNTIME === "nodejs") {
    console.error = createConsoleMethod(logger, "error");
    console.log = createConsoleMethod(logger, "log");
    console.info = createConsoleMethod(logger, "info");
    console.warn = createConsoleMethod(logger, "warn");
    console.debug = createConsoleMethod(logger, "debug");
  }
}

Will output something like:

{"level":50,"time":1738026622949,"pid":84337,"hostname":"local.home","err":{"type":"Object","message":"test","stack":"Error: test\n    at BrowserSessionsPage (webpack-internal:///(rsc)/./src/app/page.tsx:12:11)","digest":"699232626","name":"Error"},"msg":"test"}

and also send to DataDog as well.

[Netail]

Would also be nice to somehow expose the pathname & certain headers (e.g. user-agent, remote addr). So it's easier to understand on which path the error occurs and what an external cause could be

[tom-sherman]

Im looking into implementing this.

Do people care about passing a custom logger for the build phase? Personally I'm only interested in customising the logger at runtime.

[Netail]

No preference on my side (we use the same config for all our deployment environments, but don't use it when developing locally), thanks Tom