TypeError: Cannot read properties of null (reading 'points') using mongoose

[Kinuseka]

Getting an error issue with ratelimiter, this error happens once if the IP connects for the first time and any subsequent connection the rate-limiter will work correctly.

TypeError: Cannot read properties of null (reading 'points')
    at RateLimiterMongo._getRateLimiterRes (\user\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:124:33)
    at RateLimiterMongo._afterConsume (\user\node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:51:22)
    at \user\node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:263:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I am using mongoose 6.5.3 under serverless instance, (do note that this issue also occurs on shared instances.)

[Kinuseka]

code setup:

const ratelimit_db = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true});
var ratelimiter = new RateLimiterMongo({
    storeClient: ratelimit_db,
    points: 10,
    duration: 15 * 60 //15 minutes
});
async function ratelimitPage(req,res,next) {
    ratelimiter.consume(req.ip, 2).then((ratelimitResponse)=>{
        res.locals.ratelimited = false;
        next();
    })
    .catch((ratelimitResponse)=>{
        console.log(ratelimitResponse);
        res.locals.ratelimited = true;
        res.locals.ratelimit = ratelimitResponse.msBeforeNext;
        next();
    })
}

[animir]

@Kinuseka Hi, please take a look at this Wiki page https://github.com/animir/node-rate-limiter-flexible/wiki/Mongo#note-about-buffering-in-mongoose. It is about Buffering and connection. Also, check this part https://github.com/animir/node-rate-limiter-flexible/wiki/Mongo#connection-to-mongo-and-errors.

[Kinuseka]

This seems pretty difficult situation to deal with,
I have tried multiple methods

1

var mongo_ratelimit = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true});

mongo_ratelimit.set('bufferCommands', false);
mongo_ratelimit.set('autoCreate', false);

still results in the same error

2

var mongo_ratelimit = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true, bufferCommands: false, autoCreate: false});

or

var mongo_ratelimit = async ()=>{
    return await mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true, bufferCommands: false, autoCreate: false});
} 

Results in an error:

MongooseError: Cannot call `rlflx.createIndex()` before initial connection is complete if `bufferCommands = false`. Make sure you `await mongoose.connect()` if you have `bufferCommands = false`.
    at NativeCollection.<computed> [as createIndex] (user\node_modules\mongoose\lib\drivers\node-mongodb-native\collection.js:219:15)
    at RateLimiterMongo._initCollection (user\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:108:16)
    at user\node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:54:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

3

according to this it seems that disabling buffering is not recommended. I also tried the unpopular solution

mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options={useNewUrlParser: true, bufferCommands: false, autoCreate: false, bufferMaxEntries:0});
} 

Which just results in:

MongoParseError: option buffermaxentries is not supported

okay

If anything, these feels like a bandaid solution than a permanent fix.
using insuranceLimiter with RateLimiterMemory will remediate the issue (still not a permanent fix though).

[animir]

@Kinuseka Could you start your server after the connection is established?
Like in this answer https://stackoverflow.com/a/42186818/4444520.

[Kinuseka]

Found the root cause, mongoDB in the examples works flawlessly due to .connect() actually creates multiple connection pools and manages the database on those multiple connections. The downside is that we can only use it once. To mitigate this issue we would have to use createConnection().

Unlike .connect(), createConnection() only creates 1 connection, so we would lose out on the automatic connection pool management from the .connect() therefore we would need to manage it ourselves.

Although this solution above works, it is a stretch especially when you already have your workflow arranged.

my solution to this problem is to simply wait for the database to connect before creating a RateLimitMongo instance

//mongo.js
var mongo_db_rt = mongoose.createConnection(`mongodb+srv://${config.DB_USER}:${config.DB_PASSWORD}@${config.ENDPOINT}/ratelimiter?retryWrites=true&w=majority`, options);

async function DB_wait(db) {
    function sleep(ms) {
        return new Promise((resolve) => {
            setTimeout(resolve, ms);
        });
    }
    /*
    0: disconnected
    1: connected
    2: connecting
    3: disconnecting
    */
    var state = { 0: "Disconnected", 1: "Connected", 2: "Connecting", 3: "Disconnecting" };
    while (db.readyState !== 1) {
        console.log(`Waiting for connection on db: ${db.name} | State: ${state[db.readyState]}`);
        await sleep(1000);
    }
    console.log(`Connection established with: ${db.name} | State: ${state[db.readyState]}`);
    return db;
}
var mongo_ratelimit = DB_wait(mongo_db_rt); // this assigns the variable into an unresolved promise
module.exports = {mongo_ratelimit};

const {mongo_ratelimit} = require('./database/mongo');

var ratelimiter = async ()=>{
    await mongo_ratelimit //since this is a promise, we wait for it to become resolved
    return new RateLimiterMongo({
        storeClient: mongo_ratelimit,
        points: 10,
        duration: 10 * 60 //10 minutes
        });
}
async function ratelimitPage(req,res,next) {
    (await ratelimiter()).consume(req.ip, 2).then((ratelimitResponse)=>{
        next();
    }
    ...
}

[Kinuseka]

I have read past issues, and this mistake happens quite pretty often. I feel like it is worth noting the differences between
.connect and .createConnection where the latter requires connection initialization

[o-ali]

jumping on this question because im seeing a similar issue and was wondering if it had to do with buffering.

I see error Cannot read properties of null (reading 'value') when the first call is made, and every call after succeeds as long as the entry in the database hasnt been deleted for that key.

Is this a bug with the rate limiter that isnt handling null from the value properly on the first call, it should expect that the first call will not have any entry in the database.

edit: adding context

TypeError: Cannot read properties of null (reading 'value')
    at RateLimiterMongo._getRateLimiterRes (node_modules\rate-limiter-flexible\lib\RateLimiterMongo.js:118:23)
    at RateLimiterMongo._afterConsume (node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:51:22)
    at node_modules\rate-limiter-flexible\lib\RateLimiterStoreAbstract.js:205:16
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

using rate limiter with the following setup

        const opts: IRateLimiterMongoOptions = {
            storeClient: db, // await mongoose.connect(uri).then((res) => res.connection)
            points: 3, 
            duration: 5, 
            tableName: 'rate-limiter',
        };
        rateLimiter = new RateLimiterMongo(opts);

    "mongoose": "^8.0.1",
    "mongodb": "^6.2.0",

@animir for vis

[issam-seghir]

I have encountered the same error mentioned by @o-ali . My rate limiter middleware looks like this:

const mongoConn = mongoose.connection;

const options = new RateLimiterMongo({
	storeClient: mongoConn,
	dbName: ENV.DATABASE_NAME,
	keyPrefix: "middleware",
	points: 2, // 10 requests
	duration: 1, // per 1 second by IP
	tableName: "rate_limits", // Name of the collection to use for storing rate limit data
});
const rateLimiterMiddleware = async (req, res, next) => {
	try {
		const rateLimiterRes = await options.consume(req.ip); // Consume 1 point for each request
		log.debug("RateLimit-Limit Response .....");
		console.log(rateLimiterRes);
		res.setHeader("Retry-After", rateLimiterRes.msBeforeNext / 1000);
		res.setHeader("X-RateLimit-Limit", options.points);
		res.setHeader("X-RateLimit-Remaining", rateLimiterRes.remainingPoints);
		res.setHeader("X-RateLimit-Reset", new Date(Date.now() + rateLimiterRes.msBeforeNext).toISOString());

		next();
	// eslint-disable-next-line unicorn/catch-error-name
	} catch (rateLimiterRes) {
		if (rateLimiterRes instanceof RateLimiterRes) {
			log.warning("RateLimit-Limit Error .....");
			console.log(rateLimiterRes);

			res.setHeader("Retry-After", rateLimiterRes.msBeforeNext / 1000);
			res.setHeader("X-RateLimit-Limit", options.points);
			res.setHeader("X-RateLimit-Remaining", rateLimiterRes.remainingPoints);
			res.setHeader("X-RateLimit-Reset", new Date(Date.now() + rateLimiterRes.msBeforeNext).toISOString());

			log.error("rate-limiter-flexible : ", "Too Many Requests");
			res.status(429).send("Too Many Requests");
		} else {
			// Handle other types of errors
			console.error(rateLimiterRes);
			res.status(500).send("Internal Server Error");
		}
	}
};

module.exports = rateLimiterMiddleware;

[o-ali]

my issue was fixed in 4.0.1, see: #251.

[alert-justin]

I'm still experiencing this issue with 5.0.4.

TypeError: Cannot read properties of null (reading 'points')\n at RateLimiterMongo._getRateLimiterRes (/home/justin/git/webapp/node_modules/rate-limiter-flexible/lib/RateLimiterMongo.js:131:33)\n at RateLimiterMongo._afterConsume (/home/justin/git/webapp/node_modules/rate-limiter-flexible/lib/RateLimiterStoreAbstract.js:51:22)\n at /home/justin/git/webapp/node_modules/rate-limiter-flexible/lib/RateLimiterStoreAbstract.js:205:16\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

This is what the result looks like in _getRateLimiterRes:

  lastErrorObject: {
    n: 1,
    updatedExisting: false,
    upserted: new ObjectId("673600a99e250aeed0665f7a")
  },
  value: null,
  ok: 1,
  '$clusterTime': {
    clusterTime: new Timestamp({ t: 1731592361, i: 1 }),
    signature: {
      hash: new Binary(Buffer.from("ec50fc82ea86ef68d657e29164df39d495aa5a0d", "hex"), 0),
      keyId: new Long("7391986883644883016")
    }
  },
  operationTime: new Timestamp({ t: 1731592361, i: 1 })
}

As you can see result.value is null, so the doc.points below fails

"mongoose": "^6.12.0",
"mongodb": "^4.1.4",

[animir]

@alert-justin Could you share your code?

[alert-justin]

@animir Here is a minimal example that can reproduce the issue:

const mongoose = require('mongoose');
const { RateLimiterMongo } = require('rate-limiter-flexible');

const DATABASE_URL = "";

const RATE_LIMIT_OPTIONS = {
	points: 10000,
	duration: 3600,
	storeClient: mongoose.connection,
	tableName: 'RateLimitEntries',
	keyPrefix: 'data-rate-limit'
};

const rateLimiter = new RateLimiterMongo(RATE_LIMIT_OPTIONS);

const main = async() => {
    const args = process.argv.slice(2);

    try {
        await mongoose.connect(DATABASE_URL);
    } catch (err) {
        console.log('Error connecting to mongoose');
        console.log(err);
        return;
    }
    console.log('Successfully connected to mongoose');

    try {
        const res = await rateLimiter.consume('test-id');
        console.log('Successfully called consume', res);
    } catch (err) {
        console.log('Error calling consume');
        console.log(err);
    }
};

main().then(() => {
    process.exit(0);
}).catch((err) => {
    console.log(err);
    process.exit(-1);
});

"dependencies": {
    "mongoose": "^6.12.0",
    "rate-limiter-flexible": "^5.0.4"
  }

Node: v20.17.0

Run this script once to see the error, run it again to see it work. Delete the entry in the DB or change the ID to produce the error again.

[alert-justin]

Investigating this further, there is an error being thrown in RateLimiterMongo.getDriverVersion which is causing the version to be 0.0.0 and not setting the returnDocument: 'after' option in RateLimiterMongo._upsert.

I think this is because of my use of mongoose.connection before the mongoose.connect call. I tried experimenting a little bit with the _client object in RateLimiterMongo.getDriverVersion but couldn't find anything that would get the driver version. However, if you just force all options into upsertOptions like below it works (for my version of mongoose at least).

const upsertOptions = {
      upsert: true,
      returnDocument: 'after',
      returnOriginal: false
};