Skip to content

Feat/flatten #226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 19, 2025
Merged

Feat/flatten #226

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0e3357b
adding tests for basic flatten code
Feb 19, 2025
3723ec4
getting flatten to work with unset
Feb 19, 2025
7c58382
4.2.10
Feb 19, 2025
5fd6c22
updating docs
Feb 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 34 additions & 34 deletions docs/docs/sql-syntax/objects.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,16 @@ Specifying `$$ROOT` as a column alias sets the value to root object but only wor
???+ example "Example `$$ROOT` usage"

```sql
SELECT
t AS `$$ROOT`
FROM
SELECT
t AS `$$ROOT`
FROM
(
SELECT
SELECT
id
,`First Name`
,`Last Name`
,LENGTHOFARRAY(Rentals,'id') AS numRentals
FROM customers)
,LENGTHOFARRAY(Rentals,'id') AS numRentals
FROM customers)
AS t
```

Expand All @@ -27,9 +27,9 @@ Only available in aggregates. Use a `SELECT` without specifying a table to creat
???+ example "Creating a new object"

```sql
SELECT
(SELECT id,`First Name` AS Name) AS t
FROM
SELECT
(SELECT id,`First Name` AS Name) AS t
FROM
customers
```

Expand All @@ -38,18 +38,16 @@ Create a new Object and assign to root
???+ example "Creating a new object and assigning to root"

```sql
SELECT
SELECT
(SELECT id,`First Name` AS Name) AS t1
,(SELECT id,`Last Name` AS LastName) AS t2
,MERGE_OBJECTS(t1,t2) AS `$$ROOT`
FROM
,MERGE_OBJECTS(t1,t2) AS `$$ROOT`
FROM
customers
```

## Supported Object Functions



### PARSE_JSON

`PARSE_JSON(expr)`
Expand All @@ -59,19 +57,20 @@ Parses the JSON string. Use in conjunction with `ARRAY_TO_OBJECT` to convert an
???+ example "Example `PARSE_JSON` usage"

```sql
SELECT
SELECT
id,
ARRAY_TO_OBJECT(PARSE_JSON('[{"k":"val","v":1}]')) AS test
FROM `customers`;
```

### MERGE_OBJECTS

`MERGE_OBJECTS(expr)`

???+ example "Example `MERGE_OBJECTS` usage"

```sql
SELECT
SELECT
id,
MERGE_OBJECTS(`Address`,PARSE_JSON('{"val":1}')) AS test
FROM `customers`;
Expand All @@ -80,7 +79,7 @@ Parses the JSON string. Use in conjunction with `ARRAY_TO_OBJECT` to convert an
???+ example "Example `MERGE_OBJECTS` usage with sub select"

```sql
SELECT
SELECT
id,
MERGE_OBJECTS(`Address`,(SELECT 1 AS val)) AS test
FROM `customers`;
Expand Down Expand Up @@ -115,27 +114,28 @@ Creates an empty object.
FROM `customers`;
```

[//]: # (todo add back when flatten implemented)
[//]: # (### FLATTEN)

[//]: # ()
[//]: # (`FLATTEN(field, prefix)`)

[//]: # ()
[//]: # (Flattens an object into a set of fields.)
### FLATTEN

[//]: # ()
[//]: # (???+ example "Example `FLATTEN` usage")
`FLATTEN(field, prefix)`

[//]: # ()
[//]: # ( ```sql)
Flattens an object into a set of fields. You can optionally add a

[//]: # ( SELECT)
???+ example "Example `FLATTEN` usage"

[//]: # ( id,)
```sql'
SELECT
id,
FLATTEN(`address`,'addr_')
FROM `customers`;
```

[//]: # ( FLATTEN(`address`,'addr_'))
???+ example "Example `FLATTEN` usage with unset"

[//]: # ( FROM `customers`;)
```sql'
SELECT
id,
FLATTEN(`address`,'addr_',true)
FROM `customers`;
```

[//]: # ( ```)
> Will remove the `address` field from the output and will only have the `addr_` prefixed fields.
211 changes: 114 additions & 97 deletions lib/MongoFunctions.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
// eslint-disable-next-line jsdoc/require-returns
/**
* The type mapping from sql to mongo for cast
* @returns {import("./types").JsonSchemaTypeMap}

Check warning on line 39 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Missing JSDoc @returns description
*/
static get _jsonSchemaTypeMapping() {
return {
Expand All @@ -63,11 +63,11 @@

/**
* Gets an allowed function by name, returns null if not found
* @param name

Check warning on line 66 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Missing JSDoc @param "name" description

Check warning on line 66 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Missing JSDoc @param "name" type
* @readonly
* @static
* @memberof AllowableFunctions
* @returns {import("./types").MongoQueryFunction|null}

Check warning on line 70 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Missing JSDoc @returns description
*/
static functionByName(name) {
if (!name) {
Expand Down Expand Up @@ -114,7 +114,7 @@

/**
* Gets the list of function mappings between sql and mongo
* @returns {import("./types").MongoQueryFunction[]}

Check warning on line 117 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Missing JSDoc @returns description
*/
static get functionMappings() {
return [
Expand Down Expand Up @@ -614,7 +614,7 @@
type: 'aggr_func',
parse: (parameters) => {
if (!$check.array(parameters)) {
throw new Error('Invalid parameters for FirstN');

Check warning on line 617 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Define a constant instead of duplicating this literal 4 times
}
if (parameters.length < 1) {
throw new Error('Invalid parameters for FirstN');
Expand All @@ -631,7 +631,7 @@
},
jsonSchemaReturnType: (parameters) => {
if (!$check.array(parameters)) {
throw new Error('Invalid parameters for substring');

Check warning on line 634 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Define a constant instead of duplicating this literal 11 times
}
if (parameters.length < 1) {
throw new Error('Invalid parameters for FirstN');
Expand Down Expand Up @@ -841,7 +841,7 @@
throw new Error('Invalid parameters for substring');
if (parameters.length !== 3) {
throw new Error(
'Invalid parameters required for substring'

Check warning on line 844 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Define a constant instead of duplicating this literal 5 times
);
}

Expand Down Expand Up @@ -870,7 +870,7 @@
if (!$check.array(parameters))
throw new Error('Invalid parameters for substring');
if (parameters.length !== 2) {
throw new Error('Invalid parameters starts_with');

Check warning on line 873 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Define a constant instead of duplicating this literal 3 times
}

return {
Expand Down Expand Up @@ -2248,7 +2248,7 @@
name: 'sum_array',
description: 'Sums the elements in an array',
allowQuery: true,
parse: (parameters, depth = 0) => {

Check warning on line 2251 in lib/MongoFunctions.js

View workflow job for this annotation

GitHub Actions / Build 

Refactor this function to reduce its Cognitive Complexity from 21 to the 15 allowed
// if(parameters.length<2)throw new Error("Invalid parameters, requires at least the array field and ids");
if (
!(
Expand Down Expand Up @@ -2586,103 +2586,120 @@
},
},
// todo add flatten back, has a error with no as which we're handling for unset
// {
// name: 'flatten',
// allowQuery: false,
// parse: (parameters) => {
// // todo fix return type
//
// if (!$check.array(parameters))
// throw new Error('Invalid parameters for flatten');
// if (parameters.length !== 2) {
// throw new Error(
// `Invalid parameter length for flatten, should be two but was ${parameters.length}`
// );
// }
// const field = parameters[0];
// const prefix = parameters[1];
// if ($check.emptyString(field) || !$check.string(field)) {
// throw new Error(
// `The first parameter passed to flatten should be a non empty string but was ${field}`
// );
// }
// if ($check.emptyString(prefix) || !$check.string(prefix)) {
// throw new Error(
// `The second parameter passed to flatten should be a non empty string but was ${prefix}`
// );
// }
// // todo decide how to unset or if we should? Leave to user?
//
// if (field.indexOf('.') > -1) {
// const fieldParts = field.split('.');
// const fieldName = fieldParts
// .slice(0, fieldParts.length - 1)
// .join('.');
//
// return {
// $set: {
// [fieldName]: {
// $mergeObjects: [
// '$' + fieldName,
// {
// $arrayToObject: {
// $map: {
// input: {
// $objectToArray:
// '$' + field,
// },
// as: 'temp_flatten',
// in: {
// k: {
// $concat: [
// prefix,
// '$$temp_flatten.k',
// ],
// },
// v: '$$temp_flatten.v',
// },
// },
// },
// },
// ],
// },
// },
// };
// } else {
// return {
// $replaceRoot: {
// newRoot: {
// $mergeObjects: [
// '$$ROOT',
// {
// $arrayToObject: {
// $map: {
// input: {
// $objectToArray:
// '$' + field,
// },
// as: 'temp_flatten',
// in: {
// k: {
// $concat: [
// prefix,
// '$$temp_flatten.k',
// ],
// },
// v: '$$temp_flatten.v',
// },
// },
// },
// },
// ],
// },
// },
// };
// }
// },
// requiresAs: false,
// jsonSchemaReturnType: 'null',
// },
{
name: 'flatten',
allowQuery: true,
parse: (parameters) => {
// todo fix return type

if (!$check.array(parameters))
throw new Error(
'Invalid parameters for flatten, should be an array'
);
if (parameters.length < 2) {
throw new Error(
`Invalid parameter length for flatten, should be two but was ${parameters.length}`
);
}
const unset = !!AllowableFunctions._getLiteral(
parameters[2]
);
let field = parameters[0];
if (field.startsWith('$')) {
field = field.substring(1);
}
const prefix = AllowableFunctions._getLiteral(
parameters[1]
);
if ($check.emptyString(field) || !$check.string(field)) {
throw new Error(
`The first parameter passed to flatten should be a non empty string but was ${field}`
);
}
if ($check.emptyString(prefix) || !$check.string(prefix)) {
throw new Error(
`The second parameter passed to flatten should be a non empty string but was ${prefix}`
);
}

// todo decide how to unset or if we should? Leave to user?

if (field.indexOf('.') > -1) {
const fieldParts = field.split('.');
const fieldName = fieldParts
.slice(0, fieldParts.length - 1)
.join('.');

const result = {
$set: {
[fieldName]: {
$mergeObjects: [
'$' + fieldName,
{
$arrayToObject: {
$map: {
input: {
$objectToArray:
'$' + field,
},
as: 'temp_flatten',
in: {
k: {
$concat: [
prefix,
'$$temp_flatten.k',
],
},
v: '$$temp_flatten.v',
},
},
},
},
],
},
},
};
if (unset) {
result.unsetAfterReplaceOrSet = {$unset: [field]};
}
return result;
}
const result = {
$replaceRoot: {
newRoot: {
$mergeObjects: [
'$$ROOT',
{
$arrayToObject: {
$map: {
input: {
$objectToArray: '$' + field,
},
as: 'temp_flatten',
in: {
k: {
$concat: [
prefix,
'$$temp_flatten.k',
],
},
v: '$$temp_flatten.v',
},
},
},
},
],
},
},
};
if (unset) {
result.unsetAfterReplaceOrSet = {$unset: [field]};
}
return result;
},
requiresAs: false,
jsonSchemaReturnType: 'null',
},
/* #endregion */

// separate action
Expand Down
2 changes: 2 additions & 0 deletions lib/make/createResultObject.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ function createResultObject() {
countDistinct: null,
windowFields: [],
subQueryRootProjections: [],
set: null,
unsetAfterReplaceOrSet: null,
};
}

Expand Down
9 changes: 9 additions & 0 deletions lib/make/makeAggregatePipeline.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -452,6 +452,12 @@ function makeAggregatePipeline(ast, context = {}) {
if (result.count.length > 0) {
result.count.forEach((countStep) => pipeline.push(countStep));
}
if (result.set) {
pipeline.push(result.set);
if (result.unsetAfterReplaceOrSet) {
pipeline.push(result.unsetAfterReplaceOrSet);
}
}
if (result.unset) {
pipeline.push(result.unset);
}
Expand Down Expand Up @@ -555,6 +561,9 @@ function makeAggregatePipeline(ast, context = {}) {

if (result.replaceRoot) {
pipeline.push(result.replaceRoot);
if (result.unsetAfterReplaceOrSet) {
pipeline.push(result.unsetAfterReplaceOrSet);
}
}

if (result.unwind && result.unwind.length > 0) {
Expand Down
Loading
Loading