google cloud nightscout data copy of profile data seems broken

[tynbendad]

on google cloud nightscout vm ssh, i go to the data option and select copy from another site. it copies fine (i can see device and entry data on the site) up until it starts copying profile.json... then it takes a very very long time (i have not yet successfully finished it, but its still running.)

for some reason its getting ~80MB data on every request. i think nightscout itself may be broken and returning too much data for each request? (site it is copying from is on 14.2.6.)

maybe a good idea to add an option to specify how many months of nightscout data you want to copy to this tool (it looks like its getting 11 years of everything else...)

here's what the profile copy looks like now:

cloning <profile>
https://redacted-sitename.up.railway.app
2023-04-22 2023-08-16
--2023-08-15 00:09:53--  https://redacted-sitename.up.railway.app/api/v1/profile.json?find[created_at][$gte]=2023-04-22&find[created_at][$lte]=2023-08-16&count=1000000
Resolving redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)... 104.196.232.237
Connecting to redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83333375 (79M) [application/json]
Saving to: ‘/tmp/profile.json’

/tmp/profile.json                               100%[=======================================================================================================>]  79.47M  58.4MB/s    in 1.4s    

2023-08-15 00:10:18 (58.4 MB/s) - ‘/tmp/profile.json’ saved [83333375/83333375]

2022-12-27 2023-04-22
--2023-08-15 00:18:35--  https://redacted-sitename.up.railway.app/api/v1/profile.json?find[created_at][$gte]=2022-12-27&find[created_at][$lte]=2023-04-22&count=1000000
Resolving redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)... 104.196.232.237
Connecting to redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83333375 (79M) [application/json]
Saving to: ‘/tmp/profile.json’

/tmp/profile.json                               100%[=======================================================================================================>]  79.47M  60.4MB/s    in 1.3s    

2023-08-15 00:18:59 (60.4 MB/s) - ‘/tmp/profile.json’ saved [83333375/83333375]

2022-09-02 2022-12-27
--2023-08-15 00:29:11--  https://redacted-sitename.up.railway.app/api/v1/profile.json?find[created_at][$gte]=2022-09-02&find[created_at][$lte]=2022-12-27&count=1000000
Resolving redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)... 104.196.232.237
Connecting to redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83336247 (79M) [application/json]
Saving to: ‘/tmp/profile.json’

/tmp/profile.json                               100%[=======================================================================================================>]  79.48M  57.1MB/s    in 1.4s    

2023-08-15 00:29:35 (57.1 MB/s) - ‘/tmp/profile.json’ saved [83336247/83336247]

2022-05-10 2022-09-02
--2023-08-15 00:33:45--  https://redacted-sitename.up.railway.app/api/v1/profile.json?find[created_at][$gte]=2022-05-10&find[created_at][$lte]=2022-09-02&count=1000000
Resolving redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)... 104.196.232.237
Connecting to redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83336247 (79M) [application/json]
Saving to: ‘/tmp/profile.json’

/tmp/profile.json                               100%[=======================================================================================================>]  79.48M  63.1MB/s    in 1.3s    

2023-08-15 00:34:08 (63.1 MB/s) - ‘/tmp/profile.json’ saved [83336247/83336247]

2022-01-14 2022-05-10
--2023-08-15 00:46:37--  https://redacted-sitename.up.railway.app/api/v1/profile.json?find[created_at][$gte]=2022-01-14&find[created_at][$lte]=2022-05-10&count=1000000
Resolving redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)... 104.196.232.237
Connecting to redacted-sitename.up.railway.app (redacted-sitename.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83336247 (79M) [application/json]
Saving to: ‘/tmp/profile.json’

/tmp/profile.json                               100%[=======================================================================================================>]  79.48M  59.8MB/s    in 1.3s    

2023-08-15 00:47:02 (59.8 MB/s) - ‘/tmp/profile.json’ saved [83336247/83336247]

<it is still running here, i'll just leave it overnight to see what happens>

[tynbendad]

when i go to: /api/v1/profile.json?find[created_at][$gte]=2022-01-14&find[created_at][$lte]=2022-05-10&count=1
i get the same data as: /api/v1/profile.json?find[created_at][$gte]=2023-04-22&find[created_at][$lte]=2023-08-16&count=1
the data for both is:
[{"_id":"64dac5477eb5234ec74a6c04","defaultProfile":"Default","store":{"Default":{"dia":"6","carbratio":[{"time":"00:00","value":"8.589","timeAsSeconds":"0"}],"carbs_hr":"20","delay":"20","sens":[{"time":"00:00","value":"45.758","timeAsSeconds":"0"}],"timezone":"US/Mountain","basal":[{"time":"00:00","value":"1.135","timeAsSeconds":"0"}],"target_low":[{"time":"00:00","value":"82","timeAsSeconds":"0"}],"target_high":[{"time":"00:00","value":"90","timeAsSeconds":"0"}],"startDate":"1970-01-01T00:00:00.000Z"},"AutoNormal":{"dia":"6","carbratio":[{"time":"00:00","value":"8.589","timeAsSeconds":"0"}],"carbs_hr":"20","delay":"20","sens":[{"time":"00:00","value":"45.758","timeAsSeconds":"0"}],"timezone":"US/Mountain","basal":[{"time":"00:00","value":"1.135","timeAsSeconds":"0"}],"target_low":[{"time":"00:00","value":"82","timeAsSeconds":"0"}],"target_high":[{"time":"00:00","value":"90","timeAsSeconds":"0"}],"startDate":"1970-01-01T00:00:00.000Z"},"AutoStrong":{"dia":"6","carbratio":[{"time":"00:00","value":"8.589","timeAsSeconds":"0"}],"carbs_hr":"20","delay":"20","sens":[{"time":"00:00","value":"34.95","timeAsSeconds":"0"}],"timezone":"US/Mountain","basal":[{"time":"00:00","value":"1.135","timeAsSeconds":"0"}],"target_low":[{"time":"00:00","value":"82","timeAsSeconds":"0"}],"target_high":[{"time":"00:00","value":"90","timeAsSeconds":"0"}],"startDate":"1970-01-01T00:00:00.000Z"},"AutoLudicrous":{"dia":"6","carbratio":[{"time":"00:00","value":"8.589","timeAsSeconds":"0"}],"carbs_hr":"20","delay":"20","sens":[{"time":"00:00","value":"30.32","timeAsSeconds":"0"}],"timezone":"US/Mountain","basal":[{"time":"00:00","value":"1.135","timeAsSeconds":"0"}],"target_low":[{"time":"00:00","value":"81","timeAsSeconds":"0"}],"target_high":[{"time":"00:00","value":"91","timeAsSeconds":"0"}],"startDate":"1970-01-01T00:00:00.000Z","units":"mg/dl"},"AutoRage":{"dia":"6","carbratio":[{"time":"00:00","value":"8.589","timeAsSeconds":"0"}],"carbs_hr":"20","delay":"20","sens":[{"time":"00:00","value":"26.58","timeAsSeconds":"0"}],"timezone":"US/Mountain","basal":[{"time":"00:00","value":"1.135","timeAsSeconds":"0"}],"target_low":[{"time":"00:00","value":"81","timeAsSeconds":"0"}],"target_high":[{"time":"00:00","value":"91","timeAsSeconds":"0"}],"startDate":"1970-01-01T00:00:00.000Z","units":"mg/dl"},"Active":{"dia":"6","carbratio":[{"time":"00:00","value":"8.589","timeAsSeconds":"0"}],"carbs_hr":"20","delay":"20","sens":[{"time":"00:00","value":"45.758","timeAsSeconds":"0"}],"timezone":"US/Mountain","basal":[{"time":"00:00","value":"1.135","timeAsSeconds":"0"}],"target_low":[{"time":"00:00","value":"110","timeAsSeconds":"0"}],"target_high":[{"time":"00:00","value":"110","timeAsSeconds":"0"}],"startDate":"1970-01-01T00:00:00.000Z","units":"mg/dl"}},"startDate":"2023-08-15T00:22:30+0000","mills":"0","units":"mg/dl","created_at":"2019-09-07T07:39:54.174Z"}]

so it does look like a nightscout issue or maybe the request is formed wrong for profile.json?

[Navid200]

Have you allowed the process to complete? You state "I have not yet successfully finished it".

How many years of data is present in the source?

[Navid200]

If you have 11 years worth of data, I wouldn't be surprised that it would take more than an hour to import everything.

It will take only a few minutes if you use the alternative method assuming the source is Atlas:
https://navid200.github.io/xDrip/docs/Nightscout/HerokuGone.html

[tynbendad]

there is several years of data there, but if you see my other reply above i think there is a nightscout or cloud tool request problem going on.

[Navid200]

Well, you didn't answer my question.
We didn't put this together without testing.

[Navid200]

I just ran a test importing from 14.2.6 using the "Copy from another Nightscout" utility.
And it worked fine.

[tynbendad]

fwiw the data copy did not complete overnight, it crashed and didn't copy treatments.

so i tried deleting the bogus profile records in my db just to see if i could get this to work and there were a ton on a single date (2019-09-07T07:39:54.174Z), too many to manually delete and nightscout REST API couldn't delete them either. i could have tried mongo shell but just deleted the whole profile collection, restarted the site, and created a new default profile in nightscout. with that the full nightscout data copy completed.

but, i think there's still a problem in nightscout, since i see all the profile copy requests are getting the same size data returned, the find parameters don't seem to work - this is likely the root of the problem i had, rather than all the old bogus records. i.e., "site/api/v1/profile.json?find[created_at][$gte]=2011-11-25&find[created_at][$lte]=2012-03-20&count=1000000" should be empty but instead has the new profile (with date outside the range requested) in it. i guess i'll file a nightscout issue.

[tynbendad]

nightscout/cgm-remote-monitor#8075