[Testing] PS4 fpkg (*.pkg) support for HB-Store

[Gkiokan]

Hi mate,

I just got a feedback that my HB-Store CDN Server tool would be great if it could supports RetroNAS as base path.
I will test the RetroNAS once I have time for it to figure out how to implement it though.

Currently there are a few endpoints and stuff to be made to get the pkgs working on the hb-store app on the ps4 side.
So therefore my Tool would only act as a proxy here and prepare the necessary data structure and provide the needen stuff.

As my tool just deep scans the given base path for all *.pkg files, I assume that it would be enough if users just put in the NAS address as base path and let it do it's job. Which seems to be working fine by the user. But apperently it scans all the NAS folders I guess which adds a bit delay.

[sairuk]

Converting to discussion

Please let us know when this tool is ready to be added

[sairuk]

Added to Ideas

[Gkiokan]

@sairuk HI mate. Thanks for the reply. I almost forgot about that part.

The tool is already finished and released and is working.
You can find the latest release at HB-Store CDN Server v1.7.2-RC
Also I've created a CLI version Hb-Store CDN Server CLI v1.2.0

Both work in the same way, so when we got it running on the one we can patch it on the other, too.

Is there any changes to be done to load the files from the retroNAS itself?
It would be enough if the retroNAS path would just return the *.fpkg files as I make a deep scan of the given path.

If you need samples you can call me on discord Gkiokan#0735

[sairuk]

Can you publish build instructions for this application and an arm build? I see there is a linux-arm target in the git tree package.json but not in the last release version.

I have no experience with nodejs, nor the motivation to learn it at this time.

I have attempted to build the git version with pkg package.json -o test but its been churning a single core at 100% and not doing much of anything that i can see for well over 20min which is too slow to build on device as part of install.

If i am doing something wrong please let me know.

[sairuk]

OK worked it out, should be able to automate it from here

$ git clone https://github.com/Gkiokan/hb-store-cdn-cli-server.git
$ cd hb-store-cdn-cli-server/
$ npm install #installs app deps
$ npm install pkg
$ npm run build:linux # or whatever target
$ cd release
$ ./hb-store-cdn-cli-server

------------------------------------------------------
  _   _ ____       ____  _                 
 | | | | __ )     / ___|| |_ ___  _ __ ___ 
 | |_| |  _ \ ____\___ \| __/ _ \| '__/ _ \
 |  _  | |_) |_____|__) | || (_) | | |  __/
 |_| |_|____/     |____/ \__\___/|_|  \___|
                                           
   ____ ____  _   _   ____                           
  / ___|  _ \| \ | | / ___|  ___ _ ____   _____ _ __ 
 | |   | | | |  \| | \___ \ / _ \ '__\ \ / / _ \ '__|
 | |___| |_| | |\  |  ___) |  __/ |   \ V /  __/ |   
  \____|____/|_| \_| |____/ \___|_|    \_/ \___|_|   
                                                     
------------------------------------------------------
 Version: 1.2.0
 HB-Store CDN Server [CLI version] Made by Gkiokan.NET
 GitHub: https://github.com/Gkiokan/hb-store-cdn-cli-server

[Gkiokan]

I would suggest to use the build release binaries directly.
Building them require at least node v14.17.* and installing the depencendies first.
I will apply the build instructions soon. Usually you would need to do 2 commands
npm installfor installing the node depencendies.
npm run builddoes build for the current system that you are on.
You may also build for different OS / arch with npm run build:*. A list of the build commands are below.

    "build": "npm run clean && npm run dist && pkg .",
    "build:all": "npm run clean && npm run dist && pkg . -t linux,macos,win",
    "build:win": "npm run clean && npm run dist && pkg . -t node14-win-x64",
    "build:mac": "npm run clean && npm run dist && pkg . -t node14-macos-x64",
    "build:linux": "npm run clean && npm run dist && pkg . -t node14-linux-x64",
    "build:linux-arm":

[sairuk]

thanks for the info

we can look at switching to the appropriate pre-built binaries if we run intoproblems. I couldn't do that here because our primary platform for RetroNAS is arm and there was no asset released for that build at 1.2.0 from what I could see support to build arm was added to packages.json after the 1.2.0 release.

the binary built fine without errors on debian 11 but this was with npm 7.5.2 and nodejs 12.22 (assuming this equates to node12)

ii  nodejs                            12.22.12~dfsg-1~deb11u1          amd64        evented I/O for V8 javascript - runtime executable
ii  nodejs-doc                        12.22.12~dfsg-1~deb11u1          all          API documentation for Node.js, the javascript platform
ii  npm                               7.5.2+ds-2                       all          package manager for Node.js

you mention we need to build on node14 but i see no difference in the functionality for the minimal comparison i am able to do when built against what i assume as per above is node12

what problems can we expect to run into?

is there a way to shrink this app down from 170mb? that is a lot of space for what it does, it blew the storage on my test RetroNAS install.

[Gkiokan]

I will also look into that. By the nature of node it does compile it's runtime into the final binary, too, so it is cross compatible to be run without dependencies. There are some compression settings that I can set and I will analyse the packages on build process to see what I can remove to shrink the size. Compression Task

Once the build is done, you can remove the node_modules server as it takes also some space.

Regarding node v12 <> v14 <> v17 there are some deprecations but luckly it does seem to work on v12, too. So it's fine then.

[sairuk]

This has now been added to the testing branch and can be installed from the Experimental menu

This needs testing with someone who has an applicable setup, i can see the server running with pkg info listed

root@testronas:~# systemctl status hb-store-cdn.service 
● hb-store-cdn.service - Homebrew Store for PlayStation4
     Loaded: loaded (/lib/systemd/system/hb-store-cdn.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-10-17 23:00:53 AEDT; 29min ago
   Main PID: 730176 (hb-store-cdn-cl)
      Tasks: 10 (limit: 4672)
     Memory: 43.5M
        CPU: 1.736s
     CGroup: /system.slice/hb-store-cdn.service
             └─730176 /opt/retronas/bin/hb-store-cdn/hb-store-cdn-cli-server start

Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]: │ id         │ name         │ version │ size      │
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]: │ NPXS39041  │ Store        │ 01.00   │ 19.66 MB  │
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]: │ LAPY20001  │ PS4-Xplorer  │ 01.00   │ 73.53 MB  │
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]: └────────────┴──────────────┴─────────┴───────────┘
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]: ? ## Server ## HB-Store CDN CLI Server (Use arrow keys)
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]: ❯ CDN Server is: running
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]:   CDN Address: http://192.168.0.123:6449 
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]:   ──────────────
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]:   [Server] Start the server
Oct 17 23:00:54 testronas hb-store-cdn-cli-server[730176]:   [Server] Restart the server
root@testronas:~# 

[coldjim]

Ok, i might not be able to do any testing today but is there anything you'd like me to try when I next get some time? I will delete the logs and pull them for you @Gkiokan

[coldjim]

When i type '192.16.8.4.30:6449/store.db' into my browser, it downloads the store.db file fine. It must be the ps4 end that isn't seeing this. Unsure how that bit works

[Gkiokan]

So the CDN does in fact works correctly. You did a great job on digging into that.
The StoreHAX header is only used in the public api, but does have no meaning on the CDN.
Yea we must see why the ps4 have issues with that. Luckly @LightningMods does a full log of this HB-Store app so we can see whats happening.

We would need a clean log file from your ps4 of the hb-store logs.
Also do you have any other configuration set besides the CDN address for the hb-store app on the config.ini (ps4)?

[sairuk]

When i type '192.16.8.4.30:6449/store.db' into my browser, it downloads the store.db file fine. It must be the ps4 end that isn't seeing this. Unsure how that bit works

your previous retronas ip address was 192.168.4.99 as per your ps4 config and it is now .30? can you please confirm.

[coldjim]

When i type '192.16.8.4.30:6449/store.db' into my browser, it downloads the store.db file fine. It must be the ps4 end that isn't seeing this. Unsure how that bit works

your previous retronas ip address was 192.168.4.99 as per your ps4 config and it is now .30? can you please confirm.

Sorry this was a typo on my part. I meant to put .99 rather than .30 (.30 is the unraid server, .99 is the retronas VM). I've just tested again to confirm and it works downloading the store typing "http://192.168.4.99:6449/store.db". I've got a little bit of time now, so i will pull the logs from the PS4 again and do some testing

[LightningMods]

Hello, if you are having issues be sure the ini has the keyword http:// like http://192.16.8.4.30:6449/store.db if it still doesn't work you can send me the log from your PS4 using FTP so I can find the issue for you, PS4 Path: /user/app/NPXS39041/logs/log.txt

[Gkiokan]

@sairuk Have a look into the new release v1.3.0

I've managed to add the binary check to the start parameter so it will be checked always before start.
But you can also check the exposed params that allows you to handle the underlying methods for e.g. trigger binary download.
I wasn't sure if I should send a exit code in specific cases, but the logs itself does contain either Download Finished or You are on the latest version if this may help already?

But anyways as the code does now does check the binary on startup I would recommend only to restart the service for updates instead of doing an interval check. The watch method isn't implemented yet, but this could be done on my side within the app that it will a) watch for changes in the basePath folder and b) updates on the binary files and does the update itself without extra triggering.

[sairuk]

that looks like a good update, i will take a look in the next day or so

exit codes are helpful to determine the result of commands without the additional overhead of querying logs, but i will take a look how it works now and see.

[sairuk]

i've updated our implementation with a check script that runs every 12h and will download/restart if required

[coldjim]

@sairuk great work, thanks. I will update to this soon. Is this in the testing branch?

[sairuk]

yeh testing branch, let me know how you go.