Skip to content

Commit 06b843d

Browse files
marcello3dsrubincvanwinklescriptoriansparkpunkd
authored
Prebuild (#1)
* experimental prebuild * skip preinstall, install native deps on linux * npm run lint * run install_ffmpeg, prebuild differently? * try to fix mac (update homebrew) and windows builds (gh token) * maybe fix macos build, add electron builds * update deps, files list, install script * update package version * include dll files in build, remove electron ones (doesn't seem like we need them with napi) * try to get build working on windows, cleanup github action script * try another approach to prebuilding * copy mac libraries in * try caret escape * let's go windows * come onnnnn * if this doesn't work… * update copy_mac_libs to recurse * change logging in copy_mac_libs * do not try to rebuild with prebuild version * set -id in copy_mac_libs * better log output in copy_mac_libs * disable copy_mac_libs to test * use loader_path instead of rpath (to search for library relative to the .node file) * fix stream.metadata type definition * try arm64 build * 0.6.10-rc.8 * Use custom shared libraries from ffmpeg-build-scripts fork instead of homebrew (#2) * use custom shared libraries from ffmpeg-build-scripts fork instead of homebrew * add ffmpegPath and ffprobePath on main module * include dirs * skip arm64 build * rc10 * handle symlinks when unzipping * Update package.json * skip prebuild install when trying to build the prebuilds :-) * Update package.json * Update package.json * Update package.json * patch-package is dev dep * dev patch-packages * paths entry point * path types * Update package.json * fixes * version bump * don't patch-package for normal install * Update package.json * console error if --enable-small * try to include ffmpeg/ffprobe from ffmpeg-ffprobe-static for linux/windows builds * try regex again * windows escaping courtesy of https://octopus.com/blog/powershell-pipe-escaping * bash?? * rc4 -- should have a better way to update that * Update package.json * cleanup * Update paths.js * Update paths.d.ts * Update package.json * Update .github/workflows/prebuild.yml Co-authored-by: Marcello Bastéa-Forte <marcello@descript.com> * Update .github/workflows/prebuild.yml Co-authored-by: Marcello Bastéa-Forte <marcello@descript.com> * PR feedback * Create .node-version Co-authored-by: Marcello Bastéa-Forte <marcello@descript.com> * Attempt to fix types for paths * ... * treat warnings as errors * fix config error * bump version number * do not add .exe to beamcoder.node on windows * bump versions (#5) * ffmpeg-ffprobe-static update (#6) * Update to latest ffmpeg-ffprobe-static * use ubuntu 18.04 (ffmpeg ppa not available for 20.04) * Cvanwink/arm 3 (#9) * Fix definition for metadata property * Remove unneeded import * Stop input and output links sorting by name * Ensure decoder is created from codec_id if available * v0.6.11 * build with debug symbols (temporary) * chore: upgrade dependencies to FFmpeg 4.4 - fixes broken build * chore: version bump for publish * Improved handling of FFmpeg minor version changes * Remove redundant setter leading to channel layout warning message * Bump path-parse from 1.0.6 to 1.0.7 Bumps [path-parse](https://github.com/jbgutierrez/path-parse) from 1.0.6 to 1.0.7. - [Release notes](https://github.com/jbgutierrez/path-parse/releases) - [Commits](https://github.com/jbgutierrez/path-parse/commits/v1.0.7) --- updated-dependencies: - dependency-name: path-parse dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Add support for hardware accelerated decoding * Fix zero size frame, update version * Update CircleCI test image * Update cache tag * Build info for CircleCI test image * Simplify decoder hwaccel flag * better stab at versioning * bump version * one more version * hack for missing definitions in latest ffmpeg? * Download proper build of ffmpeg from our repo for arm64 * Add arm64 paths for building beamcoder * update node version to run as a native arm64 process when building beamcoder https://descript-inc.slack.com/archives/C01G891V16Z/p1637013049098700 * fix const vs let usage * Bump package version to pick up new arm64 build of ffmpeg * Fix getting ffmpeg build on x86_64 vs x64 * Add new package-lock * Build release for both x86_64 and arm64 * Build beamcoder based off of target_arch and not host_arch * Try to get both ffmpeg versions downloaded on macOS * Try to clean build directory in-between x64 and arm64 * Try verbose output to see why CI can't build arm64 * fix position of --verbose flag * bump node version to 17.0.1 so it can maybe cross-compile * Try CI node version 16.13.0 since that matches what is on the GitHub OS image * Don't clean build directory * Fix node packaging for x64 on macOS * Update to latest FFmpeg build * Fix version string to not have extra 'v' * Update Beamcoder to 1.33rc3 * v0.6.10-rc.41 * Update Windows ffmpeg to 4.4 * v0.6.10-rc.42 * Revert "Update Windows ffmpeg to 4.4" This reverts commit 1073fb2. * fix bad merge * Fix another merge error * Fix ffmpeg extraction on windows * add verbose tag for windows * try prebuild on windows without bash * v0.6.15-rc.2 * Revert "try prebuild on windows without bash" This reverts commit d5784ae. * try git-bash instead of bash * try specific bash * quote the temp path * no quote and no escape for temp path * go back to using bash on windows * temporarily comment out ffprobe which doesn't seem to be available * restore ffmpeg copies * Fix regex on windows * v0.6.15-rc.3 * 0.6.15-rc.4 * Try using FFmpeg shared instead of static on Windows * Revert "Try using FFmpeg shared instead of static on Windows" This reverts commit 721001f. * Update ffmpeg-ffprobe-static to 4.4.0-rc.11 * v0.6.15-rc.6 * Streampunk#85 Add arm64 to supported linux platforms * add linux arm64 prebuild --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Simon Rogers <scriptorian@streampunk.media> Co-authored-by: Steve Rubin <nerdforlife@gmail.com> Co-authored-by: Richard Cartwright <spark@streampunk.media> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simon Rogers <scriptorian@users.noreply.github.com> * update package lock * Update package-lock.json * try including libzimg and checking deps * Update prebuild.yml * Update install_ffmpeg.js * Update install_ffmpeg.js * force latest node-gyp to fix incompat between python 3.11 and node-gyp 6 * try statically linking libstdc++ * Update package.json * Update binding.gyp * Update package.json * build on ubuntu 20 * i don't think these were doing anything --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Steve Rubin <nerdforlife@gmail.com> Co-authored-by: Charles Van Winkle <1332142+cvanwinkle@users.noreply.github.com> Co-authored-by: Simon Rogers <scriptorian@streampunk.media> Co-authored-by: Richard Cartwright <spark@streampunk.media> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simon Rogers <scriptorian@users.noreply.github.com>
1 parent ffb4b56 commit 06b843d

15 files changed

+5370
-868
lines changed

‎.github/workflows/prebuild.yml

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
name: prebuild
2+
on: [push]
3+
4+
jobs:
5+
package:
6+
runs-on: ${{ matrix.os }}
7+
strategy:
8+
fail-fast: false
9+
matrix:
10+
os:
11+
- ubuntu-20.04
12+
- macos-latest
13+
- windows-latest
14+
steps:
15+
- uses: actions/checkout@v2
16+
with:
17+
lfs: true
18+
19+
- name: Use nodejs
20+
uses: actions/setup-node@v1
21+
with:
22+
node-version: 16.13.0
23+
24+
- name: npm install
25+
run: npm install
26+
env:
27+
PREBUILD: true
28+
29+
- name: install Linux dependencies
30+
run: |
31+
sudo add-apt-repository ppa:savoury1/ffmpeg4
32+
sudo apt-get update
33+
sudo apt-get install libavcodec-dev libavformat-dev libavdevice-dev libavfilter-dev libavutil-dev libpostproc-dev libswresample-dev libswscale-dev libzimg-dev
34+
if: ${{ runner.os == 'Linux' }}
35+
36+
- name: check Linux dependencies
37+
run: node ./install_ffmpeg.js
38+
if: ${{ runner.os == 'Linux'}}
39+
40+
- name: install Mac build dependencies (brew)
41+
run: brew install nasm pkg-config
42+
if: ${{ runner.os == 'macOS'}}
43+
44+
- name: install Mac dependencies (shared release x64)
45+
run: node ./install_ffmpeg.js --arch x64
46+
if: ${{ runner.os == 'macOS'}}
47+
48+
- name: install Mac dependencies (shared release arm64)
49+
run: node ./install_ffmpeg.js --arch arm64
50+
if: ${{ runner.os == 'macOS'}}
51+
52+
- name: install Windows dependencies
53+
run: node ./install_ffmpeg.js
54+
if: ${{ runner.os == 'Windows'}}
55+
56+
- name: prebuild Release for macOS (x64)
57+
run: npx prebuild --runtime napi --arch x64 --include-regex '(\.node|\.dylib|ffmpeg|ffprobe)$' --verbose --upload ${{ secrets.GITHUB_TOKEN }}
58+
if: ${{ runner.os == 'macOS'}}
59+
60+
# - name: prebuild Debug for macOS (x64)
61+
# run: npx prebuild --debug --runtime napi --arch x64 --include-regex '(\.node|\.dylib|ffmpeg|ffprobe)$' --upload ${{ secrets.GITHUB_TOKEN }}
62+
# if: ${{ runner.os == 'macOS'}}
63+
64+
- name: clean build directory for cross-compiling
65+
run: rm ./build/Release/*.dylib
66+
if: ${{ runner.os == 'macOS'}}
67+
68+
- name: prebuild Release for macOS (arm64)
69+
run: npx prebuild --runtime napi --arch arm64 --include-regex '(\.node|\.dylib|ffmpeg|ffprobe)$' --verbose --upload ${{ secrets.GITHUB_TOKEN }}
70+
if: ${{ runner.os == 'macOS'}}
71+
72+
- name: prebuild for Windows
73+
run: npx prebuild --runtime napi --include-regex '.node|.dll|.exe' --verbose --upload ${{ secrets.GITHUB_TOKEN }}
74+
shell: bash
75+
if: ${{ runner.os == 'Windows'}}
76+
77+
- name: prebuild for Linux (x64)
78+
run: npx prebuild --runtime napi --arch x64 --include-regex '(\.node|\.dylib|ffmpeg|ffprobe)$' --upload ${{ secrets.GITHUB_TOKEN }}
79+
if: ${{ runner.os == 'Linux'}}
80+
81+
- name: prebuild for Linux (arm64)
82+
run: npx prebuild --runtime napi --arch arm64 --include-regex '(\.node|\.dylib|ffmpeg|ffprobe)$' --upload ${{ secrets.GITHUB_TOKEN }}
83+
if: ${{ runner.os == 'Linux'}}

‎.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ typings/
6262

6363
# Native module build dir
6464
build/
65+
prebuilds/
6566

6667
# FFmpeg install folder
6768
ffmpeg/
@@ -73,3 +74,5 @@ ffmpeg/
7374
# Editors and IDE's
7475
*.swp
7576
.vscode/
77+
78+
.DS_Store

‎.node-version

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
17.0.1

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ On Linux, use the appropriate package manager to install the FFmpeg 4.1 developm
104104

105105
sudo add-apt-repository ppa:jonathonf/ffmpeg-4
106106
sudo apt-get update
107-
sudo apt-get install libavcodec-dev libavformat-dev libavdevice-dev libavfilter-dev libavutil-dev libpostproc-dev libswresample-dev libswscale-dev
107+
sudo apt-get install libavcodec-dev libavformat-dev libavdevice-dev libavfilter-dev libavutil-dev libpostproc-dev libswresample-dev libswscale-dev libzimg-dev
108108

109109
An example for Debian / Redhat / Fedora / CentOs will follow.
110110

‎binding.gyp

Lines changed: 71 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,11 @@
88
"src/packet.cc", "src/frame.cc",
99
"src/codec_par.cc", "src/format.cc",
1010
"src/codec.cc", "src/hwcontext.cc" ],
11+
"defines": [
12+
"NAPI_VERSION=<(napi_build_version)"
13+
],
1114
"conditions": [
12-
['OS!="win"', {
15+
['OS=="linux"', {
1316
"defines": [
1417
"__STDC_CONSTANT_MACROS"
1518
],
@@ -30,16 +33,76 @@
3033
"-lavutil",
3134
"-lpostproc",
3235
"-lswresample",
33-
"-lswscale"
36+
"-lswscale",
37+
"-lzimg"
38+
]
39+
},
40+
"copies": [
41+
{
42+
"destination": "<(PRODUCT_DIR)",
43+
"files": [
44+
"node_modules/ffmpeg-ffprobe-static/ffmpeg",
45+
"node_modules/ffmpeg-ffprobe-static/ffprobe",
46+
]
47+
}
3448
]
35-
}
3649
}],
50+
["OS=='mac'", {
51+
"variables": {
52+
"ffmpeg_version": "1.33rc3",
53+
"target_arch_override": "<!(node -p \"'<(target_arch)' === 'x64' ? 'x86_64' : '<(target_arch)'\")",
54+
},
55+
"defines": [
56+
"__STDC_CONSTANT_MACROS"
57+
],
58+
"cflags_cc!": [
59+
"-fno-rtti",
60+
"-fno-exceptions"
61+
],
62+
"cflags_cc": [
63+
"-std=c++11",
64+
"-fexceptions"
65+
],
66+
"include_dirs": [
67+
"<(module_root_dir)/ffmpeg/ffmpeg-ffprobe-shared-darwin-<(target_arch_override).<(ffmpeg_version)/include/"
68+
],
69+
"link_settings": {
70+
"library_dirs": [
71+
"<(module_root_dir)/ffmpeg/ffmpeg-ffprobe-shared-darwin-<(target_arch_override).<(ffmpeg_version)/"
72+
],
73+
"libraries": [
74+
"-Wl,-rpath,@loader_path",
75+
"-lavcodec",
76+
"-lavdevice",
77+
"-lavfilter",
78+
"-lavformat",
79+
"-lavutil",
80+
"-lpostproc",
81+
"-lswresample",
82+
"-lswscale",
83+
"-lzimg"
84+
],
85+
},
86+
'xcode_settings': {
87+
'MACOSX_DEPLOYMENT_TARGET': '10.11',
88+
},
89+
"copies": [
90+
{
91+
"destination": "<(PRODUCT_DIR)",
92+
"files": [
93+
"<!@(node -p \"require('fs').readdirSync('ffmpeg/ffmpeg-ffprobe-shared-darwin-<(target_arch_override).<(ffmpeg_version)').map(f => 'ffmpeg/ffmpeg-ffprobe-shared-darwin-<(target_arch_override).<(ffmpeg_version)/' + f).join(' ')\")"
94+
]
95+
}
96+
]
97+
}
98+
],
3799
['OS=="win"', {
38100
"configurations": {
39101
"Release": {
40102
"msvs_settings": {
41103
"VCCLCompilerTool": {
42-
"RuntimeTypeInfo": "true"
104+
"RuntimeTypeInfo": "true",
105+
"AdditionalOptions": ['/WX']
43106
}
44107
}
45108
}
@@ -59,7 +122,7 @@
59122
],
60123
"copies": [
61124
{
62-
"destination": "build/Release/",
125+
"destination": "<(PRODUCT_DIR)",
63126
"files": [
64127
"ffmpeg/ffmpeg-4.x-win64-shared/bin/avcodec-58.dll",
65128
"ffmpeg/ffmpeg-4.x-win64-shared/bin/avdevice-58.dll",
@@ -68,7 +131,9 @@
68131
"ffmpeg/ffmpeg-4.x-win64-shared/bin/avutil-56.dll",
69132
"ffmpeg/ffmpeg-4.x-win64-shared/bin/postproc-55.dll",
70133
"ffmpeg/ffmpeg-4.x-win64-shared/bin/swresample-3.dll",
71-
"ffmpeg/ffmpeg-4.x-win64-shared/bin/swscale-5.dll"
134+
"ffmpeg/ffmpeg-4.x-win64-shared/bin/swscale-5.dll",
135+
"node_modules/ffmpeg-ffprobe-static/ffmpeg.exe",
136+
"node_modules/ffmpeg-ffprobe-static/ffprobe.exe"
72137
]
73138
}
74139
]

‎index.js

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,12 @@ https://github.com/Streampunk/beamcoder/blob/master/LICENSE`;
3535
console.log(splash);
3636
console.log('Using FFmpeg version', beamcoder.avVersionInfo());
3737

38+
if (Object.values(beamcoder.configurations()).some(value => value.includes('--enable-small'))) {
39+
console.error('beamcoder will crash because ffmpeg is compiled' +
40+
' with --enable-small (because, for example, codec->long_name is ' +
41+
'NULL and beamcoder expects it to be defined)');
42+
}
43+
3844
beamcoder.demuxerStream = beamstreams.demuxerStream;
3945
beamcoder.muxerStream = beamstreams.muxerStream;
4046

‎install.js

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
const {execSync} = require('child_process');
2+
3+
if (!process.env.PREBUILD) {
4+
execSync('npx prebuild-install --runtime napi');
5+
}

‎install_ffmpeg.js

100644100755
Lines changed: 50 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ const fs = require('fs');
2424
const util = require('util');
2525
const https = require('https');
2626
const cp = require('child_process');
27+
const { argv } = require('process');
2728
const [ mkdir, access, rename, execFile, exec ] = // eslint-disable-line
2829
[ fs.mkdir, fs.access, fs.rename, cp.execFile, cp.exec ].map(util.promisify);
2930

@@ -83,7 +84,7 @@ async function inflate(rs, folder, name) {
8384
return new Promise((comp, err) => {
8485
console.log(`Unzipping '${folder}/${name}.zip'.`);
8586
rs.pipe(unzip.Extract({ path: folder }).on('close', () => {
86-
fs.rename(`./${folder}/${directoryName}`, `./${folder}/${name}`, () => {
87+
fs.rename(`${folder}/${directoryName}`, `${folder}/${name}`, () => {
8788
console.log(`Unzipping of '${folder}/${name}.zip' completed.`);
8889
comp();
8990
});
@@ -166,46 +167,65 @@ async function linux() {
166167
console.error('libswscale.so.5 is not installed.');
167168
result = 1;
168169
}
170+
if (stdout.indexOf('libzimg.so.2') < 0) {
171+
console.error('libzimg.so.2 is not installed.');
172+
result = 1;
173+
}
169174

170175
if (result === 1) {
171176
console.log(`Try running the following (Ubuntu/Debian):
172-
sudo add-apt-repository ppa:jonathonf/ffmpeg-4
173-
sudo apt-get install libavcodec-dev libavformat-dev libavdevice-dev libavfilter-dev libavutil-dev libpostproc-dev libswresample-dev libswscale-dev`);
177+
sudo add-apt-repository ppa:savoury1/ffmpeg4
178+
sudo apt-get install libavcodec-dev libavformat-dev libavdevice-dev libavfilter-dev libavutil-dev libpostproc-dev libswresample-dev libswscale-dev libzimg-dev`);
174179
process.exit(1);
175180
}
176181
return result;
177182
}
178183

179184
async function darwin() {
180-
console.log('Checking for FFmpeg dependencies via HomeBrew.');
181-
let output;
182-
let returnMessage;
185+
console.log('Checking/downloading ffmpeg shared libraries');
186+
187+
await mkdir('ffmpeg').catch(e => {
188+
if (e.code === 'EEXIST') return;
189+
else throw e;
190+
});
191+
192+
const version = '1.33rc3';
193+
194+
// default to platform-architecture
195+
let arch = os.arch()
196+
197+
// but if the '--arch' argument is provided
198+
// use the next argument as the value (e.g. 'x64' or 'arm64')
199+
const overrideArchIndex = process.argv.indexOf('--arch');
200+
if (0 < overrideArchIndex && overrideArchIndex < process.argv.length - 1) {
201+
arch = process.argv[overrideArchIndex + 1];
202+
}
183203

184-
try {
185-
output = await exec('brew list ffmpeg');
186-
returnMessage = 'FFmpeg already present via Homebrew.';
187-
} catch (err) {
188-
if (err.stderr !== 'Error: No such keg: /usr/local/Cellar/ffmpeg\n') {
189-
console.error(err);
190-
console.log('Either Homebrew is not installed or something else is wrong.\nExiting');
191-
process.exit(1);
192-
}
193-
194-
console.log('FFmpeg not installed. Attempting to install via Homebrew.');
195-
try {
196-
output = await exec('brew install nasm pkg-config texi2html ffmpeg');
197-
returnMessage = 'FFmpeg installed via Homebrew.';
198-
} catch (err) {
199-
console.log('Failed to install ffmpeg:\n');
200-
console.error(err);
201-
process.exit(1);
202-
}
204+
if (arch === 'x64') {
205+
arch = 'x86_64';
203206
}
204207

205-
console.log(output.stdout);
206-
console.log(returnMessage);
208+
const ffmpegFilename = `ffmpeg-ffprobe-shared-darwin-${arch}.${version}`;
209+
const tag = `v${version}`
207210

208-
return 0;
211+
await access(`ffmpeg/${ffmpegFilename}`, fs.constants.R_OK).catch(async () => {
212+
const ws = fs.createWriteStream(`ffmpeg/${ffmpegFilename}.zip`);
213+
await get(
214+
ws,
215+
`https://github.com/descriptinc/ffmpeg-build-script/releases/download/${tag}/${ffmpegFilename}.zip`,
216+
`${ffmpegFilename}.zip`
217+
).catch(async (err) => {
218+
if (err.name === 'RedirectError') {
219+
const redirectURL = err.message;
220+
await get(ws, redirectURL, `${ffmpegFilename}.zip`);
221+
} else {
222+
console.error(err);
223+
throw err;
224+
}
225+
});
226+
227+
await exec(`unzip ffmpeg/${ffmpegFilename}.zip -d ffmpeg/${ffmpegFilename}/`);
228+
});
209229
}
210230

211231
switch (os.platform()) {
@@ -218,15 +238,15 @@ case 'win32':
218238
}
219239
break;
220240
case 'linux':
221-
if (os.arch() != 'x64') {
241+
if (os.arch() != 'x64' && os.arch() != 'arm64') {
222242
console.error('Only 64-bit platforms are supported.');
223243
process.exit(1);
224244
} else {
225245
linux();
226246
}
227247
break;
228248
case 'darwin':
229-
if (os.arch() != 'x64') {
249+
if (os.arch() != 'x64' && os.arch() != 'arm64') {
230250
console.error('Only 64-bit platforms are supported.');
231251
process.exit(1);
232252
} else {

0 commit comments

Comments
 (0)