This commit is contained in:
Tomáš Dvořák
2025-10-03 16:26:44 +02:00
parent 0fc92f8464
commit cdb8bcd3e9
5928 changed files with 1054476 additions and 72 deletions
+131
View File
@@ -0,0 +1,131 @@
# Changelog
## [7.1.3] - 2025-01-22
### Fixed
- Bump napi-build-utils from 1 to 2 ([#204](https://github.com/prebuild/prebuild-install/issues/204)) ([`1bf4a15`](https://github.com/prebuild/prebuild-install/commit/1bf4a15)) (Bailey Pearson)
## [7.1.2] - 2024-02-29
### Fixed
- Support environments where MD5 is prohibited ([#191](https://github.com/prebuild/prebuild-install/issues/191)) ([`9140468`](https://github.com/prebuild/prebuild-install/commit/9140468)) (Tomasz Szuba)
## [7.1.1] - 2022-06-07
### Fixed
- Replace use of npmlog dependency with console.error ([#182](https://github.com/prebuild/prebuild-install/issues/182)) ([`4e2284c`](https://github.com/prebuild/prebuild-install/commit/4e2284c)) (Lovell Fuller)
- Ensure script output can be captured by tests ([#181](https://github.com/prebuild/prebuild-install/issues/181)) ([`d1853cb`](https://github.com/prebuild/prebuild-install/commit/d1853cb)) (Lovell Fuller)
## [7.1.0] - 2022-04-20
### Changed
- Allow setting libc to glibc on non-glibc platform ([#176](https://github.com/prebuild/prebuild-install/issues/176)) ([`f729abb`](https://github.com/prebuild/prebuild-install/commit/f729abb)) (Joona Heinikoski)
## [7.0.1] - 2022-01-28
### Changed
- Upgrade to the latest version of `detect-libc` ([#166](https://github.com/prebuild/prebuild-install/issues/166)) ([`f71c6b9`](https://github.com/prebuild/prebuild-install/commit/f71c6b9)) (Lovell Fuller)
## [7.0.0] - 2021-11-12
### Changed
- **Breaking:** bump `node-abi` so that Electron 14+ gets correct ABI ([#161](https://github.com/prebuild/prebuild-install/issues/161)) ([`477f347`](https://github.com/prebuild/prebuild-install/commit/477f347)) (csett86). Drops support of Node.js < 10.
- Bump `simple-get` ([`7468c14`](https://github.com/prebuild/prebuild-install/commit/7468c14)) (Vincent Weevers).
## [6.1.4] - 2021-08-11
### Fixed
- Move auth token to header instead of query param ([#160](https://github.com/prebuild/prebuild-install/issues/160)) ([`b3fad76`](https://github.com/prebuild/prebuild-install/commit/b3fad76)) (nicolai-nordic)
- Remove `_` prefix as it isn't allowed by npm config ([#153](https://github.com/prebuild/prebuild-install/issues/153)) ([`a964e5b`](https://github.com/prebuild/prebuild-install/commit/a964e5b)) (Tom Boothman)
- Make `rc.path` absolute ([#158](https://github.com/prebuild/prebuild-install/issues/158)) ([`57bcc06`](https://github.com/prebuild/prebuild-install/commit/57bcc06)) (George Waters).
## [6.1.3] - 2021-06-03
### Changed
- Inline no longer maintained `noop-logger` ([#155](https://github.com/prebuild/prebuild-install/issues/155)) ([`e08d75a`](https://github.com/prebuild/prebuild-install/commit/e08d75a)) (Alexandru Dima)
- Point users towards `prebuildify` in README ([#150](https://github.com/prebuild/prebuild-install/issues/150)) ([`5ee1a2f`](https://github.com/prebuild/prebuild-install/commit/5ee1a2f)) (Vincent Weevers)
## [6.1.2] - 2021-04-24
### Fixed
- Support URL-safe strings in scoped packages ([#148](https://github.com/prebuild/prebuild-install/issues/148)) ([`db36c7a`](https://github.com/prebuild/prebuild-install/commit/db36c7a)) (Marco)
## [6.1.1] - 2021-04-04
### Fixed
- Support `force` & `buildFromSource` options in yarn ([#140](https://github.com/prebuild/prebuild-install/issues/140)) ([`8cb1ced`](https://github.com/prebuild/prebuild-install/commit/8cb1ced)) (João Moreno)
- Bump `node-abi` to prevent dedupe (closes [#135](https://github.com/prebuild/prebuild-install/issues/135)) ([`2950fb2`](https://github.com/prebuild/prebuild-install/commit/2950fb2)) (Vincent Weevers)
## [6.1.0] - 2021-04-03
### Added
- Restore local prebuilds feature ([#137](https://github.com/prebuild/prebuild-install/issues/137)) ([`dc4e5ea`](https://github.com/prebuild/prebuild-install/commit/dc4e5ea)) (Wes Roberts). Previously removed in [#81](https://github.com/prebuild/prebuild-install/issues/81) / [`a069253`](https://github.com/prebuild/prebuild-install/commit/a06925378d38ca821bfa93aa4c1fdedc253b2420).
## [6.0.1] - 2021-02-14
### Fixed
- Fixes empty `--tag-prefix` ([#143](https://github.com/prebuild/prebuild-install/issues/143)) ([**@mathiask88**](https://github.com/mathiask88))
## [6.0.0] - 2020-10-23
### Changed
- **Breaking:** don't skip downloads in standalone mode ([`b6f3b36`](https://github.com/prebuild/prebuild-install/commit/b6f3b36)) ([**@vweevers**](https://github.com/vweevers))
### Added
- Document cross platform options ([`e5c9a5a`](https://github.com/prebuild/prebuild-install/commit/e5c9a5a)) ([**@fishbone1**](https://github.com/fishbone1))
### Removed
- **Breaking:** remove `--compile` and `--prebuild` options ([`94f2492`](https://github.com/prebuild/prebuild-install/commit/94f2492)) ([**@vweevers**](https://github.com/vweevers))
### Fixed
- Support npm 7 ([`8acccac`](https://github.com/prebuild/prebuild-install/commit/8acccac), [`08eaf6d`](https://github.com/prebuild/prebuild-install/commit/08eaf6d), [`22175b8`](https://github.com/prebuild/prebuild-install/commit/22175b8)) ([**@vweevers**](https://github.com/vweevers))
## [5.3.6] - 2020-10-20
### Changed
- Replace `mkdirp` dependency with `mkdirp-classic` ([**@ralphtheninja**](https://github.com/ralphtheninja))
[7.1.3]: https://github.com/prebuild/prebuild-install/releases/tag/v7.1.3
[7.1.2]: https://github.com/prebuild/prebuild-install/releases/tag/v7.1.2
[7.1.1]: https://github.com/prebuild/prebuild-install/releases/tag/v7.1.1
[7.1.0]: https://github.com/prebuild/prebuild-install/releases/tag/v7.1.0
[7.0.1]: https://github.com/prebuild/prebuild-install/releases/tag/v7.0.1
[7.0.0]: https://github.com/prebuild/prebuild-install/releases/tag/v7.0.0
[6.1.4]: https://github.com/prebuild/prebuild-install/releases/tag/v6.1.4
[6.1.3]: https://github.com/prebuild/prebuild-install/releases/tag/v6.1.3
[6.1.2]: https://github.com/prebuild/prebuild-install/releases/tag/v6.1.2
[6.1.1]: https://github.com/prebuild/prebuild-install/releases/tag/v6.1.1
[6.1.0]: https://github.com/prebuild/prebuild-install/releases/tag/v6.1.0
[6.0.1]: https://github.com/prebuild/prebuild-install/releases/tag/v6.0.1
[6.0.0]: https://github.com/prebuild/prebuild-install/releases/tag/v6.0.0
[5.3.6]: https://github.com/prebuild/prebuild-install/releases/tag/v5.3.6
+6
View File
@@ -0,0 +1,6 @@
# Contributing to prebuild
- no commits direct to master
- all commits as pull requests (one or several per PR)
- each commit solves one identifiable problem
- never merge one's own PRs, another contributor does this
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2015 Mathias Buus
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+163
View File
@@ -0,0 +1,163 @@
# prebuild-install
> **A command line tool to easily install prebuilt binaries for multiple versions of Node.js & Electron on a specific platform.**
> By default it downloads prebuilt binaries from a GitHub release.
[![npm](https://img.shields.io/npm/v/prebuild-install.svg)](https://www.npmjs.com/package/prebuild-install)
![Node version](https://img.shields.io/node/v/prebuild-install.svg)
[![Test](https://img.shields.io/github/workflow/status/prebuild/prebuild-install/Test?label=test)](https://github.com/prebuild/prebuild-install/actions/workflows/test.yml)
[![Standard](https://img.shields.io/badge/standard-informational?logo=javascript\&logoColor=fff)](https://standardjs.com)
[![Common Changelog](https://common-changelog.org/badge.svg)](https://common-changelog.org)
## Note
**Instead of [`prebuild`](https://github.com/prebuild/prebuild) paired with [`prebuild-install`](https://github.com/prebuild/prebuild-install), we recommend [`prebuildify`](https://github.com/prebuild/prebuildify) paired with [`node-gyp-build`](https://github.com/prebuild/node-gyp-build).**
With `prebuildify`, all prebuilt binaries are shipped inside the package that is published to npm, which means there's no need for a separate download step like you find in `prebuild`. The irony of this approach is that it is faster to download all prebuilt binaries for every platform when they are bundled than it is to download a single prebuilt binary as an install script.
Upsides:
1. No extra download step, making it more reliable and faster to install.
2. Supports changing runtime versions locally and using the same install between Node.js and Electron. Reinstalling or rebuilding is not necessary, as all prebuilt binaries are in the npm tarball and the correct one is simply picked on runtime.
3. The `node-gyp-build` runtime dependency is dependency-free and will remain so out of principle, because introducing dependencies would negate the shorter install time.
4. Prebuilt binaries work even if npm install scripts are disabled.
5. The npm package checksum covers prebuilt binaries too.
Downsides:
1. The installed npm package is larger on disk. Using [Node-API](https://nodejs.org/api/n-api.html) alleviates this because Node-API binaries are runtime-agnostic and forward-compatible.
2. Publishing is mildly more complicated, because `npm publish` must be done after compiling and fetching prebuilt binaries (typically in CI).
## Usage
Use [`prebuild`](https://github.com/prebuild/prebuild) to create and upload prebuilt binaries. Then change your package.json install script to:
```json
{
"scripts": {
"install": "prebuild-install || node-gyp rebuild"
}
}
```
When a consumer then installs your package with npm thus triggering the above install script, `prebuild-install` will download a suitable prebuilt binary, or exit with a non-zero exit code if there is none, which triggers `node-gyp rebuild` in order to build from source.
Options (see below) can be passed to `prebuild-install` like so:
```json
{
"scripts": {
"install": "prebuild-install -r napi || node-gyp rebuild"
}
}
```
### Help
```
prebuild-install [options]
--download -d [url] (download prebuilds, no url means github)
--target -t version (version to install for)
--runtime -r runtime (Node runtime [node, napi or electron] to build or install for, default is node)
--path -p path (make a prebuild-install here)
--token -T gh-token (github token for private repos)
--arch arch (target CPU architecture, see Node OS module docs, default is current arch)
--platform platform (target platform, see Node OS module docs, default is current platform)
--tag-prefix <prefix> (github tag prefix, default is "v")
--build-from-source (skip prebuild download)
--verbose (log verbosely)
--libc (use provided libc rather than system default)
--debug (set Debug or Release configuration)
--version (print prebuild-install version and exit)
```
When `prebuild-install` is run via an `npm` script, options `--build-from-source`, `--debug`, `--download`, `--target`, `--runtime`, `--arch` `--platform` and `--libc` may be passed through via arguments given to the `npm` command.
Alternatively you can set environment variables `npm_config_build_from_source=true`, `npm_config_platform`, `npm_config_arch`, `npm_config_target` `npm_config_runtime` and `npm_config_libc`.
### Libc
On non-glibc Linux platforms, the Libc name is appended to platform name. For example, musl-based environments are called `linuxmusl`. If `--libc=glibc` is passed as option, glibc is discarded and platform is called as just `linux`. This can be used for example to build cross-platform packages on Alpine Linux.
### Private Repositories
`prebuild-install` supports downloading prebuilds from private GitHub repositories using the `-T <github-token>`:
```
$ prebuild-install -T <github-token>
```
If you don't want to use the token on cli you can put it in `~/.prebuild-installrc`:
```
token=<github-token>
```
Alternatively you can specify it in the `prebuild-install_token` environment variable.
Note that using a GitHub token uses the API to resolve the correct release meaning that you are subject to the ([GitHub Rate Limit](https://developer.github.com/v3/rate_limit/)).
### Create GitHub Token
To create a token:
- Go to [this page](https://github.com/settings/tokens)
- Click the `Generate new token` button
- Give the token a name and click the `Generate token` button, see below
![prebuild-token](https://cloud.githubusercontent.com/assets/13285808/20844584/d0b85268-b8c0-11e6-8b08-2b19522165a9.png)
The default scopes should be fine.
### Custom binaries
The end user can override binary download location through environment variables in their .npmrc file.
The variable needs to meet the mask `% your package name %_binary_host` or `% your package name %_binary_host_mirror`. For example:
```
leveldown_binary_host=http://overriden-host.com/overriden-path
```
Note that the package version subpath and file name will still be appended.
So if you are installing `leveldown@1.2.3` the resulting url will be:
```
http://overriden-host.com/overriden-path/v1.2.3/leveldown-v1.2.3-node-v57-win32-x64.tar.gz
```
#### Local prebuilds
If you want to use prebuilds from your local filesystem, you can use the `% your package name %_local_prebuilds` .npmrc variable to set a path to the folder containing prebuilds. For example:
```
leveldown_local_prebuilds=/path/to/prebuilds
```
This option will look directly in that folder for bundles created with `prebuild`, for example:
```
/path/to/prebuilds/leveldown-v1.2.3-node-v57-win32-x64.tar.gz
```
Non-absolute paths resolve relative to the directory of the package invoking prebuild-install, e.g. for nested dependencies.
### Cache
All prebuilt binaries are cached to minimize traffic. So first `prebuild-install` picks binaries from the cache and if no binary could be found, it will be downloaded. Depending on the environment, the cache folder is determined in the following order:
- `${npm_config_cache}/_prebuilds`
- `${APP_DATA}/npm-cache/_prebuilds`
- `${HOME}/.npm/_prebuilds`
## Install
With [npm](https://npmjs.org) do:
```
npm install prebuild-install
```
## License
[MIT](./LICENSE)
+44
View File
@@ -0,0 +1,44 @@
const get = require('simple-get')
const util = require('./util')
const proxy = require('./proxy')
function findAssetId (opts, cb) {
const downloadUrl = util.getDownloadUrl(opts)
const apiUrl = util.getApiUrl(opts)
const log = opts.log || util.noopLogger
log.http('request', 'GET ' + apiUrl)
const reqOpts = proxy({
url: apiUrl,
json: true,
headers: {
'User-Agent': 'simple-get',
Authorization: 'token ' + opts.token
}
}, opts)
const req = get.concat(reqOpts, function (err, res, data) {
if (err) return cb(err)
log.http(res.statusCode, apiUrl)
if (res.statusCode !== 200) return cb(err)
// Find asset id in release
for (const release of data) {
if (release.tag_name === opts['tag-prefix'] + opts.pkg.version) {
for (const asset of release.assets) {
if (asset.browser_download_url === downloadUrl) {
return cb(null, asset.id)
}
}
}
}
cb(new Error('Could not find GitHub release for version'))
})
req.setTimeout(30 * 1000, function () {
req.abort()
})
}
module.exports = findAssetId
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env node
const path = require('path')
const fs = require('fs')
const napi = require('napi-build-utils')
const pkg = require(path.resolve('package.json'))
const rc = require('./rc')(pkg)
const log = require('./log')(rc, process.env)
const download = require('./download')
const asset = require('./asset')
const util = require('./util')
const prebuildClientVersion = require('./package.json').version
if (rc.version) {
console.log(prebuildClientVersion)
process.exit(0)
}
if (rc.path) process.chdir(rc.path)
if (rc.runtime === 'electron' && rc.target[0] === '4' && rc.abi === '64') {
log.error(`Electron version ${rc.target} found - skipping prebuild-install work due to known ABI issue`)
log.error('More information about this issue can be found at https://github.com/lgeiger/node-abi/issues/54')
process.exit(1)
}
if (!fs.existsSync('package.json')) {
log.error('setup', 'No package.json found. Aborting...')
process.exit(1)
}
if (rc.help) {
console.error(fs.readFileSync(path.join(__dirname, 'help.txt'), 'utf-8'))
process.exit(0)
}
log.info('begin', 'Prebuild-install version', prebuildClientVersion)
const opts = Object.assign({}, rc, { pkg: pkg, log: log })
if (napi.isNapiRuntime(rc.runtime)) napi.logUnsupportedVersion(rc.target, log)
const origin = util.packageOrigin(process.env, pkg)
if (opts.force) {
log.warn('install', 'prebuilt binaries enforced with --force!')
log.warn('install', 'prebuilt binaries may be out of date!')
} else if (origin && origin.length > 4 && origin.substr(0, 4) === 'git+') {
log.info('install', 'installing from git repository, skipping download.')
process.exit(1)
} else if (opts.buildFromSource) {
log.info('install', '--build-from-source specified, not attempting download.')
process.exit(1)
}
const startDownload = function (downloadUrl) {
download(downloadUrl, opts, function (err) {
if (err) {
log.warn('install', err.message)
return process.exit(1)
}
log.info('install', 'Successfully installed prebuilt binary!')
})
}
if (opts.token) {
asset(opts, function (err, assetId) {
if (err) {
log.warn('install', err.message)
return process.exit(1)
}
startDownload(util.getAssetUrl(opts, assetId))
})
} else {
startDownload(util.getDownloadUrl(opts))
}
+142
View File
@@ -0,0 +1,142 @@
const path = require('path')
const fs = require('fs')
const get = require('simple-get')
const pump = require('pump')
const tfs = require('tar-fs')
const zlib = require('zlib')
const util = require('./util')
const error = require('./error')
const proxy = require('./proxy')
const mkdirp = require('mkdirp-classic')
function downloadPrebuild (downloadUrl, opts, cb) {
let cachedPrebuild = util.cachedPrebuild(downloadUrl)
const localPrebuild = util.localPrebuild(downloadUrl, opts)
const tempFile = util.tempFile(cachedPrebuild)
const log = opts.log || util.noopLogger
if (opts.nolocal) return download()
log.info('looking for local prebuild @', localPrebuild)
fs.access(localPrebuild, fs.R_OK | fs.W_OK, function (err) {
if (err && err.code === 'ENOENT') {
return download()
}
log.info('found local prebuild')
cachedPrebuild = localPrebuild
unpack()
})
function download () {
ensureNpmCacheDir(function (err) {
if (err) return onerror(err)
log.info('looking for cached prebuild @', cachedPrebuild)
fs.access(cachedPrebuild, fs.R_OK | fs.W_OK, function (err) {
if (!(err && err.code === 'ENOENT')) {
log.info('found cached prebuild')
return unpack()
}
log.http('request', 'GET ' + downloadUrl)
const reqOpts = proxy({ url: downloadUrl }, opts)
if (opts.token) {
reqOpts.headers = {
'User-Agent': 'simple-get',
Accept: 'application/octet-stream',
Authorization: 'token ' + opts.token
}
}
const req = get(reqOpts, function (err, res) {
if (err) return onerror(err)
log.http(res.statusCode, downloadUrl)
if (res.statusCode !== 200) return onerror()
mkdirp(util.prebuildCache(), function () {
log.info('downloading to @', tempFile)
pump(res, fs.createWriteStream(tempFile), function (err) {
if (err) return onerror(err)
fs.rename(tempFile, cachedPrebuild, function (err) {
if (err) return cb(err)
log.info('renaming to @', cachedPrebuild)
unpack()
})
})
})
})
req.setTimeout(30 * 1000, function () {
req.abort()
})
})
function onerror (err) {
fs.unlink(tempFile, function () {
cb(err || error.noPrebuilts(opts))
})
}
})
}
function unpack () {
let binaryName
const updateName = opts.updateName || function (entry) {
if (/\.node$/i.test(entry.name)) binaryName = entry.name
}
log.info('unpacking @', cachedPrebuild)
const options = {
readable: true,
writable: true,
hardlinkAsFilesFallback: true
}
const extract = tfs.extract(opts.path, options).on('entry', updateName)
pump(fs.createReadStream(cachedPrebuild), zlib.createGunzip(), extract,
function (err) {
if (err) return cb(err)
let resolved
if (binaryName) {
try {
resolved = path.resolve(opts.path || '.', binaryName)
} catch (err) {
return cb(err)
}
log.info('unpack', 'resolved to ' + resolved)
if (opts.runtime === 'node' && opts.platform === process.platform && opts.abi === process.versions.modules && opts.arch === process.arch) {
try {
require(resolved)
} catch (err) {
return cb(err)
}
log.info('unpack', 'required ' + resolved + ' successfully')
}
}
cb(null, resolved)
})
}
function ensureNpmCacheDir (cb) {
const cacheFolder = util.npmCache()
fs.access(cacheFolder, fs.R_OK | fs.W_OK, function (err) {
if (err && err.code === 'ENOENT') {
return makeNpmCacheDir()
}
cb(err)
})
function makeNpmCacheDir () {
log.info('npm cache directory missing, creating it...')
mkdirp(cacheFolder, cb)
}
}
}
module.exports = downloadPrebuild
+14
View File
@@ -0,0 +1,14 @@
exports.noPrebuilts = function (opts) {
return new Error([
'No prebuilt binaries found',
'(target=' + opts.target,
'runtime=' + opts.runtime,
'arch=' + opts.arch,
'libc=' + opts.libc,
'platform=' + opts.platform + ')'
].join(' '))
}
exports.invalidArchive = function () {
return new Error('Missing .node file in archive')
}
+16
View File
@@ -0,0 +1,16 @@
prebuild-install [options]
--download -d [url] (download prebuilds, no url means github)
--target -t version (version to install for)
--runtime -r runtime (Node runtime [node or electron] to build or install for, default is node)
--path -p path (make a prebuild-install here)
--token -T gh-token (github token for private repos)
--arch arch (target CPU architecture, see Node OS module docs, default is current arch)
--platform platform (target platform, see Node OS module docs, default is current platform)
--tag-prefix <prefix> (github tag prefix, default is "v")
--force (always use prebuilt binaries when available)
--build-from-source (skip prebuild download)
--verbose (log verbosely)
--libc (use provided libc rather than system default)
--debug (set Debug or Release configuration)
--version (print prebuild-install version and exit)
+1
View File
@@ -0,0 +1 @@
exports.download = require('./download')
+33
View File
@@ -0,0 +1,33 @@
const levels = {
silent: 0,
error: 1,
warn: 2,
notice: 3,
http: 4,
timing: 5,
info: 6,
verbose: 7,
silly: 8
}
module.exports = function (rc, env) {
const level = rc.verbose
? 'verbose'
: env.npm_config_loglevel || 'notice'
const logAtLevel = function (messageLevel) {
return function (...args) {
if (levels[messageLevel] <= levels[level]) {
console.error(`prebuild-install ${messageLevel} ${args.join(' ')}`)
}
}
}
return {
error: logAtLevel('error'),
warn: logAtLevel('warn'),
http: logAtLevel('http'),
info: logAtLevel('info'),
level
}
}
@@ -0,0 +1,6 @@
language: node_js
node_js:
- 8
- 10
- 12
- 14
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Mathias Buus
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+165
View File
@@ -0,0 +1,165 @@
# tar-fs
filesystem bindings for [tar-stream](https://github.com/mafintosh/tar-stream).
```
npm install tar-fs
```
[![build status](https://secure.travis-ci.org/mafintosh/tar-fs.png)](http://travis-ci.org/mafintosh/tar-fs)
## Usage
tar-fs allows you to pack directories into tarballs and extract tarballs into directories.
It doesn't gunzip for you, so if you want to extract a `.tar.gz` with this you'll need to use something like [gunzip-maybe](https://github.com/mafintosh/gunzip-maybe) in addition to this.
``` js
var tar = require('tar-fs')
var fs = require('fs')
// packing a directory
tar.pack('./my-directory').pipe(fs.createWriteStream('my-tarball.tar'))
// extracting a directory
fs.createReadStream('my-other-tarball.tar').pipe(tar.extract('./my-other-directory'))
```
To ignore various files when packing or extracting add a ignore function to the options. `ignore`
is also an alias for `filter`. Additionally you get `header` if you use ignore while extracting.
That way you could also filter by metadata.
``` js
var pack = tar.pack('./my-directory', {
ignore: function(name) {
return path.extname(name) === '.bin' // ignore .bin files when packing
}
})
var extract = tar.extract('./my-other-directory', {
ignore: function(name) {
return path.extname(name) === '.bin' // ignore .bin files inside the tarball when extracing
}
})
var extractFilesDirs = tar.extract('./my-other-other-directory', {
ignore: function(_, header) {
// pass files & directories, ignore e.g. symlinks
return header.type !== 'file' && header.type !== 'directory'
}
})
```
You can also specify which entries to pack using the `entries` option
```js
var pack = tar.pack('./my-directory', {
entries: ['file1', 'subdir/file2'] // only the specific entries will be packed
})
```
If you want to modify the headers when packing/extracting add a map function to the options
``` js
var pack = tar.pack('./my-directory', {
map: function(header) {
header.name = 'prefixed/'+header.name
return header
}
})
var extract = tar.extract('./my-directory', {
map: function(header) {
header.name = 'another-prefix/'+header.name
return header
}
})
```
Similarly you can use `mapStream` incase you wanna modify the input/output file streams
``` js
var pack = tar.pack('./my-directory', {
mapStream: function(fileStream, header) {
// NOTE: the returned stream HAS to have the same length as the input stream.
// If not make sure to update the size in the header passed in here.
if (path.extname(header.name) === '.js') {
return fileStream.pipe(someTransform)
}
return fileStream;
}
})
var extract = tar.extract('./my-directory', {
mapStream: function(fileStream, header) {
if (path.extname(header.name) === '.js') {
return fileStream.pipe(someTransform)
}
return fileStream;
}
})
```
Set `options.fmode` and `options.dmode` to ensure that files/directories extracted have the corresponding modes
``` js
var extract = tar.extract('./my-directory', {
dmode: parseInt(555, 8), // all dirs should be readable
fmode: parseInt(444, 8) // all files should be readable
})
```
It can be useful to use `dmode` and `fmode` if you are packing/unpacking tarballs between *nix/windows to ensure that all files/directories unpacked are readable.
Alternatively you can set `options.readable` and/or `options.writable` to set the dmode and fmode to readable/writable.
``` js
var extract = tar.extract('./my-directory', {
readable: true, // all dirs and files should be readable
writable: true, // all dirs and files should be writable
})
```
Set `options.strict` to `false` if you want to ignore errors due to unsupported entry types (like device files)
To dereference symlinks (pack the contents of the symlink instead of the link itself) set `options.dereference` to `true`.
## Copy a directory
Copying a directory with permissions and mtime intact is as simple as
``` js
tar.pack('source-directory').pipe(tar.extract('dest-directory'))
```
## Interaction with [`tar-stream`](https://github.com/mafintosh/tar-stream)
Use `finalize: false` and the `finish` hook to
leave the pack stream open for further entries (see
[`tar-stream#pack`](https://github.com/mafintosh/tar-stream#packing)),
and use `pack` to pass an existing pack stream.
``` js
var mypack = tar.pack('./my-directory', {
finalize: false,
finish: function(sameAsMypack) {
mypack.entry({name: 'generated-file.txt'}, "hello")
tar.pack('./other-directory', {
pack: sameAsMypack
})
}
})
```
## Performance
Packing and extracting a 6.1 GB with 2496 directories and 2398 files yields the following results on my Macbook Air.
[See the benchmark here](https://gist.github.com/mafintosh/8102201)
* tar-fs: 34.261 seconds
* [node-tar](https://github.com/isaacs/node-tar): 366.123 seconds (or 10x slower)
## License
MIT
+363
View File
@@ -0,0 +1,363 @@
var chownr = require('chownr')
var tar = require('tar-stream')
var pump = require('pump')
var mkdirp = require('mkdirp-classic')
var fs = require('fs')
var path = require('path')
var os = require('os')
var win32 = os.platform() === 'win32'
var noop = function () {}
var echo = function (name) {
return name
}
var normalize = !win32 ? echo : function (name) {
return name.replace(/\\/g, '/').replace(/[:?<>|]/g, '_')
}
var statAll = function (fs, stat, cwd, ignore, entries, sort) {
var queue = entries || ['.']
return function loop (callback) {
if (!queue.length) return callback()
var next = queue.shift()
var nextAbs = path.join(cwd, next)
stat.call(fs, nextAbs, function (err, stat) {
if (err) return callback(err)
if (!stat.isDirectory()) return callback(null, next, stat)
fs.readdir(nextAbs, function (err, files) {
if (err) return callback(err)
if (sort) files.sort()
for (var i = 0; i < files.length; i++) {
if (!ignore(path.join(cwd, next, files[i]))) queue.push(path.join(next, files[i]))
}
callback(null, next, stat)
})
})
}
}
var strip = function (map, level) {
return function (header) {
header.name = header.name.split('/').slice(level).join('/')
var linkname = header.linkname
if (linkname && (header.type === 'link' || path.isAbsolute(linkname))) {
header.linkname = linkname.split('/').slice(level).join('/')
}
return map(header)
}
}
exports.pack = function (cwd, opts) {
if (!cwd) cwd = '.'
if (!opts) opts = {}
var xfs = opts.fs || fs
var ignore = opts.ignore || opts.filter || noop
var map = opts.map || noop
var mapStream = opts.mapStream || echo
var statNext = statAll(xfs, opts.dereference ? xfs.stat : xfs.lstat, cwd, ignore, opts.entries, opts.sort)
var strict = opts.strict !== false
var umask = typeof opts.umask === 'number' ? ~opts.umask : ~processUmask()
var dmode = typeof opts.dmode === 'number' ? opts.dmode : 0
var fmode = typeof opts.fmode === 'number' ? opts.fmode : 0
var pack = opts.pack || tar.pack()
var finish = opts.finish || noop
if (opts.strip) map = strip(map, opts.strip)
if (opts.readable) {
dmode |= parseInt(555, 8)
fmode |= parseInt(444, 8)
}
if (opts.writable) {
dmode |= parseInt(333, 8)
fmode |= parseInt(222, 8)
}
var onsymlink = function (filename, header) {
xfs.readlink(path.join(cwd, filename), function (err, linkname) {
if (err) return pack.destroy(err)
header.linkname = normalize(linkname)
pack.entry(header, onnextentry)
})
}
var onstat = function (err, filename, stat) {
if (err) return pack.destroy(err)
if (!filename) {
if (opts.finalize !== false) pack.finalize()
return finish(pack)
}
if (stat.isSocket()) return onnextentry() // tar does not support sockets...
var header = {
name: normalize(filename),
mode: (stat.mode | (stat.isDirectory() ? dmode : fmode)) & umask,
mtime: stat.mtime,
size: stat.size,
type: 'file',
uid: stat.uid,
gid: stat.gid
}
if (stat.isDirectory()) {
header.size = 0
header.type = 'directory'
header = map(header) || header
return pack.entry(header, onnextentry)
}
if (stat.isSymbolicLink()) {
header.size = 0
header.type = 'symlink'
header = map(header) || header
return onsymlink(filename, header)
}
// TODO: add fifo etc...
header = map(header) || header
if (!stat.isFile()) {
if (strict) return pack.destroy(new Error('unsupported type for ' + filename))
return onnextentry()
}
var entry = pack.entry(header, onnextentry)
if (!entry) return
var rs = mapStream(xfs.createReadStream(path.join(cwd, filename), { start: 0, end: header.size > 0 ? header.size - 1 : header.size }), header)
rs.on('error', function (err) { // always forward errors on destroy
entry.destroy(err)
})
pump(rs, entry)
}
var onnextentry = function (err) {
if (err) return pack.destroy(err)
statNext(onstat)
}
onnextentry()
return pack
}
var head = function (list) {
return list.length ? list[list.length - 1] : null
}
var processGetuid = function () {
return process.getuid ? process.getuid() : -1
}
var processUmask = function () {
return process.umask ? process.umask() : 0
}
exports.extract = function (cwd, opts) {
if (!cwd) cwd = '.'
if (!opts) opts = {}
var xfs = opts.fs || fs
var ignore = opts.ignore || opts.filter || noop
var map = opts.map || noop
var mapStream = opts.mapStream || echo
var own = opts.chown !== false && !win32 && processGetuid() === 0
var extract = opts.extract || tar.extract()
var stack = []
var now = new Date()
var umask = typeof opts.umask === 'number' ? ~opts.umask : ~processUmask()
var dmode = typeof opts.dmode === 'number' ? opts.dmode : 0
var fmode = typeof opts.fmode === 'number' ? opts.fmode : 0
var strict = opts.strict !== false
if (opts.strip) map = strip(map, opts.strip)
if (opts.readable) {
dmode |= parseInt(555, 8)
fmode |= parseInt(444, 8)
}
if (opts.writable) {
dmode |= parseInt(333, 8)
fmode |= parseInt(222, 8)
}
var utimesParent = function (name, cb) { // we just set the mtime on the parent dir again everytime we write an entry
var top
while ((top = head(stack)) && name.slice(0, top[0].length) !== top[0]) stack.pop()
if (!top) return cb()
xfs.utimes(top[0], now, top[1], cb)
}
var utimes = function (name, header, cb) {
if (opts.utimes === false) return cb()
if (header.type === 'directory') return xfs.utimes(name, now, header.mtime, cb)
if (header.type === 'symlink') return utimesParent(name, cb) // TODO: how to set mtime on link?
xfs.utimes(name, now, header.mtime, function (err) {
if (err) return cb(err)
utimesParent(name, cb)
})
}
var chperm = function (name, header, cb) {
var link = header.type === 'symlink'
/* eslint-disable node/no-deprecated-api */
var chmod = link ? xfs.lchmod : xfs.chmod
var chown = link ? xfs.lchown : xfs.chown
/* eslint-enable node/no-deprecated-api */
if (!chmod) return cb()
var mode = (header.mode | (header.type === 'directory' ? dmode : fmode)) & umask
if (chown && own) chown.call(xfs, name, header.uid, header.gid, onchown)
else onchown(null)
function onchown (err) {
if (err) return cb(err)
if (!chmod) return cb()
chmod.call(xfs, name, mode, cb)
}
}
extract.on('entry', function (header, stream, next) {
header = map(header) || header
header.name = normalize(header.name)
var name = path.join(cwd, path.join('/', header.name))
if (ignore(name, header)) {
stream.resume()
return next()
}
var stat = function (err) {
if (err) return next(err)
utimes(name, header, function (err) {
if (err) return next(err)
if (win32) return next()
chperm(name, header, next)
})
}
var onsymlink = function () {
if (win32) return next() // skip symlinks on win for now before it can be tested
xfs.unlink(name, function () {
var dst = path.resolve(path.dirname(name), header.linkname)
if (!inCwd(dst, cwd)) return next(new Error(name + ' is not a valid symlink'))
xfs.symlink(header.linkname, name, stat)
})
}
var onlink = function () {
if (win32) return next() // skip links on win for now before it can be tested
xfs.unlink(name, function () {
var srcpath = path.join(cwd, path.join('/', header.linkname))
xfs.realpath(srcpath, function (err, dst) {
if (err || !inCwd(dst, cwd)) return next(new Error(name + ' is not a valid hardlink'))
xfs.link(dst, name, function (err) {
if (err && err.code === 'EPERM' && opts.hardlinkAsFilesFallback) {
stream = xfs.createReadStream(srcpath)
return onfile()
}
stat(err)
})
})
})
}
var onfile = function () {
var ws = xfs.createWriteStream(name)
var rs = mapStream(stream, header)
ws.on('error', function (err) { // always forward errors on destroy
rs.destroy(err)
})
pump(rs, ws, function (err) {
if (err) return next(err)
ws.on('close', stat)
})
}
if (header.type === 'directory') {
stack.push([name, header.mtime])
return mkdirfix(name, {
fs: xfs, own: own, uid: header.uid, gid: header.gid
}, stat)
}
var dir = path.dirname(name)
validate(xfs, dir, path.join(cwd, '.'), function (err, valid) {
if (err) return next(err)
if (!valid) return next(new Error(dir + ' is not a valid path'))
mkdirfix(dir, {
fs: xfs, own: own, uid: header.uid, gid: header.gid
}, function (err) {
if (err) return next(err)
switch (header.type) {
case 'file': return onfile()
case 'link': return onlink()
case 'symlink': return onsymlink()
}
if (strict) return next(new Error('unsupported type for ' + name + ' (' + header.type + ')'))
stream.resume()
next()
})
})
})
if (opts.finish) extract.on('finish', opts.finish)
return extract
}
function validate (fs, name, root, cb) {
if (name === root) return cb(null, true)
fs.lstat(name, function (err, st) {
if (err && err.code !== 'ENOENT') return cb(err)
if (err || st.isDirectory()) return validate(fs, path.join(name, '..'), root, cb)
cb(null, false)
})
}
function mkdirfix (name, opts, cb) {
mkdirp(name, { fs: opts.fs }, function (err, made) {
if (!err && made && opts.own) {
chownr(made, opts.uid, opts.gid, cb)
} else {
cb(err)
}
})
}
function inCwd (dst, cwd) {
cwd = path.resolve(cwd)
return cwd === dst || dst.startsWith(cwd + path.sep)
}
+41
View File
@@ -0,0 +1,41 @@
{
"name": "tar-fs",
"version": "2.1.4",
"description": "filesystem bindings for tar-stream",
"dependencies": {
"chownr": "^1.1.1",
"mkdirp-classic": "^0.5.2",
"pump": "^3.0.0",
"tar-stream": "^2.1.4"
},
"keywords": [
"tar",
"fs",
"file",
"tarball",
"directory",
"stream"
],
"devDependencies": {
"rimraf": "^2.6.3",
"standard": "^13.0.1",
"tape": "^4.9.2"
},
"scripts": {
"test": "standard && tape test/index.js"
},
"bugs": {
"url": "https://github.com/mafintosh/tar-fs/issues"
},
"homepage": "https://github.com/mafintosh/tar-fs",
"main": "index.js",
"directories": {
"test": "test"
},
"author": "Mathias Buus",
"license": "MIT",
"repository": {
"type": "git",
"url": "https://github.com/mafintosh/tar-fs.git"
}
}
@@ -0,0 +1 @@
hello world
@@ -0,0 +1 @@
test
+346
View File
@@ -0,0 +1,346 @@
var test = require('tape')
var rimraf = require('rimraf')
var tar = require('../index')
var tarStream = require('tar-stream')
var path = require('path')
var fs = require('fs')
var os = require('os')
var win32 = os.platform() === 'win32'
var mtime = function (st) {
return Math.floor(st.mtime.getTime() / 1000)
}
test('copy a -> copy/a', function (t) {
t.plan(5)
var a = path.join(__dirname, 'fixtures', 'a')
var b = path.join(__dirname, 'fixtures', 'copy', 'a')
rimraf.sync(b)
tar.pack(a)
.pipe(tar.extract(b))
.on('finish', function () {
var files = fs.readdirSync(b)
t.same(files.length, 1)
t.same(files[0], 'hello.txt')
var fileB = path.join(b, files[0])
var fileA = path.join(a, files[0])
t.same(fs.readFileSync(fileB, 'utf-8'), fs.readFileSync(fileA, 'utf-8'))
t.same(fs.statSync(fileB).mode, fs.statSync(fileA).mode)
t.same(mtime(fs.statSync(fileB)), mtime(fs.statSync(fileA)))
})
})
test('copy b -> copy/b', function (t) {
t.plan(8)
var a = path.join(__dirname, 'fixtures', 'b')
var b = path.join(__dirname, 'fixtures', 'copy', 'b')
rimraf.sync(b)
tar.pack(a)
.pipe(tar.extract(b))
.on('finish', function () {
var files = fs.readdirSync(b)
t.same(files.length, 1)
t.same(files[0], 'a')
var dirB = path.join(b, files[0])
var dirA = path.join(a, files[0])
t.same(fs.statSync(dirB).mode, fs.statSync(dirA).mode)
t.same(mtime(fs.statSync(dirB)), mtime(fs.statSync(dirA)))
t.ok(fs.statSync(dirB).isDirectory())
var fileB = path.join(dirB, 'test.txt')
var fileA = path.join(dirA, 'test.txt')
t.same(fs.readFileSync(fileB, 'utf-8'), fs.readFileSync(fileA, 'utf-8'))
t.same(fs.statSync(fileB).mode, fs.statSync(fileA).mode)
t.same(mtime(fs.statSync(fileB)), mtime(fs.statSync(fileA)))
})
})
test('symlink', function (t) {
if (win32) { // no symlink support on win32 currently. TODO: test if this can be enabled somehow
t.plan(1)
t.ok(true)
return
}
t.plan(5)
var a = path.join(__dirname, 'fixtures', 'c')
rimraf.sync(path.join(a, 'link'))
fs.symlinkSync('.gitignore', path.join(a, 'link'))
var b = path.join(__dirname, 'fixtures', 'copy', 'c')
rimraf.sync(b)
tar.pack(a)
.pipe(tar.extract(b))
.on('finish', function () {
var files = fs.readdirSync(b).sort()
t.same(files.length, 2)
t.same(files[0], '.gitignore')
t.same(files[1], 'link')
var linkA = path.join(a, 'link')
var linkB = path.join(b, 'link')
t.same(mtime(fs.lstatSync(linkB)), mtime(fs.lstatSync(linkA)))
t.same(fs.readlinkSync(linkB), fs.readlinkSync(linkA))
})
})
test('follow symlinks', function (t) {
if (win32) { // no symlink support on win32 currently. TODO: test if this can be enabled somehow
t.plan(1)
t.ok(true)
return
}
t.plan(5)
var a = path.join(__dirname, 'fixtures', 'c')
rimraf.sync(path.join(a, 'link'))
fs.symlinkSync('.gitignore', path.join(a, 'link'))
var b = path.join(__dirname, 'fixtures', 'copy', 'c-dereference')
rimraf.sync(b)
tar.pack(a, { dereference: true })
.pipe(tar.extract(b))
.on('finish', function () {
var files = fs.readdirSync(b).sort()
t.same(files.length, 2)
t.same(files[0], '.gitignore')
t.same(files[1], 'link')
var file1 = path.join(b, '.gitignore')
var file2 = path.join(b, 'link')
t.same(mtime(fs.lstatSync(file1)), mtime(fs.lstatSync(file2)))
t.same(fs.readFileSync(file1), fs.readFileSync(file2))
})
})
test('strip', function (t) {
t.plan(2)
var a = path.join(__dirname, 'fixtures', 'b')
var b = path.join(__dirname, 'fixtures', 'copy', 'b-strip')
rimraf.sync(b)
tar.pack(a)
.pipe(tar.extract(b, { strip: 1 }))
.on('finish', function () {
var files = fs.readdirSync(b).sort()
t.same(files.length, 1)
t.same(files[0], 'test.txt')
})
})
test('strip + map', function (t) {
t.plan(2)
var a = path.join(__dirname, 'fixtures', 'b')
var b = path.join(__dirname, 'fixtures', 'copy', 'b-strip')
rimraf.sync(b)
var uppercase = function (header) {
header.name = header.name.toUpperCase()
return header
}
tar.pack(a)
.pipe(tar.extract(b, { strip: 1, map: uppercase }))
.on('finish', function () {
var files = fs.readdirSync(b).sort()
t.same(files.length, 1)
t.same(files[0], 'TEST.TXT')
})
})
test('map + dir + permissions', function (t) {
t.plan(win32 ? 1 : 2) // skip chmod test, it's not working like unix
var a = path.join(__dirname, 'fixtures', 'b')
var b = path.join(__dirname, 'fixtures', 'copy', 'a-perms')
rimraf.sync(b)
var aWithMode = function (header) {
if (header.name === 'a') {
header.mode = parseInt(700, 8)
}
return header
}
tar.pack(a)
.pipe(tar.extract(b, { map: aWithMode }))
.on('finish', function () {
var files = fs.readdirSync(b).sort()
var stat = fs.statSync(path.join(b, 'a'))
t.same(files.length, 1)
if (!win32) {
t.same(stat.mode & parseInt(777, 8), parseInt(700, 8))
}
})
})
test('specific entries', function (t) {
t.plan(6)
var a = path.join(__dirname, 'fixtures', 'd')
var b = path.join(__dirname, 'fixtures', 'copy', 'd-entries')
var entries = ['file1', 'sub-files/file3', 'sub-dir']
rimraf.sync(b)
tar.pack(a, { entries: entries })
.pipe(tar.extract(b))
.on('finish', function () {
var files = fs.readdirSync(b)
t.same(files.length, 3)
t.notSame(files.indexOf('file1'), -1)
t.notSame(files.indexOf('sub-files'), -1)
t.notSame(files.indexOf('sub-dir'), -1)
var subFiles = fs.readdirSync(path.join(b, 'sub-files'))
t.same(subFiles, ['file3'])
var subDir = fs.readdirSync(path.join(b, 'sub-dir'))
t.same(subDir, ['file5'])
})
})
test('check type while mapping header on packing', function (t) {
t.plan(3)
var e = path.join(__dirname, 'fixtures', 'e')
var checkHeaderType = function (header) {
if (header.name.indexOf('.') === -1) t.same(header.type, header.name)
}
tar.pack(e, { map: checkHeaderType })
})
test('finish callbacks', function (t) {
t.plan(3)
var a = path.join(__dirname, 'fixtures', 'a')
var b = path.join(__dirname, 'fixtures', 'copy', 'a')
rimraf.sync(b)
var packEntries = 0
var extractEntries = 0
var countPackEntry = function (header) { packEntries++ }
var countExtractEntry = function (header) { extractEntries++ }
var pack
var onPackFinish = function (passedPack) {
t.equal(packEntries, 2, 'All entries have been packed') // 2 entries - the file and base directory
t.equal(passedPack, pack, 'The finish hook passes the pack')
}
var onExtractFinish = function () { t.equal(extractEntries, 2) }
pack = tar.pack(a, { map: countPackEntry, finish: onPackFinish })
pack.pipe(tar.extract(b, { map: countExtractEntry, finish: onExtractFinish }))
.on('finish', function () {
t.end()
})
})
test('not finalizing the pack', function (t) {
t.plan(2)
var a = path.join(__dirname, 'fixtures', 'a')
var b = path.join(__dirname, 'fixtures', 'b')
var out = path.join(__dirname, 'fixtures', 'copy', 'merged-packs')
rimraf.sync(out)
var prefixer = function (prefix) {
return function (header) {
header.name = path.join(prefix, header.name)
return header
}
}
tar.pack(a, {
map: prefixer('a-files'),
finalize: false,
finish: packB
})
function packB (pack) {
tar.pack(b, { pack: pack, map: prefixer('b-files') })
.pipe(tar.extract(out))
.on('finish', assertResults)
}
function assertResults () {
var containers = fs.readdirSync(out)
t.deepEqual(containers, ['a-files', 'b-files'])
var aFiles = fs.readdirSync(path.join(out, 'a-files'))
t.deepEqual(aFiles, ['hello.txt'])
}
})
test('do not extract invalid tar', function (t) {
var a = path.join(__dirname, 'fixtures', 'invalid.tar')
var out = path.join(__dirname, 'fixtures', 'invalid')
rimraf.sync(out)
fs.createReadStream(a)
.pipe(tar.extract(out))
.on('error', function (err) {
t.ok(/is not a valid symlink/i.test(err.message))
fs.stat(path.join(out, '../bar'), function (err) {
t.ok(err)
t.end()
})
})
})
test('no abs hardlink targets', function (t) {
var out = path.join(__dirname, 'fixtures', 'invalid')
var outside = path.join(__dirname, 'fixtures', 'outside')
rimraf.sync(out)
var s = tarStream.pack()
fs.writeFileSync(outside, 'something')
s.entry({
type: 'link',
name: 'link',
linkname: outside
})
s.entry({
name: 'link'
}, 'overwrite')
s.finalize()
s.pipe(tar.extract(out))
.on('error', function (err) {
t.ok(err, 'had error')
fs.readFile(outside, 'utf-8', function (err, str) {
t.error(err, 'no error')
t.same(str, 'something')
t.end()
})
})
})
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2014 Mathias Buus
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
+168
View File
@@ -0,0 +1,168 @@
# tar-stream
tar-stream is a streaming tar parser and generator and nothing else. It is streams2 and operates purely using streams which means you can easily extract/parse tarballs without ever hitting the file system.
Note that you still need to gunzip your data if you have a `.tar.gz`. We recommend using [gunzip-maybe](https://github.com/mafintosh/gunzip-maybe) in conjunction with this.
```
npm install tar-stream
```
[![build status](https://secure.travis-ci.org/mafintosh/tar-stream.png)](http://travis-ci.org/mafintosh/tar-stream)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](http://opensource.org/licenses/MIT)
## Usage
tar-stream exposes two streams, [pack](https://github.com/mafintosh/tar-stream#packing) which creates tarballs and [extract](https://github.com/mafintosh/tar-stream#extracting) which extracts tarballs. To [modify an existing tarball](https://github.com/mafintosh/tar-stream#modifying-existing-tarballs) use both.
It implementes USTAR with additional support for pax extended headers. It should be compatible with all popular tar distributions out there (gnutar, bsdtar etc)
## Related
If you want to pack/unpack directories on the file system check out [tar-fs](https://github.com/mafintosh/tar-fs) which provides file system bindings to this module.
## Packing
To create a pack stream use `tar.pack()` and call `pack.entry(header, [callback])` to add tar entries.
``` js
var tar = require('tar-stream')
var pack = tar.pack() // pack is a streams2 stream
// add a file called my-test.txt with the content "Hello World!"
pack.entry({ name: 'my-test.txt' }, 'Hello World!')
// add a file called my-stream-test.txt from a stream
var entry = pack.entry({ name: 'my-stream-test.txt', size: 11 }, function(err) {
// the stream was added
// no more entries
pack.finalize()
})
entry.write('hello')
entry.write(' ')
entry.write('world')
entry.end()
// pipe the pack stream somewhere
pack.pipe(process.stdout)
```
## Extracting
To extract a stream use `tar.extract()` and listen for `extract.on('entry', (header, stream, next) )`
``` js
var extract = tar.extract()
extract.on('entry', function(header, stream, next) {
// header is the tar header
// stream is the content body (might be an empty stream)
// call next when you are done with this entry
stream.on('end', function() {
next() // ready for next entry
})
stream.resume() // just auto drain the stream
})
extract.on('finish', function() {
// all entries read
})
pack.pipe(extract)
```
The tar archive is streamed sequentially, meaning you **must** drain each entry's stream as you get them or else the main extract stream will receive backpressure and stop reading.
## Headers
The header object using in `entry` should contain the following properties.
Most of these values can be found by stat'ing a file.
``` js
{
name: 'path/to/this/entry.txt',
size: 1314, // entry size. defaults to 0
mode: 0o644, // entry mode. defaults to to 0o755 for dirs and 0o644 otherwise
mtime: new Date(), // last modified date for entry. defaults to now.
type: 'file', // type of entry. defaults to file. can be:
// file | link | symlink | directory | block-device
// character-device | fifo | contiguous-file
linkname: 'path', // linked file name
uid: 0, // uid of entry owner. defaults to 0
gid: 0, // gid of entry owner. defaults to 0
uname: 'maf', // uname of entry owner. defaults to null
gname: 'staff', // gname of entry owner. defaults to null
devmajor: 0, // device major version. defaults to 0
devminor: 0 // device minor version. defaults to 0
}
```
## Modifying existing tarballs
Using tar-stream it is easy to rewrite paths / change modes etc in an existing tarball.
``` js
var extract = tar.extract()
var pack = tar.pack()
var path = require('path')
extract.on('entry', function(header, stream, callback) {
// let's prefix all names with 'tmp'
header.name = path.join('tmp', header.name)
// write the new entry to the pack stream
stream.pipe(pack.entry(header, callback))
})
extract.on('finish', function() {
// all entries done - lets finalize it
pack.finalize()
})
// pipe the old tarball to the extractor
oldTarballStream.pipe(extract)
// pipe the new tarball the another stream
pack.pipe(newTarballStream)
```
## Saving tarball to fs
``` js
var fs = require('fs')
var tar = require('tar-stream')
var pack = tar.pack() // pack is a streams2 stream
var path = 'YourTarBall.tar'
var yourTarball = fs.createWriteStream(path)
// add a file called YourFile.txt with the content "Hello World!"
pack.entry({name: 'YourFile.txt'}, 'Hello World!', function (err) {
if (err) throw err
pack.finalize()
})
// pipe the pack stream to your file
pack.pipe(yourTarball)
yourTarball.on('close', function () {
console.log(path + ' has been written')
fs.stat(path, function(err, stats) {
if (err) throw err
console.log(stats)
console.log('Got file info successfully!')
})
})
```
## Performance
[See tar-fs for a performance comparison with node-tar](https://github.com/mafintosh/tar-fs/blob/master/README.md#performance)
# License
MIT
@@ -0,0 +1,257 @@
var util = require('util')
var bl = require('bl')
var headers = require('./headers')
var Writable = require('readable-stream').Writable
var PassThrough = require('readable-stream').PassThrough
var noop = function () {}
var overflow = function (size) {
size &= 511
return size && 512 - size
}
var emptyStream = function (self, offset) {
var s = new Source(self, offset)
s.end()
return s
}
var mixinPax = function (header, pax) {
if (pax.path) header.name = pax.path
if (pax.linkpath) header.linkname = pax.linkpath
if (pax.size) header.size = parseInt(pax.size, 10)
header.pax = pax
return header
}
var Source = function (self, offset) {
this._parent = self
this.offset = offset
PassThrough.call(this, { autoDestroy: false })
}
util.inherits(Source, PassThrough)
Source.prototype.destroy = function (err) {
this._parent.destroy(err)
}
var Extract = function (opts) {
if (!(this instanceof Extract)) return new Extract(opts)
Writable.call(this, opts)
opts = opts || {}
this._offset = 0
this._buffer = bl()
this._missing = 0
this._partial = false
this._onparse = noop
this._header = null
this._stream = null
this._overflow = null
this._cb = null
this._locked = false
this._destroyed = false
this._pax = null
this._paxGlobal = null
this._gnuLongPath = null
this._gnuLongLinkPath = null
var self = this
var b = self._buffer
var oncontinue = function () {
self._continue()
}
var onunlock = function (err) {
self._locked = false
if (err) return self.destroy(err)
if (!self._stream) oncontinue()
}
var onstreamend = function () {
self._stream = null
var drain = overflow(self._header.size)
if (drain) self._parse(drain, ondrain)
else self._parse(512, onheader)
if (!self._locked) oncontinue()
}
var ondrain = function () {
self._buffer.consume(overflow(self._header.size))
self._parse(512, onheader)
oncontinue()
}
var onpaxglobalheader = function () {
var size = self._header.size
self._paxGlobal = headers.decodePax(b.slice(0, size))
b.consume(size)
onstreamend()
}
var onpaxheader = function () {
var size = self._header.size
self._pax = headers.decodePax(b.slice(0, size))
if (self._paxGlobal) self._pax = Object.assign({}, self._paxGlobal, self._pax)
b.consume(size)
onstreamend()
}
var ongnulongpath = function () {
var size = self._header.size
this._gnuLongPath = headers.decodeLongPath(b.slice(0, size), opts.filenameEncoding)
b.consume(size)
onstreamend()
}
var ongnulonglinkpath = function () {
var size = self._header.size
this._gnuLongLinkPath = headers.decodeLongPath(b.slice(0, size), opts.filenameEncoding)
b.consume(size)
onstreamend()
}
var onheader = function () {
var offset = self._offset
var header
try {
header = self._header = headers.decode(b.slice(0, 512), opts.filenameEncoding, opts.allowUnknownFormat)
} catch (err) {
self.emit('error', err)
}
b.consume(512)
if (!header) {
self._parse(512, onheader)
oncontinue()
return
}
if (header.type === 'gnu-long-path') {
self._parse(header.size, ongnulongpath)
oncontinue()
return
}
if (header.type === 'gnu-long-link-path') {
self._parse(header.size, ongnulonglinkpath)
oncontinue()
return
}
if (header.type === 'pax-global-header') {
self._parse(header.size, onpaxglobalheader)
oncontinue()
return
}
if (header.type === 'pax-header') {
self._parse(header.size, onpaxheader)
oncontinue()
return
}
if (self._gnuLongPath) {
header.name = self._gnuLongPath
self._gnuLongPath = null
}
if (self._gnuLongLinkPath) {
header.linkname = self._gnuLongLinkPath
self._gnuLongLinkPath = null
}
if (self._pax) {
self._header = header = mixinPax(header, self._pax)
self._pax = null
}
self._locked = true
if (!header.size || header.type === 'directory') {
self._parse(512, onheader)
self.emit('entry', header, emptyStream(self, offset), onunlock)
return
}
self._stream = new Source(self, offset)
self.emit('entry', header, self._stream, onunlock)
self._parse(header.size, onstreamend)
oncontinue()
}
this._onheader = onheader
this._parse(512, onheader)
}
util.inherits(Extract, Writable)
Extract.prototype.destroy = function (err) {
if (this._destroyed) return
this._destroyed = true
if (err) this.emit('error', err)
this.emit('close')
if (this._stream) this._stream.emit('close')
}
Extract.prototype._parse = function (size, onparse) {
if (this._destroyed) return
this._offset += size
this._missing = size
if (onparse === this._onheader) this._partial = false
this._onparse = onparse
}
Extract.prototype._continue = function () {
if (this._destroyed) return
var cb = this._cb
this._cb = noop
if (this._overflow) this._write(this._overflow, undefined, cb)
else cb()
}
Extract.prototype._write = function (data, enc, cb) {
if (this._destroyed) return
var s = this._stream
var b = this._buffer
var missing = this._missing
if (data.length) this._partial = true
// we do not reach end-of-chunk now. just forward it
if (data.length < missing) {
this._missing -= data.length
this._overflow = null
if (s) return s.write(data, cb)
b.append(data)
return cb()
}
// end-of-chunk. the parser should call cb.
this._cb = cb
this._missing = 0
var overflow = null
if (data.length > missing) {
overflow = data.slice(missing)
data = data.slice(0, missing)
}
if (s) s.end(data)
else b.append(data)
this._overflow = overflow
this._onparse()
}
Extract.prototype._final = function (cb) {
if (this._partial) return this.destroy(new Error('Unexpected end of data'))
cb()
}
module.exports = Extract
@@ -0,0 +1,295 @@
var alloc = Buffer.alloc
var ZEROS = '0000000000000000000'
var SEVENS = '7777777777777777777'
var ZERO_OFFSET = '0'.charCodeAt(0)
var USTAR_MAGIC = Buffer.from('ustar\x00', 'binary')
var USTAR_VER = Buffer.from('00', 'binary')
var GNU_MAGIC = Buffer.from('ustar\x20', 'binary')
var GNU_VER = Buffer.from('\x20\x00', 'binary')
var MASK = parseInt('7777', 8)
var MAGIC_OFFSET = 257
var VERSION_OFFSET = 263
var clamp = function (index, len, defaultValue) {
if (typeof index !== 'number') return defaultValue
index = ~~index // Coerce to integer.
if (index >= len) return len
if (index >= 0) return index
index += len
if (index >= 0) return index
return 0
}
var toType = function (flag) {
switch (flag) {
case 0:
return 'file'
case 1:
return 'link'
case 2:
return 'symlink'
case 3:
return 'character-device'
case 4:
return 'block-device'
case 5:
return 'directory'
case 6:
return 'fifo'
case 7:
return 'contiguous-file'
case 72:
return 'pax-header'
case 55:
return 'pax-global-header'
case 27:
return 'gnu-long-link-path'
case 28:
case 30:
return 'gnu-long-path'
}
return null
}
var toTypeflag = function (flag) {
switch (flag) {
case 'file':
return 0
case 'link':
return 1
case 'symlink':
return 2
case 'character-device':
return 3
case 'block-device':
return 4
case 'directory':
return 5
case 'fifo':
return 6
case 'contiguous-file':
return 7
case 'pax-header':
return 72
}
return 0
}
var indexOf = function (block, num, offset, end) {
for (; offset < end; offset++) {
if (block[offset] === num) return offset
}
return end
}
var cksum = function (block) {
var sum = 8 * 32
for (var i = 0; i < 148; i++) sum += block[i]
for (var j = 156; j < 512; j++) sum += block[j]
return sum
}
var encodeOct = function (val, n) {
val = val.toString(8)
if (val.length > n) return SEVENS.slice(0, n) + ' '
else return ZEROS.slice(0, n - val.length) + val + ' '
}
/* Copied from the node-tar repo and modified to meet
* tar-stream coding standard.
*
* Source: https://github.com/npm/node-tar/blob/51b6627a1f357d2eb433e7378e5f05e83b7aa6cd/lib/header.js#L349
*/
function parse256 (buf) {
// first byte MUST be either 80 or FF
// 80 for positive, FF for 2's comp
var positive
if (buf[0] === 0x80) positive = true
else if (buf[0] === 0xFF) positive = false
else return null
// build up a base-256 tuple from the least sig to the highest
var tuple = []
for (var i = buf.length - 1; i > 0; i--) {
var byte = buf[i]
if (positive) tuple.push(byte)
else tuple.push(0xFF - byte)
}
var sum = 0
var l = tuple.length
for (i = 0; i < l; i++) {
sum += tuple[i] * Math.pow(256, i)
}
return positive ? sum : -1 * sum
}
var decodeOct = function (val, offset, length) {
val = val.slice(offset, offset + length)
offset = 0
// If prefixed with 0x80 then parse as a base-256 integer
if (val[offset] & 0x80) {
return parse256(val)
} else {
// Older versions of tar can prefix with spaces
while (offset < val.length && val[offset] === 32) offset++
var end = clamp(indexOf(val, 32, offset, val.length), val.length, val.length)
while (offset < end && val[offset] === 0) offset++
if (end === offset) return 0
return parseInt(val.slice(offset, end).toString(), 8)
}
}
var decodeStr = function (val, offset, length, encoding) {
return val.slice(offset, indexOf(val, 0, offset, offset + length)).toString(encoding)
}
var addLength = function (str) {
var len = Buffer.byteLength(str)
var digits = Math.floor(Math.log(len) / Math.log(10)) + 1
if (len + digits >= Math.pow(10, digits)) digits++
return (len + digits) + str
}
exports.decodeLongPath = function (buf, encoding) {
return decodeStr(buf, 0, buf.length, encoding)
}
exports.encodePax = function (opts) { // TODO: encode more stuff in pax
var result = ''
if (opts.name) result += addLength(' path=' + opts.name + '\n')
if (opts.linkname) result += addLength(' linkpath=' + opts.linkname + '\n')
var pax = opts.pax
if (pax) {
for (var key in pax) {
result += addLength(' ' + key + '=' + pax[key] + '\n')
}
}
return Buffer.from(result)
}
exports.decodePax = function (buf) {
var result = {}
while (buf.length) {
var i = 0
while (i < buf.length && buf[i] !== 32) i++
var len = parseInt(buf.slice(0, i).toString(), 10)
if (!len) return result
var b = buf.slice(i + 1, len - 1).toString()
var keyIndex = b.indexOf('=')
if (keyIndex === -1) return result
result[b.slice(0, keyIndex)] = b.slice(keyIndex + 1)
buf = buf.slice(len)
}
return result
}
exports.encode = function (opts) {
var buf = alloc(512)
var name = opts.name
var prefix = ''
if (opts.typeflag === 5 && name[name.length - 1] !== '/') name += '/'
if (Buffer.byteLength(name) !== name.length) return null // utf-8
while (Buffer.byteLength(name) > 100) {
var i = name.indexOf('/')
if (i === -1) return null
prefix += prefix ? '/' + name.slice(0, i) : name.slice(0, i)
name = name.slice(i + 1)
}
if (Buffer.byteLength(name) > 100 || Buffer.byteLength(prefix) > 155) return null
if (opts.linkname && Buffer.byteLength(opts.linkname) > 100) return null
buf.write(name)
buf.write(encodeOct(opts.mode & MASK, 6), 100)
buf.write(encodeOct(opts.uid, 6), 108)
buf.write(encodeOct(opts.gid, 6), 116)
buf.write(encodeOct(opts.size, 11), 124)
buf.write(encodeOct((opts.mtime.getTime() / 1000) | 0, 11), 136)
buf[156] = ZERO_OFFSET + toTypeflag(opts.type)
if (opts.linkname) buf.write(opts.linkname, 157)
USTAR_MAGIC.copy(buf, MAGIC_OFFSET)
USTAR_VER.copy(buf, VERSION_OFFSET)
if (opts.uname) buf.write(opts.uname, 265)
if (opts.gname) buf.write(opts.gname, 297)
buf.write(encodeOct(opts.devmajor || 0, 6), 329)
buf.write(encodeOct(opts.devminor || 0, 6), 337)
if (prefix) buf.write(prefix, 345)
buf.write(encodeOct(cksum(buf), 6), 148)
return buf
}
exports.decode = function (buf, filenameEncoding, allowUnknownFormat) {
var typeflag = buf[156] === 0 ? 0 : buf[156] - ZERO_OFFSET
var name = decodeStr(buf, 0, 100, filenameEncoding)
var mode = decodeOct(buf, 100, 8)
var uid = decodeOct(buf, 108, 8)
var gid = decodeOct(buf, 116, 8)
var size = decodeOct(buf, 124, 12)
var mtime = decodeOct(buf, 136, 12)
var type = toType(typeflag)
var linkname = buf[157] === 0 ? null : decodeStr(buf, 157, 100, filenameEncoding)
var uname = decodeStr(buf, 265, 32)
var gname = decodeStr(buf, 297, 32)
var devmajor = decodeOct(buf, 329, 8)
var devminor = decodeOct(buf, 337, 8)
var c = cksum(buf)
// checksum is still initial value if header was null.
if (c === 8 * 32) return null
// valid checksum
if (c !== decodeOct(buf, 148, 8)) throw new Error('Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?')
if (USTAR_MAGIC.compare(buf, MAGIC_OFFSET, MAGIC_OFFSET + 6) === 0) {
// ustar (posix) format.
// prepend prefix, if present.
if (buf[345]) name = decodeStr(buf, 345, 155, filenameEncoding) + '/' + name
} else if (GNU_MAGIC.compare(buf, MAGIC_OFFSET, MAGIC_OFFSET + 6) === 0 &&
GNU_VER.compare(buf, VERSION_OFFSET, VERSION_OFFSET + 2) === 0) {
// 'gnu'/'oldgnu' format. Similar to ustar, but has support for incremental and
// multi-volume tarballs.
} else {
if (!allowUnknownFormat) {
throw new Error('Invalid tar header: unknown format.')
}
}
// to support old tar versions that use trailing / to indicate dirs
if (typeflag === 0 && name && name[name.length - 1] === '/') typeflag = 5
return {
name,
mode,
uid,
gid,
size,
mtime: new Date(1000 * mtime),
type,
linkname,
uname,
gname,
devmajor,
devminor
}
}
@@ -0,0 +1,2 @@
exports.extract = require('./extract')
exports.pack = require('./pack')
+255
View File
@@ -0,0 +1,255 @@
var constants = require('fs-constants')
var eos = require('end-of-stream')
var inherits = require('inherits')
var alloc = Buffer.alloc
var Readable = require('readable-stream').Readable
var Writable = require('readable-stream').Writable
var StringDecoder = require('string_decoder').StringDecoder
var headers = require('./headers')
var DMODE = parseInt('755', 8)
var FMODE = parseInt('644', 8)
var END_OF_TAR = alloc(1024)
var noop = function () {}
var overflow = function (self, size) {
size &= 511
if (size) self.push(END_OF_TAR.slice(0, 512 - size))
}
function modeToType (mode) {
switch (mode & constants.S_IFMT) {
case constants.S_IFBLK: return 'block-device'
case constants.S_IFCHR: return 'character-device'
case constants.S_IFDIR: return 'directory'
case constants.S_IFIFO: return 'fifo'
case constants.S_IFLNK: return 'symlink'
}
return 'file'
}
var Sink = function (to) {
Writable.call(this)
this.written = 0
this._to = to
this._destroyed = false
}
inherits(Sink, Writable)
Sink.prototype._write = function (data, enc, cb) {
this.written += data.length
if (this._to.push(data)) return cb()
this._to._drain = cb
}
Sink.prototype.destroy = function () {
if (this._destroyed) return
this._destroyed = true
this.emit('close')
}
var LinkSink = function () {
Writable.call(this)
this.linkname = ''
this._decoder = new StringDecoder('utf-8')
this._destroyed = false
}
inherits(LinkSink, Writable)
LinkSink.prototype._write = function (data, enc, cb) {
this.linkname += this._decoder.write(data)
cb()
}
LinkSink.prototype.destroy = function () {
if (this._destroyed) return
this._destroyed = true
this.emit('close')
}
var Void = function () {
Writable.call(this)
this._destroyed = false
}
inherits(Void, Writable)
Void.prototype._write = function (data, enc, cb) {
cb(new Error('No body allowed for this entry'))
}
Void.prototype.destroy = function () {
if (this._destroyed) return
this._destroyed = true
this.emit('close')
}
var Pack = function (opts) {
if (!(this instanceof Pack)) return new Pack(opts)
Readable.call(this, opts)
this._drain = noop
this._finalized = false
this._finalizing = false
this._destroyed = false
this._stream = null
}
inherits(Pack, Readable)
Pack.prototype.entry = function (header, buffer, callback) {
if (this._stream) throw new Error('already piping an entry')
if (this._finalized || this._destroyed) return
if (typeof buffer === 'function') {
callback = buffer
buffer = null
}
if (!callback) callback = noop
var self = this
if (!header.size || header.type === 'symlink') header.size = 0
if (!header.type) header.type = modeToType(header.mode)
if (!header.mode) header.mode = header.type === 'directory' ? DMODE : FMODE
if (!header.uid) header.uid = 0
if (!header.gid) header.gid = 0
if (!header.mtime) header.mtime = new Date()
if (typeof buffer === 'string') buffer = Buffer.from(buffer)
if (Buffer.isBuffer(buffer)) {
header.size = buffer.length
this._encode(header)
var ok = this.push(buffer)
overflow(self, header.size)
if (ok) process.nextTick(callback)
else this._drain = callback
return new Void()
}
if (header.type === 'symlink' && !header.linkname) {
var linkSink = new LinkSink()
eos(linkSink, function (err) {
if (err) { // stream was closed
self.destroy()
return callback(err)
}
header.linkname = linkSink.linkname
self._encode(header)
callback()
})
return linkSink
}
this._encode(header)
if (header.type !== 'file' && header.type !== 'contiguous-file') {
process.nextTick(callback)
return new Void()
}
var sink = new Sink(this)
this._stream = sink
eos(sink, function (err) {
self._stream = null
if (err) { // stream was closed
self.destroy()
return callback(err)
}
if (sink.written !== header.size) { // corrupting tar
self.destroy()
return callback(new Error('size mismatch'))
}
overflow(self, header.size)
if (self._finalizing) self.finalize()
callback()
})
return sink
}
Pack.prototype.finalize = function () {
if (this._stream) {
this._finalizing = true
return
}
if (this._finalized) return
this._finalized = true
this.push(END_OF_TAR)
this.push(null)
}
Pack.prototype.destroy = function (err) {
if (this._destroyed) return
this._destroyed = true
if (err) this.emit('error', err)
this.emit('close')
if (this._stream && this._stream.destroy) this._stream.destroy()
}
Pack.prototype._encode = function (header) {
if (!header.pax) {
var buf = headers.encode(header)
if (buf) {
this.push(buf)
return
}
}
this._encodePax(header)
}
Pack.prototype._encodePax = function (header) {
var paxHeader = headers.encodePax({
name: header.name,
linkname: header.linkname,
pax: header.pax
})
var newHeader = {
name: 'PaxHeader',
mode: header.mode,
uid: header.uid,
gid: header.gid,
size: paxHeader.length,
mtime: header.mtime,
type: 'pax-header',
linkname: header.linkname && 'PaxHeader',
uname: header.uname,
gname: header.gname,
devmajor: header.devmajor,
devminor: header.devminor
}
this.push(headers.encode(newHeader))
this.push(paxHeader)
overflow(this, paxHeader.length)
newHeader.size = header.size
newHeader.type = header.type
this.push(headers.encode(newHeader))
}
Pack.prototype._read = function (n) {
var drain = this._drain
this._drain = noop
drain()
}
module.exports = Pack
@@ -0,0 +1,58 @@
{
"name": "tar-stream",
"version": "2.2.0",
"description": "tar-stream is a streaming tar parser and generator and nothing else. It is streams2 and operates purely using streams which means you can easily extract/parse tarballs without ever hitting the file system.",
"author": "Mathias Buus <mathiasbuus@gmail.com>",
"dependencies": {
"bl": "^4.0.3",
"end-of-stream": "^1.4.1",
"fs-constants": "^1.0.0",
"inherits": "^2.0.3",
"readable-stream": "^3.1.1"
},
"devDependencies": {
"concat-stream": "^2.0.0",
"standard": "^12.0.1",
"tape": "^4.9.2"
},
"scripts": {
"test": "standard && tape test/extract.js test/pack.js",
"test-all": "standard && tape test/*.js"
},
"keywords": [
"tar",
"tarball",
"parse",
"parser",
"generate",
"generator",
"stream",
"stream2",
"streams",
"streams2",
"streaming",
"pack",
"extract",
"modify"
],
"bugs": {
"url": "https://github.com/mafintosh/tar-stream/issues"
},
"homepage": "https://github.com/mafintosh/tar-stream",
"main": "index.js",
"files": [
"*.js",
"LICENSE"
],
"directories": {
"test": "test"
},
"license": "MIT",
"repository": {
"type": "git",
"url": "git+https://github.com/mafintosh/tar-stream.git"
},
"engines": {
"node": ">=6"
}
}
@@ -0,0 +1,11 @@
const tar = require('tar-stream')
const fs = require('fs')
const path = require('path')
const pipeline = require('pump') // eequire('stream').pipeline
fs.createReadStream('test.tar')
.pipe(tar.extract())
.on('entry', function (header, stream, done) {
console.log(header.name)
pipeline(stream, fs.createWriteStream(path.join('/tmp', header.name)), done)
})
+67
View File
@@ -0,0 +1,67 @@
{
"name": "prebuild-install",
"version": "7.1.3",
"description": "A command line tool to easily install prebuilt binaries for multiple version of node/iojs on a specific platform",
"scripts": {
"test": "standard && hallmark && tape test/*-test.js",
"hallmark": "hallmark --fix"
},
"keywords": [
"prebuilt",
"binaries",
"native",
"addon",
"module",
"c",
"c++",
"bindings",
"devops",
"napi"
],
"dependencies": {
"detect-libc": "^2.0.0",
"expand-template": "^2.0.3",
"github-from-package": "0.0.0",
"minimist": "^1.2.3",
"mkdirp-classic": "^0.5.3",
"napi-build-utils": "^2.0.0",
"node-abi": "^3.3.0",
"pump": "^3.0.0",
"rc": "^1.2.7",
"simple-get": "^4.0.0",
"tar-fs": "^2.0.0",
"tunnel-agent": "^0.6.0"
},
"devDependencies": {
"a-native-module": "^1.0.0",
"hallmark": "^4.0.0",
"nock": "^10.0.6",
"rimraf": "^2.5.2",
"standard": "^16.0.4",
"tape": "^5.3.1",
"tempy": "0.2.1"
},
"bin": "./bin.js",
"repository": {
"type": "git",
"url": "https://github.com/prebuild/prebuild-install.git"
},
"author": "Mathias Buus (@mafintosh)",
"contributors": [
"Julian Gruber <julian@juliangruber.com> (https://github.com/juliangruber)",
"Brett Lawson <brett19@gmail.com> (https://github.com/brett19)",
"Pieter Hintjens <ph@imatix.com> (https://github.com/hintjens)",
"Lars-Magnus Skog <ralphtheninja@riseup.net> (https://github.com/ralphtheninja)",
"Jesús Leganés Combarro <piranna@gmail.com> (https://github.com/piranna)",
"Mathias Küsel <mathiask@hotmail.de> (https://github.com/mathiask88)",
"Lukas Geiger <lukas.geiger94@gmail.com> (https://github.com/lgeiger)"
],
"license": "MIT",
"bugs": {
"url": "https://github.com/prebuild/prebuild-install/issues"
},
"homepage": "https://github.com/prebuild/prebuild-install",
"engines": {
"node": ">=10"
}
}
+35
View File
@@ -0,0 +1,35 @@
const url = require('url')
const tunnel = require('tunnel-agent')
const util = require('./util')
function applyProxy (reqOpts, opts) {
const log = opts.log || util.noopLogger
const proxy = opts['https-proxy'] || opts.proxy
if (proxy) {
// eslint-disable-next-line node/no-deprecated-api
const parsedDownloadUrl = url.parse(reqOpts.url)
// eslint-disable-next-line node/no-deprecated-api
const parsedProxy = url.parse(proxy)
const uriProtocol = (parsedDownloadUrl.protocol === 'https:' ? 'https' : 'http')
const proxyProtocol = (parsedProxy.protocol === 'https:' ? 'Https' : 'Http')
const tunnelFnName = [uriProtocol, proxyProtocol].join('Over')
reqOpts.agent = tunnel[tunnelFnName]({
proxy: {
host: parsedProxy.hostname,
port: +parsedProxy.port,
proxyAuth: parsedProxy.auth
}
})
log.http('request', 'Proxy setup detected (Host: ' +
parsedProxy.hostname + ', Port: ' +
parsedProxy.port + ', Authentication: ' +
(parsedProxy.auth ? 'Yes' : 'No') + ')' +
' Tunneling with ' + tunnelFnName)
}
return reqOpts
}
module.exports = applyProxy
+64
View File
@@ -0,0 +1,64 @@
const path = require('path')
const minimist = require('minimist')
const getAbi = require('node-abi').getAbi
const detectLibc = require('detect-libc')
const napi = require('napi-build-utils')
const env = process.env
const libc = env.LIBC || process.env.npm_config_libc ||
(detectLibc.isNonGlibcLinuxSync() && detectLibc.familySync()) || ''
// Get the configuration
module.exports = function (pkg) {
const pkgConf = pkg.config || {}
const buildFromSource = env.npm_config_build_from_source
const rc = require('rc')('prebuild-install', {
target: pkgConf.target || env.npm_config_target || process.versions.node,
runtime: pkgConf.runtime || env.npm_config_runtime || 'node',
arch: pkgConf.arch || env.npm_config_arch || process.arch,
libc: libc,
platform: env.npm_config_platform || process.platform,
debug: env.npm_config_debug === 'true',
force: false,
verbose: env.npm_config_verbose === 'true',
buildFromSource: buildFromSource === pkg.name || buildFromSource === 'true',
path: '.',
proxy: env.npm_config_proxy || env.http_proxy || env.HTTP_PROXY,
'https-proxy': env.npm_config_https_proxy || env.https_proxy || env.HTTPS_PROXY,
'local-address': env.npm_config_local_address,
'local-prebuilds': 'prebuilds',
'tag-prefix': 'v',
download: env.npm_config_download
}, minimist(process.argv, {
alias: {
target: 't',
runtime: 'r',
help: 'h',
arch: 'a',
path: 'p',
version: 'v',
download: 'd',
buildFromSource: 'build-from-source',
token: 'T'
}
}))
rc.path = path.resolve(rc.path === true ? '.' : rc.path || '.')
if (napi.isNapiRuntime(rc.runtime) && rc.target === process.versions.node) {
rc.target = napi.getBestNapiBuildVersion()
}
rc.abi = napi.isNapiRuntime(rc.runtime) ? rc.target : getAbi(rc.target, rc.runtime)
rc.libc = rc.platform !== 'linux' || rc.libc === detectLibc.GLIBC ? '' : rc.libc
return rc
}
// Print the configuration values when executed standalone for testing purposses
if (!module.parent) {
console.log(JSON.stringify(module.exports({}), null, 2))
}
+143
View File
@@ -0,0 +1,143 @@
const path = require('path')
const github = require('github-from-package')
const home = require('os').homedir
const crypto = require('crypto')
const expandTemplate = require('expand-template')()
function getDownloadUrl (opts) {
const pkgName = opts.pkg.name.replace(/^@[a-zA-Z0-9_\-.~]+\//, '')
return expandTemplate(urlTemplate(opts), {
name: pkgName,
package_name: pkgName,
version: opts.pkg.version,
major: opts.pkg.version.split('.')[0],
minor: opts.pkg.version.split('.')[1],
patch: opts.pkg.version.split('.')[2],
prerelease: opts.pkg.version.split('-')[1],
build: opts.pkg.version.split('+')[1],
abi: opts.abi || process.versions.modules,
node_abi: process.versions.modules,
runtime: opts.runtime || 'node',
platform: opts.platform,
arch: opts.arch,
libc: opts.libc || '',
configuration: (opts.debug ? 'Debug' : 'Release'),
module_name: opts.pkg.binary && opts.pkg.binary.module_name,
tag_prefix: opts['tag-prefix']
})
}
function getApiUrl (opts) {
return github(opts.pkg).replace('github.com', 'api.github.com/repos') + '/releases'
}
function getAssetUrl (opts, assetId) {
return getApiUrl(opts) + '/assets/' + assetId
}
function urlTemplate (opts) {
if (typeof opts.download === 'string') {
return opts.download
}
const packageName = '{name}-v{version}-{runtime}-v{abi}-{platform}{libc}-{arch}.tar.gz'
const hostMirrorUrl = getHostMirrorUrl(opts)
if (hostMirrorUrl) {
return hostMirrorUrl + '/{tag_prefix}{version}/' + packageName
}
if (opts.pkg.binary && opts.pkg.binary.host) {
return [
opts.pkg.binary.host,
opts.pkg.binary.remote_path,
opts.pkg.binary.package_name || packageName
].map(function (path) {
return trimSlashes(path)
}).filter(Boolean).join('/')
}
return github(opts.pkg) + '/releases/download/{tag_prefix}{version}/' + packageName
}
function getEnvPrefix (pkgName) {
return 'npm_config_' + (pkgName || '').replace(/[^a-zA-Z0-9]/g, '_').replace(/^_/, '')
}
function getHostMirrorUrl (opts) {
const propName = getEnvPrefix(opts.pkg.name) + '_binary_host'
return process.env[propName] || process.env[propName + '_mirror']
}
function trimSlashes (str) {
if (str) return str.replace(/^\.\/|^\/|\/$/g, '')
}
function cachedPrebuild (url) {
const digest = crypto.createHash('sha512').update(url).digest('hex').slice(0, 6)
return path.join(prebuildCache(), digest + '-' + path.basename(url).replace(/[^a-zA-Z0-9.]+/g, '-'))
}
function npmCache () {
const env = process.env
return env.npm_config_cache || (env.APPDATA ? path.join(env.APPDATA, 'npm-cache') : path.join(home(), '.npm'))
}
function prebuildCache () {
return path.join(npmCache(), '_prebuilds')
}
function tempFile (cached) {
return cached + '.' + process.pid + '-' + Math.random().toString(16).slice(2) + '.tmp'
}
function packageOrigin (env, pkg) {
// npm <= 6: metadata is stored on disk in node_modules
if (pkg._from) {
return pkg._from
}
// npm 7: metadata is exposed to environment by arborist
if (env.npm_package_from) {
// NOTE: seems undefined atm (npm 7.0.2)
return env.npm_package_from
}
if (env.npm_package_resolved) {
// NOTE: not sure about the difference with _from, but it's all we have
return env.npm_package_resolved
}
}
function localPrebuild (url, opts) {
const propName = getEnvPrefix(opts.pkg.name) + '_local_prebuilds'
const prefix = process.env[propName] || opts['local-prebuilds'] || 'prebuilds'
return path.join(prefix, path.basename(url))
}
const noopLogger = {
http: function () {},
silly: function () {},
debug: function () {},
info: function () {},
warn: function () {},
error: function () {},
critical: function () {},
alert: function () {},
emergency: function () {},
notice: function () {},
verbose: function () {},
fatal: function () {}
}
exports.getDownloadUrl = getDownloadUrl
exports.getApiUrl = getApiUrl
exports.getAssetUrl = getAssetUrl
exports.urlTemplate = urlTemplate
exports.cachedPrebuild = cachedPrebuild
exports.localPrebuild = localPrebuild
exports.prebuildCache = prebuildCache
exports.npmCache = npmCache
exports.tempFile = tempFile
exports.packageOrigin = packageOrigin
exports.noopLogger = noopLogger