Skip to content

Deno rework #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jul 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
root = true

[*]
charset = utf-8
end_of_line = lf
trim_trailing_whitespace = true
insert_final_newline = true

[*.md]
insert_final_newline = false
trim_trailing_whitespace = false

[*.{js,json,ts,mts,yml,yaml}]
indent_size = 2
indent_style = space
2 changes: 0 additions & 2 deletions .eslintignore

This file was deleted.

23 changes: 0 additions & 23 deletions .eslintrc.json

This file was deleted.

20 changes: 0 additions & 20 deletions .github/workflows/nodejs.yml

This file was deleted.

39 changes: 24 additions & 15 deletions .github/workflows/publish.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

name: Publish to NPM

on:
Expand All @@ -11,18 +12,26 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: volta-cli/action@v1
- run: npm ci --no-audit
- run: npm run lint --if-present
- run: npm test
- run: npm run build --if-present
env:
CI: true
- name: Setup .npmrc file to publish to npm
run: |
echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > .npmrc
env:
NPM_TOKEN: ${{secrets.NPM_AUTOMATION_TOKEN}}
- name: Publish to NPM
run: npm publish
- name: Clone repository
uses: actions/checkout@v3

- name: Install Deno
uses: denoland/setup-deno@v1
with:
deno-version: v1.x

- name: Run Deno build script
run: deno task build-npm

- name: Navigate to npm directory
run: cd ./npm

- name: Setup .npmrc file to publish to npm
run: |
echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > .npmrc
env:
NPM_TOKEN: ${{ secrets.NPM_AUTOMATION_TOKEN }}

- name: Publish to NPM
run: npm publish --dry-run
working-directory: ./npm
28 changes: 28 additions & 0 deletions .github/workflows/run-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
name: Run tests

on:
workflow_dispatch:
pull_request:


jobs:
build:
runs-on: ubuntu-latest

steps:
- name: Clone repository
uses: actions/checkout@v3

- name: Install Deno
uses: denoland/setup-deno@v1
with:
deno-version: v1.x

- name: Run Deno Test
run: deno task test

- name: Build JSR
run: deno task build-jsr

- name: Build NPM
run: deno task build-npm
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
node_modules
coverage
coverage
scripts
npm
dev
cov_profile
.vscode
.history
3 changes: 0 additions & 3 deletions .prettierignore

This file was deleted.

14 changes: 0 additions & 14 deletions .prettierrc

This file was deleted.

47 changes: 33 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# Scrapfly SDK

`npm install scrapfly-sdk`
`npm install scrapfly-sdk`
`deno add jsr:@scrapfly/scrapfly-sdk`
`bun jsr add @scrapfly/scrapfly-sdk`

Typescript/NodeJS SDK for [Scrapfly.io](https://scrapfly.io/) web scraping API which allows to:
Typescript/Javascript SDK for [Scrapfly.io](https://scrapfly.io/) web scraping API which allows to:

- Scrape the web without being blocked.
- Use headless browsers to access Javascript-powered page data.
Expand All @@ -11,14 +13,21 @@ Typescript/NodeJS SDK for [Scrapfly.io](https://scrapfly.io/) web scraping API w

For web scraping guides see [our blog](https://scrapfly.io/blog/) and [#scrapeguide](https://scrapfly.io/blog/tag/scrapeguide/) tag for how to scrape specific targets.

The SDK is distributed through:
- [npmjs.com/package/scrapfly-sdk](https://www.npmjs.com/package/scrapfly-sdk)
- [jsr.io/@scrapfly/scrapfly-sdk](https://jsr.io/@scrapfly/scrapfly-sdk)

## Quick Intro

1. Register a [Scrapfly account for free](https://scrapfly.io/register)
2. Get your API Key on [scrapfly.io/dashboard](https://scrapfly.io/dashboard)
3. Start scraping: 🚀

```javascript
// node or bun:
import { ScrapflyClient, ScrapeConfig } from 'scrapfly-sdk';
// deno:
import { ScrapflyClient, ScrapeConfig } from 'jsr:@scrapfly/scrapfly-sdk';

const key = 'YOUR SCRAPFLY KEY';
const client = new ScrapflyClient({ key });
Expand Down Expand Up @@ -70,15 +79,25 @@ new ScrapeConfig({

## Development

Install and setup environment:

```shell
$ npm install
```

Build and test:

```shell
$ npm run build
$ npm run test
```
This is a Deno Typescript project that builds to NPM through [DNT](https://github.com/denoland/dnt).

- `/src` directory contains all of the source code with `main.ts` being the entry point.
- `__tests__` directory contains tests for the source code.
- `deno.json` contains meta information
- `build.ts` is the build script that builds the project to nodejs ESM package.
- `/npm` directory will be produced when `built.ts` is executed for building node package.

```bash
# make modifications and run tests
$ deno task test
# format
$ deno fmt
# lint
$ deno lint
# publish JSR:
$ deno publish
# build NPM package:
$ deno build-npm
# publish NPM:
$ cd npm && npm publish
```
Loading
Loading