Skip to content

Commit 20d1124

Browse files
d4mrd4mr
committed
update readme
1 parent 6ab3548 commit 20d1124

File tree

1 file changed

+189
-98
lines changed

1 file changed

+189
-98
lines changed

‎README.md

Lines changed: 189 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -1,136 +1,227 @@
11
# Thirdweb Engine Core
22

3-
A high-performance, Rust-based blockchain transaction engine that provides robust infrastructure for Web3 applications. Engine Core handles smart contract interactions, Account Abstraction (ERC-4337), and transaction management with enterprise-grade reliability.
3+
Production-grade blockchain transaction infrastructure built in Rust. Engine Core is a high-performance, horizontally-scalable system designed for developers building serious Web3 applications that require reliable smart contract interactions, Account Abstraction support, and enterprise-level transaction processing.
4+
5+
## Why Engine Core?
6+
7+
### Performance & Scalability
8+
Built with Rust's zero-cost abstractions and memory safety guarantees. The architecture supports horizontal scaling through Redis-backed job queues with configurable worker pools and lease-based concurrency control.
9+
10+
### Production-Ready Infrastructure
11+
- Redis-backed message queues with atomic operations and retry logic
12+
- Graceful shutdown handling with job completion guarantees
13+
- Comprehensive error handling and transaction rollback mechanisms
14+
- Built-in monitoring and observability through structured logging
15+
16+
### Developer Experience
17+
- Complete OpenAPI specification with interactive documentation
18+
- Type-safe configuration system with environment variable overrides
19+
- Modular architecture allowing selective component deployment
20+
- Extensive test coverage including integration tests with Redis
21+
22+
## 🏗️ Architecture
23+
24+
Engine Core implements a microservices-like architecture within a single binary, using Rust's workspace system for clean module separation:
25+
26+
### Core Infrastructure (`core/`)
27+
**Purpose**: Fundamental blockchain operations and abstractions
28+
- **Chain Management** (`chain.rs`): Multi-chain RPC client management with automatic failover
29+
- **Transaction Primitives** (`transaction.rs`): Raw transaction building, signing, and broadcasting
30+
- **UserOperation Support** (`userop.rs`): Complete ERC-4337 implementation with v0.6/v0.7 compatibility
31+
- **RPC Clients** (`rpc_clients/`): Specialized clients for bundlers, paymasters, and JSON-RPC endpoints
32+
- **Error Handling** (`error.rs`): Comprehensive error types with context preservation
33+
34+
### Account Abstraction Engine (`aa-core/`)
35+
**Purpose**: Complete ERC-4337 Account Abstraction implementation
36+
- **Smart Account Management** (`smart_account/`): Account factory integrations and deployment
37+
- **UserOperation Builder** (`userop/`): Gas estimation, signature aggregation, and bundler submission
38+
- **Account Factory Support** (`account_factory/`): Pluggable factory implementations (default, chained)
39+
- **Signature Handling** (`signer.rs`): Multi-signature support with Vault integration
40+
41+
### HTTP API Server (`server/`)
42+
**Purpose**: REST API layer with comprehensive endpoint coverage
43+
- **Contract Operations**: Read, write, and encode smart contract functions
44+
- **Transaction Management**: Raw transaction sending with AA support
45+
- **Message Signing**: EIP-712 typed data and personal message signing
46+
- **Dynamic ABI**: Runtime contract introspection and interaction
47+
- **OpenAPI Documentation**: Auto-generated specs with Scalar UI
48+
49+
### Background Job System (`executors/` + `twmq/`)
50+
**Purpose**: Reliable asynchronous processing with Redis persistence
51+
52+
#### TWMQ (Thirdweb Message Queue)
53+
Advanced Redis-backed job queue with enterprise features:
54+
- **Lease-Based Concurrency**: Prevents job duplication across worker instances
55+
- **Atomic Operations**: All queue operations use Lua scripts for consistency
56+
- **Retry Logic**: Configurable backoff strategies with failure categorization
57+
- **Job Lifecycle Management**: Pending → Active → Success/Failed with full audit trail
58+
- **Delayed Jobs**: Schedule jobs for future execution with precise timing
59+
- **Cancellation Support**: Cancel jobs in any state with immediate or pending cancellation
60+
61+
#### Executor Types
62+
- **Webhook Delivery**: Reliable HTTP webhook notifications with configurable retries
63+
- **Transaction Confirmation**: Block confirmation tracking with reorganization handling
64+
- **External Bundler Integration**: UserOperation submission and status monitoring
65+
66+
### Thirdweb Service Integration (`thirdweb-core/`)
67+
**Purpose**: First-party service integrations
68+
- **Vault SDK**: Hardware-backed private key management
69+
- **IAW (In-App Wallets)**: Embedded wallet creation and management
70+
- **ABI Service**: Dynamic contract ABI resolution and caching
71+
72+
## 🚀 Getting Started
73+
74+
### System Requirements
75+
76+
- **Rust 1.70+** (2021 edition with async support)
77+
- **Redis 6.0+** (required for job queue persistence and atomic operations)
78+
- **Thirdweb API Credentials** (secret key and client ID from dashboard)
79+
80+
### Quick Setup
481

5-
## 🏗️ Architecture Overview
6-
7-
Thirdweb Engine Core is built as a modular Rust workspace with the following key components:
8-
9-
### Core Components
10-
11-
- **`server/`** - Main HTTP API server (`thirdweb-engine`)
12-
- **`core/`** - Core blockchain functionality (`engine-core`)
13-
- **`aa-core/`** - Account Abstraction support (`engine-aa-core`)
14-
- **`executors/`** - Background job processors (`engine-executors`)
15-
- **`thirdweb-core/`** - Thirdweb-specific integrations
16-
- **`twmq/`** - Thirdweb Message Queue - Redis-backed job queue system
17-
18-
### Key Features
19-
20-
- ✅ **Account Abstraction (ERC-4337)** - Full support for UserOperations v0.6 and v0.7
21-
- ✅ **Multi-chain Support** - Works with any EVM-compatible blockchain
22-
- ✅ **Smart Contract Interactions** - Read, write, and encode contract calls
23-
- ✅ **Background Job Processing** - Reliable webhook delivery and transaction confirmation
24-
- ✅ **Secure Wallet Management** - Integration with Thirdweb Vault for key management
25-
- ✅ **REST API** - OpenAPI-documented endpoints with Scalar documentation
26-
- ✅ **Enterprise Reliability** - Redis-backed job queues with retry logic and error handling
27-
28-
## 🚀 Quick Start
29-
30-
### Prerequisites
82+
```bash
83+
# Clone and build
84+
git clone <repo-url> && cd engine-core
85+
cargo build --release
3186

32-
- **Rust** (latest stable version)
33-
- **Redis** server running locally or accessible remotely
34-
- **Thirdweb API credentials** (secret key and client ID)
87+
# Start Redis (Docker recommended for development)
88+
docker run -d --name redis -p 6379:6379 redis:7-alpine
3589

36-
### Environment Setup
90+
# Configure credentials
91+
export APP__THIRDWEB__SECRET="your_secret_key"
92+
export APP__THIRDWEB__CLIENT_ID="your_client_id"
3793

38-
1. **Clone the repository:**
94+
# Launch engine
95+
RUST_LOG=info ./target/release/thirdweb-engine
96+
```
3997

40-
```bash
41-
git clone <repo-url>
42-
cd engine-core
43-
```
98+
### Configuration System
4499

45-
2. **Install dependencies:**
100+
Engine Core uses a hierarchical configuration system: YAML files + environment variables with full type safety and validation.
46101

47-
```bash
48-
cargo build
49-
```
102+
#### Configuration Layers
103+
1. **Base Configuration** (`server_base.yaml`) - Default values
104+
2. **Environment-Specific** (`server_development.yaml`, `server_production.yaml`)
105+
3. **Environment Variables** - Highest priority, prefix with `APP__`
50106

51-
3. **Set up Redis:**
107+
#### Essential Configuration
52108

53-
```bash
54-
# Using Docker
55-
docker run -d --name redis -p 6379:6379 redis:7-alpine
109+
```yaml
110+
# server/configuration/server_local.yaml
111+
server:
112+
host: "0.0.0.0"
113+
port: 3069
56114

57-
# Or install locally (macOS)
58-
brew install redis
59-
brew services start redis
60-
```
115+
thirdweb:
116+
secret: "your_thirdweb_secret_key"
117+
client_id: "your_thirdweb_client_id"
118+
urls:
119+
vault: "https://vault.thirdweb.com"
120+
bundler: "bundler.thirdweb.com"
121+
paymaster: "bundler.thirdweb.com"
61122

62-
### Configuration
123+
redis:
124+
url: "redis://localhost:6379"
63125

64-
The server uses a layered configuration system with YAML files and environment variables.
126+
queue:
127+
webhook_workers: 50
128+
external_bundler_send_workers: 20
129+
userop_confirm_workers: 10
130+
local_concurrency: 100
131+
polling_interval_ms: 100
132+
lease_duration_seconds: 600
133+
```
65134
66-
#### Configuration Files
135+
#### Environment Variable Override Examples
67136
68-
Create configuration files in the `server/configuration/` directory:
137+
```bash
138+
# Scale worker pools for high throughput
139+
export APP__QUEUE__WEBHOOK_WORKERS=200
140+
export APP__QUEUE__LOCAL_CONCURRENCY=500
69141

70-
**`server/configuration/server_local.yaml`:**
142+
# Custom Redis configuration
143+
export APP__REDIS__URL="redis://redis-cluster:6379"
71144

72-
```yaml
73-
thirdweb:
74-
secret: "your_thirdweb_secret_key"
75-
client_id: "your_thirdweb_client_id"
145+
# Debug logging for development
146+
export RUST_LOG="thirdweb_engine=debug,twmq=debug"
76147
```
77148

78-
#### Or with Environment Variables
149+
### Docker Deployment
79150

80-
You can also override any configuration using environment variables with the prefix `APP__`:
151+
```dockerfile
152+
FROM rust:1.70 as builder
153+
WORKDIR /app
154+
COPY . .
155+
RUN cargo build --release
81156

82-
```bash
83-
export APP__THIRDWEB__SECRET="your_secret_key"
84-
export APP__THIRDWEB__CLIENT_ID="your_client_id"
85-
export APP__REDIS__URL="redis://localhost:6379"
86-
export APP__SERVER__PORT=3069
157+
FROM debian:bookworm-slim
158+
RUN apt-get update && apt-get install -y ca-certificates
159+
COPY --from=builder /app/target/release/thirdweb-engine /usr/local/bin/
160+
EXPOSE 3069
161+
CMD ["thirdweb-engine"]
87162
```
88163

89-
#### Required Configuration
90-
91-
- **`thirdweb.secret`** - Your Thirdweb secret key
92-
- **`thirdweb.client_id`** - Your Thirdweb client ID
93-
- **`redis.url`** - Redis connection URL (default: `redis://localhost:6379`)
164+
### Health Checks & Monitoring
94165

95-
### Running the Server
96-
97-
1. **Start Redis** (if not already running):
166+
```bash
167+
# API health check
168+
curl http://localhost:3069/v1/api.json
98169

99-
```bash
100-
redis-server
101-
```
170+
# Queue statistics (if monitoring endpoint enabled)
171+
curl http://localhost:3069/v1/queue/stats
102172

103-
2. **Run the server:**
173+
# Structured logging output
174+
RUST_LOG="thirdweb_engine=info,twmq=warn" ./thirdweb-engine
175+
```
104176

105-
```bash
106-
# Development mode with debug logging
107-
RUST_LOG=debug cargo run --bin thirdweb-engine
177+
## 🌩️ Thirdweb Engine Cloud
108178

109-
# Or build and run the optimized binary
110-
cargo build --release
111-
./target/release/thirdweb-engine
112-
```
179+
**Want Engine without the ops overhead?** [**Thirdweb Engine Cloud**](https://thirdweb.com/engine) is our fully-managed, production-ready service built on Engine Core with enterprise enhancements:
113180

114-
3. **Verify the server is running:**
115-
```bash
116-
curl http://localhost:3069/v1/api.json
117-
```
181+
### ⚡ Enhanced Features Beyond Core
182+
- **Auto Execution Resolution**: Smart execution strategy selection with cached Account Abstraction details
183+
- **Streamlined Wallet Management**: Convenient wallet creation and management through dashboard
184+
- **Smart Account Cache**: Pre-resolved AA configurations (signer addresses, factory details, gas policies)
185+
- **Global Edge Network**: Optimized RPC routing and intelligent caching for sub-100ms response times
186+
- **Advanced Analytics**: Real-time transaction monitoring, gas usage insights, and performance metrics
187+
- **Zero-Config Account Abstraction**: Automatic paymaster selection and gas sponsorship
118188

119-
The server will start on `http://localhost:3069` by default.
189+
### 🛡️ Production-Ready Operations
190+
- **High Availability** with automated failover and disaster recovery
191+
- **Horizontal Auto-Scaling** based on transaction volume and queue depth
192+
- **Enterprise Security**: Encryption at rest/transit, comprehensive audit logging
193+
- **Expert Support** with dedicated technical assistance
194+
- **Custom Rate Limits** and priority processing for high-volume applications
120195

121-
## 📚 API Documentation
196+
### 🚀 Get Started Instantly
197+
```bash
198+
# No infrastructure setup required
199+
curl -X POST "https://api.engine.thirdweb.com/contract/write" \
200+
-H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
201+
-H "Content-Type: application/json" \
202+
-d '{
203+
"executionOptions": {
204+
"chainId": 137,
205+
"type": "auto",
206+
"from": "0x..."
207+
},
208+
"params": [{
209+
"contractAddress": "0x...",
210+
"functionName": "mint",
211+
"args": ["0x...", "1"]
212+
}]
213+
}'
214+
```
122215

123-
Once the server is running, you can access:
216+
**[Start Building →](https://thirdweb.com/engine)** | **[View Cloud API Reference →](https://engine.thirdweb.com/reference)**
124217

125-
- **API Documentation**: `http://localhost:3069/v1/reference`
126-
- **OpenAPI Spec**: `http://localhost:3069/v1/api.json`
218+
---
127219

128-
### Available Endpoints
220+
## 📚 Self-Hosted API Documentation
129221

130-
- `POST /v1/read/contract` - Read from smart contracts
131-
- `POST /v1/write/contract` - Write to smart contracts (with AA support)
132-
- `POST /v1/encode/contract` - Encode contract function calls
133-
- `POST /v1/write/transaction` - Send raw transactions
222+
For self-hosted Engine Core instances:
223+
- **Interactive Documentation**: `http://localhost:3069/reference`
224+
- **OpenAPI Specification**: `http://localhost:3069/api.json`
134225

135226
## 🔧 Development
136227

@@ -278,7 +369,7 @@ MIT
278369

279370
For issues and questions:
280371

281-
1. Check the API documentation at `/v1/reference`
372+
1. Check the API documentation at `/reference`
282373
2. Review server logs for error details
283374
3. Ensure Redis is running and accessible
284375
4. Verify Thirdweb credentials are valid

0 commit comments

Comments
 (0)