revamp target #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## What Was Added
I've successfully added functionality to the target-elasticsearch Meltano target to create or update index field mappings based on a tap config parameter.
## Files Modified
### 1. `target_elasticsearch/common.py`
- Added `INDEX_MAPPINGS = "index_mappings"` constant
### 2. `target_elasticsearch/target.py`
- Added import for `INDEX_MAPPINGS` constant
- Added new configuration property for `index_mappings` with detailed documentation
### 3. `target_elasticsearch/sinks.py`
- Added import for `INDEX_MAPPINGS` constant
- Added `create_or_update_mapping()` method to handle index mapping creation/updates
- Modified `create_indices()` method to use the new mapping functionality
### 4. `tests/unit/test_target.py`
- Added comprehensive tests for the new mapping functionality:
- Test with mapping configuration
- Test without mapping configuration
- Test with existing indices (updates)
## Files Created
### 1. `MAPPING_EXAMPLES.md`
- Comprehensive documentation with examples
- Configuration options for meltano.yml and environment variables
- Advanced mapping features and use cases
### 2. `example_usage.py`
- Runnable example script demonstrating the new functionality
- Shows both YAML and environment variable configuration methods
### 3. Updated `README.md`
- Added documentation for the new `index_mappings` parameter
- Added quick example and reference to detailed documentation
## How It Works
1. **Configuration**: Users define field mappings in the `index_mappings` config parameter
2. **Per-Stream Mappings**: Each stream can have its own mapping configuration
3. **Smart Creation**:
- Creates new indices with mappings if they don't exist
- Updates mappings for existing indices where possible
- Falls back to default behavior for streams without mapping config
4. **Error Handling**: Gracefully handles mapping conflicts and other Elasticsearch errors
## Example Configuration
```yaml
config:
index_mappings:
users:
properties:
email:
type: keyword
created_at:
type: date
full_name:
type: text
analyzer: standard
```
## Benefits
- **Performance**: Proper field types improve query performance
- **Storage**: Appropriate field types reduce storage requirements
- **Search**: Better text analysis and keyword fields for search
- **Data Integrity**: Field types enforce consistency
- **Backward Compatible**: Existing configurations work unchanged
## Testing
The implementation includes comprehensive unit tests covering:
- Index creation with mappings
- Index updates with mappings
- Graceful handling when no mappings are configured
- Error handling for various scenarios
All changes maintain backward compatibility - existing configurations will work exactly as before.
…ping-config-property Index Mappings Feature -- to be added at v1.3.0
- move from poetry to uv - update dependencies - drop `common.py` and move functionality to `sink.py` - use lowercase for tap config - only create an index once if `index_schema_fields` in None because then max one index will be set per stream. - move `template_index` and `build_fields` to the class - simplify overall function structure - simplify the create index functionality
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
common.pyand move functionality tosink.pyindex_schema_fieldsin None because then max one index will be set per stream.template_indexandbuild_fieldsto the class