Michael Lando | 451a340 | 2017-02-19 10:28:42 +0200 | [diff] [blame] | 1 | ASDC elasticsearch tool |
| 2 | ======================== |
| 3 | |
| 4 | This tool purpose is to ease and allow updating elasticsearch indices. |
| 5 | |
| 6 | In order to use the scripts, you need to verify Python is installed and to install the elasticsearc-py library: |
| 7 | Verify pip is installed: $command -v pip |
| 8 | if not installed: |
| 9 | Download https://bootstrap.pypa.io/get-pip.py |
| 10 | $python get-pip.py (see instruction: https://pip.pypa.io/en/latest/installing/#installing-with-get-pip-py) |
| 11 | $pip install elasticsearch |
| 12 | |
| 13 | |
| 14 | Tool contains: |
| 15 | - index_ops.py |
| 16 | This script includes operations on elasticsearch index: |
| 17 | |
| 18 | create index: |
| 19 | $python index_ops.py -o create -a <elasticsearch hostname> -n <indexName> -f <index mapping file> |
| 20 | |
| 21 | delete index: |
| 22 | $python index_ops.py -o delete -a <elasticsearch hostname> -n <indexName> |
| 23 | |
| 24 | copy index (assumes destination index already exists): |
| 25 | $python index_ops.py -o move -a <elasticsearch hostname> -n <indexName> -t <toIndex> |
| 26 | |
| 27 | |
| 28 | - file_utils.py |
| 29 | This script includes operations on files |
| 30 | |
| 31 | - audit_migration_1602.py |
| 32 | This script run full flow to migrate audit information from previous versions to ASDC 1602 |
| 33 | It has 2 inputs: |
| 34 | 1. config_properties.py - this file holds configuration (hostname, index name, index mapping file etc.) |
| 35 | 2. folder of fields mapping per elasticsearch type (map old field to new field) |
| 36 | The flow of this script is as follow: |
| 37 | * create temp index with correct index mapping |
| 38 | * scan the audit index to get all records |
| 39 | * manipulate fields data and insert it to temp index |
| 40 | * delete audit index |
| 41 | * create audit index with correct mapping |
| 42 | * copy from temp index to newly created audit index |
| 43 | * delete temp index |