Migrate data from one AWS Elasticsearch cluster to another.

If you’re having trouble migrating your indexes from one AWS ElasticSearch cluster to another ES cluster. Have a look into the Python Dash app that I have created. It will do all the heavy lifting for you. You only need to fill in your details and click Submit. As easy as it sounds.

What is Dash Python?

Dash is a Python framework for building web applications. It built on top of Flask, Plotly. js, React, and React Js. It enables you to build dashboards using pure Python. Dash is open-source, and its apps run on the web browser.

“Small Tweak before we begin”

You have to create an IAM role and delegate permissions to it.
Follow this AWS link & Also, make sure you meet the prerequisite here. (only the 3 Steps of the resolution)
After going through the above link you will have -
Selected S3 bucket
IAM role

“Modify Source Code-”

Populate the CAPITAL field with your own data.

def attach_snapshot_repo(self, vpc_host, snapshot_repo_name):
    print(vpc_host,snapshot_repo_name)
    path = '_snapshot/'+snapshot_repo_name
    url = vpc_host+ path
    payload = {
        "type": "s3",
        "settings": {
            "bucket": "BUCKET-NAME",
            "region": "REGION",
            "role_arn": "IAM ROLE ARN"
        }
    }

    headers = {"Content-Type": "application/json"}
    r = requests.put(url, auth=self.awsauth, json=payload, headers=headers)
    print(r.status_code)
    print(r.text)

How does my 3 step code work?

Step -1 Connect Pipeline

This will connect the new ElasticSearch and the Old ElasticSearch to a snapshot that will be stored inside a common S3 bucket (you will be required to make the code changes and add the bucket details).
Fill in the details so that the app can communicate with your AWS account.
Region will be like “us-west-2” as described by the AWS
New host ( where you want the data to be restored)
Old Host ( From where the data should be backed up)
Snapshot_repo_name ( Repository name where the snapshot should be taken, keep it simple and don't forget it)

Click Submit. If done successfully it will print “Pipeline connected successfully”

Step -2 Snapshot

It will take a snapshot of either the complete Elasticsearch cluster or of selected indexes depending on your choice.
Backup Name — (choose any name preferably the date, it will be used for restoring in the next step)
Snapshot Type — (Full or Partial )
If you choose Full it will take a backup of the whole cluster.
If you choose Partial you will be asked to Enter the IndexList
Index List — ( List the indices you want to transfer in x,y,z format a comma after every index name is a must)

Enter submit. You could see the backup progress in the Python terminal in the backend.

Step 3 -Restore

It will restore either the complete Elasticsearch cluster or selected indexes depending on your choice from the backup you just took.
Backup Name — (Enter the backup name that you want to restore from)
Restore Type — (Full or Partial )
If you choose Full it will restore the whole backup.
If you choose Partial you will be asked to Enter the IndexList that you want to restore from the taken backup
Index List — ( List the indices you want to restore in x,y,z format a comma after every index name is a must)

Enter submit. The restore process will start.

Source code-

The GitHub link to the source file is here.
The application is also dockerized.

PYTHON Package Requirements —

flask
pandas
numpy
dash
requests_aws4auth
boto3
dash_core_components
dash_html_components

Note —

In case there are any errors it will be presented in the error section of the application . Certain mistakes like restoring and index from “X ElasticSearch” to “Y ElasticSearch” can cause error if there is already an index with the same name.
Or if you dont give a comma “,” between the index name.
If you face any other errors reach out to me.

But ensure you have done “Small Tweak before we begin” successfully.