Backup Elastic Search DB and Elastic Search Logging Data

Need the ability to backup the following 2 types of data:

1. Elasticsearch DB
2. Elasticsearch DB logs

Topics for discussion:

- Elasticsearch and restore
- lvm snapshot available or not
- For how long the data needs to be kept
- The timeline by when the feature needs to be delivered
- The data set size if possible
- On how many nodes the data is sharded
- What is the file system path where the elasticsearch data is stored.

Additional topics for discussion:

<Please fill in here>

What versions of ES will be supported?

Need to investigate using Elastic Search plugin that is initiated by Freezer.

Link to Elasticsearch Docs:
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Link to ES Backup Instructions:
https://www.elastic.co/guide/en/elasticsearch/reference/current/backup.html

Link to Elasticsearch Snapshot and Restore Info:
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

Snapshot is only to shared volume. Plugin can be used to snapshot to something other than a shared volume.

Configure ES log file path and logging options:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html

ES data and log locations:
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-dir-layout.html

ES Plugin Docs:
https://www.elastic.co/guide/en/elasticsearch/plugins/2.1/index.html

Plugins must be written in Java.

Existing OpenStack Swift Plugin:
https://github.com/wikimedia/search-repository-swift

It looks like the OpenStack Swift Plugin can be used to do the backup and restore.

Freezer needs to be able to call the ES API to invoke the backups and restores. There is some coding effort needed for this.

Need to decide on naming conventions for snapshots and retention periods.
Snapshots do incremental backups by default.
ES is available while snapshot is being done.

Looks like version of ES in the Freezer API DevStack environment is 1.4.2. Current version of ES is 2.1.1.

The swift repository does not seem to work with v1.4.2

dek@dek-VirtualBox:~/Downloads$ curl localhost:9200
{
  "status" : 200,
  "name" : "Bennet du Paris",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.4.2",
    "build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c",
    "build_timestamp" : "2014-12-16T14:11:12Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.2"
  },
  "tagline" : "You Know, for Search"
}

dek@dek-VirtualBox:/usr/share/elasticsearch$ sudo bin/plugin -install org.wikimedia.elasticsearch.swift/swift-repository-plugin/1.7.0
-> Installing org.wikimedia.elasticsearch.swift/swift-repository-plugin/1.7.0...
Trying http://download.elasticsearch.org/org.wikimedia.elasticsearch.swift/swift-repository-plugin/swift-repository-plugin-1.7.0.zip...
Trying http://search.maven.org/remotecontent?filepath=org/wikimedia/elasticsearch/swift/swift-repository-plugin/1.7.0/swift-repository-plugin-1.7.0.zip...
Downloading ..............................................................................................................................................................................DONE
Installed org.wikimedia.elasticsearch.swift/swift-repository-plugin/1.7.0 into /usr/share/elasticsearch/plugins/swift-repository-plugin

dek@dek-VirtualBox:/usr/share/elasticsearch$ curl -XPUT 'http://localhost:9200/_snapshot/es_backup' -d@/home/dek/Documents/swift-plugin-create-repo
{"error":"RepositoryException[[es_backup] failed to create repository]; nested: NoClassSettingsException[failed to load class with value [swift]; tried [swift, org.elasticsearch.repositories.SwiftRepositoryModule, org.elasticsearch.repositories.swift.SwiftRepositoryModule, org.elasticsearch.repositories.swift.SwiftRepositoryModule]]; nested: ClassNotFoundException[org.elasticsearch.repositories.swift.SwiftRepositoryModule]; ","status":500}

Tried with latest version of Elasticsearch and it failed.

dek@dek-VirtualBox:~/Downloads/elasticsearch-2.1.1$ curl localhost:9200
{
  "name" : "Rom the Spaceknight",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.1.1",
    "build_hash" : "40e2c53a6b6c2972b3d13846e450e66f4375bd71",
    "build_timestamp" : "2015-12-15T13:05:55Z",
    "build_snapshot" : false,
    "lucene_version" : "5.3.1"
  },
  "tagline" : "You Know, for Search"
}

ek@dek-VirtualBox:~/Downloads/elasticsearch-2.1.1$ bin/plugin install org.wikimedia.elasticsearch.swift/swift-repository-plugin/1.7.0
-> Installing org.wikimedia.elasticsearch.swift/swift-repository-plugin/1.7.0...
Trying https://download.elastic.co/org.wikimedia.elasticsearch.swift/swift-repository-plugin/swift-repository-plugin-1.7.0.zip ...
Trying https://search.maven.org/remotecontent?filepath=org/wikimedia/elasticsearch/swift/swift-repository-plugin/1.7.0/swift-repository-plugin-1.7.0.zip ...
Downloading ..............................................................................................................................................................................DONE
Verifying https://search.maven.org/remotecontent?filepath=org/wikimedia/elasticsearch/swift/swift-repository-plugin/1.7.0/swift-repository-plugin-1.7.0.zip checksums if available ...
Downloading .DONE
ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip