Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems.
|-- Hive website|
Hive abstracts Hadoop by abstracting it through SQL-like language, called HiveQL so that users can apply data defining and manipulating operations to it, just like with SQL. In Hive data set are defined through tables (that expose type information) in which data can be loaded, selected and transformed through built-in operators or custom/user defined functions (or UDFs).
Make elasticsearch-hadoop jar available in the Hive classpath. Depending on your options, there are various ways to achieve that. Use ADD command to add files, jars (what we want) or archives to the classpath:
the command expects a proper URI that can be found either on the local file-system or remotely. Typically it’s best to use a distributed file-system (like HDFS or Amazon S3) and use that since the script might be executed on various machines.
As an alternative, one can use the command-line:
$ bin/hive -hiveconf hive.aux.jars.path=/path/elasticsearch-hadoop.jar
or if the
hive-site.xml configuration can be modified, one can register additional jars through the
hive.aux.jars.path option (that accepts an URI as well):
<property> <name>hive.aux.jars.path</name> <value>/path/elasticsearch-hadoop.jar</value> <description>A comma separated list (with no spaces) of the jar files</description> </property>
When using Hive, one can use
TBLPROPERTIES to specify the configuration properties (as an alternative to Hadoop
Configuration object) when declaring the external table backed by Elasticsearch:
CREATE EXTERNAL TABLE artists (...) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'radio/artists/', 'es.index.auto.create' = 'false') ;
By default, elasticsearch-hadoop uses the Hive table schema to map the data in Elasticsearch, using both the field names and types in the process. There are cases however when the names in Hive cannot
be used with Elasticsearch (the field name can contain characters accepted by Elasticsearch but not by Hive). For such cases, one can use the
es.mapping.names setting which accepts a comma-separated list of names mapping in the following format:
Hive field name:
Elasticsearch field name
CREATE EXTERNAL TABLE artists (...) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'radio/artists/', 'es.mapping.names' = 'date:@timestamp , url:url_123 ');
name mapping for two fields
Elasticsearch accepts only lower-case field name and, as such, elasticsearch-hadoop will always convert Hive column names to lower-case. This poses no issue as Hive is case insensitive however it is recommended to use the default Hive style and use upper-case names only for Hive commands and avoid mixed-case names.
If automatic index creation is used, please review this section for more information.
Hive provides various types for defining data and internally uses different implementations depending on the target environment (from JDK native types to binary-optimized ones). Elasticsearch integrates with all of them, including and Serde2 lazy and lazy binary:
|Hive type||Elasticsearch type|
not supported yet
Available in Hive 0.11 or higher
Available in Hive 0.12 or higher
While Elasticsearch understands Hive types up to version 0.12, it is backwards compatible with Hive 0.9
With elasticsearch-hadoop, Elasticsearch becomes just an external table in which data can be loaded or read from:
CREATE EXTERNAL TABLE artists ( id BIGINT, name STRING, links STRUCT<url:STRING, picture:STRING>) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'radio/artists/'); -- insert data to Elasticsearch from another table called 'source' INSERT OVERWRITE TABLE artists SELECT NULL, s.name, named_struct('url', s.url, 'picture', s.picture) FROM source s;
Elasticsearch resource (index and type) associated with the given storage
For cases where the job output data is already in JSON, elasticsearch-hadoop allows direct indexing without applying any transformation; the data is taken as is and sent directly to Elasticsearch. In such cases, one needs to indicate the json input by setting
es.input.json parameter. As such, in this case elasticsearch-hadoop expects the output table to contain only one field, who
s content is used as the JSON document. That is, the library will recognize specific textual types (such as
binary) or simply call (
Table 3. Hive types to use for JSON representation
use this when the JSON data is represented as a
use this if the JSON data is represented as a
make sure the
Available in Hive 0.12 or higher
use this as an alternative to Hive
Make sure the data is properly encoded, in
UTF-8. The field content is considered the final form of the document sent to Elasticsearch.
CREATE EXTERNAL TABLE json (data STRING) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = '...', 'es.json.input` = 'yes'); ...
The table declaration only one field of type
Indicate elasticsearch-hadoop the table content is in JSON format
Reading from Elasticsearch is strikingly similar: