Install Apache Hive:
Step 1: Download Apache Hive
https://hive.apache.org/downloads.htmlClick on "Download a release now!" hyperlink
Click on https://mirrors.estointernet.in/apache/hive/ hyperlink
Click on hive-3.1.2 hyperlink
apache-hive-3.1.2-bin.tar.gz
https://mirrors.estointernet.in/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
wget https://mirrors.estointernet.in/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
apache-hive-3.1.2-bin.tar.gz
Copy apache-hive-3.1.2-bin.tar.gz file into /home/datamaking/softwares/ directory
cp apache-hive-3.1.2-bin.tar.gz /home/datamaking/softwares/
cd /home/datamaking/softwares
tar -xvzf apache-hive-3.1.2-bin.tar.gz
Step 2: Add the HIVE_HOME path in the bash file (.bashrc)
nano ~/.bashrcexport HIVE_HOME=/home/datamaking/softwares/apache-hive-3.1.2-bin
export HIVE_CONF_DIR=/home/datamaking/softwares/apache-hive-3.1.2-bin/conf
export PATH=$HIVE_HOME/bin:$PATH
source ~/.bashrc
Step 3: Create "warehouse" directory in HDFS
Use below HDFS commands to create /tmp and /user/hive/warehouse (aka hive.metastore.warehouse.dir) and set them chmod 777 before you can create a table in Hive.jps
hdfs dfs -mkdir /tmp
hdfs dfs -chmod 777 /tmp
hdfs dfs -mkdir -p /user/hive/warehouse
hdfs dfs -chmod 777 /user/hive/warehouse
Step 4: Configure Hive with MySQL as Metastore
Install MySQLUpdate repository index
sudo apt-get update
Install MySQL Server with apt-get
sudo apt-get install mysql-server
Verify installation (optional)
mysql --version
Set password for root user using below command
sudo mysql_secure_installation
Configuring Root to use MySQL shell
sudo mysql
SELECT user,authentication_string,plugin,host FROM mysql.user;
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'datamaking';
FLUSH PRIVILEGES;
SELECT user,authentication_string,plugin,host FROM mysql.user;
exit;
sudo systemctl status mysql
sudo systemctl stop mysql
sudo systemctl start mysql
sudo systemctl status mysql
mysql -u root -p
CREATE DATABASE metastore;
USE metastore;
CREATE USER 'hive'@'%' IDENTIFIED BY 'datamaking';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
exit;
Configure hive-site.xml in ${HIVE_HOME}/conf/
cd ${HIVE_HOME}/conf/
nano hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<!-- WARNING!!! This file is auto generated for documentation purposes ONLY! -->
<!-- WARNING!!! Any changes you make to this file will be ignored by Hive. -->
<!-- WARNING!!! You must make your changes in hive-site.xml instead. -->
<!-- Hive Execution Parameters -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore?createDatabaseIfNotExist=true&autoReconnect=true&useSSL=false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>datamaking</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>true</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>True</value>
</property>
<property>
<name>hive.server2.transport.mode</name>
<value>http</value>
</property>
<property>
<name> hive.server2.thrift.http.port </name>
<value>10000</value>
</property>
<property>
<name>hive.server2.http.endpoint</name>
<value>cliservice</value>
</property>
</configuration>
Installing the MySQL JDBC Driver
Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html (in .tar.gz format).
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.49.tar.gz
tar -xvzf mysql-connector-java-5.1.49.tar.gz
Copy the JDBC driver, renamed, to /usr/share/java/.
If the target directory does not yet exist, create it. For example:
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.49
sudo cp mysql-connector-java-5.1.49.jar /usr/share/java/mysql-connector-java.jar
sudo cp mysql-connector-java-5.1.49.jar /home/datamaking/softwares/apache-hive-3.1.2-bin/lib/mysql-connector-java.jar
Create Hive schema (MySQL)
${HIVE_HOME}/bin/schematool -initSchema -dbType mysql
If you get the below error,
Exception in thread "main" java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
To fix the issue, find guava installed with hadoop and hive
find /home/datamaking/softwares/hadoop-3.2.1/ -type f -name "guava-*.jar"
/home/datamaking/softwares/hadoop-3.2.1/share/hadoop/hdfs/lib/guava-27.0-jre.jar
find /home/datamaking/softwares/apache-hive-3.1.2-bin/ -type f -name "guava-*.jar"
mv /home/datamaking/softwares/apache-hive-3.1.2-bin/lib/guava-19.0.jar /home/datamaking/softwares/apache-hive-3.1.2-bin/lib/guava-19.0.jar.bak
cp /home/datamaking/softwares/hadoop-3.2.1/share/hadoop/hdfs/lib/guava-27.0-jre.jar /home/datamaking/softwares/apache-hive-3.1.2-bin/lib/
${HIVE_HOME}/bin/schematool -initSchema -dbType mysql
datamaking@datamakingvm:~/Downloads/mysql-connector-java-5.1.49$ ${HIVE_HOME}/bin/schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/datamaking/softwares/apache-hive-3.1.2-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/datamaking/softwares/hadoop-3.2.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:mysql://localhost/metastore?createDatabaseIfNotExist=true&autoReconnect=true&useSSL=false
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: hive
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql
.
.
.
.
.
Initialization script completed
schemaTool completed
datamaking@datamakingvm:~/Downloads/mysql-connector-java-5.1.49$
Upgrade metastore database in mysql (Optional)
cd ${HIVE_HOME}/scripts/metastore/upgrade/mysql/
mysql -u root -p
USE metastore;
source hive-schema-3.1.0.mysql.sql;
exit;
nohup hive --service metastore &
nohup hive --service hiveserver2 &
Verify port no
netstat -an | grep 9083
sudo apt install net-tools
open Hive CLI
hive
show databases;
show tables;
Hiveserver2 Web UI:
http://localhost:10002/
Happy Learning !!!
0 Comments