Skip to main content
Lab - Working with Hive Tables
Deliverable: 6 Screenshot in one word document. I have mentioned where these screenshot are to be taken.

  1. In the cloudera Quickstart vm. Download the SalesData.csv from Blackboard.
  2. Open the Hue UI.  
  3. Click Hue menu and select Files → Go to /user/hive/warehouse/
  4. Click Upload button and select Files. Select the SalesData.csv file that you downloaded from Downloads folder.
  5. Click Hue menu and select Tables to create a new table
  6. We will create a table from a file. Click on the + button on top right side of the screen.
  7. We’ll use a file to create our table. Ensure that File is selected as Type.
  8. Select the path by selecting the ellipse button. Click Upload a file, and select the /user/hive/warehouse/SalesData.csv
  9. Click Next. Review Field Names and Field Types listed and click submit.
  10. Hive will create table using the file you uploaded. Take a look at the newly created table in Hive. Take screenshot 1 here.
  11. When uploading CSV file to create table it is important to ensure that there is no commas in field data. If there are commas in the data, the table columns in Hive will have wrong data. Custom SerDe (Serializer/Deserializer) code can be used to fix this problem.  
  12. In the cloudera Quickstart vm. Download the SalesData-WithCommas.csv and csv-serde-1.1.2-0.11.0-all.jar from Blackboard.
  13. Open Hive Menu → Files → Goto folder /user/hive/warehouse
  14. Click Upload button and select Files. Select the SalesData-WithCommas.csv file that you downloaded from Downloads folder.
  15. Click Hue menu and select Tables to create a new table
  16. We will create a table from a file. Click on the + button on top right side of the screen.
  17. We’ll use a file to create our table. Ensure that File is selected as Type.
  18. Select the path by selecting the ellipse button. Click Upload a file, and select the /user/hive/warehouse/SalesData-WithCommas.csv
  19. Click Next. Review Field Name and Field Type and click submit.
  20. Open the salesdatawithcommas table and check the company name you will notice “Netflix, inc.” has been separated into two columns. Sector and columns after Sector have wrong data. Take screenshot 2 here.
  21. We will fix this using custom serialization and deserialization. Open Hive Query editor and select all from the salesdatawithcommas table.  
  22. Lets alter the table using a custom serde code. Type the following query, do not execute it yet.
ALTER TABLE salesdatawithcommas SET SERDE ‘com.bizo.hive.serde.csv.CSVSerde’
  1. Click the session button (the one with gears icon) → Select files → click on ellipses and select the jar file. Close the window and hit the play button to run the query. Take screenshot 3 here.
  2. Now you should be able to notice that we have fixed our issue with commas in data field.
  3. The Hive performance can be improved by partitioning the table into multiple tables. Partitions are based on one or more partition column. Let’s create a table that is partitioned by year. Download YearlySales.zip file and extract it.
  4. Open Hive Editior and use the following script to create the table definitions in Hive. Take screenshot 4 here.
create table sales_all_years (RowID smallint, OrderID int, OrderDate date, OrderMonthYear date, Quantity int, Quote float, DiscountPct float, Rate float, SaleAmount float, CustomerName string, CompanyName string, Sector string, Industry string, City string, ZipCode string, State string, Region string, ProjectCompleteDate date, DaystoComplete int, ProductKey string, ProductCategory string, ProductSubCategory string, Consultant string, Manager string, HourlyWage float, RowCount int, WageMargin float)
partitioned by (yr int)
row format serde ‘com.bizo.hive.serde.csv.CSVSerde’
stored as textfile;
  1. Click Hue Menu → select Files → go to /user/hive/warehouse/sale_all-years
  2. Create 4 directories 2009, 2010, 2011 and 2012. Then, upload appropriate csv file into each directory. For example, add SalesData-2009.csv to 2009 folder.
  3. Open Hive editor and run the following scripts: After it executes, take screenshot 5 here.
-- add the partitions
alter table sales_all_years
add partition (yr=2009)
location '2009/';
alter table sales_all_years
add partition (yr=2010)
location '2010/';
alter table sales_all_years
add partition (yr=2011)
location '2011/';
alter table sales_all_years
add partition (yr=2012)
location '2012/';
  1. Open the sales_all_year table. You should be able to see the data that you uploaded. Take ScreenShot 6 of the data.
  2. Submit the screenshots in a word file.

Labels:

Comments

Post a Comment

Popular posts from this blog

A Comprehensive Evaluation of the Internal Consulting Process: Steps and Considerations

Introduction Internal consulting has emerged as a critical function within organizations, offering in-house expertise to solve complex business problems and drive change. It closely mirrors external consulting in methodology but is differentiated by the consultant's intimate knowledge of the organization and a vested interest in its long-term success. This article aims to evaluate the key steps involved in the internal consulting process, offering insights into each phase's significance and challenges. Steps in the Internal Consulting Process The internal consulting process can generally be segmented into five distinct stages: Initial Assessment, Data Collection and Analysis, Solution Development, Implementation, and Evaluation. Below is an evaluation of each step: Step 1: Initial Assessment Objective: To understand the problem or opportunity area and define the scope of the project. Significance: A well-defined scope ensures that the consulting project stays focused and manage...
Post a Comment
Read more

The Evolving Landscape of Consulting Practice: Changes and Implications

Introduction Consulting is a field that thrives on its ability to adapt to market demands and emerging trends. As businesses evolve due to technological advancements, shifts in consumer behavior, and fluctuations in global markets, consulting practices must keep pace. This article explores some of the significant changes currently transforming the consulting industry and discusses their implications for both consultants and clients. Technological Disruption Data Analytics and Artificial Intelligence Consulting firms are increasingly integrating data analytics and artificial intelligence into their service offerings. These technologies allow consultants to offer data-driven insights that can significantly enhance strategic decision-making. This evolution means consultants now need skills in data interpretation and analysis, alongside their traditional expertise in business strategy. Virtual Consulting Platforms The advent of digital platforms enables consulting services to be offered re...
Post a Comment
Read more

The Skillset of Internal Consultants: A Comparative Analysis

Introduction In the organizational landscape, the role of internal consultants has gained prominence due to the increasing complexity of business problems and the need for specialized in-house expertise. While many skills required for internal consulting overlap with those of external consultants, there are distinct abilities that set them apart. This article aims to compare and contrast these skill sets to provide a clearer understanding of what makes an effective internal consultant. Skills Common to Both Internal and External Consultants Problem-Solving Both types of consultants need to excel at identifying issues and creating viable solutions. Critical thinking and analytical skills are paramount for dissecting complex situations and recommending actionable strategies. Communication Excellent communication skills are a must for any consultant. Whether it’s making a presentation to stakeholders, writing a report, or simply discussing ideas with a team, effective communication is key...
Post a Comment
Read more