Convert csv data to Avro data

In one of my previous post I explained how we can convert json data to avro data and vice versa using avro tools command line option. Today I was trying to see what options we have for converting csv data to avro format, as of now we don't have any avro tool option to accomplish this . Now, we can either write our own java program (MapReduce program or a simple java program) or we can use various SerDe's available with Hive to do this quickly and without writing any code :)

To convert csv data to Avro data using Hive we need to follow below steps:

1. Create a Hive table stored as textfile and specify your csv delimiter also.
2. Load csv file to above table using "load data" command.
3. Create another Hive table using AvroSerDe.
4. Insert data from former table to new Avro Hive table using "insert overwrite" command.

To demonstrate this I will use use below data (student.csv):

0,38,91
0,65,28
0,78,16
1,34,96
1,78,14
1,11,43

Now execute below queries in Hive:

	--1. Create a Hive table stored as textfile
	USE test;
	CREATE TABLE csv_table (
	student_id INT,
	subject_id INT,
	marks INT)
	ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
	STORED AS TEXTFILE;
	--2. Load csv_table with student.csv data
	LOAD DATA LOCAL INPATH "/path/to/student.csv" OVERWRITE INTO TABLE test.csv_table;
	--3. Create another Hive table using AvroSerDe
	CREATE TABLE avro_table
	ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
	STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
	OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
	TBLPROPERTIES (
	'avro.schema.literal'='{
	"namespace": "com.rishav.avro",
	"name": "student_marks",
	"type": "record",
	"fields": [ { "name":"student_id","type":"int"}, { "name":"subject_id","type":"int"}, { "name":"marks","type":"int"}]
	}');
	--4. Load avro_table with data from csv_table
	INSERT OVERWRITE TABLE avro_table SELECT student_id, subject_id, marks FROM csv_table;

view raw csv_avro.hql hosted with ❤ by GitHub

Now you can get data in Avro format from Hive warehouse folder. To dump this file to local file system use below command:
hadoop fs -cat /path/to/warehouse/test.db/avro_table/* > student.avro

If you want to get json data from this avro file you can use avro tools command:
java -jar avro-tools-1.7.5.jar tojson student.avro > student.json

So we can easily convert csv to avro and csv to json also by just writing 4 HQLs.