Shiva Krishna

Hadoop developer

PROFESSIONAL SUMMARY:
Overall 8 years of professional experience in Software Development and Requirement Analysis in Agile work
environment with 4+ years of Big Data Ecosystems experience in ingestion, storage, querying, processing and
analysis of Big Data.
Experience in dealing with Apache Hadoop components like HDFS, MapReduce, Hive, HBase, Pig, Sqoop,
Oozier, Mahout, Python, Spark, Cassandra, MongoDB,
Good understanding/knowledge of Hadoop Architecture and various components such as HDFS, Job Tracker,
Task Tracker, Name Node, Data Node, Secondary Name node, and MapReduce concepts.
Experienced managing No-SQL DB on large Hadoop distribution Systems such as: Cloudera, Hortonworks
HDP, Map M series etc.
Experienced developing Hadoop integration for data ingestion, data mapping and data process capabilities.
Worked with various data sources such as Flat files and RDBMS-Teradata, SQL server 2005, Netezza and
Oracle. Extensive work in ETL process consisting of data transformation, data sourcing, mapping, conversion.
Hands on experience in installing, configuring, and using Hadoop ecosystem components like Hadoop
MapReduce, HDFS, HBase, Hive, Sqoop, Pig, Zookeeper, Storm, Spark, Kafka and Flume.
Strong understanding of Data Modeling and experience with Data Cleansing, Data Profiling and Data
analysis.
Experience in ETL (Data stage) analysis, designing, developing, testing and implementing ETL processes
including performance tuning and query optimizing of databases.
Experience in extracting source data from Sequential files, XML files, Excel files, transforming and loading it
into the target data warehouse.
Strong experience with Java/J2EE technologies such as Core Java, JDBC, JSP, JSTL, HTML, JavaScript,
JSON
Strong experience in Scala programming for developing the Spark applications.
Extensive coding experience in Java and Mainframes - COBOL, CICS and JCL
Experience of working in all the phases of Software Development in various methodologies
Strong base in writing the Test plans, perform Unit Testing, User Acceptance testing, Integration Testing, System
Testing
Proficient in software documentation and technical report writing.
Worked coherently with multiple teams. Conducted peer reviews, organized and participated in knowledge
transfer (technical and domain) sessions.
Experience in working with Onsite-Offshore model.
Developed various UDFs in Map-Reduce and Python for Pig and Hive.
Decent experience and knowledge in other SQL and NoSQL Databases like MySQL, MS SQL, MongoDB,
HBase, Accumulo, Neo4j and Cassandra.
Good Data Warehouse experience in MS SQL.
Proficiency in programming with different IDE's like Eclipse, NetBeans.
Involved in database design, creating Tables, Views, Stored Procedures, Functions, Triggers and Indexes.
Good understanding of service oriented architecture (SOA) and web services like XML, XSD, XSDL, SOAP.
Good Knowledge about scalable, secure cloud architecture based on Amazon Web Services (leveraging AWS
cloud services: EC2, Cloud Formation, VPC, S3, etc.
Good Knowledge on Hadoop Cluster architecture and monitoring the cluster.
In-depth understanding of Data Structure and Algorithms.
Experience in managing and troubleshooting Hadoop related issues.
Expertise in setting up standards and processes for Hadoop based application design and implementation.
Experience in importing and exporting data using Sqoop from Relational Database Systems to HDFS and
vice-versa.
Experience in managing Hadoop clusters using Cloudera Manager.
Hands on experience in VPN, Putty, wisp, Unviewed, etc.
Expertise in setting up standards and processes for Hadoop based application design and implementation
EDUCATION:
Bachelors of Technology

TOOLA AND TECHNOLOGIES:

Hadoop/Big Data Map Reduce, Hive, Pig, Impala, Sqoop, Flume, HDFS, Oozie, Hue,
HBase, Zookeeper, Spark
Operating Systems Windows, Ubuntu, RedHat Linux, Unix

Java & J2EE Technologies Core Java, Servlets, JSP, JDBC

Frameworks Hibernate
Databases/Database Languages Oracle 11g/10g/9i, MySQL, DB2, SQLServer, SQL, HQL, NoSQL
(HBase)
Web Technologies JavaScript, HTML, XML, REST, CSS
Programming Languages Java, Scala, Unix shell scripting, COBOL, CICS, JCL
IDEs Eclipse, Net beans
Web Servers Apache Tomcat 6
Methodologies Waterfall, Agile and Scrum

PROFESSIONAL EXPERIENCE:

Client: AT&T Dallas, Texas Aug 2015 Present

Role: Hadoop Developer
Description:
AT&T Corp. provides voice, video, and data communications services to businesses, consumers, and governments
worldwide. It operates in two segments, Business Services and Consumer Services. The Business Services segment offers
domestic and international long distance and toll-free voice services, local voice services, audio and video teleconferencing
services, Web-based video conferencing services, and calling cards. Its local voice services include local voice and data
telecommunications services, such as basic local exchange service, exchange access, private line, and high speed data and
pay phone
Responsibilities:
Installed, configured, and maintained Apache Hadoop clusters for application development and major
components of Hadoop Ecosystem: Hive, Pig, HBase, Sqoop, Flume, Oozier and Zookeeper.
Used Sqoop to transfer data between RDBMS and HDFS.
Involved in collecting and aggregating large amounts of streaming data into HDFS using Flume and defined
channel selectors to multiplex data into different sinks.
Responsible for troubleshooting issues in the execution of MapReduce jobs by inspecting and reviewing log
files.
Converted existing SQL queries into Hive QL queries.
Implemented UDFs, UDAFs, UDTFs in java for hive to process the data that cant be performed using Hive
inbuilt functions.
Effectively used Oozier to develop automatic workflows of Sqoop, MapReduce and Hive jobs.
Exported the analyzed data into relational databases using Sqoop for visualization and to generate reports for
the BI team.
Gathered the business requirements from the Business Partners and Subject Matter Experts.
Utilized Agile Scrum Methodology to help manage and organize a team of 4 developers with regular code
review sessions.
Weekly meetings with technical collaborators and active participation in code review sessions with senior and
junior developers.
Loaded and analyzed Omniture logs generated by different web applications
Loaded and transformed large sets of structured, semi structured and unstructured data in various formats like
text, zip, XML and JSON.
Refined the Website clickstream data from Omniture logs and moved it into Hive.
Wrote multiple MapReduce programs to power data for extraction, transformation and aggregation from
multiple file formats including XML, JSON, CSV & other compressed file formats.
 Defined job flows and developed simple to complex Map Reduce jobs as per the requirement.
Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms
Developed PIG UDFs for manipulating the data according to Business Requirements and also worked on
developing custom PIG Loaders.
Developed Scala code for Spark Streaming and Spark SQL.
Used different Scala Collection API for Spark Programming.
Developed Spark code using Scala for faster data processing using RDD's and Data frame API.
Developed Spark Programs for Batch and real time Processing.
Involved in creating POCs to ingest and process streaming data using Spark streaming and Kafka.
Worked on Creating Kafka topics, partitions, writing custom partitioner classes.
Used Kafka to store events from various system and processed using Spark Streaming to perform near real
time analytics.
Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
Developed Spark Streaming applications for real time Processing
Involved in creating POCs to ingest and process streaming data using Spark streaming and Kafka.

Environment: Hadoop, CDH4, Map Reduce, HDFS, Pig, Hive, Impala, oozy, Java, spark, Kafka, Flume, Storm, Knox,
Linux, Scala, Maven, Java Scripting, Oracle 11g/10g, SVN

Client: T-Mobile, Bellevue, WA. Dec 2014 Jul 2015

Role: Hadoop Developer
Description:
T-Mobile US, Inc., together with its subsidiaries, provides mobile communications services in the United States, Puerto
Rico, and the U.S. Virgin Islands. The company offers voice, messaging, and data services to approximately 71 million
customers in the postpaid, prepaid, and wholesale markets. It also provides wireless devices, including smartphones, tablets,
and other mobile communication devices, as well as accessories that are manufactured by various suppliers
Responsibilities:
Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hbase database and
Sqoop.
Responsible for building scalable distributed data solutions using Hadoop.
Implemented nine nodes CDH3 Hadoop cluster on CentOS
Involved in loading data from LINUX file system to HDFS.
Worked on installing cluster, commissioning & decommissioning of data node, name node recovery, capacity
planning, and slots configuration.
Implemented a script to transmit suspiring information from Oracle to HBase using Sqoop.
Implemented best income logic using Pig scripts and UDFs.
Implemented test scripts to support test driven development and continuous integration.
Worked on tuning the performance Pig queries.
Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as
required.
Responsible to manage data coming from different sources.
Involved in loading data from file system to HDFS.
Load and transform large sets of structured, semi structured and unstructured data
Cluster coordination services through Zookeeper.
Experience in managing and reviewing Hadoop log files.
Job management using Fair scheduler.
Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports
for the BI team.
Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and
troubleshooting.
Installed Oozie workflow engine to run multiple Hive and pig jobs.
Supported in setting up QA environment and updating configurations for implementing scripts with Pig and
Sqoop.
Implemented custom interceptors for flume to filter data as per requirement.
Created internal and external Hive tables and defined static and dynamic partitions for optimized
performance.
Wrote Pig Latin scripts for running advanced analytics on the data collected.
 Configured daily workflow for extraction, processing and analysis of data using Oozie Scheduler.
Proactively involved in ongoing maintenance, support and improvements in Hadoop cluster.
Wrote Pig Latin scripts for running advanced analytics on the data collected.
Configured daily workflow for extraction, processing and analysis of data using Oozie Scheduler.
Implemented custom interceptors for flume to filter data as per requirement.
Used Hive and Pig to analyze data in HDFS to identify issues and behavioral patterns.
Environment: Hadoop, HDFS, Pig, Sqoop, HBase, Shell Scripting, CDH3, CentOS

Client: Core metrics Inc, San Mateo, CA Jan 2012 Nov 2014
Role: Hadoop Developer
Description:
Coremetrics, Inc., a software as a service provider, offers digital marketing optimization solutions. It offers Coremetrics
Continuous Optimization Platform that gives insight into the behavior of customers and prospects; Coremetrics Analytics to
deliver intuitive collaboration capabilities to share performance insights; and Coremetrics for Mobile that provides access to
the marketing metrics on handheld devices. The company also provides Coremetrics Explore that provides a complete
picture of visitor and customer behavior.
Responsibilities:
Involved in review of functional and non-functional requirements.
Installed and configured Hadoop Mapreduce, HDFS, Developed multiple MapReduce jobs in java for data
cleaning and pre-processing.
Importing and exporting data into HDFS and Hive using Sqoop.
Supported Map Reduce Programs those are running on the cluster.
Involved in loading data from UNIX file system to HDFS.
Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map
reduce way.
Setup and benchmarked Hadoop/HBase clusters for internal use
Involved in Loading process into the Hadoop distributed File System and Pig in order to preprocess the data.
Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box
(such as MapReduce, Pig, Hive, Sqoop, flume) as well as some system specific jobs like (shell scripts)
Involved in Data modeling sessions to develop models for Hive tables
Imported and exported large sets of data into HDFS and vice-versa using Sqoop.
Transferred log files from the log generating servers into HDFS.
Installed Hive, Pig, Flume, Sqoop and Oozie on the Hadoop cluster.
Created tasks, workflows and sessions using Workflow manager. Worked with scheduling team to come up
Production schedule
Worked on Hive partition and bucketing concepts and created hive External and Internal tables with Hive
partition
Assisted the project manager in problem solving with Big Data technologies for integration of Hive with
HBASE and Sqoop with HBase.
Solved performance issues in Hive and pig with understanding of Joins, Group and aggregation and how does
it transfer to Map-Reduce.
Moved the data from traditional databases like MySQL, MS SQL Server and Oracle into Hadoop
Worked on Integrating Talend and SSIS with Hadoop and performed ETL operations.
Used Flume to collect, aggregate and push log data from different log servers
Performed unit testing using JUnit testing framework and used Log4j to monitor the error log
Environment: Hadoop, MapReduce, HDFS, Hive, Sqoop, HBase, UNIX Shell Scripting

Client: Citrix Systems, Hyderabad, India May 2010 Dec 2011

Role: Sr. JAVA Developer
Description:
Citrix Systems, Inc., incorporated on April 17, 1989, offers Enterprise and Service Provider products, which include
Workspace Services solutions and Delivery Networking products. The Company's Enterprise and Service Provider products
include Cloud Services solutions, and related license updates and maintenance, support and professional services. The
Company's NetScaler nCore Technology is an architecture that enables execution of multiple packet engines in parallel.
Responsibilities:
Installation, Configuration & Upgrade of Solaris and Linux operating system.
Actively participated in requirements gathering, analysis, design, and testing phases
 Designed use case diagrams, class diagrams, and sequence diagrams as a part of Design Phase
Developed the entire application implementing MVC Architecture integrating JSF with Hibernate and Spring
frameworks.
Developed the Enterprise Java Beans (Stateless Session beans) to handle different transactions such as online
funds transfer, bill payments to the service providers.
Implemented Service Oriented Architecture (SOA) using JMS for sending and receiving messages while
creating web services
Developed XML documents and generated XSL files for Payment Transaction and Reserve Transaction
systems.
Developed SQL queries and stored procedures.
Developed Web Services for data transfer from client to server and vice versa using Apache Axis, SOAP and
WSDL.
Used JUnit Framework for the unit testing of all the java classes.
Implemented various J2EE Design patterns like Singleton, Service Locator, DAO, and SOA.
Worked on AJAX to develop an interactive Web Application and JavaScript for Data Validations.
Developed the application under JEE architecture, developed Designed dynamic and browser compatible user
interfaces using JSP, Custom Tags, HTML, CSS, and JavaScript.
Deployed & maintained the JSP, Servlets components on Web logic 8.0
Developed Application Servers persistence layer using, JDBC, SQL, Hibernate.
Used JDBC to connect the web applications to Data Bases.
Implemented Test First unit testing framework driven using Junit.
Developed and utilized J2EE Services and JMS components for messaging communication in Web Logic.
Configured development environment using Web logic application server for developers integration testing
Environment: Java/J2EE, SQL, Oracle 10g, JSP 2.0, EJB, AJAX, Java Script, Web Logic 8.0, HTML, JDBC 3.0, XML,
JMS, log4j, Junit, Servlets, MVC, My Eclipse

Client: Ixigo,Gurgaon, India Aug 2008 Apr 2010

Role: Jr. Java Developer
Description:
Ixigo is a travel search company that gives the deals about the air tickets, hotels, train tickets and bus tickets. It provides all
the information about the travel.
Responsibilities:
Installation, Configuration & Upgrade of Solaris and Linux operating system.
Actively participated in requirements gathering, analysis, design, and testing phases
Designed use case diagrams, class diagrams, and sequence diagrams as a part of Design Phase
Developed the entire application implementing MVC Architecture integrating JSF with Hibernate and Spring
frameworks.
Developed the Enterprise Java Beans (Stateless Session beans) to handle different transactions such as online
funds transfer, bill payments to the service providers.
Implemented Service Oriented Architecture (SOA) using JMS for sending and receiving messages while
creating web services
Developed XML documents and generated XSL files for Payment Transaction and Reserve Transaction
systems.
Developed SQL queries and stored procedures.
Developed Web Services for data transfer from client to server and vice versa using Apache Axis, SOAP and
WSDL.
Used JUnit Framework for the unit testing of all the java classes.
Implemented various J2EE Design patterns like Singleton, Service Locator, DAO, and SOA.
Worked on AJAX to develop an interactive Web Application and JavaScript for Data Validations.
Developed the application under JEE architecture, developed Designed dynamic and browser compatible user
interfaces using JSP, Custom Tags, HTML, CSS, and JavaScript.
Deployed & maintained the JSP, Servlets components on Web logic 8.0
Developed Application Servers persistence layer using, JDBC, SQL, Hibernate.
Used JDBC to connect the web applications to Data Bases.
Implemented Test First unit testing framework driven using Junit.
Developed and utilized J2EE Services and JMS components for messaging communication in Web Logic.
Configured development environment using Web logic application server for developers integration testing
Environment: Java/J2EE, SQL, Oracle 10g, JSP 2.0, EJB, AJAX, Java Script, Web Logic 8.0, HTML, JDBC 3.0, XML,
JMS, log4j, Junit, Servlets, MVC, My Eclipse