The Right (big data) Tool for the Job with Jay Shankar

FromData Skeptic

Start listening View podcast show

The Right (big data) Tool for the Job with Jay Shankar

FromData Skeptic

ratings:

Length:

50 minutes

Released:

Jul 7, 2014

Format:

Podcast episode

Description

In this week's episode, we discuss applied solutions to big data problem with big data engineer Jay Shankar. The episode explores approaches and design philosophy to solving real world big data business problems, and the exploration of the wide array of tools available.

Released:

Jul 7, 2014

Format:

Podcast episode

Titles in the series (100)

Data Skeptic is a data science podcast exploring machine learning, statistics, artificial intelligence, and other data topics through short tutorials and interviews with domain experts.

Skip carousel

Advertising Attribution with Nathan Janos: A conversation with Convertro's Nathan Janos about methodologies used to help advertisers understand the affect each of their marketing efforts (print, SEM, display, skywriting, etc.) contributes to their overall return. by Data Skeptic
76 min listen
Streetlight Outage and Crime Rate Analysis with Zach Seeskin: This episode features a discussion with statistics PhD student Zach Seeskin about a project he was involved in as part of the Eric and Wendy Schmidt Data Science for Social Good Summer Fellowship. The project involved exploring the... by Data Skeptic
33 min listen
Introduction: The Data Skeptic Podcast features conversations with topics related to data science, statistics, machine learning, artificial intelligence and the like, all from the perspective of applying critical thinking and the scientific method to evaluate the... by Data Skeptic
4 min listen
[MINI] Decision Tree Learning: Linhda and Kyle talk about Decision Tree Learning in this miniepisode. Decision Tree Learning is the algorithmic process of trying to generate an optimal decision tree to properly classify or forecast some future unlabeled element based by... by Data Skeptic
13 min listen
Guerilla Skepticism on Wikipedia with Susan Gerbic: Our guest this week is Susan Gerbic. Susan is a skeptical activist involved in many activities, the one we focus on most in this episode is Guerrilla Skepticism on Wikipedia, an organization working to improve the content and citations of... by Data Skeptic
70 min listen
[MINI] type i / type ii errors: In this first mini-episode of the Data Skeptic Podcast, we define and discuss type i and type ii errors (a.k.a. false positives and false negatives). by Data Skeptic
11 min listen
[MINI] p-values: In this mini, we discuss p-values and their use in hypothesis testing, in the context of an hypothetical experiment on plant flowering, and end with a reference to the Particle Fever documentary and how statistical significance played a role. by Data Skeptic
17 min listen
Data in Healthcare IT with Shahid Shah: Our guest this week is Shahid Shah. Shahid is CEO at Netspective, and writes three blogs: Health Care Guy, Shahid Shah, and HitSphere - the Healthcare IT Supersite. During the program, Kyle recommended a talk from the 2014 MIT Sloan CIO Symposium... by Data Skeptic
57 min listen
Economic Modeling and Prediction, Charitable Giving, and a Follow Up with Peter Backus: Economist Peter Backus joins me in this episode to discuss a few interesting topics. You may recall Linhda and I previously discussed his paper "The Girlfriend Equation" on a recent mini-episode. We start by touching base on this fun paper and get a... by Data Skeptic
24 min listen
Game Science Dice with Louis Zocchi: In this bonus episode, guest Louis Zocchi discusses his background in the gaming industry, specifically, how he became a manufacturer of dice designed to produce statistically uniform outcomes. During the show Louis mentioned a two part video... by Data Skeptic
47 min listen
Personalized Medicine with Niki Athanasiadou: In the second full length episode of the podcast, we discuss the current state of personalized medicine and the advancements in genetics that have made it possible. by Data Skeptic
57 min listen
[MINI] Monkeys on Typewriters: What is randomness? How can we determine if some results are randomly generated or not? Why are random numbers important to us in our everyday life? These topics and more are discussed in this mini-episode on random numbers. Many readers will be... by Data Skeptic
3 min listen
Jackson Pollock Authentication Analysis with Kate Jones-Smith: Our guest this week is Hamilton physics professor Kate Jones-Smith who joins us to discuss the evidence for the claim that drip paintings of Jackson Pollock contain fractal patterns. This hypothesis originates in a paper by Taylor, Micolich,... by Data Skeptic
50 min listen
$Jackson Pollock Authentication Analysis with Kate Jones-Smith: Our guest this week is Hamilton physics professor Kate Jones-Smith who joins us to discuss the evidence for the claim that drip paintings of Jackson Pollock contain fractal patterns. This hypothesis originates in a paper by Taylor, Micolich,...$
[MINI] Cross Validation: This miniepisode discusses the technique called Cross Validation - a process by which one randomly divides up a dataset into numerous small partitions. Next, (typically) one is held out, and the rest are used to train some model. The hold out set can... by Data Skeptic
1 min listen
The Right (big data) Tool for the Job with Jay Shankar: In this week's episode, we discuss applied solutions to big data problem with big data engineer Jay Shankar. The episode explores approaches and design philosophy to solving real world big data business problems, and the exploration of the wide... by Data Skeptic
50 min listen
[MINI] The Battle of the Sexes: Love and Data is the continued theme in this mini-episode as we discuss the game theory example of The Battle of the Sexes. In this textbook example, a couple must strategize about how to spend their Friday night. One partner prefers football games... by Data Skeptic
18 min listen
[MINI] Bayesian Updating: In this minisode, we discuss Bayesian Updating - the process by which one can calculate the most likely hypothesis might be true given one's older / prior belief and all new evidence. by Data Skeptic
11 min listen
[MINI] Experimental Design: This episode loosely explores the topic of Experimental Design including hypothesis testing, the importance of statistical tests, and an everyday and business example. by Data Skeptic
16 min listen
Computer-based Personality Judgments: Guest Youyou Wu discuses the work she and her collaborators did to measure the accuracy of computer based personality judgments. Using Facebook "like" data, they found that machine learning approaches could be used to estimate user's self assessment... by Data Skeptic
32 min listen
[MINI] Ant Colony Optimization: In this week's mini episode, Linhda and Kyle discuss Ant Colony Optimization - a numerical / stochastic optimization technique which models its search after the process ants employ in using random walks to find a goal (food) and then leaving a... by Data Skeptic
15 min listen
[MINI] Value of Information: A discussion about getting ready in the morning, negotiating a used car purchase, and selecting the best AirBnB place to stay at help frame a conversation about the decision theoretic principal known as the Value of Information equation. by Data Skeptic
14 min listen
Data Fest 2015: This episode contains converage of the 2015 Data Fest hosted at UCLA. Data Fest is an analysis competition that gives teams of students 48 hours to explore a new dataset and present novel findings. This year, data from Edmunds.com... by Data Skeptic
27 min listen
[MINI] Data Provenance: This episode introduces a high level discussion on the topic of Data Provenance, with more MINI episodes to follow to get into specific topics. Thanks to listener Sara L who wrote in to point out the Data Skeptic Podcast has focused alot... by Data Skeptic
11 min listen
[MINI] Selection Bias: A discussion about conducting US presidential election polls helps frame a converation about selection bias. by Data Skeptic
15 min listen
[MINI] Noise!!: Our topic for this week is "noise" as in signal vs. noise. This is not a signal processing discussions, but rather a brief introduction to how the work noise is used to describe how much information in a dataset is useless (as opposed to... by Data Skeptic
16 min listen
Data Science at ZestFinance with Marick Sinay: Marick Sinay from ZestFianance is our guest this weel. This episode explores how data science techniques are applied in the financial world, specifically in assessing credit worthiness. by Data Skeptic
31 min listen
Data Myths with Karl Mamer: Black out baby boom | Subliminal messages | Beer and Diapers by Data Skeptic
48 min listen
Easily Fooling Deep Neural Networks: My guest this week is Anh Nguyen, a PhD student at the University of Wyoming working in the Evolving AI lab. The episode discusses the paper Deep Neural Networks are Easily Fooled [pdf] by Anh Nguyen, Jason Yosinski, and Jeff Clune. It... by Data Skeptic
28 min listen
[MINI] The T-Test: The t-test is this week's mini-episode topic. The t-test is a statistical testing procedure used to determine if the mean of two datasets differs by a statistically significant amount. We discuss how a wine manufacturer might apply a t-test to... by Data Skeptic
17 min listen
[MINI] Confidence Intervals: Commute times and BBQ invites help frame a discussion about the statistical concept of confidence intervals. by Data Skeptic
12 min listen

More Episodes from Data Skeptic

Skip carousel

Pose Tracking: Many researchers and students have painstakingly labeled precise details about the body positions of the creatures they study. Can AI be used for this labeling? Of course it can! Today's episode discusses Social LEAP Estimates Animal Poses (SLEAP), a... by Data Skeptic
51 min listen
Modeling Group Behavior: Our guest in this episode is Sebastien Motsch, an assistant professor at Arizona State University, working in the School of Mathematical and Statistical Science. He works on modeling self-organized biological systems to understand how complex patterns... by Data Skeptic
41 min listen
Advances in Data Loggers: Our guest in this episode is Ryan Hanscom. Ryan is a Ph.D. candidate in a joint doctoral evolution program at San Diego State University and the University of California, Riverside. He is a terrestrial ecologist with a focus on herpetology and... by Data Skeptic
36 min listen
What You Know About Intelligence is Wrong (fixed): We are joined by Hank Schlinger, a professor of psychology at California State University, Los Angeles. His research revolves around theoretical issues in psychology and behavioral analysis. Hank establishes that words have references and... by Data Skeptic
42 min listen
Animal Decision Making: On today’s episode, we are joined by Aimee Dunlap. Aimee is an assistant professor at the University of Missouri–St. Louis and the interim director at the Whitney R. Harris World Ecology Center. Aimee discussed how animals perceive information and... by Data Skeptic
37 min listen
Octopus Cognition: We are joined by Tamar Gutnick, a visiting professor at the University of Naples Federico II, Napoli, Italy. She studies the octopus nervous system and their behavior, focusing on cognition and learning behaviors. Tamar gave a background to the kind... by Data Skeptic
38 min listen
Optimal Foraging: Claire Hemmingway, an assistant professor in the Department of Psychology and Ecology and Evolutionary Biology at the University of Tennessee in Knoxville, is our guest today. Her research is on decision-making in animal cognition, focusing on... by Data Skeptic
38 min listen
Memory in Chess: On today’s show, we are joined by our co-host, Becky Hansis-O’Neil. Becky is a Ph.D. student at the University of Missouri, St Louis, where she studies bumblebees and tarantulas to understand their learning and cognitive work. She joins us... by Data Skeptic
49 min listen
OpenWorm: On this episode, we are joined by Stephen Larson, the CEO of MetaCell and an affiliate of the OpenWorm foundation. Stephen discussed what the Openworm project is about. They hope to use a digital C. elegans nematode (C. elegans for... by Data Skeptic
34 min listen
What the Antlion Knows: Our guest is Becky Hansis-O’Neil, a Ph.D. student at the University of Missouri, St Louis, and our co-host for the new "Animal Intelligence" season. Becky shares her background on how she got into the field of behavioral intelligence and biology. by Data Skeptic
42 min listen
AI Roundtable: Kyle is joined by friends and former guests Pramit Choudhary and Frank Bell to have an open discussion of the impacts LLMs and machine learning have had in the past year on industry, and where things may go in the current year. by Data Skeptic
51 min listen
Uncontrollable AI Risks: We are joined by Darren McKee, a Policy Advisor and the host of Reality Check — a critical thinking podcast. Darren gave a background about himself and how he got into the AI space. Darren shared his thoughts on AGI's achievements in the coming... by Data Skeptic
39 min listen
I LLM and You Can Too: It took a massive financial investment for the first large language models (LLMs) to be created. Did their corporate backers lock these tools away for all but the richest? No. They provided comodity priced API options for using... by Data Skeptic
24 min listen
Q&A with Kyle: We celebrate episode 1000000000 with some Q&A from host Kyle Polich. We boil this episode down to four key questions: 1) How do you find guests 2) What is Data Skeptic all about? 3) What is Kyle all about? 4) What are Kyle's thoughts on AGI?... by Data Skeptic
40 min listen
LLMs for Data Analysis: In this episode, we are joined by Amir Netz, a Technical Fellow at Microsoft and the CTO of Microsoft Fabric. He discusses how companies can use Microsoft's latest tools for business intelligence. Amir started by discussing how business intelligence... by Data Skeptic
29 min listen
AI Platforms: Our guest today is Eric Boyd, the Corporate Vice President of AI at Microsoft. Eric joins us to share how organizations can leverage AI for faster development. Eric shared the benefits of using natural language to build products. He discussed the... by Data Skeptic
34 min listen
Deploying LLMs: We are excited to be joined by Aaron Reich and Priyanka Shah. Aaron is the CTO at Avanade, while Priyanka leads their AI/IoT offering for the SEA Region. Priyanka is also the MVP for Microsoft AI. They join us to discuss how LLMs are deployed in... by Data Skeptic
35 min listen
A Survey Assessing Github Copilot: In this episode, we are joined by Jenny Liang, a PhD student at Carnegie Mellon University, where she studies the usability of code generation tools. She discusses her recent survey on the usability of AI programming assistants. Jenny discussed the... by Data Skeptic
26 min listen
Program Aided Language Models: We are joined by Aman Madaan and Shuyan Zhou. They are both PhD students at the Language Technology Institute at Carnegie Mellon University. They join us to discuss their latest published paper, PAL: Program-aided Language Models. Aman and Shuyan... by Data Skeptic
32 min listen
Which Programming Language is ChatGPT Best At: In this episode, we have Alessio Buscemi, a software engineer at Lifeware SA. Alessio was a post-doctoral researcher at the University of Luxembourg. He joins us to discuss his paper, A Comparative Study of Code Generation using ChatGPT 3.5 across 10... by Data Skeptic
40 min listen
GraphText: On the show today, we are joined by Jianan Zhao, a Computer Science student at Mila and the University of Montreal. His research focus is on graph databases and natural language processing. He joins us to discuss how to use graphs with LLMs... by Data Skeptic
31 min listen
arXiv Publication Patterns: Today, we are joined by Rajiv Movva, a PhD student in Computer Science at Cornell Tech University. His research interest lies in the intersection of responsible AI and computational social science. He joins to discuss the findings of this work that... by Data Skeptic
28 min listen
Do LLMs Make Ethical Choices: We are excited to be joined by Josh Albrecht, the CTO of Imbue. Imbue is a research company whose mission is to create AI agents that are more robust, safer, and easier to use. He joins us to share findings of his work; Despite "super-human"... by Data Skeptic
29 min listen
Emergent Deception in LLMs: On today’s show, we are joined by Thilo Hagendorff, a Research Group Leader of Ethics of Generative AI at the University of Stuttgart. He joins us to discuss his research, . Thilo discussed how machine psychology is useful in machine learning... by Data Skeptic
27 min listen
Agents with Theory of Mind Play Hanabi: Nieves Montes, a Ph.D. student at the Artificial Intelligence Research Institute in Barcelona, Spain, joins us. Her PhD research revolves around value-based reasoning in relation to norms. She shares her latest study, Combining theory of mind and... by Data Skeptic
38 min listen
LLMs for Evil: We are joined by Maximilian Mozes, a PhD student at the University College, London. His PhD research focuses on Natural Language Processing (NLP), particularly the intersection of adversarial machine learning and NLP. He joins us to discuss his latest... by Data Skeptic
26 min listen
The Defeat of the Winograd Schema Challenge: Our guest today is Vid Kocijan, a Machine Learning Engineer at Kumo AI. Vid has a Ph.D. in Computer Science at the University of Oxford. His research focused on common sense reasoning, pre-training in LLMs, pretraining in knowledge-based completion,... by Data Skeptic
31 min listen
LLMs in Social Science: Today, We are joined by Petter Törnberg, an Assistant Professor in Computational Social Science at the University of Amsterdam and a Senior Researcher at the University of Neuchatel. His research is centered on the intersection of computational... by Data Skeptic
34 min listen
LLMs in Music Composition: In this episode, we are joined by Carlos Hernández Oliván, a Ph.D. student at the University of Zaragoza. Carlos’s interest focuses on building new models for symbolic music generation. Carlos shared his thoughts on whether these models are... by Data Skeptic
34 min listen

Related podcast episodes

Skip carousel

Modern Web Podcast S11E18- ChatGPT Can't Solve These Problems! with Chris Gardner: Ever been curious about neural networks? Chris Gardner talks neural networks, covering fundamental concepts, historical development, and practical applications. What’s the difference between AI (artificial intelligence) and machine learning? Learn ab... by Modern Web
48 min listen
Ideas to Action Ep. 3 | The Venture Pitch | Prof. Svafa Grönfeldt & Gilad Rosenzweig by The Digital Transformation Journey
23 min listen
How LLMs and Generative AI are Revolutionizing AI for Science with Anima Anandkumar - #614 by The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
62 min listen
411: Follow-upisode: This week, we follow up on last week’s conversation about designing at agencies, hiring contractors, Yelp’s app icons, and Bock’s Constant. In the Sidebar, we talk about designing with your mind’s eye and how learning to brain-prototype will help fast-track your growth as a designer. by Design Details
33 min listen
Rockstar Problem Solving Part I: The solutions to novel problems are rarely obvious. That’s why experts in complex problem solving apply data-driven methods to break down problems and creatively craft solutions. In this episode, Jay sits with two world-class problem solvers: Bob... by Think Deeply, Speak Simply
30 min listen
109. Danijar Hafner - Gaming our way to AGI by Towards Data Science
50 min listen
Deciphering Data Architectures with James Serra by Insights Tomorrow
32 min listen
JSJ 465:The Power of Micro Front-Ends with Michael Geers: Micro-services have been very popular on the backend for a while now. But can this architecture be applied to the front-end as well? Should it be applied to the front-end? Michael Geers, who literally wrote the book on micro-fronts ends, explains what they are and why they can be a powerful architectural pattern especially when implementing large-scale projects. He also describes how Web Components can be used to enable this type of application architecture. by JavaScript Jabber
44 min listen
Building and Retaining a Data Science Team: This week on AI in Industry, we speak with Equifax's Dr. Rajkumar Bondugula about how the dynamics, composition and requirements of the data science team have evolved over the years. Raj also shares valuable insights on how to build a robust data... by The AI in Business Podcast
29 min listen
Developing an Enterprise Data Strategy the Right Way - with Dora Boussias of Stryker: In this week’s episode, we’re discussing a topic that applies across industries for AI projects: where is the data coming from? Our guest is Dora Boussias, Senior Director for Data Strategy and Architecture at Stryker. Stryker is a $17B per year... by The AI in Business Podcast
29 min listen
Walking Fast Into the Future: Evolvable Technical Reference Frameworks for Mixed-Criticality Systems: In this SEI Podcast, Nickolas Guertin, a senior systems engineer with the SEI’s Software Solutions Division, and Douglas Schmidt, associate provost of research at Vanderbilt University and former chief technical officer at the SEI, discuss... by Software Engineering Institute (SEI) Podcast Series
40 min listen
Ideas to Action Ep. 2 | The Human Perspective | Prof. Svafa Grönfeldt & Gilad Rosenzweig by The Digital Transformation Journey
16 min listen
Practical Differential Privacy at LinkedIn with Ryan Rogers - #346: Today we’re joined by Ryan Rogers, Senior Software Engineer at LinkedIn. We caught up with Ryan at NeurIPS, where he presented the paper “Practical Differentially Private Top-k Selection with Pay-what-you-get Composition” as a spotlight talk. In... by The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
34 min listen
Graphite Metrics Stack with Jason Dixon and Dan Cech: Capture, Store, and Analyze Your System Metrics with Python (Interview) by The Python Podcast.__init__
74 min listen
Augmenting Ourselves with AI: A Design Revolution with Mirit Cohen and Andrew Weigand: In this episode, Mike MacDade and Alex, the AI co-host, dives into the groundbreaking world of AI's influence on design with industry experts Mirit Cohen, an Interactive Designer, and Andrew Weigand, a Multimedia Professional. The episode navigates... by Your Career: Is it Choice or Chance?
16 min listen
194: The future of AI/ML in design: As artificial intelligence and machine learning begin to impact the design industry in new ways, the future of design may differ from the way we work today. If that is the case, how can you remain relevant as a designer? by Design Life
26 min listen
Hey Reseller, What's the "Value" You're Adding?: All links and images for this episode can be found on CISO Series () It seems that you're offering so much more when you add the VA ("value added") in front of your title. What is that? Why am I working with you rather than buying directly from the... by CISO Series Podcast
37 min listen
Databricks’ Rao on Harnessing AI at Scale by Tech Disruptors
47 min listen
How to Leverage Data Assets for Business - with Kenneth Cukier: In this episode, we speak with Senior Editor for the Economist in digital and data products and Co-author of ", Kenneth Cukier, who speaks on the technologies that underlie big data and make it what it is today. Cukier addresses and dives... by The AI in Business Podcast
26 min listen
Ideas to Action Ep. 1 | Framing Problems, Designing Solutions | Prof. Svafa Grönfeldt & Gilad Rosenzweig by The Digital Transformation Journey
15 min listen
From startup to Google and back again: Ben and Cassidy talk with programmer and developer advocate Sean Falconer, Head of Developer Relations and Marketing at Skyflow, ex-Googler, and host of the podcasts Partially Redacted and Software Engineering Daily. They cover why Sean pivoted from academic to cofounder, why AI is sparking excitement and anxiety in the DevRel space, and how working at Google compares to life at a Series B startup. by The Stack Overflow Podcast
30 min listen
How deliberate design facilitates communication | Karel Vredenburg: Design thinking is a methodology that starts with identifying the real problem, recognizing the needs of users, prototyping solutions, and working through an iterative process. When leveraged properly, it has the potential to unlock amazing ideas and... by Think Deeply, Speak Simply
25 min listen
Magic.dev CEO Eric Steinberger on making developers bionic | E1744 by This Week in Startups
57 min listen
Revisiting The Technical And Social Benefits Of The Data Mesh: An interview with Zhamak Dehghani about her experience working with the community that has grown up around her idea of the data mesh and the lessons that she has learned. by Data Engineering Podcast
71 min listen
432: One of the world's leading experts on strategy and innovation, Vijay “VG” Govindarajan. How Real-Time Data and AI Will Power the Industrial Future: Welcome to Strategy Skills episode 432, an interview with the coauthor of Fusion Strategy: How Real-Time Data and AI Will Power the Industrial Future. In Fusion Strategy, world-renowned innovation guru Vijay Govindarajan and digital strategy expert... by The Strategy Skills Podcast: Strategy | Leadership | Critical Thinking | Problem-Solving
56 min listen
Nodes of Design#97: Designing for AI by Christopher Reardon by Nodes of Design
56 min listen
Problem Formulation for Machine Learning with Romer Rosales - TWiML Talk #149: In this episode, i'm joined by Romer Rosales, Dir… by The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
50 min listen
Episode 95: The New Guardian.co.uk website with Matt Wall and Erik DoernenBurg: In this episode we talk to Matthew Wall (Guardian News and Media) and Erik Doernenburg (Thoughtworks) about their work on the new guardian.co.uk website. We discuss the challenge of scalability and interactivity, their use of Domain Driven Design, by Software Engineering Radio - the podcast for professional software developers
44 min listen
#33 Top 6 High-Level Skills for Designer by Future of UX | Your Design, Tech and User Experience Podcast | AI Design
26 min listen
Data and Analytics Summit Preps Leaders for the Future: Gartner’s Data and Analytics Summit is one of the world’s largest gatherings of business and IT leaders across this important discipline. Gartner’s leading analysts came together to share a postcard from the future to enable everyone to be ready for the coming technology wave. by TechWave: A Gartner Podcast for IT Leaders
28 min listen

Discover this podcast and so much more

The Right (big data) Tool for the Job with Jay Shankar

The Right (big data) Tool for the Job with Jay Shankar

Description

Titles in the series (100)

More Episodes from Data Skeptic

Related podcast episodes